Containers

Pre-built Apptainer/Singularity containers let you run THOR without compiling from source.

Quick Start

thor-generic is the recommended image — same SIF runs on CPU, NVIDIA, or AMD GPU.

Pick the backend at exec time:

# Download
apptainer pull oras://ghcr.io/cbyrohl/thor-generic:dev

# CPU (OpenMP via SSCP JIT)
apptainer run --env ACPP_VISIBILITY_MASK=omp thor-generic_dev.sif config.yaml

# NVIDIA GPU
apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda thor-generic_dev.sif config.yaml

# AMD GPU
apptainer run --rocm --env ACPP_VISIBILITY_MASK=hip thor-generic_dev.sif config.yaml

Set device: in your config YAML to match — cpu, cpu-openmp, or gpu.

apptainer pull --force oras://ghcr.io/cbyrohl/thor-generic:dev

Setting Thread Count

Control the number of OpenMP threads with OMP_NUM_THREADS:

apptainer run --env OMP_NUM_THREADS=8 --env ACPP_VISIBILITY_MASK=omp \
    thor-generic_dev.sif config.yaml

MPI (multiple ranks)

THOR parallelises MCRT across MPI ranks. The thor-generic image ships a full Open MPI 4.1 stack (launchers included), so a single-node, multi-rank job runs entirely inside one container — the container's own mpirun launches the ranks:

apptainer exec --cleanenv --env OMP_NUM_THREADS=8 --env ACPP_VISIBILITY_MASK=omp \
    thor-generic_dev.sif mpirun --map-by slot:PE=8 --bind-to core -np 3 /opt/thor/bin/thor config.yaml

• Rank↔core binding (--map-by slot:PE=N --bind-to core) — this is the part that matters for hybrid MPI+OpenMP. By default Open MPI pins each rank to a single core, so the OMP_NUM_THREADS OpenMP threads stack onto that one core (measured ~2.6× slower). --map-by slot:PE=N instead gives each rank its own N physical cores (set N = OMP_NUM_THREADS), and AdaptiveCpp honours the resulting cpuset — so the threads spread with proper locality. Constraint: N × ranks ≤ physical cores. Simpler fallback: --bind-to none leaves ranks unpinned so the threads spread across the whole node — correct thread count and it fixes the slowdown, but with no per-rank NUMA locality (fine on a single socket, suboptimal across sockets). Use it when the core count is unknown or PE-mapping is awkward.
• --cleanenv stops the host's OpenMPI MCA environment from leaking into the container, which can crash MPI_Init on GPU-less nodes.
• --env OMP_NUM_THREADS: bound the OpenMP threads per rank. --cleanenv wipes any host OMP_NUM_THREADS, so without this each of the -np 3 ranks defaults to all cores and oversubscribes the node. Set it to roughly cores-per-node ÷ ranks.
• Cold JIT cache: with the SSCP/JIT thor-generic image, ranks racing on AdaptiveCpp's shared kernel cache can spam Could not remove kernel cache file. Give each rank its own cache with ACPP_APPDB_DIR=…_rank_${OMPI_COMM_WORLD_RANK} (e.g. wrap the launch in bash -c) — see regression_tests/CI/shellmodel/test_mpi_container.sh for the pattern. (The AOT thor-omp image has no JIT cache and is unaffected.)
• Set device: cpu-openmp in your config to match ACPP_VISIBILITY_MASK=omp.
• Multi-rank support is per dataset: uniform grid and point-cloud (gas) domain-decompose across ranks; analytical models (e.g. the shell model, as the container MPI regression exercises) replicate the domain and split photons across ranks; octree and PSS abort with more than one rank.

For multi-node runs the launcher must sit outside the container so it can span nodes (srun --mpi=pmix apptainer exec … thor). That path needs the host MPI to be PMIx-compatible with the container's Open MPI 4.1 — see Running on HPC Clusters.

GPU usage

thor-generic carries both the CUDA and ROCm runtimes in one image; backend selection happens at exec time, not build time.

NVIDIA

apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda thor-generic_dev.sif config.yaml

Requirements:

• NVIDIA driver installed on the host
• The --nv flag (binds host GPU drivers into the container)
• device: gpu in your config YAML

AMD

apptainer run --rocm --env ACPP_VISIBILITY_MASK=hip thor-generic_dev.sif config.yaml

Requirements:

• ROCm-supported AMD GPU (Instinct MI-series, supported Radeon Pro / RX)
• amdgpu kernel module loaded (/dev/kfd and /dev/dri/renderD* must exist)
• The --rocm flag
• device: gpu in your config YAML

apptainer run --rocm \
    --env LD_PRELOAD=/lib/x86_64-linux-gnu/libnuma.so.1 \
    --env ACPP_VISIBILITY_MASK=hip \
    thor-generic_dev.sif config.yaml

Selecting FP32 or FP64

The GPU-capable images ship two binaries side-by-side — FP64 (thor-fp64, default) and FP32 (thor-fp32) — and a wrapper that picks one based on the THOR_PRECISION env var:

# FP64 (default)
apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda thor-generic_dev.sif config.yaml

# FP32 — typically much faster on consumer GPUs (RTX, GTX, RDNA),
# which have crippled FP64 throughput
apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda --env THOR_PRECISION=fp32 \
    thor-generic_dev.sif config.yaml

Available Images

Building Custom Containers

pip install typer pyyaml jinja2
cd apptainer

# List configurations
./build.py list

# Build container
./build.py build omp

# Build from local source (requires initialized submodules)
./build.py build omp --local

Cluster Configurations

Configurations in apptainer/clusters/:

Create a custom config:

cp apptainer/clusters/example-hpc.yaml apptainer/clusters/my-cluster.yaml
# Edit march for your CPU (e.g., znver3 for AMD EPYC Milan)
./build.py build my-cluster

Common march values: x86-64-v3 (generic), znver2/znver3/znver4 (AMD EPYC), skylake-avx512/icelake-server (Intel Xeon).

Shipping FP64 + FP32 binaries

Set dual_precision: true in your cluster YAML to build both binaries into the same image with a THOR_PRECISION runtime dispatch wrapper (see Selecting FP32 or FP64 above). The second cmake build runs in build-fp32/ with -DTHOR_USE_FP32=ON; on-disk overhead is small (~10 MB compressed) because the two binaries share most of their code. Useful for any GPU-targeted config; off by default for CPU configs since FP32 yields little benefit on the CPU path.

Environment Container

For development without rebuilding containers:

./build.py build env-only

# Build and run THOR from source
apptainer exec build/thor-env-only.sif bash -c \
    "cd /path/to/thor && cmake -B build && cmake --build build -j\$(nproc)"
apptainer exec build/thor-env-only.sif /path/to/thor/build/src/thor config.yaml

Accessing Host Data

By default, Apptainer only mounts your home directory and a few system paths inside the container. If your simulation data or Cloudy tables live elsewhere (e.g. /virgotng/, /scratch/, /data/), you must bind-mount those paths:

# Bind a single path
apptainer run --bind /virgotng thor-generic_dev.sif config.yaml

# Bind multiple paths
apptainer run --bind /virgotng,/scratch thor-generic_dev.sif config.yaml

Alternatively, set the APPTAINER_BIND environment variable (useful in job scripts):

export APPTAINER_BIND="/virgotng,/scratch"
apptainer run thor-generic_dev.sif config.yaml

For full details, see the Apptainer Bind Paths documentation.

Troubleshooting

fuse2fs not found / gocryptfs not found warnings: These informational messages from Apptainer can be safely ignored. THOR containers use Docker-based images and do not require EXT3 filesystem mounting or encrypted overlays.

Slow performance: Check march matches your CPU. Set OMP_NUM_THREADS appropriately.