Containers

Experimental

Container support is experimental. MPI is not supported in containers. Single-node CPU, NVIDIA GPU, and AMD GPU execution have been tested.

Pre-built Apptainer/Singularity containers let you run THOR without compiling from source.

Quick Start

thor-generic is the recommended image — same SIF runs on CPU, NVIDIA, or AMD GPU. Pick the backend at exec time:

# Download
apptainer pull oras://ghcr.io/cbyrohl/thor-generic:dev

# CPU (OpenMP via SSCP JIT)
apptainer run --env ACPP_VISIBILITY_MASK=omp thor-generic_dev.sif config.yaml

# NVIDIA GPU
apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda thor-generic_dev.sif config.yaml

# AMD GPU
apptainer run --rocm --env ACPP_VISIBILITY_MASK=hip thor-generic_dev.sif config.yaml

Set device: in your config YAML to match — cpu, cpu-openmp, or gpu.

Updating an existing image

The :dev tag is updated in place whenever a new build is published, so your local .sif can fall behind. apptainer pull won't overwrite the existing file by default — pass --force:

apptainer pull --force oras://ghcr.io/cbyrohl/thor-generic:dev

Setting Thread Count

Control the number of OpenMP threads with OMP_NUM_THREADS:

apptainer run --env OMP_NUM_THREADS=8 --env ACPP_VISIBILITY_MASK=omp \
    thor-generic_dev.sif config.yaml

GPU usage

thor-generic carries both the CUDA and ROCm runtimes in one image; backend selection happens at exec time, not build time.

NVIDIA

apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda thor-generic_dev.sif config.yaml

Requirements:

NVIDIA driver installed on the host
The --nv flag (binds host GPU drivers into the container)
device: gpu in your config YAML

AMD

apptainer run --rocm --env ACPP_VISIBILITY_MASK=hip thor-generic_dev.sif config.yaml

Requirements:

ROCm-supported AMD GPU (Instinct MI-series, supported Radeon Pro / RX)
amdgpu kernel module loaded (/dev/kfd and /dev/dri/renderD* must exist)
The --rocm flag
device: gpu in your config YAML

Fedora hosts: libnuma ABI conflict with --rocm

Apptainer's --rocm binds the host's libnuma.so.1. On Fedora ≥ 39 (glibc ≥ 2.40) this conflicts with the Ubuntu 24.04 glibc inside the container (GLIBC_ABI_GNU2_TLS not found) and every acpp backend fails to load. Workaround:

apptainer run --rocm \
    --env LD_PRELOAD=/lib/x86_64-linux-gnu/libnuma.so.1 \
    --env ACPP_VISIBILITY_MASK=hip \
    thor-generic_dev.sif config.yaml

Unsupported AMD targets

ROCm ships device-libs bitcode only for officially supported gfxes. Consumer iGPUs (e.g. gfx1036 on Ryzen 9000 desktop) hit Code object construction failed at JIT time. Sometimes --env HSA_OVERRIDE_GFX_VERSION=10.3.0 (gfx1030) works as a hack; for production use, target a supported device.

Selecting FP32 or FP64

The GPU-capable images ship two binaries side-by-side — FP64 (thor-fp64, default) and FP32 (thor-fp32) — and a wrapper that picks one based on the THOR_PRECISION env var:

# FP64 (default)
apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda thor-generic_dev.sif config.yaml

# FP32 — typically much faster on consumer GPUs (RTX, GTX, RDNA),
# which have crippled FP64 throughput
apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda --env THOR_PRECISION=fp32 \
    thor-generic_dev.sif config.yaml

FP32 is reduced-precision

Float32 is sufficient for many MCRT runs but can degrade results in long photon chains, high optical depths, or tight numerical tolerances. Validate against an FP64 reference for your problem before relying on FP32 production runs.

Available Images

Image	Description	Precision
`thor-generic`	Portable SYCL build. CUDA + ROCm runtimes bundled; backend chosen at exec time.	FP64 + FP32
`thor-omp`	CPU-only AOT build (AdaptiveCpp OMP backend). Smaller; best raw CPU perf.	FP64
`thor-env-only`	Build environment (no `thor` binary) — for compiling THOR yourself.	—

Building Custom Containers

For Development Only

Most users should use the pre-built thor-omp image above.

pip install typer pyyaml jinja2
cd apptainer

# List configurations
./build.py list

# Build container
./build.py build omp

# Build from local source (requires initialized submodules)
./build.py build omp --local

Cluster Configurations

Configurations in apptainer/clusters/:

Configuration	Description
`generic.yaml`	Portable SSCP build with CUDA + ROCm runtimes; FP64 + FP32 binaries
`omp.yaml`	OMP AOT backend, FP64, smaller image
`env-only.yaml`	Build environment only (no THOR binary)
`example-hpc.yaml`	Template for custom cluster-specific configs

Create a custom config:

cp apptainer/clusters/example-hpc.yaml apptainer/clusters/my-cluster.yaml
# Edit march for your CPU (e.g., znver3 for AMD EPYC Milan)
./build.py build my-cluster

Common march values: x86-64-v3 (generic), znver2/znver3/znver4 (AMD EPYC), skylake-avx512/icelake-server (Intel Xeon).

Shipping FP64 + FP32 binaries

Set dual_precision: true in your cluster YAML to build both binaries into the same image with a THOR_PRECISION runtime dispatch wrapper (see Selecting FP32 or FP64 above). The second cmake build runs in build-fp32/ with -DTHOR_USE_FP32=ON; on-disk overhead is small (~10 MB compressed) because the two binaries share most of their code. Useful for any GPU-targeted config; off by default for CPU configs since FP32 yields little benefit on the CPU path.

Environment Container

For development without rebuilding containers:

./build.py build env-only

# Build and run THOR from source
apptainer exec build/thor-env-only.sif bash -c \
    "cd /path/to/thor && cmake -B build && cmake --build build -j\$(nproc)"
apptainer exec build/thor-env-only.sif /path/to/thor/build/src/thor config.yaml

Accessing Host Data

By default, Apptainer only mounts your home directory and a few system paths inside the container. If your simulation data or Cloudy tables live elsewhere (e.g. /virgotng/, /scratch/, /data/), you must bind-mount those paths:

# Bind a single path
apptainer run --bind /virgotng thor-omp_dev.sif config.yaml

# Bind multiple paths
apptainer run --bind /virgotng,/scratch thor-omp_dev.sif config.yaml

Alternatively, set the APPTAINER_BIND environment variable (useful in job scripts):

export APPTAINER_BIND="/virgotng,/scratch"
apptainer run thor-omp_dev.sif config.yaml

For full details, see the Apptainer Bind Paths documentation.

Troubleshooting

fuse2fs not found / gocryptfs not found warnings: These informational messages from Apptainer can be safely ignored. THOR containers use Docker-based images and do not require EXT3 filesystem mounting or encrypted overlays.

Slow performance: Check march matches your CPU. Set OMP_NUM_THREADS appropriately.