Containers

Pre-built Apptainer/Singularity containers let you run THOR without compiling from source.

Quick Start

thor-generic is the recommended image — same SIF runs on CPU, NVIDIA, or AMD GPU.

Pick the backend at exec time:

# Download
apptainer pull oras://ghcr.io/cbyrohl/thor-generic:dev

# CPU (OpenMP via SSCP JIT)
apptainer run --env ACPP_VISIBILITY_MASK=omp thor-generic_dev.sif config.yaml

# NVIDIA GPU
apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda thor-generic_dev.sif config.yaml

# AMD GPU
apptainer run --rocm --env ACPP_VISIBILITY_MASK=hip thor-generic_dev.sif config.yaml

Set device: in your config YAML to match — cpu, cpu-openmp, or gpu.

apptainer pull --force oras://ghcr.io/cbyrohl/thor-generic:dev

Updating from Python or thor-tools

The base thor-rt Python package can download and update images for batch runs and live sessions. Install Apptainer, then run:

# Create or refresh thor-generic_dev.sif in the current directory
thor-tools container update

# List the supported images and their registry URLs
thor-tools container list

# Select the CPU-only image (defaults to thor-omp_dev.sif)
thor-tools container update --image omp

# Optionally choose a local destination
thor-tools container update ./thor-omp.sif --image omp

The same operation is available from Python:

from thor.container import update_image

image = update_image()  # returns an absolute pathlib.Path
image = update_image(image="omp")  # writes thor-omp_dev.sif
image = update_image("./thor-omp.sif", image="omp")

The commands support these two image variants, defaulting to their development tags:

Unknown image names and custom registry URLs are not accepted. Listing shows the supported catalog; it does not query registry availability or access permissions.

The destination's parent directory must already exist. An existing regular file is replaced; symlinks and directories are rejected. Updates download into a temporary directory beside the destination, check the downloaded SIF structure, and atomically replace the destination. A failed download or validation preserves the previous file. Allow space for both the old and new images, plus Apptainer's cache.

Each call pulls the selected public image using Apptainer's cache; registry login is not required. The default image is always generic; the variant is not inferred from the filename or existing image. Repeat --image when updating a nondefault image. The default filename is thor-<image>_dev.sif in the current directory. Use --digest sha256:<64-hex-digits> (Python: digest="sha256:...") to pull an exact registry artifact of the selected variant. The default filename then becomes thor-<image>_sha256-<64-hex-digits>.sif, keeping it separate from the development image. Digests must use lowercase hexadecimal. For release tags, other variants, or other registries, use apptainer pull directly.

Use runtime="/path/to/apptainer" or --runtime /path/to/apptainer for a custom executable. SIF validation checks structure, not signatures or THOR compatibility.

Before replacing the image, the updater compares the installed thor-rt source revision with the container's org.opencontainers.image.revision label. A mismatch emits a RevisionWarning showing both revisions and continues the update. Missing metadata or an unsuccessful metadata inspection also warns, reporting the unknown revision. Standard Python warning filters apply.

Wheels and source distributions carry their build revision. Editable installs read the current HEAD of their own checkout, so switching branches changes the reported revision without reinstalling. You can inspect both revisions from Python:

from thor.container import python_revision, container_revision

print(python_revision())
print(container_revision(image))

This check covers the source revision of thor-rt. It does not compare uncommitted edits or independently installed thor-live, thor-mcp, or thor-bindings. Matching revisions do not establish compatibility of runtime settings or dependencies.

The updated image takes effect the next time it is opened. Restart a live session to use it. Apptainer itself and the Python packages are unchanged.

Container identity and reproducible runs

The org.opencontainers.image.revision label records the full THOR Git commit. This is sufficient to identify the source revision. Rebuilding that commit can still produce a different image: the toolchain base and apt packages can change. Three identifiers serve different purposes:

The SIF checksum and registry manifest digest are different hashes. Do not substitute the SIF checksum into an oras://…@sha256:… reference.

The Release Containers workflow publishes a thor-<variant>-provenance-<attempt> artifact containing thor-<variant>.provenance.json. The same JSON is included in each build job's summary. It records the three identifiers above, the SIF size, and the build-attempt URL. Reruns get separate artifacts. After upload, the workflow fetches the raw registry manifest with ORAS, checks that its SIF checksum and size match the local image, and hashes that manifest for the pinned reference. A competing tag update to a different image fails verification; later updates of :dev cannot change an already recorded reference. Publication manifests are available for builds using this workflow; older builds may not have one.

For a production run, download that manifest from the chosen workflow run, then pull its pinned reference and verify the local image before starting THOR:

# Requires jq. Keep these files in the run directory with the configuration.
REFERENCE=$(jq -er '.registry_reference' thor-generic.provenance.json) &&
apptainer pull thor.sif "$REFERENCE" &&
thor-tools container provenance thor.sif \
    --manifest thor-generic.provenance.json > container-provenance.json &&
apptainer run --env ACPP_VISIBILITY_MASK=omp thor.sif config.yaml

The provenance command hashes the entire SIF and compares its checksum, size, and embedded THOR revision against the publication manifest. A mismatch exits with an error and prints no JSON. Use && as above so a failed verification prevents the simulation from starting. Treat the manifest as trusted publication metadata; this comparison does not authenticate its author or verify signatures.

The equivalent Python inspection is:

import json
from pathlib import Path
from thor.container import image_provenance

record = image_provenance("thor.sif", manifest="thor-generic.provenance.json")
Path("container-provenance.json").write_text(json.dumps(record, indent=2) + "\n")

Without --manifest (or the Python manifest argument), inspection still records the SIF checksum and any available source revision, but registry_reference is null. A local SIF does not reliably reveal its original registry manifest. Keep a copy of such an image to reproduce the run. Inspection is offline, reads the whole file, and does not execute the container. It reports unavailable source metadata as null unless a manifest was supplied, in which case verification fails.

The provenance command is explicit; invoking apptainer run or launching a Python live session does not automatically create this record. Keep the record with the run's configuration, random seeds, precision/backend settings, and relevant host hardware and driver information. The same container does not guarantee identical numerical results on different devices.

Publication and retention policy

• :dev is a moving development alias. A local copy updates only when pulled.
• Preserve the embedded THOR commit and use registry digests to pin exact images; a separate permanent tag for every build is not required.
• Retain all published THOR image digests, including versions that become untagged when :dev moves. The repository has no registry cleanup workflow. Any future cleanup must preserve images referenced by simulations or releases; being untagged is not evidence that an image is unused.
• Workflow provenance artifacts have a requested retention of 90 days, subject to repository limits. Archive the JSON with simulation outputs rather than relying on Actions artifact retention. Archive the SIF as well when long-term availability must be independent of the registry.

Setting Thread Count

Control the number of OpenMP threads with OMP_NUM_THREADS:

apptainer run --env OMP_NUM_THREADS=8 --env ACPP_VISIBILITY_MASK=omp \
    thor-generic_dev.sif config.yaml

MPI (multiple ranks)

THOR parallelises MCRT across MPI ranks. The thor-generic image ships a full Open MPI 4.1 stack (launchers included), so a single-node, multi-rank job runs entirely inside one container — the container's own mpirun launches the ranks:

apptainer exec --cleanenv --env OMP_NUM_THREADS=8 --env ACPP_VISIBILITY_MASK=omp \
    thor-generic_dev.sif mpirun --map-by slot:PE=8 --bind-to core -np 3 /opt/thor/bin/thor config.yaml

• Rank↔core binding (--map-by slot:PE=N --bind-to core) — this is the part that matters for hybrid MPI+OpenMP. By default Open MPI pins each rank to a single core, so the OMP_NUM_THREADS OpenMP threads stack onto that one core (measured ~2.6× slower). --map-by slot:PE=N instead gives each rank its own N physical cores (set N = OMP_NUM_THREADS), and AdaptiveCpp honours the resulting cpuset — so the threads spread with proper locality. Constraint: N × ranks ≤ physical cores. Simpler fallback: --bind-to none leaves ranks unpinned so the threads spread across the whole node — correct thread count and it fixes the slowdown, but with no per-rank NUMA locality (fine on a single socket, suboptimal across sockets). Use it when the core count is unknown or PE-mapping is awkward.
• --cleanenv stops the host's OpenMPI MCA environment from leaking into the container, which can crash MPI_Init on GPU-less nodes.
• --env OMP_NUM_THREADS: bound the OpenMP threads per rank. --cleanenv wipes any host OMP_NUM_THREADS, so without this each of the -np 3 ranks defaults to all cores and oversubscribes the node. Set it to roughly cores-per-node ÷ ranks.
• Cold JIT cache: with the SSCP/JIT thor-generic image, ranks racing on AdaptiveCpp's shared kernel cache can spam Could not remove kernel cache file. Give each rank its own cache with ACPP_APPDB_DIR=…_rank_${OMPI_COMM_WORLD_RANK} (e.g. wrap the launch in bash -c) — see regression_tests/CI/shellmodel/test_mpi_container.sh for the pattern. (The AOT thor-omp image has no JIT cache and is unaffected.)
• Set device: cpu-openmp in your config to match ACPP_VISIBILITY_MASK=omp.
• Multi-rank support is per dataset: uniform grid and point-cloud (gas) domain-decompose across ranks; analytical models (e.g. the shell model, as the container MPI regression exercises) replicate the domain and split photons across ranks; octree and PSS abort with more than one rank.

For multi-node runs the launcher must sit outside the container so it can span nodes (srun --mpi=pmix apptainer exec … thor). That path needs the host MPI to be PMIx-compatible with the container's Open MPI 4.1 — see Running on HPC Clusters.

GPU usage

thor-generic carries both the CUDA and ROCm runtimes in one image; backend selection happens at exec time, not build time.

NVIDIA

apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda thor-generic_dev.sif config.yaml

Requirements:

• NVIDIA driver installed on the host
• The --nv flag (binds host GPU drivers into the container)
• device: gpu in your config YAML

AMD

apptainer run --rocm --env ACPP_VISIBILITY_MASK=hip thor-generic_dev.sif config.yaml

Requirements:

• ROCm-supported AMD GPU (Instinct MI-series, supported Radeon Pro / RX)
• amdgpu kernel module loaded (/dev/kfd and /dev/dri/renderD* must exist)
• The --rocm flag
• device: gpu in your config YAML

apptainer run --rocm \
    --env LD_PRELOAD=/lib/x86_64-linux-gnu/libnuma.so.1 \
    --env ACPP_VISIBILITY_MASK=hip \
    thor-generic_dev.sif config.yaml

Selecting FP32 or FP64

The GPU-capable images ship two binaries side-by-side — FP64 (thor-fp64, default) and FP32 (thor-fp32) — and a wrapper that picks one based on the THOR_PRECISION env var:

# FP64 (default)
apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda thor-generic_dev.sif config.yaml

# FP32 — typically much faster on consumer GPUs (RTX, GTX, RDNA),
# which have crippled FP64 throughput
apptainer run --nv --env ACPP_VISIBILITY_MASK=cuda --env THOR_PRECISION=fp32 \
    thor-generic_dev.sif config.yaml

Available Images

Building Custom Containers

pip install typer pyyaml jinja2
cd apptainer

# List configurations
./build.py list

# Build container
./build.py build omp

# Build from local source (requires initialized submodules)
./build.py build omp --local

Cluster Configurations

Configurations in apptainer/clusters/:

Create a custom config:

cp apptainer/clusters/example-hpc.yaml apptainer/clusters/my-cluster.yaml
# Edit march for your CPU (e.g., znver3 for AMD EPYC Milan)
./build.py build my-cluster

Common march values: x86-64-v3 (generic), znver2/znver3/znver4 (AMD EPYC), skylake-avx512/icelake-server (Intel Xeon).

Shipping FP64 + FP32 binaries

Set dual_precision: true in your cluster YAML to build both binaries into the same image with a THOR_PRECISION runtime dispatch wrapper (see Selecting FP32 or FP64 above). The second cmake build runs in build-fp32/ with -DTHOR_USE_FP32=ON; on-disk overhead is small (~10 MB compressed) because the two binaries share most of their code. Useful for any GPU-targeted config; off by default for CPU configs since FP32 yields little benefit on the CPU path.

Environment Container

The published thor-env-only package is private. Downloading it requires access and apptainer registry login --username <github-user> docker://ghcr.io with a GitHub personal access token (classic) scoped to read:packages.

For development without rebuilding containers:

./build.py build env-only

# Build and run THOR from source
apptainer exec build/thor-env-only.sif bash -c \
    "cd /path/to/thor && cmake -B build && cmake --build build -j\$(nproc)"
apptainer exec build/thor-env-only.sif /path/to/thor/build/src/thor config.yaml

Accessing Host Data

By default, Apptainer only mounts your home directory and a few system paths inside the container. If your simulation data or Cloudy tables live elsewhere (e.g. /virgotng/, /scratch/, /data/), you must bind-mount those paths:

# Bind a single path
apptainer run --bind /virgotng thor-generic_dev.sif config.yaml

# Bind multiple paths
apptainer run --bind /virgotng,/scratch thor-generic_dev.sif config.yaml

Alternatively, set the APPTAINER_BIND environment variable (useful in job scripts):

export APPTAINER_BIND="/virgotng,/scratch"
apptainer run thor-generic_dev.sif config.yaml

For full details, see the Apptainer Bind Paths documentation.

Troubleshooting

fuse2fs not found / gocryptfs not found warnings: These informational messages from Apptainer can be safely ignored. THOR containers use Docker-based images and do not require EXT3 filesystem mounting or encrypted overlays.

Slow performance: Check march matches your CPU. Set OMP_NUM_THREADS appropriately.