Running on HPC Clusters

Submit a THOR job in three steps: copy a job script below, edit the paths and wall-clock limit, and sbatch it. Each example has a Binary tab (you compiled THOR yourself, see Installation) and an Apptainer tab (you're using a container image).

Scheduler scope

Examples use SLURM. On PBS, LSF, or other schedulers, request the same resources (ranks, threads-per-rank, GPUs) using your scheduler's flags — the device: and ACPP_VISIBILITY_MASK settings stay the same.

Quick Start (CPU)

The simplest way to get THOR running on a cluster: one node, OpenMP threads, no GPU required. By default these scripts request the whole node (--exclusive + --mem=0), which gives reproducible benchmarks and avoids contention with other users' jobs.

Why no srun or mpirun invocation for single-node runs?

Call the THOR binary directly in the sbatch script. The script runs inside the job's cgroup; with --exclusive, that cgroup holds every core on the node, and the binary inherits all of them. Wrapping in srun adds nothing — and since SLURM 22.05, srun no longer inherits --cpus-per-task from sbatch, so it can silently bind your task to a single core and bottleneck OpenMP.

BinaryApptainer

run_thor.sbatch

#!/bin/bash
#SBATCH --job-name=thor
#SBATCH --time=04:00:00              # wall-clock limit
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --exclusive                  # reserve the whole node
#SBATCH --mem=0                      # all available memory

# paths to your THOR binary and config
THOR=/path/to/thor/cmake-build-release-omp/src/thor
CONFIG=config.yaml

export ACPP_VISIBILITY_MASK=omp
export OMP_NUM_THREADS=$(nproc)      # or set explicitly, e.g. 32

# direct invocation — see note above; do NOT wrap in srun
"$THOR" "$CONFIG"

run_thor.sbatch

#!/bin/bash
#SBATCH --job-name=thor
#SBATCH --time=04:00:00              # wall-clock limit
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --exclusive                  # reserve the whole node
#SBATCH --mem=0                      # all available memory

IMAGE=/path/to/thor-omp_dev.sif
CONFIG=config.yaml

# Filesystems THOR must read/write that live outside $HOME and the current
# directory (input snapshots, Cloudy tables, output dirs, etc.) — Apptainer
# auto-mounts $HOME and $PWD but nothing else. Adjust to your cluster.
export APPTAINER_BIND=/scratch,/data

# direct invocation — see note above; do NOT wrap in srun
# OMP_NUM_THREADS=$(nproc) below uses every reserved core; set
# explicitly (e.g. OMP_NUM_THREADS=32) to cap the thread count.
apptainer run \
    --env ACPP_VISIBILITY_MASK=omp \
    --env OMP_NUM_THREADS=$(nproc) \
    "$IMAGE" "$CONFIG"

In your config.yaml, set:

device: "cpu-openmp"

Submit:

sbatch run_thor.sbatch

Sanity-Check Your Configuration

The commands below confirm the job is using the cores you reserved and is not silently crashing or running at a fraction of the expected speed. They are not for tracking simulation progress.

While the job is running:

squeue -u $USER lists your queued and running jobs.
tail -f slurm-<jobid>.out follows the live log.
tail -f <output_dir>/log_monitor_rank0.txt follows THOR's per-rank monitor log (CPU% and RSS, every 30 s). During an MCRT kernel the cpu= field should sit near N × 100% on an N-core node; much lower means cores are idle.
The startup line -- enabled (omp_get_max_threads=N) should report the core count you reserved. 1 instead of 64 means OMP_NUM_THREADS was not picked up.

After the job finishes:

seff <jobid> prints a short efficiency report. CPU Efficiency should be above 80% on a CPU-bound MCRT run.
If seff is missing on your cluster, fall back to sacct -j <jobid> --format=JobID,Elapsed,TotalCPU,MaxRSS.

Multi-node (MPI)

To scale across nodes, THOR domain-decomposes MCRT over MPI ranks. Here the launcher must wrap the program (unlike the single-node case above) so it can span nodes. Use one rank per node and let OpenMP fill each node's cores.

BinaryApptainer

run_thor_mpi.sbatch

#!/bin/bash
#SBATCH --job-name=thor
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1          # 1 rank/node
#SBATCH --cpus-per-task=128          # OpenMP threads/rank — set to cores per node
#SBATCH --exclusive

THOR=/path/to/thor/cmake-build-release-omp/src/thor
CONFIG=config.yaml
export ACPP_VISIBILITY_MASK=omp
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-$(nproc)}

# Pass --cpus-per-task to the step too: recent Slurm (23.11+) no longer
# propagates it from #SBATCH automatically, and without it each rank is
# confined to a single CPU (OpenMP then can't fill the node).
srun --mpi=pmix --cpus-per-task="${SLURM_CPUS_PER_TASK}" "$THOR" "$CONFIG"

run_thor_mpi.sbatch

#!/bin/bash
#SBATCH --job-name=thor
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=128          # OpenMP threads/rank — set to cores per node
#SBATCH --exclusive

IMAGE=/path/to/thor-generic_dev.sif
CONFIG=config.yaml
export APPTAINER_BIND=/scratch,/data
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-$(nproc)}

# Launcher OUTSIDE the container so it spans nodes; the container's Open MPI
# connects back over PMIx. `--mpi=pmix` is the portable choice because PMIx
# tolerates version skew between the host Slurm and the container's MPI.
#
# This path can't use --cleanenv (the host PMIx/Slurm env must reach the
# container), so OMPI_MCA_opal_cuda_support=0 is forwarded explicitly: the
# generic image is GPU-capable and bakes HWLOC_COMPONENTS=-gl but NOT that
# guard, so a host CUDA-aware OpenMPI would otherwise take the CUDA MPI_Init
# path and crash on GPU-less nodes. Drop it for a genuine multi-node GPU run.
srun --mpi=pmix --cpus-per-task="${SLURM_CPUS_PER_TASK}" apptainer exec \
    --env ACPP_VISIBILITY_MASK=omp \
    --env OMP_NUM_THREADS="${OMP_NUM_THREADS}" \
    --env OMPI_MCA_opal_cuda_support=0 \
    "$IMAGE" /opt/thor/bin/thor "$CONFIG"

In your config.yaml, set device: "cpu-openmp".

Per-rank JIT cache

The generic image compiles kernels at runtime (SSCP/JIT). On a cold cache, ranks sharing AdaptiveCpp's default cache dir race and spam Could not remove kernel cache file. Give each rank its own with ACPP_APPDB_DIR=…_rank_${SLURM_PROCID} (wrap the launch in bash -c) — see regression_tests/CI/shellmodel/test_mpi_container.sh. The AOT thor-omp image is unaffected.

If every rank prints the startup banner

Seeing N startup banners (N rank-0 processes instead of one N-rank job) means the host launcher and the container's MPI didn't connect. Prefer srun --mpi=pmix; if the cluster lacks the pmix plugin, fall back to mpirun --bind-to none -n N apptainer exec … (the --bind-to none avoids pinning each hybrid rank to a single core) with a host Open MPI matching the container's 4.1.x. Uniform-grid and point-cloud (gas) datasets domain-decompose across ranks; analytical models (e.g. the shell model) replicate and split photons across ranks; octree and PSS abort with more than one rank.

Troubleshooting

If your job exits immediately, look at slurm-<jobid>.out. Common causes:

The device in your config doesn't match ACPP_VISIBILITY_MASK. For example, device: cpu-openmp paired with ACPP_VISIBILITY_MASK=cuda will fail to start. Make sure they agree.
Apptainer can't find your input data. Apptainer auto-mounts only your home directory and the working directory. Add any other paths your config reads from to APPTAINER_BIND.