Skip to content

Preparing reference data

thor-tools prepare downloads versioned reference files and checks that a complete release is ready to use. Preparation runs before a simulation, for example on a cluster login node with internet access. Verification is offline.

Install the base Python package; preparation requires no THOR build or analysis extras:

uv tool install "/path/to/thor/python/thor-rt"
thor-tools prepare --help

Choose a catalog

thor-tools prepare list
thor-tools prepare list --manifest /path/to/catalog.json

The list shows release versions, download sizes, provenance, and citations. Choose the Cloudy model appropriate to your calculation, including its UV background, shielding assumptions, abundances, and density/temperature/redshift coverage. A table release is independent of the THOR software version.

The bundled cloudy-c17.03-fg11-rahmati2013 catalog contains ion fractions and line emissivities from Cloudy C17.03 with FG11, GASS10 abundances, and temet's Rahmati2013 shielding using fixed FG09 fit coefficients. Its native lg grid spans redshift 0–8, log10(nH/cm^-3) −7–4, log10(Z/Zsun) −3–1, and log10(T/K) 3–9. It is licensed under CC BY 4.0.

Version v2 preserves molecular hydrogen (H₂) separately from the atomic ion fractions. Its atomic fractions and emissivities are unchanged from v1; THOR does not yet expose an H₂ field. Version v1 remains available for older THOR installations.

The v2 download is approximately 309 MiB; the installed HDF5 files occupy 5.39 GiB. Select the version explicitly; no default alias is configured:

thor-tools prepare fetch cloudy-c17.03-fg11-rahmati2013 --version v2

The bundled catalog references the published manifests for v1 and v2 instead of embedding their complete generation records. Full records remain in those manifests and each HDF5 file's provenance_json attribute. The historical DRAFT status in v1 describes preparation time; both releases are published.

generated_files_reference.sha256 checks the referenced manifest's bytes; records_sha256 checks its generated_files object serialized with Python's json.dumps(sort_keys=True, separators=(",", ":"), allow_nan=False) defaults and UTF-8 encoding. Existing receipts containing full records remain valid offline when the records match this digest and every other catalog field agrees. No additional download or receipt rewrite is required.

A data provider can also supply a JSON manifest as described below. There are no automatic substitutions with tables from other projects.

AtomDB cluster Fe-K catalog

The AtomDB 3.1.3 CIE catalog uses AG89 reference abundances and covers 6.3–8.1 keV at 200 native temperatures, approximately 10⁴–9.44 × 10⁸ K. It includes 365 explicit transitions, ion fractions, and continuum emission. Download the compressed table (37 MiB; checksums). See the Fe XXV example for unpacking and configuration.

Alternatives and pinned defaults

Each table set has an explicit provider (for example cloudy or atomdb), provider version, contents, and scientific provenance. Use distinct catalog IDs for different physical assumptions, such as cloudy-c17.03-fg11-rahmati2013 and cloudy-c17.03-fg11-unshielded. The release version identifies the exact files within that set. Provider metadata does not declare reader compatibility: that is determined separately by each file's format and validation.

prepare list displays the provider, contents, assumptions, and defaults. prepare fetch prints the selected ID/version and physics to stderr before any download; stdout remains usable as YAML (or an installed path for generic files).

A manifest can optionally include purpose aliases pinned to exact releases:

"defaults": {
  "cloudy": {"id": "cloudy-c17.03-fg11-unshielded", "version": "v1"}
}

This is an illustrative selection, not a scientific recommendation. With that entry and matching catalog, prepare fetch cloudy resolves to that exact set. Aliases cannot shadow catalog IDs, chain to other aliases, or select a moving latest release. To override a default, name the actual catalog ID and version; --version is rejected with aliases. Without a default, existing explicit selection rules apply. Default choices must be scientifically reviewed for the intended purpose before publication.

Receipts retain the resolved catalog and its metadata, and generated YAML uses its absolute, versioned paths. A later change to a default does not change those files or configurations. To verify an older installation after a default changes, use its explicit catalog ID and --version (and retained manifest if necessary).

Self-shielding is an object with a model and parameters. Use none for an explicitly unshielded model and not-applicable when shielding is outside the data model; both require empty parameters. UVB likewise uses explicit strings such as FG11, none, or not-applicable. Named shielding prescriptions also require a citation, for example:

"self_shielding": {
  "model": "rahmati2013",
  "citation": "https://doi.org/10.1093/mnras/stt066",
  "parameters": {"uvb_fit": "FG09", "implementation": "temet; pin commit in generator"}
}

This refers to Rahmati et al. (2013), whose preprint appeared in 2012. The temet grid adapter calls the FG09-based attenuation routine even when the incident spectrum is FG11. Publishers must record the implementation and any settings; the prescription name alone does not reproduce a table. This example is not a complete production provenance record.

Schema 2 replaces the unpublished schema-1 prototype. Local prototype manifests need the provider, contents, equilibrium, and structured shielding fields added before loading.

Download and configure

For a provider's manifest containing cloudy release v1:

thor-tools prepare fetch cloudy --version v1 --manifest /path/to/catalog.json
thor-tools prepare verify cloudy --version v1 --manifest /path/to/catalog.json

If there is only one release for a catalog, --version can be omitted. Otherwise the command requires a choice; it never selects a moving latest.

Preparation writes files under <data-root>/<catalog>/<version>/ and prints YAML to stdout, with progress and citations on stderr. For Cloudy, paste the YAML under the dataset section of your simulation configuration:

pointcloud_voronoi:
  # Your existing loader and dataset settings...
  cloudy:
    enabled: true
    redshift_interp: true
    ion_table_path: /absolute/data/root/cloudy/v1/grid_ions.hdf5
    emission_table_path: /absolute/data/root/cloudy/v1/grid_emissivities.hdf5

The command supplies absolute paths to the actual installed filenames. It sets redshift_interp to match the table's dimensionality. Other interpolation, temperature, and physics settings remain choices in your simulation config. The generated snippet is a dataset subsection, not a complete runnable config.

Choose storage

The data root is selected in this order:

  1. --data-dir.
  2. The THOR_DATA environment variable.
  3. THOR_DATA in ~/.config/thor/paths.yaml.
  4. $XDG_DATA_HOME/thor, or ~/.local/share/thor when XDG_DATA_HOME is unset.

For shared cluster storage:

thor-tools prepare fetch cloudy --version v1 --manifest /path/to/catalog.json \
    --data-dir /shared/thor-data --register

--register adds an absolute THOR_DATA entry to ~/.config/thor/paths.yaml, preserving existing entries and comments. An existing entry or environment variable pointing elsewhere produces an error instead of retargeting existing simulations. You can also edit that paths file yourself. THOR already supports references such as ${THOR_DATA}/cloudy/v1/grid_ions.hdf5 in YAML values.

Registration with --register requires HOME to be nonempty and absolute, matching THOR’s paths-file loader. Shared THOR_DATA values in the environment or paths.yaml must be absolute literal paths; relative paths and ~ are rejected. Explicit --data-dir paths can still be relative. A nonempty XDG_DATA_HOME must also be absolute. If HOME is unavailable or invalid, the paths-file lookup is skipped and an absolute XDG_DATA_HOME can still supply the default root.

Verification and recovery

  • Listings show download bytes (the sum of transport parts for multipart releases) separately from installed bytes.
  • Every file must match the manifest's byte count and SHA-256 checksum.
  • Catalog ID/version pairs and artifact names must be unique without regard to case; receipt.json is reserved.
  • Metadata must be strict JSON: NaN, infinities, and unpaired Unicode surrogates are rejected. Offline verification requires a supported receipt schema version.
  • Cloudy files must have nonempty, finite, strictly increasing grid coordinates and dataset shapes matching THOR's readers. Singleton axes are supported. Ion datasets must use the exact element names supported by THOR (for example, Hydrogen, not H or hydrogen) and declare ion counts from 1 through Z + 1, where Z is the atomic number. Fluorine and Sulfur are canonical; the legacy Flourine and alternate Sulphur spellings are also accepted. THOR resolves these aliases in memory without rewriting the HDF5 files or changing numerical values. A file containing both names for the same element is rejected as ambiguous. Emission datasets must use full line names rather than recognized aliases such as Lya. Both tables must use the same redshift-dimension convention; their grid resolutions can differ.
  • Validation reads every dataset payload in blocks, catching unreadable or corrupt filtered chunks. NumPy blocks are capped at 8 MiB; HDF5 may additionally allocate a full compressed chunk. Validation therefore incurs a full data read.
  • HDF5 datasets may use gzip, shuffle, Fletcher32, n-bit, and scale-offset filters. SZIP and external filters such as LZF are rejected for portability.
  • With --register, registration constraints are checked before downloading. The paths file is updated only after successful installation; constraints are rechecked then to catch intervening changes.
  • Both Cloudy tables are installed together because THOR's dataset loaders initialize the ion and emission readers together.
  • A release becomes visible at its final path only after every file passes. Interrupted transfers are restarted on the next fetch invocation.
  • Repeating fetch verifies an installed release and makes no download requests.
  • Concurrent installers for the same release use a file lock. Shared storage must support file locking and atomic directory renames. A client waiting more than 60 seconds for the lock reports an error and can be retried.
  • Existing damaged or mismatched releases are reported without replacement. Move the affected release directory aside explicitly, then fetch again.
  • receipt.json records the exact manifest and installation time alongside the files. Keep it with the release for offline verification and provenance.

AtomDB tables

The AtomDB preprocessing container exports CIE ion fractions, line emissivities, and continuum into one HDF5 file. Choose temperature and energy ranges that cover your gas and spectral band. PyAtomDB is needed to generate the table, but not to run THOR with it.

See dataset configuration to select lines and ions, or the Fe XXV example for a complete run.

Publishing a catalog

A catalog is a UTF-8 JSON file with schema_version: 2 and a catalogs list. The following is a template, not a downloadable release. Replace the URLs, checksums, sizes, and scientific metadata with verified values before use:

{
  "schema_version": 2,
  "catalogs": [
    {
      "id": "cloudy",
      "version": "v1",
      "description": "Describe the model, grid ranges, units, and log conventions",
      "citation": "Data publication or DOI, plus the appropriate Cloudy citation",
      "provider": {"name": "cloudy", "version": "17.03"},
      "contents": ["ion-fractions", "line-emissivities"],
      "provenance": {
        "uvb": "UV background model",
        "self_shielding": {"model": "none", "parameters": {}},
        "equilibrium": "Describe ionization and thermal assumptions",
        "abundances": "Abundance model and solar metallicity convention",
        "generator": "Generation script commit and configuration reference"
      },
      "files": [
        {
          "name": "grid_ions.hdf5",
          "url": "https://data.example.org/cloudy/v1/grid_ions.hdf5",
          "sha256": "<64 lowercase hexadecimal characters>",
          "size_bytes": 1234,
          "format": "cloudy-ion-v1"
        },
        {
          "name": "grid_emissivities.hdf5",
          "url": "https://data.example.org/cloudy/v1/grid_emissivities.hdf5",
          "sha256": "<64 lowercase hexadecimal characters>",
          "size_bytes": 5678,
          "format": "cloudy-emission-v1"
        }
      ]
    }
  ]
}

Use immutable, directly downloadable HTTPS URLs and checksums calculated from the final files. HTTP is supported only for localhost testing or local mirrors. Downloads follow redirects subject to the same transport rules. Direct files are installed as supplied. Ordered transport parts are verified, concatenated, and optionally decompressed with XZ or gzip; the installed file must then match its own size and checksum. General archive extraction is not supported. Changes to files or release metadata require a new version.

The recognized formats are cloudy-ion-v1, cloudy-ion-v2, cloudy-emission-v1, atomdb-cie-v1, and file. file supports additional reference catalogs with checksum verification only; it does not imply compatibility with a particular scientific reader. Cloudy releases require one ion/emission pair and the provenance fields illustrated above. Additional provenance fields are preserved and displayed.

The cloudy-ion-v2 format stores HI and HII in /Hydrogen and optionally H₂ in /molecules/H2. THOR reads the atomic ions but does not yet expose an H₂ field. Existing v1 files remain supported.

The datasets share the same grid, with an extra dimension for the two atomic ion stages:

/Hydrogen      [2, redshift, dens, metal, temp]
/molecules/H2     [redshift, dens, metal, temp]

Omit the redshift dimension from both datasets for tables without redshift interpolation. H₂ values are the log10 fraction of hydrogen nuclei bound in H₂, 2*n(H2)/n(H).

To convert a temet table with HI, HII, and H₂ components, run this from the THOR repository root with NumPy and h5py installed. Replace the paths below with your source file and a new destination in an existing directory:

python - <<'PYTHON'
from python.data_generation.cloudy.catalog_layout import convert_temet_ions

convert_temet_ions("/path/to/temet/grid_ions.hdf5", "/path/to/grid_ions_v2.hdf5")
PYTHON

The converter separates the third hydrogen component into H2, preserving values and dtypes without renormalizing HI or HII. It writes a new file with conversion provenance and leaves the source unchanged.

Publish converted files under a new catalog version.

For Cloudy, the HDF5 root attributes dens, metal, and temp define the coordinate arrays, with optional redshift. Emission datasets have shape [redshift?, dens, metal, temp]; ion datasets have shape [ion, redshift?, dens, metal, temp] and a positive num_ions or NumIons attribute. Tables must be self-contained. Format validation cannot establish that a model is scientifically suitable; the publisher must validate the physical assumptions and numerical values before releasing it.

To add an approved release to THOR's default list, add its manifest entry to python/thor-rt/src/thor/prepare/catalog.json. Tests use tiny synthetic tables and a local HTTP server, so CI does not download production catalogs.

Package tables for GitHub Releases

The publisher command prepares local assets; it does not contact GitHub or upload anything. Start with scientifically validated tables and a reviewed metadata.json containing one schema-2 catalog entry, without files (not the surrounding catalogs list). Include provenance.license describing the data's distribution license in addition to the scientific fields above. Packaging does not choose a license or default selection for you.

thor-tools prepare package \
  --ions /path/to/grid_ions.hdf5 \
  --emission /path/to/grid_emissivities.hdf5 \
  --metadata metadata.json \
  --release-url https://github.com/OWNER/REPO/releases/download/cloudy-c17-fg11-v1 \
  --output release-cloudy-c17-fg11-v1

The output directory must not already exist. It contains:

  • grid_ions.hdf5.xz.part0000, and more numbered parts if needed.
  • Equivalent emission-table assets.
  • catalog.json, with ordered asset URLs/checksums and original HDF5 checksums.
  • SHA256SUMS, covering the transport assets and catalog.

Each transport part defaults to at most 1,900,000,000 bytes, below GitHub's 2 GiB limit. --asset-bytes can select a smaller bound. The package checks all assets against the strict 2 GiB limit and limits the release to 1,000 assets including the manifest and checksum list. These checks assume a dedicated release; account for any other assets if combining packages in one GitHub release.

Lossless transport compression preserves the original HDF5 byte-for-byte. Embedded provenance_json is required in both inputs and retained in the manifest. For THOR/temet generation records (run.schema_version must be the integer 1, with a run.identity.temet_commit identifier), packaging also rejects different generation identities or contradictory provider/version, UVB, abundances, and shielding metadata. Full physical validation and review of the remaining metadata are the publisher's responsibility. Provenance may contain internal paths or host details; review it before public distribution.

Packaging uses temporary copies and compressed streams: allow free space for both original tables plus roughly twice the compressed size. Fetching needs space for installed tables plus the compressed stream and one part. Transfers use bounded memory, restart after interruption, and reuse verified installations. Final HDF5 paths and THOR configuration are independent of transport splitting.

Publication workflow:

  1. Prepare and review the local directory and catalog.json.
  2. Create a draft GitHub release at the tag used in --release-url.
  3. Upload all files in that directory as release assets.
  4. Publish after review, preferably with release immutability enabled.
  5. Test prepare fetch <id> --version <version> --manifest catalog.json from a fresh data directory, followed by prepare verify.
  6. Add the reviewed catalog entry to THOR's bundled registry. Select a default separately, if desired. Changes to published bytes or metadata need a new version.

An authorized human or trusted publishing workflow performs the GitHub upload; the packaging command does not bypass repository write policies. Private-release authentication is not implemented by the downloader: use public downloadable assets for the current distribution path. Packaging accepts either a Cloudy ion/emission pair or one AtomDB CIE table.

Run the focused tests from the repository root:

PYTHONPATH=python/thor-rt/src python -m pytest \
  python/thor-rt/tests/common/test_prepare.py \
  python/thor-rt/tests/common/test_prepare_atomdb.py

Set THOR_PREPARE_TEST_BINARY to a Release thor executable to also load the downloaded tables through a small Gadget/Voronoi simulation. The PythonCLI_prepare CTest entry enables this automatically for builds with Python regressions and Voronoi support enabled.

Transport compression

New packages default to XZ (LZMA2, preset 6). Python's standard-library lzma module decompresses assets during prepare fetch; no external xz executable is needed. Existing gzip and uncompressed catalogs remain supported. Use prepare package --compression gzip when targeting older THOR fetchers that only understand gzip; the manifest records encoding: "xz" or "gzip" explicitly. Both compressed formats use ordered transport parts and the same size and SHA-256 checks before installation. Compression changes download size; THOR still reads the reconstructed HDF5 files and installed disk usage is unchanged.

One rolling data release

Multiple catalog versions can share a single preprocessing-data release in thor-rt/preprocessing-data. Give each package a unique versioned prefix:

thor-tools prepare package \
  --ions /path/to/grid_ions.hdf5 \
  --emission /path/to/grid_emissivities.hdf5 \
  --metadata metadata.json \
  --release-url https://github.com/thor-rt/preprocessing-data/releases/download/preprocessing-data \
  --asset-prefix cloudy-c17.03-fg11-rahmati2013-lg-v1 \
  --output release-cloudy-c17-fg11-v1

The prefix applies to every upload: compressed transport parts, catalog.json, and SHA256SUMS. Installed HDF5 filenames stay unchanged; the manifest maps the versioned assets to them. Upload each catalog's manifest under its own versioned name, so adding a second catalog does not replace the first one's metadata.

Once published, keep every version's files and contents unchanged. Add a new versioned prefix for corrections; do not overwrite existing assets. THOR's bundled registry can aggregate these versioned catalog entries for discovery. The 1,000-asset limit is shared by the entire rolling release: the local package check cannot count previously uploaded files, so publishers must account for them.

AtomDB catalogs

atomdb-cie-v1 contains one exported CIE table with ion fractions, line emissivities, and continuum. Fetching it prints an atomdb configuration with table_path and atomdb_version; choose the lines and ions in your dataset configuration. Fetch and offline verification check the HDF5 schema, units, nonnegative coefficients, normalized fractions, and stored reconstruction errors. No PyAtomDB installation is required to consume a catalog.

Package a table locally with the same command used for Cloudy:

thor-tools prepare package \
  --atomdb cluster-cie.h5 \
  --metadata metadata.json \
  --release-url https://github.com/thor-rt/preprocessing-data/releases/download/preprocessing-data \
  --asset-prefix atomdb-3.1.3-cie-ag89-fe-k-v1 \
  --output release-draft

Use --atomdb instead of the --ions/--emission pair. Publisher metadata uses provider atomdb, the exact database version, and contents ["ion-fractions", "line-emissivities", "continuum"]. Record uvb: "none", equilibrium: "low-density CIE", abundances: "AG89", and self_shielding: {"model": "none", "parameters": {}}, plus the generator, citation, and applicable data license. Packaging checks these conventions against the table and retains its exporter configuration and input hashes.

The command creates compressed parts, a catalog manifest, and checksums; it does not upload. After publication, use prepare fetch with the published manifest. A release should enter the bundled catalog only after its assets are available.