Testing

This project employs a comprehensive testing strategy utilizing CTest to organize both regression and unit tests.

• Unit Tests: Verify the correctness of individual components (functions, classes) in isolation. These are written in C++ using the Catch2 framework.
• Regression Tests: End-to-end simulations that ensure overall simulation results are consistent and correct over time. These are typically bash scripts that run the full thor executable against known configurations.

See the respective subsections to learn more.

The repository is build and tested on a self-hosted machine via github actions. Some results with automatically show as pull request messages, but most information about build, tests, and coverage is submitted to CDash. More information on CDash below.

Unit Tests

Unit tests are written using the Catch2 framework and are designed to verify individual components of the codebase. Test code is located in the tests/ directory.

Tags

Tests are grouped with specific tags for granular control:

• [cpu] - CPU-only tests (for benchmarks)
• [gpu] - GPU-required tests (for benchmarks)
• [.benchmark] - Performance benchmarks (hidden by default)
• [.LongRunning] - Long-running tests (hidden by default)

Running Unit Tests

First, build the thor_tests target. Afterwards, you can run the tests from your build directory:

cd <build_directory>
./tests/thor_tests

You can also run specific tests using tags or names:

# Specific test categories
./tests/thor_tests "[cpu]"
./tests/thor_tests "[gpu]"
./tests/thor_tests "[.benchmark]"  # Explicit for hidden tags

# Specific test names
./tests/thor_tests "UniformGrid*"

Regression Tests

Regression tests are end-to-end tests that ensure the overall system behaves as expected. These tests are primarily bash scripts located in regression_tests/CI that execute the thor application with specific configurations and validate its output.

The scripts with a test_*.sh prefix are automatically registered as CTest tests. These scripts must accept an optional argument for the path to the thor executable.

Running Regression Tests

Manually

You can run a regression test script directly:

cd regression_tests/CI/shellmodel
bash test_peel.sh PATH_TO_THOR_BUILD/thor

With CTest

CTest is the recommended way to run the entire test suite. From your build directory:

cd <build_directory>
ctest

This will automatically discover and run all registered tests, including regression tests.

You can also run specific tests or use other CTest options:

# Run tests with names matching a regex
ctest -R <test_name_regex>

# See verbose output from tests
ctest --verbose

CDash

CDash is a web-based software testing dashboard. This project is configured to submit test results to a public CDash instance, providing a visual overview of the project's health.

CDash Submission

The CI is configured to automatically submit results to CDash. It is also possible to trigger this manually. The .github/workflows/cmake-cdash.yml workflow is responsible for this.

To manually trigger a CDash submission, you can use the manual_cdash.sh script. This script uses gh act to run the GitHub Actions workflow locally.

bash manual_cdash.sh

You need to have gh and act installed for this to work. You will also need a .env_cdash file with a CDASH_AUTH_TOKEN.

The CDash server is set to the kitware-owned my.cdash.org instance.