Variance and Resource Usage of Computational Experiments

This repository contains the thesis documents and experimental software developed for the Bachelor's Thesis "Variance and Resource Usage of Computational Experiments" at the University of Basel.

The project investigates runtime variation and allocated resource usage in repeated Fast Downward experiments on a shared Slurm-based HPC system. The experimental pipeline uses JSON configurations, Slurm job arrays, Apptainer containers and Python scripts for parsing and plotting results.

Repository structure

.
├── README.md
├── LICENSE
├── CITATION.cff
├── requirements.txt
├── docs/
│   ├── thesis.pdf
│   └── presentation.pdf
├── experiment-runner/
│   ├── benchrun.py
│   ├── experiments/
│   ├── generation/
│   ├── parsers/
│   └── plots/
└── container/
    ├── Apptainer.fd_24_06_1_cplex
    └── cplex.local.example

Requirements

• Python 3.10 or newer
• Slurm
• Apptainer
• Fast Downward 24.06.1
• Fast Downward benchmark tasks
• Matplotlib for plotting

Operator-counting experiments additionally require IBM ILOG CPLEX. CPLEX is proprietary and is not included in this repository.

Install the Python plotting dependency with:

python3 -m pip install -r requirements.txt

Running an experiment

The main command-line interface is experiment-runner/benchrun.py.

cd experiment-runner
python3 benchrun.py --help

Generate an experiment from a JSON configuration:

python3 benchrun.py generate \
  --config experiments/container_lmcut_all.json \
  --benchmarks /path/to/downward-benchmarks \
  --container /path/to/fd_24_06_1_cplex.sif

The supplied configurations contain the Slurm partition and QOS used for the original experiments. Override them when running on another cluster:

python3 benchrun.py generate \
  --config experiments/container_lmcut_all.json \
  --benchmarks /path/to/downward-benchmarks \
  --container /path/to/container.sif \
  --partition your_partition \
  --qos your_qos

Submit the generated Slurm array:

python3 benchrun.py submit --name container_lmcut_all

Check its status:

python3 benchrun.py status --name container_lmcut_all

Parse completed results and create the basic plots:

python3 benchrun.py parse-plot --name container_lmcut_all

Create the final analysis plots used in the thesis:

python3 benchrun.py analysis-plots --kind all

Create the CPU-hour and estimated-energy summary:

python3 benchrun.py resource-summary

Planner configurations

• Uniform Cost Search (astar_blind internally)
• A* with the LM-cut heuristic (astar_lmcut)
• LAMA first (lama_first)
• Operator counting with state-equation constraints (astar_opcount_se)

Container and CPLEX

The repository includes an Apptainer definition file but no built .sif image and no CPLEX installer or binaries.

To build the CPLEX-enabled image, obtain the CPLEX installer separately, place it in the container directory as cplex.bin, and run:

cd container
apptainer build --fakeroot fd_24_06_1_cplex.sif Apptainer.fd_24_06_1_cplex

Review the IBM licence terms before building or distributing an image containing CPLEX.

Reproducibility notes

Benchmark tasks, raw experiment outputs and built container images are not included. Paths, Slurm settings and resource limits may need to be adapted for another system. The stored configuration files and scripts document the original experiment definitions, but identical runtimes cannot be guaranteed on different hardware or under different cluster load.

Licence

The original software in this repository is released under the MIT License. Third-party software, including Fast Downward, Apptainer and IBM ILOG CPLEX, remains subject to its own licence terms. The thesis and presentation documents are not covered by the software licence.

Author

Nillan Sivarasa