Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 0 additions & 8 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,3 @@ updates:
dependencies:
patterns:
- "*"
- package-ecosystem: pip
directory: /docs
schedule:
interval: monthly
groups:
docs:
patterns:
- "*"
6 changes: 2 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ jobs:
run: TOOL_BIN="$PWD/zig-out/bin/eessi-arch-guard" python -m unittest discover -s tests -v

docs:
# The scripts behind the documented output; docs.yml builds the site.
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
Expand All @@ -74,10 +75,6 @@ jobs:
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- name: build documentation
run: |
python -m pip install -r docs/requirements.txt
make -C docs html
- name: run documentation examples
run: |
python docs/examples/walkthrough.py mismatch
Expand All @@ -93,6 +90,7 @@ jobs:
python docs/examples/failure_cases.py verbose
python docs/examples/failure_cases.py bad-timeout
python docs/examples/shell_demo.py
python docs/examples/strict_workspace.py
python docs/examples/accelerator.py mismatch
python docs/examples/accelerator.py missing-module
bash docs/examples/check_install.sh
Expand Down
64 changes: 64 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Build the documentation the way `pixi run -e docs docbld` does: Org export,
# API pages from the code, Sphinx with warnings and unresolved references as
# errors, the Zig autodoc site, and the internal link check. External links
# are checked in their own step. On main the site is published to
# https://stackhands.github.io/eessi-arch-guard/.
name: docs

on:
push:
branches: [main]
pull_request:
workflow_dispatch:
schedule:
# Weekly, so a link that went dead elsewhere is noticed.
- cron: "17 5 * * 1"

permissions:
contents: read

concurrency:
group: docs-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: prefix-dev/setup-pixi@9936258fee697aff89b3fed6ca1f80a382ef694e # v0.11.3
with:
environments: docs
cache: true
locked: true

- name: Build
run: pixi run -e docs docbld
env:
DOCS_SOURCE_REF: ${{ github.sha }}

- name: External links
run: pixi run -e docs linkcheck
env:
GITHUB_TOKEN: ${{ github.token }}

- uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
with:
path: docs/build/html

deploy:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1
36 changes: 0 additions & 36 deletions .github/workflows/pages.yml

This file was deleted.

15 changes: 14 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,21 @@ dist/
build/
.ruff_cache/
.venv/
docs/_build/
docs/examples/_work/
src/eessi-arch-guard.conf
.zig-cache/
zig-out/
zig-pkg/

# Documentation build (docs-plan template): generated, never committed.
/docs/build/
/docs/source/**/*.rst
/docs/source/reference/api/
/docs/source/**/*.png
/docs/source/**/*.svg
/docs/source/**/*.jpg
/docs/source/**/*.webp
!/docs/source/_static/**
/docs/**/__pycache__/
/.pixi/
/.lycheecache
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Changelog

<!-- towncrier release notes start -->

## [0.2.0] - 2026-11-14

### Features
Expand Down
33 changes: 25 additions & 8 deletions COMPAT.md

Large diffs are not rendered by default.

24 changes: 12 additions & 12 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,17 @@ A Slurm **TaskProlog** is still the place that can fix the environment. It is th

The program:

1. Decides, in Python, whether this node is the target the loaded modules were built for. CPU detection is `eessi_archdetect.sh -a cpupath` and then the first candidate that has `<EESSI_PREFIX>/software/<EESSI_OS_TYPE>/<candidate>/software`. Accelerator detection is `eessi_archdetect.sh accelpath`, compared raw with `EESSI_ACCEL_SUBDIR`. The `ccNN` → `ccN0` fallback stays inside the EESSI module, which is what a reload is for.
2. On a mismatch, restores modules in a helper: the same file, exec'd as `python3 -I <file> --restore`, with a process environment that contains only `PATH`, `HOME`, `LANG`, `LC_ALL`, and `TMPDIR`. The task environment is data on stdin, not the helper's process environment. Module names are argv to `LMOD_CMD`, after a whitelist check. Bash is started as `bash --noprofile --norc` and only after `BASH_ENV`, `ENV`, `SHELLOPTS`, `BASHOPTS`, `BASH_XTRACEFD`, and `PS4` have been removed. Its stdout is `env -0`, not a shell script the prolog will print.
3. Emits the **delta** of that restore, and only lines Slurm can apply safely. Names match `^[A-Za-z_][A-Za-z0-9_]*$`. In the default (prolog) mode, values must not contain CR, LF, or NUL. A value that cannot be carried aborts the whole restore: the task keeps the environment it arrived with, and the prolog sets `EESSI_ARCH_GUARD_FAILED`. Prompt variables (`PS1`, `PS2`, `PS3`, `PROMPT_COMMAND`) are the exception: an unsafe new value is left unchanged instead of failing the restore. `SLURM_*`, `BASH_FUNC_*`, `TMPDIR`, `BASH_ENV`, and shell bookkeeping (`_`, `SHLVL`, `PWD`, `OLDPWD`, `SHELLOPTS`, `BASHOPTS`) are never emitted.
4. Refuses module names that fail `^[A-Za-z0-9_][A-Za-z0-9._+-]*(/[A-Za-z0-9_][A-Za-z0-9._+-]*)+$` (length at most 256). The EESSI entry must be exactly one `EESSI/<version>` name. A bad name is a failed restore, not a string interpolated into a shell.
5. Fails loud, and still exits 0. A missing software directory, a broken archdetect, an unusable `LMOD_CMD`, an unsafe name, or a module that will not load sets `EESSI_ARCH_GUARD_FAILED` and prints one line. `EESSI_ARCH_GUARD=0` is the opt-out (silent). `EESSI_ARCH_GUARD_MISSING=ignore` is the narrower opt-out for a node that has no EESSI software directory. Exit 0 is required: on current Slurm master, a non-zero TaskProlog status is `_exit(status)` and the task never runs. Loud means the user and the environment can see the failure. It does not mean killing the task.
6. `--shell` prints `export`/`unset` with `shlex.quote` on stdout, for `eval "$(eessi-arch-guard --shell)"`. The human message goes to stderr, so command substitution does not capture it and does not need an `echo` inside the eval. Prolog mode uses Slurm's `print` line, which master writes to the task's stdout. Those channels differ because Slurm has no stderr form of `print`.
7. One deadline covers the lock wait, both archdetect runs, and the restore helper. The default is 60 seconds (`DEFAULT_TIMEOUT`), Slurm's `DEFAULT_UNKILLABLE_TIMEOUT`. `--timeout SEC` overrides a `timeout = SEC` line in the root-owned config. `SEC` is a positive number of seconds at most CPython's `TIMEOUT_MAX` (`threading.TIMEOUT_MAX`); anything else keeps the default and warns. The children run in their own sessions. `SIGTERM` is held until the child process group is known, then released. At the deadline or on `SIGTERM` the guard SIGKILLs that group, sets `EESSI_ARCH_GUARD_FAILED` to `->timeout` or `->terminated`, exits 0, and does not cache that failure. After the kill the guard still waits up to `POST_KILL_WAIT_SECONDS` for the pipe to close. A result already in the cache is returned unchanged. A decision that has finished is still the result if the signal arrives while it is published.
1. Decides, in Python, whether this node is the target the loaded modules were built for. CPU detection is `eessi_archdetect.sh -a cpupath` on the processors the task may run on (the intersection of its affinity, its cpuset and the Slurm bind list) and then the first candidate that has `<EESSI_PREFIX>/software/<EESSI_OS_TYPE>/<candidate>/software`. Accelerator detection is `eessi_archdetect.sh accelpath`, compared raw with `EESSI_ACCEL_SUBDIR`. The `ccNN` → `ccN0` fallback stays inside the EESSI module, which is what a reload is for.
2. On a mismatch, restores modules in a helper: the same file, exec'd as `python3 -I -X utf8 <file> --restore` (with `--strict` added when the guard has it), with a process environment that contains only `PATH`, `LANG`, `LC_ALL`, `HOME`, `TMPDIR`, and `EESSI_ARCH_GUARD_WORKDIR`; the last three name one new private directory in the restore workspace. The task environment is data on stdin, not the helper's process environment. Module names are argv to `LMOD_CMD`, after a whitelist check. Bash is started as `bash --noprofile --norc` and only after `BASH_ENV`, `ENV`, `SHELLOPTS`, `BASHOPTS`, `BASH_XTRACEFD`, and `PS4` have been removed. Its stdout is `env -0`, not a shell script the prolog will print.
3. Emits the **delta** of that restore, and only lines Slurm can apply safely. Names match `^[A-Za-z_][A-Za-z0-9_]*$`. In the default (prolog) mode, values must not contain CR, LF, or NUL, and no line, newline included, may pass the 4094 bytes Slurm 17.11 reads. A value that cannot be carried aborts the whole restore: the task keeps the environment it arrived with, and the prolog sets `EESSI_ARCH_GUARD_FAILED`. Prompt variables (`PS1`, `PS2`, `PS3`, `PROMPT_COMMAND`) are the exception: an unsafe new value is left unchanged instead of failing the restore. `SLURM_*`, `BASH_FUNC_*`, `TMPDIR`, `BASH_ENV`, and shell bookkeeping (`_`, `SHLVL`, `PWD`, `OLDPWD`, `SHELLOPTS`, `BASHOPTS`) are never emitted.
4. Refuses module names that fail `^[A-Za-z0-9_][A-Za-z0-9._+-]*(/[A-Za-z0-9_][A-Za-z0-9._+-]*)+$` (length at most 256), except a four-digit year such as `2026`, which matches `^[0-9]{4}$`. The EESSI entry must be exactly one `EESSI/<version>` name. A bad name is a failed restore, not a string interpolated into a shell.
5. Fails loud, and still exits 0. A missing software directory, a broken archdetect, an unusable `LMOD_CMD`, an unsafe name, or a module that will not load sets `EESSI_ARCH_GUARD_FAILED` and prints one line. `EESSI_ARCH_GUARD=0` is the opt-out (silent). `EESSI_ARCH_GUARD_MISSING=ignore` is the narrower opt-out for a node that has no EESSI software directory. Exit 0 is required: on current Slurm master, a non-zero TaskProlog status is `_exit(status)` and the task never runs. Loud means the user and the environment can see the failure. It does not mean killing the task. `--strict` is for a user's job wrapper that wants a status: a failed decision or an unexpected exception exits 78 instead. Slurm starts a TaskProlog with no arguments, so a site prolog keeps exit 0.
6. `--shell` prints `export`/`unset` with `shlex.quote` on stdout, for `eval "$(eessi-arch-guard --shell)"`. The human message goes to stderr, so command substitution does not capture it and does not need an `echo` inside the eval. Prolog mode uses Slurm's `print` line, which master writes to the task's stdout, and only local task 0 (`SLURM_LOCALID` unset or `0`) writes it, so a step with many tasks on a node shows it once. Those channels differ because Slurm has no stderr form of `print`.
7. One deadline covers the lock wait, both archdetect runs, and the restore helper. The default is 60 seconds (`DEFAULT_TIMEOUT`), Slurm's `DEFAULT_UNKILLABLE_TIMEOUT`. `--timeout SEC` overrides a `timeout = SEC` line in the root-owned config. `SEC` is a positive number of seconds at most CPython's `TIMEOUT_MAX` (`threading.TIMEOUT_MAX`); anything else keeps the default and warns. Above 2147483.647 seconds the Python guard accepts the value but fails every decision that runs a command, a bug COMPAT.md lists with the unintended differences. The children run in their own sessions. `SIGTERM` is held until the child process group is known, then released. At the deadline or on `SIGTERM` the guard SIGKILLs that group, sets `EESSI_ARCH_GUARD_FAILED` to `->timeout` or `->terminated`, exits 0 (78 with `--strict`), and does not cache that failure. After the kill the guard still waits up to `POST_KILL_WAIT_SECONDS` for the pipe to close. A result already in the cache is returned unchanged. A decision that has finished is still the result if the signal arrives while it is published.

The helper command is `python3 -I -X utf8`. Each name and value crosses as one Latin-1 character per environment byte, and `-X utf8` turns that text back into those bytes for Lmod.

One decision is cached per uid, node, job, and step, under `TMPDIR` if that directory exists and otherwise `/tmp`. The cache key is the one from the original design: `EESSI_SOFTWARE_SUBDIR`, `EESSI_ACCEL_SUBDIR`, `LOADEDMODULES`, `MODULEPATH`, `EESSI_INIT_PREFIX`, `EESSI_PROC_CPUINFO`, `PATH`. `EESSI_ARCH_GUARD` is not part of the key. Adding it would change the contract; two tasks in one step that differ only by that variable share a result. The cache file is mode `0600`, created with `O_NOFOLLOW` and `O_EXCL`, and published with `rename`. The lock file is opened the same way. A symlink is not followed. If the cache cannot be used safely, the task computes its own result and does not write through the link.
One decision is cached per uid, node, job, step, and output mode (prolog or `--shell`), under `TMPDIR` if that is a safe absolute directory and otherwise `/tmp`. With `--strict` the cache is in the private restore workspace instead (`EESSI_ARCH_GUARD_WORKDIR`, a private `TMPDIR`, or `eessi-arch-guard` under `XDG_STATE_HOME` or `~/.local/state`), and without one nothing is cached. The cache key is the one from the original design, `EESSI_SOFTWARE_SUBDIR`, `EESSI_ACCEL_SUBDIR`, `LOADEDMODULES`, `MODULEPATH`, `EESSI_INIT_PREFIX`, `EESSI_PROC_CPUINFO`, `PATH`, plus the processors the task may run on and the GPU variables `CUDA_VISIBLE_DEVICES`, `ROCR_VISIBLE_DEVICES`, `GPU_DEVICE_ORDINAL`, `SLURM_STEP_GPUS`, and `SLURM_GPUS_ON_NODE`, so two tasks of one step on different processors or GPUs do not share a result. `EESSI_ARCH_GUARD` is not part of the key. Adding it would change the contract; two tasks in one step that differ only by that variable share a result. The cache file is mode `0600`, created with `O_NOFOLLOW` and `O_EXCL`, and published with `rename`. The lock file is opened the same way. A symlink is not followed. A cache or lock file is trusted only when the open file is a regular file owned by the caller, with one link and no group or other permission bits. If the cache cannot be used safely, the task computes its own result and does not write through the link.

Chaining does not trust the job environment. Slurm execs a TaskProlog with no arguments, so a user cannot pass `--chain`. A site either wraps the existing prolog, or writes `chain = /absolute/path` in `/etc/eessi-arch-guard.conf` (or a file of that name beside the program). The file is used only when it is a regular file, owned by root, and not writable by group or others. Symlinks are ignored. Each key is decoded on its own. A value that is not UTF-8 is kept with surrogateescape, and the other keys still apply. `--chain PATH` exists for a root-owned wrapper and for tests.

Expand Down Expand Up @@ -93,16 +93,16 @@ These were rejected before this repo existed. The new information does not reviv
| Module name fails the grammar | Loud failure. Restore is not started. |
| A module has no build for this target | Loud failure. Nothing from the restore is applied. When one load of all the modules fails, they are loaded again one call each, in order, so a module from a tree an earlier module adds (EESSI-extend and the user's installations) still loads, and the message names every module that fails rather than the first Lmod error. The list stops after 32. |
| A changed variable cannot be expressed as a prolog line | Loud failure. No partial delta. |
| Any unexpected exception | One skip line, exit 0. |
| The decision is still running at the deadline | `->timeout`, the waited-on process group is SIGKILLed, exit 0, nothing cached. |
| `SIGTERM` during the decision | `->terminated`, the same kill, exit 0, nothing cached. |
| Any unexpected exception | One skip line, exit 0 (78 with `--strict`). |
| The decision is still running at the deadline | `->timeout`, the waited-on process group is SIGKILLed, exit 0 (78 with `--strict`), nothing cached. |
| `SIGTERM` during the decision | `->terminated`, the same kill, exit 0 (78 with `--strict`), nothing cached. |
| Chained prolog exits non-zero | That status is returned and this program does not restore. Slurm will kill the task, which is what that prolog already did. |

`EESSI_ARCH_GUARD_RELOADED` is `was->now`. When the accelerator differs, each side is `<cpu> <accel>` or `<cpu> no accel`. `EESSI_ARCH_GUARD_FAILED` uses the same shape, with a fixed token (`no-software-dir`, `archdetect-failed`, `unsafe-module-name`, `unsafe-environment`, `lmod-unavailable`, `restore-failed`, `timeout`, `terminated`) when there is no selected target.

## Install shape

Runtime dependency is Python 3.9+ standard library, executed as `python3 -I`. The TaskProlog target is the single file `src/eessi_arch_guard.py` (shebang `#!/usr/bin/python3 -I`). `pyproject.toml` packages that module for a login-node `--shell` install. The generated console script does not pass `-I`; the prolog should be the file itself.
Runtime dependency is Python 3.9+ standard library, executed as `python3 -I`. The TaskProlog target is the single file `src/eessi_arch_guard.py` (shebang `#!/usr/bin/python3 -I`). `pyproject.toml` packages that module for a login-node `--shell` install. The generated console script does not pass `-I`; the prolog should be the file itself. The Zig build in `zig/` is a static binary with the same interface, whose restore helper is the binary itself with `--restore`; COMPAT.md lists where it differs. The Python file stays the default.

## Intel A100 CPU subdirectory

Expand Down
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,13 @@

eessi-arch-guard is a Slurm TaskProlog that reloads an EESSI module stack when the task runs on a CPU or GPU other than the target recorded at `module load`. The EESSI module chooses that target once. Lmod keeps the recorded `MODULEPATH` and the `EBROOT*` paths until the stack is purged and loaded again. The prolog does that purge and load on the compute node, before `execve`, and prints the environment delta for `slurmstepd` to apply.

Documentation: <https://stackhands.github.io/eessi-arch-guard/>, a Sphinx site built from `docs/`: a tutorial, how-to guides, topics, and a reference, including the generated Python API.
Documentation: <https://stackhands.github.io/eessi-arch-guard/>, built from the Org pages in `docs/orgmode/`: a tutorial, how-to guides, explanation, and a reference, including the API generated from the Python and Zig code.

```console
python3 -m pip install -r docs/requirements.txt
make -C docs html
pixi run -e docs docbld
```

The HTML is written to `docs/_build/html`.
The HTML is written to `docs/build/html`. `docs/orgmode/contributing.org` says how to write a page.

## Install the prolog

Expand Down
Loading
Loading