Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .github/workflows/ci_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,11 @@ jobs:
run: |
cargo test --locked --test no_helper_script
cargo test --locked --test public_metadata
# A site's names and paths belong in its own configuration; release
# notes may name what a release removed.
- name: No site names in the tree
run: |
! git grep -nIiE 'snellius|surf\.nl|/gpfs/|/projects/0|rg\.terra|scratch-shared|scratch-node|eblocalinstall' -- . ':!CHANGELOG.md' ':!changelog.d/' ':!tests/cli_ergonomics.rs' ':!.github/workflows/ci_test.yml'
- name: Cargo source package
run: cargo package --locked

Expand Down Expand Up @@ -343,7 +348,7 @@ jobs:
fi
"$BIN" --version
"$BIN" stack capacity --snapshot tests/planning_capacity_fixtures/snapshot.json \
--partition local --target example/host --feature scratch-node \
--partition local --target example/host --feature local-disk \
--cpus 4 --memory-mib 4096 --exclusive-node true --max-slots-per-node 1 \
--generated-at 2026-01-01T12:00:10Z --max-age-seconds 60 \
> target/portable-capacity-check.json
Expand Down
1 change: 1 addition & 0 deletions changelog.d/+golden-replay-builder.misc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`skills/golden-replay/run-on-builder.sh` has no default builder: set `EB_STACK_BUILDER` to the builder's ssh destination (it was `rg.terra`). Without `EB_STACK_BIN` it runs the debug build of the builder's checkout at the same place under the home directory as the local one, instead of one fixed checkout path.
1 change: 1 addition & 0 deletions changelog.d/+site-neutral.doc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The `stack shard` and `stack capacity` help examples, the shard how-to, the capacity reference and the skills name a neutral example site, not one cluster: a `zen2.list` or `zen4.list` build list, a commit cache under `$SCRATCH`, the node feature `local-disk`, and the site's build host.
4 changes: 2 additions & 2 deletions docs/orgmode/howto/stack-shard.org
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ reads each such line's tree there, downloading the ones it lacks:
eb-stack stack shard \
--build-list site-2026e.list \
--easyconfigs easybuild-easyconfigs/easybuild/easyconfigs \
--commit-cache /scratch-shared/$USER/eb-stack-commit-trees \
--commit-cache $SCRATCH/eb-stack-commit-trees \
--shards 4 \
--out-dir /tmp/shards
#+end_src
Expand Down Expand Up @@ -243,7 +243,7 @@ free:

#+begin_src bash
eb-stack stack shard \
--build-list genoa.list \
--build-list zen4.list \
--easyconfigs easyconfigs \
--shards 7 \
--inode-quota 3000000 --inode-margin 100000 \
Expand Down
2 changes: 1 addition & 1 deletion docs/orgmode/reference/planning-capacity.org
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ complete view, and minimize scheduler RPCs.
#+begin_src sh
eb-stack stack capacity \\
--snapshot tests/planning_capacity_fixtures/snapshot.json \\
--partition local --target example/host --feature scratch-node \\
--partition local --target example/host --feature local-disk \\
--cpus 4 --memory-mib 4096 --exclusive-node true --max-slots-per-node 1 \\
--generated-at 2026-01-01T12:00:10Z
#+end_src
Expand Down
6 changes: 3 additions & 3 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5961,7 +5961,7 @@ reads each such line's tree there, downloading the ones it lacks:
eb-stack stack shard \
--build-list site-2026e.list \
--easyconfigs easybuild-easyconfigs/easybuild/easyconfigs \
--commit-cache /scratch-shared/$USER/eb-stack-commit-trees \
--commit-cache $SCRATCH/eb-stack-commit-trees \
--shards 4 \
--out-dir /tmp/shards
```
Expand Down Expand Up @@ -6033,7 +6033,7 @@ free:

```bash
eb-stack stack shard \
--build-list genoa.list \
--build-list zen4.list \
--easyconfigs easyconfigs \
--shards 7 \
--inode-quota 3000000 --inode-margin 100000 \
Expand Down Expand Up @@ -9172,7 +9172,7 @@ complete view, and minimize scheduler RPCs.
```sh
eb-stack stack capacity \\
--snapshot tests/planning_capacity_fixtures/snapshot.json \\
--partition local --target example/host --feature scratch-node \\
--partition local --target example/host --feature local-disk \\
--cpus 4 --memory-mib 4096 --exclusive-node true --max-slots-per-node 1 \\
--generated-at 2026-01-01T12:00:10Z
```
Expand Down
2 changes: 1 addition & 1 deletion skills/golden-replay/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,4 @@ establish `builds`.
See `skills/annual-bump/SKILL.md` for the bump command.
See `skills/verify-recipe/SKILL.md` for checksum class and source claims.
See `skills/tool-repair/SKILL.md` when emit disagrees with the golden.
See `skills/site-consume/SKILL.md` for site generation and eblocalinstall.
See `skills/site-consume/SKILL.md` for site generation and building a recipe at a site.
29 changes: 18 additions & 11 deletions skills/golden-replay/run-on-builder.sh
Original file line number Diff line number Diff line change
@@ -1,28 +1,35 @@
#!/usr/bin/env bash
# Run argv as the current checkout's eb-stack on the configured builder.
# Paths in argv are evaluated on the builder. There is no default script.
#
# EB_STACK_BUILDER=HOST [EB_STACK_BIN=PATH] run-on-builder.sh package ...
#
# HOST is the builder's ssh destination. PATH is the eb-stack binary there;
# unset, it is the debug build of the builder's checkout at the same place
# under the home directory as this one.
set -euo pipefail

if [[ $# -eq 0 ]]; then
printf 'usage: run-on-builder.sh package bump|plan|inspect ...\n' >&2
printf 'usage: EB_STACK_BUILDER=HOST run-on-builder.sh package bump|plan|inspect ...\n' >&2
exit 2
fi

builder=${EB_STACK_BUILDER:-rg.terra}
builder=${EB_STACK_BUILDER:?set EB_STACK_BUILDER to the builder, an ssh destination}

if [[ -z ${EB_STACK_BIN:-} ]]; then
EB_STACK_BIN=$(ssh -o BatchMode=yes "$builder" 'bash -s' <<'EOS'
checkout=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd -P)
if [[ $checkout != "$HOME"/* ]]; then
printf 'this checkout is not under %s; set EB_STACK_BIN\n' "$HOME" >&2
exit 2
fi
EB_STACK_BIN=$(ssh -o BatchMode=yes "$builder" 'bash -s' -- "$(printf '%q' "${checkout#"$HOME"/}")" <<'EOS'
set -euo pipefail
if [[ -x $HOME/Git/Github/Tools/eb-stack/target/debug/eb-stack ]]; then
printf '%s\n' "$HOME/Git/Github/Tools/eb-stack/target/debug/eb-stack"
bin=$HOME/$1/target/debug/eb-stack
if [[ -x $bin ]]; then
printf '%q\n' "$bin"
exit 0
fi
found=$(rg --files "$HOME/Git" --glob 'target/debug/eb-stack' 2>/dev/null | head -1 || true)
if [[ -n ${found:-} && -x $found ]]; then
printf '%s\n' "$found"
exit 0
fi
printf 'no eb-stack debug binary on the builder\n' >&2
printf 'no eb-stack debug binary at %s on the builder; set EB_STACK_BIN\n' "$bin" >&2
exit 1
EOS
)
Expand Down
2 changes: 1 addition & 1 deletion skills/upstream-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Canonical contribution docs: https://docs.easybuild.io/contributing/

## Canonical sequence (do this order)

On the **configured EasyBuild host** (for this site: `rg.terra`, not the
On the **configured EasyBuild host** (the site's build host, not the
laptop; not a random host without the foss generation installed):

```sh
Expand Down
2 changes: 1 addition & 1 deletion src/inode_budget.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1233,7 +1233,7 @@ mod tests {
"ts": "2026-11-03T10:00:00Z",
"actor": "jenkins/run/1",
"stack": "2025",
"target": "x86_64/amd/genoa",
"target": "x86_64/amd/zen4",
"module": module,
"location": format!("/sw/{module}"),
"action": "install",
Expand Down
4 changes: 2 additions & 2 deletions src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1679,12 +1679,12 @@ enum StackCommand {
/// `inode_footprint` and `max_concurrent_shards`, the most shards that
/// fit under the quota minus `--inode-margin` at once.
#[command(
after_long_help = "Examples:\n eb-stack stack shard --build-list rome.list --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs --easyconfigs site-easyconfigs --installed-modules-dir /cvmfs/software.eessi.io/versions/2026.06/software/linux/x86_64/amd/zen2/modules/all --installed-modules-dir site-modules/all --shards 4 --weights build-seconds.tsv --out-dir shards\n eb-stack stack shard --build-list rome.list --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs --commit-cache /scratch-shared/$USER/eb-stack-commit-trees --shards 4 --out-dir shards\n eb-stack stack shard --build-list rome.list --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs --easyconfigs-repo easybuild-easyconfigs --materialize --build-robot-path /sw/easybuild/easyconfigs --shards 4 --out-dir shards"
after_long_help = "Examples:\n eb-stack stack shard --build-list zen2.list --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs --easyconfigs site-easyconfigs --installed-modules-dir /cvmfs/software.eessi.io/versions/2026.06/software/linux/x86_64/amd/zen2/modules/all --installed-modules-dir site-modules/all --shards 4 --weights build-seconds.tsv --out-dir shards\n eb-stack stack shard --build-list zen2.list --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs --commit-cache $SCRATCH/eb-stack-commit-trees --shards 4 --out-dir shards\n eb-stack stack shard --build-list zen2.list --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs --easyconfigs-repo easybuild-easyconfigs --materialize --build-robot-path /sw/easybuild/easyconfigs --shards 4 --out-dir shards"
)]
Shard(Box<ShardArgs>),
/// Inspect timestamped advisory scheduler capacity without placing jobs.
#[command(
after_long_help = "Examples:\n eb-stack stack capacity --snapshot capacity.json --partition compute --target example/host --feature scratch-node --cpus 4 --memory-mib 4096 --exclusive-node true --max-slots-per-node 1 --generated-at 2026-01-01T12:00:10Z --max-age-seconds 60"
after_long_help = "Examples:\n eb-stack stack capacity --snapshot capacity.json --partition compute --target example/host --feature local-disk --cpus 4 --memory-mib 4096 --exclusive-node true --max-slots-per-node 1 --generated-at 2026-01-01T12:00:10Z --max-age-seconds 60"
)]
Capacity(Box<CapacityArgs>),
/// Write the peak build-tree inode footprint per software name.
Expand Down
8 changes: 4 additions & 4 deletions tests/planning_capacity.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ use sha2::{Digest, Sha256};

fn node(name: &str, state: &[&str]) -> Value {
json!({"name":name,"partitions":["compute"],"target":"example/host",
"available_features":["scratch-node"],"active_features":["scratch-node"],
"available_features":["local-disk"],"active_features":["local-disk"],
"state_flags":state,"cpus_total":16,"cpus_allocatable":16,"cpus_allocated":0,
"memory_mib_total":32768,"memory_mib_allocatable":32768,"memory_mib_allocated":0,
"gpus_total":4,"gpus_allocatable":4,"gpus_allocated":0,"gpu_type":"example-gpu","cpu_load":90.0})
Expand All @@ -15,7 +15,7 @@ fn snapshot(nodes: Vec<Value>) -> Value {
}
fn request() -> CapacityRequest {
serde_json::from_value(json!({"partition":"compute","host_target":"example/host",
"features":["scratch-node"],"cpus":8,"memory_mib":8192,"gpus":1,"gpu_type":"example-gpu","exclusive_node":false}))
"features":["local-disk"],"cpus":8,"memory_mib":8192,"gpus":1,"gpu_type":"example-gpu","exclusive_node":false}))
.unwrap()
}
fn report(value: &Value) -> Value {
Expand Down Expand Up @@ -137,7 +137,7 @@ fn pending_demand_is_scoped_and_unknown_inputs_are_visible() {
let mut value = snapshot(vec![node("node-a", &["IDLE"])]);
value["pending_complete"] = json!(false);
value["pending_jobs"] = json!([
{"id":"one","state":"PENDING","partitions":["compute"],"target":"example/host","features":["scratch-node"],"cpus":8,"memory_mib":8192,"gpus":1},
{"id":"one","state":"PENDING","partitions":["compute"],"target":"example/host","features":["local-disk"],"cpus":8,"memory_mib":8192,"gpus":1},
{"id":"broad","state":"PENDING","partitions":["compute"],"target":null,"features":null,"cpus":4,"memory_mib":null,"gpus":0},
{"id":"other","state":"PENDING","partitions":["elsewhere"],"target":"example/host","features":[],"cpus":900,"memory_mib":900,"gpus":9}]);
let result = report(&value);
Expand All @@ -160,7 +160,7 @@ fn malformed_constraints_duplicates_and_inconsistent_resources_are_not_accepted(
)
.is_err());
let mut req = request();
req.features.insert("scratch-node|other".into());
req.features.insert("local-disk|other".into());
assert!(inspect(
&serde_json::to_vec(&snapshot(vec![])).unwrap(),
&req,
Expand Down
4 changes: 2 additions & 2 deletions tests/planning_capacity_fixtures/snapshot.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
"nodes": [
{
"active_features": [
"scratch-node"
"local-disk"
],
"available_features": [
"scratch-node"
"local-disk"
],
"cpu_load": 0.5,
"cpus_allocatable": 16,
Expand Down
8 changes: 4 additions & 4 deletions tests/planning_capacity_fixtures/storage-policy.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@
"capacity_sha256": "aaa0baa0e64a1b06d4aeb27960ceae593e16f6c417d6392a501e9367c7dc112b",
"capacity_snapshot": "snapshot.json",
"max_age_seconds": 60,
"node_local_feature": "scratch-node",
"node_local_feature": "local-disk",
"profiles": {
"cpu_prerequisites": {
"compile_target": "example/host",
"node_local": {
"cpus": 4,
"exclusive_node": true,
"features": [
"scratch-node"
"local-disk"
],
"gpus": 0,
"host_target": "example/host",
Expand All @@ -37,7 +37,7 @@
"cpus": 4,
"exclusive_node": true,
"features": [
"scratch-node"
"local-disk"
],
"gpus": 0,
"host_target": "example/host",
Expand All @@ -64,7 +64,7 @@
"cpus": 4,
"exclusive_node": true,
"features": [
"scratch-node"
"local-disk"
],
"gpus": 0,
"host_target": "example/host",
Expand Down
2 changes: 1 addition & 1 deletion tests/stack_admit.rs
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ fn footprints_take_the_ledger_peak_then_the_table_then_the_classes() {
"ts": "2026-11-03T10:00:00Z",
"actor": "jenkins/run/1",
"stack": "2025",
"target": "x86_64/amd/genoa",
"target": "x86_64/amd/zen4",
"module": module,
"location": format!("/sw/{module}"),
"action": "install",
Expand Down
2 changes: 1 addition & 1 deletion tests/stack_shard_fixtures/wave1.list
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Wave 1, rome
# Wave 1, zen2
OpenFOAM-13-GCCcore-15.2.0.eb --hooks=/sw/hooks/openfoam.py
ParaView-6.0.0-GCCcore-15.2.0.eb

Expand Down
Loading