diff --git a/AGENTS.md b/AGENTS.md index e6047bfc..659331bb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -71,9 +71,9 @@ lint findings remain, and do not comment on the PR. ## Version-one CLI ```text -package inspect | plan | bump +package inspect | plan | bump | latest recipe easyblock | templates | hierarchy | check | lint | format -stack solve | sbom | order | easystack | sections | from-commit | gap | provided | shadow | batch | shard | footprints | admit | arch | files | generation | cuda-cc | known-failures | oom | bot-classify +stack solve | sbom | order | easystack | sections | from-commit | gap | provided | shadow | batch | shard | capacity | footprints | admit | arch | files | generation | cuda-cc | known-failures | oom | bot-classify target list | doctor campaign run | status | finding claim | finding resolve repro summary diff --git a/changelog.d/+docs-drift.doc.md b/changelog.d/+docs-drift.doc.md new file mode 100644 index 00000000..822cce42 --- /dev/null +++ b/changelog.d/+docs-drift.doc.md @@ -0,0 +1 @@ +The docs match the code again: `AGENTS.md` lists `package latest` and `stack capacity`; the `cli.org` MCP list names `eb_stack_footprints` and `eb_stack_admit`; `docs/campaign-boundary.md` cites the current `src/campaign.rs` and `src/foreign.rs` lines and the `run_target` calls; the shard how-to says to plan with EESSI-extend's `EASYBUILD_FILTER_DEPS`, or pass that list as `--filter-deps`; the PyPI and CRAN how-tos say the robot tree is `$EBROOTEASYBUILD/easybuild/easyconfigs` from EESSI's EasyBuild module; the commented pins in the EESSI example policies name the EESSI 2026.06 bundles; and the `provides` and `campaign` API docs have no broken link or stray `///`. diff --git a/docs/campaign-boundary.md b/docs/campaign-boundary.md index b00dada5..1ee20414 100644 --- a/docs/campaign-boundary.md +++ b/docs/campaign-boundary.md @@ -1,23 +1,23 @@ # Campaign Execution Boundary Documentation -*Generated from the current repository state (commit `1fd9fb568f60a5531527e82181ee18ce14b36c10`).* +*Line references checked against commit `72ea4e5534c176ba79898f466e7cfbd9e5ff9f19`.* ## 1. Direct Executions performed by `src/campaign.rs` | Action | Command / Function | Conditions | File:Line Reference | |--------|-------------------|------------|---------------------| -| Read campaign bundle manifest | `read_json(&manifest_path)` | Always after acquiring lock | `src/campaign.rs:268-269` | -| Extract package name & version from manifest | `manifest.pointer("/package/name")` and `manifest.pointer("/package/version")` | Must exist in manifest | `src/campaign.rs:270-277` | -| Load recipes list | `load_campaign_recipes(&request.bundle)` | Must return at least one recipe | `src/campaign.rs:278-283` | -| Discover Resolvo profile locks | `discover_files(&request.bundle.join("locks"), "json")` | Must find at least one lock file | `src/campaign.rs:284-288` | -| Initialise or load campaign state JSON | `read_json(&request.state_path)` or construct new `CampaignState` | If state file exists, schema must match | `src/campaign.rs:291-319` | -| Record interrupted previous attempt (if any) | `record_interrupted_attempt(&mut state, &request.state_path)` | When previous state was `Running` | `src/campaign.rs:321`, function at `src/campaign.rs:800` | -| Write updated state to disk (atomic) | `write_state(&request.state_path, &state)` | After each state transition | `src/campaign.rs:1377-1390` | -| Pre-flight packaging gate for each recipe | `packaging_gate(&resolved, &[])` | Runs before any build; on error creates a `BuildFinding` with class `Checksum` or `Unknown` | `src/campaign.rs:356` | -| Stage bundle on target | `request.target.stage_bundle(&request.bundle)` | May fail with transport error | `src/campaign.rs:402` | -| Build each recipe on target | `request.target.build_command_with_overlay(...)` then `command.execute()` | Executes EasyBuild command; on failure creates a `BuildFinding` | `src/campaign.rs:532` | -| Verify binaries for each profile | `request.target.verification_command(&program, &args)` then `command.execute()` | Runs verification commands; on failure creates a `BuildFinding` | `src/campaign.rs:629` | -| Finalise campaign state as `Completed` | Set `state.status = CampaignStatus::Completed` and write state | After all builds and verification succeed | `src/campaign.rs:700` | +| Read campaign bundle manifest | `read_json(&manifest_path)` | Always after acquiring lock | `src/campaign.rs:279-280` | +| Extract package name & version from manifest | `manifest.pointer("/package/name")` and `manifest.pointer("/package/version")` | Must exist in manifest | `src/campaign.rs:281-288` | +| Load recipes list | `load_campaign_recipes(&request.bundle)` | Must return at least one recipe | `src/campaign.rs:289-294` | +| Discover Resolvo profile locks | `discover_files(&request.bundle.join("locks"), "json")` | Must find at least one lock file | `src/campaign.rs:295-300` | +| Initialise or load campaign state JSON | `read_json(&request.state_path)` or construct new `CampaignState` | If state file exists, schema must match | `src/campaign.rs:302-332` | +| Record interrupted previous attempt (if any) | `record_interrupted_attempt(&mut state, &request.state_path)` | When previous state was `Running` | `src/campaign.rs:349`, function at `src/campaign.rs:844` | +| Write updated state to disk (atomic) | `write_state(&request.state_path, &state)` | After each state transition | `src/campaign.rs:1515-1529` | +| Pre-flight packaging gate for each recipe | `packaging_gate(&resolved, &[])` | Runs before any build; on error creates a `BuildFinding` with class `Checksum` or `Unknown` | `src/campaign.rs:384` | +| Stage bundle on target | `run_target.stage_bundle(&request.bundle)` | May fail with transport error | `src/campaign.rs:435` | +| Build each recipe on target | `run_target.build_command_with_overlay(...)` then `command.execute()` | Executes EasyBuild command; on failure creates a `BuildFinding` | `src/campaign.rs:571-576` | +| Verify binaries for each profile | `run_target.verification_command(&program, &args)` then `command.execute()` | Runs verification commands; on failure creates a `BuildFinding` | `src/campaign.rs:669-670` | +| Finalise campaign state as `Completed` | Set `state.status = CampaignStatus::Completed` and write state | After all builds and verification succeed | `src/campaign.rs:744` | ## 2. Files Written (but not executed) and Their Consumers @@ -25,7 +25,7 @@ |------|-------------|----------------------| | `state_path` (e.g. `campaign.json`) | Persistent JSON representation of the campaign state, including history, findings, claims | Human operators, other campaign runs, debugging tools | | `state_path.lock` and `state_path.lock.guard` | Exclusive lock metadata to prevent concurrent campaigns | `CampaignLock::acquire` ensures only one process owns the lock | -| Temporary `*.tmp` files during state writes | Atomic write-then-rename to avoid partial state files | The `write_state` helper (`src/campaign.rs:1377-1390`) | +| Temporary `*.tmp` files during state writes | Atomic write-then-rename to avoid partial state files | The `write_state` helper (`src/campaign.rs:1515-1529`) | | `campaign.lock` metadata JSON (`*.lock`) | Records host, PID, start ticks for the lock holder | Lock acquisition / release logic (`CampaignLock::acquire` and `Drop`) | | `campaign.findings` entries inside the state JSON | Records failures, interruptions, and their evidences | Human reviewers, `eb-stack campaign finding` commands | | `campaign.history` entries inside the state JSON | Chronological log of actions (e.g., "build evaluation on ") | Auditing, debugging | @@ -36,19 +36,19 @@ | Error Variant | Situation | Where Raised (File:Line) | |---------------|-----------|--------------------------| -| `CampaignError::InvalidBundle` | Missing required fields in bundle (e.g., no recipes, no locks, missing manifest entries) | `src/campaign.rs:273-288` | -| `CampaignError::UnsupportedSchema` | State file schema version does not match `CAMPAIGN_SCHEMA_VERSION` | `src/campaign.rs:293-294` | -| `CampaignError::StateIdentity` | Existing state file refers to a different package/version than bundle | `src/campaign.rs:296-297` | -| `CampaignError::Busy` | Another process owns the campaign lock (`CampaignLock::acquire`) | `src/campaign.rs:1414-1416` (inside `CampaignLock::acquire`) | -| `CampaignError::Target` (wraps `TargetError`) | A target command that cannot be started is stored as a finding; `run_campaign` returns `Ok` with status `Failed` | Variant at `src/campaign.rs:1528`. Call sites `src/campaign.rs:402`, `444`, `537`, and `630` go through `record_target_command_failure` | -| `CampaignError::Io` / `Json` | File system or JSON (de)serialization failures | `read_json` at `src/campaign.rs:1371-1375` and `write_state` at `src/campaign.rs:1377-1390` | -| `CampaignError::FindingNotFound`, `FindingOwned`, `FindingState` | Operations on campaign findings that are invalid (e.g., claiming an already-owned finding) | `src/campaign.rs:843-876` (`claim_finding`) and `src/campaign.rs:881-910` (`resolve_finding`) | +| `CampaignError::InvalidBundle` | Missing required fields in bundle (e.g., no recipes, no locks, missing manifest entries) | `src/campaign.rs:284-299` | +| `CampaignError::UnsupportedSchema` | State file schema version does not match `CAMPAIGN_SCHEMA_VERSION` | `src/campaign.rs:304-305` | +| `CampaignError::StateIdentity` | Existing state file refers to a different package/version than bundle | `src/campaign.rs:307-308` | +| `CampaignError::Busy` | Another process owns the campaign lock (`CampaignLock::acquire`) | `src/campaign.rs:1553-1555` (inside `CampaignLock::acquire`) | +| `CampaignError::Target` (wraps `TargetError`) | A target command that cannot be started is stored as a finding; `run_campaign` returns `Ok` with status `Failed` | Variant at `src/campaign.rs:1668`. Call sites `src/campaign.rs:485`, `579`, and `673` go through `record_target_command_failure`; a staging failure at `src/campaign.rs:435` is recorded as a `Transport` finding in place | +| `CampaignError::Io` / `Json` | File system or JSON (de)serialization failures | `read_json` at `src/campaign.rs:1509-1513` and `write_state` at `src/campaign.rs:1515-1529` | +| `CampaignError::FindingNotFound`, `FindingOwned`, `FindingState` | Operations on campaign findings that are invalid (e.g., claiming an already-owned finding) | `src/campaign.rs:888-921` (`claim_finding`) and `src/campaign.rs:926-955` (`resolve_finding`) | ### In `src/foreign.rs` | Error Variant | Situation | Where Raised (File:Line) | |---------------|-----------|--------------------------| -| `ForeignError::Parse` | Recipe could not be parsed as YAML or Python abstract syntax tree (AST) | `src/foreign.rs:575-598` (conda parsing) and `src/foreign.rs:1418` (Spack parsing) | +| `ForeignError::Parse` | Recipe could not be parsed as YAML or Python abstract syntax tree (AST) | `src/foreign.rs:579-602` (conda parsing) and `src/foreign.rs:1474` (Spack parsing) | | `ForeignError::Unsupported` | File extension/name not recognised as a supported foreign format | `src/foreign.rs:354` (`parse_foreign_path`; detection is `detect_foreign_format` at `src/foreign.rs:289`) | | `ForeignError::Io` | Underlying file read error | `src/foreign.rs:351` (`parse_foreign_path`) | diff --git a/docs/orgmode/howto/cran-extend.org b/docs/orgmode/howto/cran-extend.org index 64bd50b2..97c3ee57 100644 --- a/docs/orgmode/howto/cran-extend.org +++ b/docs/orgmode/howto/cran-extend.org @@ -10,7 +10,10 @@ dependency on the EasyBuild ~R~ module. Packages already listed in * Inputs 1. a DESCRIPTION file, ~cran.json~, or a one-spec-per-line package list; -2. the European Environment for Scientific Software Installations (EESSI) robot tree; +2. the European Environment for Scientific Software Installations (EESSI) robot tree, + passed as ~--easyconfigs~. It is the easyconfigs directory bundled with + EESSI's EasyBuild: ~$EBROOTEASYBUILD/easybuild/easyconfigs~ after + loading EESSI's ~EasyBuild~ module; 3. a stack policy that locks ~R~ and ~R-bundle-CRAN~ to the versions in the robot. See ~examples/stacks/eessi-r-extras.toml~. diff --git a/docs/orgmode/howto/pypi-extend.org b/docs/orgmode/howto/pypi-extend.org index 9908511e..4a204d61 100644 --- a/docs/orgmode/howto/pypi-extend.org +++ b/docs/orgmode/howto/pypi-extend.org @@ -17,7 +17,10 @@ toolchain-built extensions and has no claim on the European Environment for Scie * Inputs 1. Warehouse-shaped JSON (~info~ + ~urls~) or a ~requirements.txt~; -2. the EESSI robot tree (or a fixture that mirrors its provides); +2. the EESSI robot tree (or a fixture that mirrors its provides), passed + as ~--easyconfigs~. It is the easyconfigs directory bundled with EESSI's + EasyBuild: ~$EBROOTEASYBUILD/easybuild/easyconfigs~ after loading + EESSI's ~EasyBuild~ module; 3. a stack policy that *locks* ~Python~, ~SciPy-bundle~, and ~Python-bundle-PyPI~ to the versions in the robot. See ~examples/stacks/eessi-python-extras.toml~. diff --git a/docs/orgmode/howto/stack-shard.org b/docs/orgmode/howto/stack-shard.org index ccd97725..e49f149e 100644 --- a/docs/orgmode/howto/stack-shard.org +++ b/docs/orgmode/howto/stack-shard.org @@ -88,6 +88,14 @@ in that file. ~--weights~ is ~Name/VersionFull~ or the easyconfig filename, a tab, then seconds. A module the file does not list weighs ~--default-weight~, which is 1. Build the prelude, then the shards. +On an EESSI build host, shard with the dependency filter the build +uses. EESSI-extend sets ~EASYBUILD_FILTER_DEPS~ to the packages EESSI +takes from its compatibility layer, such as binutils, zlib and XZ, and +~stack shard~ reads that variable when ~--filter-deps~ is absent. When +planning without that module loaded, pass the same comma-separated +list as ~--filter-deps~. Otherwise those packages count as missing, and +the ones several shards share land in the prelude. + * Lift CPU-only prerequisites of accelerator roots EESSI installs accelerator packages under an ~/accel/~ prefix and diff --git a/docs/orgmode/reference/cli.org b/docs/orgmode/reference/cli.org index 62d7185f..2c05f650 100644 --- a/docs/orgmode/reference/cli.org +++ b/docs/orgmode/reference/cli.org @@ -2230,7 +2230,7 @@ plus sandboxed recipe and easyblock tools: - ~eb_stack_solve~, ~eb_stack_sbom~, ~eb_stack_order~, ~eb_stack_easystack~, ~eb_stack_gap~, ~eb_stack_provided~, ~eb_stack_shadow~, ~eb_stack_batch~, ~eb_stack_shard~, ~eb_stack_capacity~, - ~eb_stack_arch~, ~eb_stack_files~, + ~eb_stack_footprints~, ~eb_stack_admit~, ~eb_stack_arch~, ~eb_stack_files~, ~eb_stack_generation~, ~eb_stack_cuda_cc~, ~eb_stack_known_failures~, ~eb_stack_oom~, ~eb_stack_bot_classify~, ~eb_stack_sections~, ~eb_stack_from_commit~; diff --git a/examples/stacks/eessi-python-extras.toml b/examples/stacks/eessi-python-extras.toml index 9a98937a..2228720e 100644 --- a/examples/stacks/eessi-python-extras.toml +++ b/examples/stacks/eessi-python-extras.toml @@ -11,10 +11,10 @@ version = "2026.1" # [[pins]] # name = "SciPy-bundle" -# version_requirement = "==2025.06" +# version_requirement = "==2026.05" # mode = "locked" # # [[pins]] # name = "Python-bundle-PyPI" -# version_requirement = "==2025.04" +# version_requirement = "==2026.04" # mode = "locked" diff --git a/examples/stacks/eessi-r-extras.toml b/examples/stacks/eessi-r-extras.toml index f82c1e98..5e924b42 100644 --- a/examples/stacks/eessi-r-extras.toml +++ b/examples/stacks/eessi-r-extras.toml @@ -10,10 +10,10 @@ version = "2026.1" # [[pins]] # name = "R" -# version_requirement = "==4.4.2" +# version_requirement = "==4.6.1" # mode = "locked" # # [[pins]] # name = "R-bundle-CRAN" -# version_requirement = "==2025.06" +# version_requirement = "==2026.07" # mode = "locked" diff --git a/llms-full.txt b/llms-full.txt index 5591be69..7c0f29f4 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -1179,7 +1179,10 @@ toolchain-built extensions and has no claim on the European Environment for Scie # Inputs 1. Warehouse-shaped JSON (`info` + `urls`) or a `requirements.txt`; -2. the EESSI robot tree (or a fixture that mirrors its provides); +2. the EESSI robot tree (or a fixture that mirrors its provides), passed + as `--easyconfigs`. It is the easyconfigs directory bundled with EESSI's + EasyBuild: `$EBROOTEASYBUILD/easybuild/easyconfigs` after loading + EESSI's `EasyBuild` module; 3. a stack policy that **locks** `Python`, `SciPy-bundle`, and `Python-bundle-PyPI` to the versions in the robot. See `examples/stacks/eessi-python-extras.toml`. @@ -1339,7 +1342,10 @@ dependency on the EasyBuild `R` module. Packages already listed in # Inputs 1. a DESCRIPTION file, `cran.json`, or a one-spec-per-line package list; -2. the European Environment for Scientific Software Installations (EESSI) robot tree; +2. the European Environment for Scientific Software Installations (EESSI) robot tree, + passed as `--easyconfigs`. It is the easyconfigs directory bundled with + EESSI's EasyBuild: `$EBROOTEASYBUILD/easybuild/easyconfigs` after + loading EESSI's `EasyBuild` module; 3. a stack policy that locks `R` and `R-bundle-CRAN` to the versions in the robot. See `examples/stacks/eessi-r-extras.toml`. @@ -5834,6 +5840,14 @@ in that file. `--weights` is `Name/VersionFull` or the easyconfig filename, a tab, then seconds. A module the file does not list weighs `--default-weight`, which is 1. Build the prelude, then the shards. +On an EESSI build host, shard with the dependency filter the build +uses. EESSI-extend sets `EASYBUILD_FILTER_DEPS` to the packages EESSI +takes from its compatibility layer, such as binutils, zlib and XZ, and +`stack shard` reads that variable when `--filter-deps` is absent. When +planning without that module loaded, pass the same comma-separated +list as `--filter-deps`. Otherwise those packages count as missing, and +the ones several shards share land in the prelude. + # Lift CPU-only prerequisites of accelerator roots EESSI installs accelerator packages under an `/accel/` prefix and @@ -8889,7 +8903,7 @@ plus sandboxed recipe and easyblock tools: - `eb_stack_solve`, `eb_stack_sbom`, `eb_stack_order`, `eb_stack_easystack`, `eb_stack_gap`, `eb_stack_provided`, `eb_stack_shadow`, `eb_stack_batch`, `eb_stack_shard`, `eb_stack_capacity`, - `eb_stack_arch`, `eb_stack_files`, + `eb_stack_footprints`, `eb_stack_admit`, `eb_stack_arch`, `eb_stack_files`, `eb_stack_generation`, `eb_stack_cuda_cc`, `eb_stack_known_failures`, `eb_stack_oom`, `eb_stack_bot_classify`, `eb_stack_sections`, `eb_stack_from_commit`; diff --git a/src/campaign.rs b/src/campaign.rs index 8fd0e025..f9065d3a 100644 --- a/src/campaign.rs +++ b/src/campaign.rs @@ -86,7 +86,7 @@ pub enum BuildFindingClass { /// The job exceeded its time limit. Timeout, /// The output did not match any known signature. Left unclassified - /// /// rather than guessed at, so it reaches a human. + /// rather than guessed at, so it reaches a human. Unknown, } @@ -155,7 +155,7 @@ pub struct BuildFinding { /// The command that failed, so it can be rerun by hand. pub command: CommandPlan, /// Exit status, when the process produced one. Absent when it was - /// /// killed or never started. + /// killed or never started. pub exit_code: Option, /// Attempt number this arose on. pub attempt: u32, diff --git a/src/provides.rs b/src/provides.rs index 06377b07..9f6b7622 100644 --- a/src/provides.rs +++ b/src/provides.rs @@ -8,7 +8,7 @@ //! `R-bundle-CRAN`. //! //! Expansion is idempotent. Synthetic candidates are marked by -//! [`crate::domain::Candidate::EXT_PROVIDE_MARKER`] in `easyconfig_path` and +//! [`EXT_PROVIDE_MARKER`] in `easyconfig_path` and //! depend on the parent bundle at its exact version. use crate::domain::{Candidate, DepReq, ExtEntry};