Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ __pycache__/
*.py[cod]
.venv/
build/
docs/_build/
.zig-cache/
zig-out/
zig-pkg/
59 changes: 49 additions & 10 deletions README.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions changelog.d/+build-secrets.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`build.secrets`, in a configuration file or a profile, and `EB_LOCAL_BUILD_SECRETS`, for one build, name the secrets a build gets: the build reads only those, so a secret it does not get never reaches `eb`, and one that cannot be read no longer stops it. Without either, every build gets every secret, as before. A name that no secret in force has is an error saying where it was set. `eb-local config` shows the setting, `config --env` carries it, `--explain` says which secrets reach `eb`, and `doctor` says FAIL only for a secret builds get, and otherwise warns. The documentation has a how-to page.
1 change: 1 addition & 0 deletions changelog.d/+install-path-check.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A build and `eb-local eb` check where EESSI-extend will install once it has loaded, and stop with exit 2, before `eb` runs, when `EASYBUILD_INSTALLPATH` is outside the tree the install variable selects, `<prefix>/versions` for a build, or when `EASYBUILD_INSTALLPATH_SOFTWARE` or `EASYBUILD_INSTALLPATH_MODULES` is set. Before, a site Lmod hook that set an install path, or an `EESSI_CVMFS_REPO` with a character such as `-`, which EESSI-extend matches as a Lua pattern, sent the install and its logs outside the prefix eb-local had checked and cleans of secrets while eb-local said it was installing into that prefix, and `eb-local eb` passed a leftover `EASYBUILD_INSTALLPATH_MODULES` on to EasyBuild, splitting an install between two trees. Both now say when loading EESSI-extend sets or unsets an install variable, as a site load hook does too late for its own load. A build names each install variable it clears from the caller's shell, and the refusal of more than one install variable by `eb-local eb` names each with its value and what EESSI-extend would do with them: with a user and a project variable, a private user install.
1 change: 1 addition & 0 deletions changelog.d/+install-variables-docs.doc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The documentation has a page on where EESSI-extend installs under each install variable and each mix of them, as the module did under Lmod, on the trap a variable left from an earlier session sets, and on what eb-local checks. Pages besides the README are written in Org under `docs/orgmode` and exported to the MyST pages Sphinx builds with `emacs --batch -Q -l docs/export.el`, which needs only the Markdown backend Org has; a test checks the committed pages against a fresh export where Emacs is installed.
1 change: 1 addition & 0 deletions changelog.d/+node-local-refusals.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The refusals of a build's node-local mount and install prefix, by a build, `eb-local clean` and `eb-local doctor`, now say what was refused and why, what the command then did not do, and, on the next line, what to change: the setting, with its value and where it was set as `eb-local config` shows it, or the variable, option or command. Before, most said only that the allocation's node-local root "must be an owned writable directory" or that paths "must use durable owned storage", without the path, the setting or the reason. A workspace that cannot be made in a mount that passed the checks names the error the kernel gave, and a setting in a 1.x `eb-local.conf` is shown with its own line. The documentation has a page listing each refusal.
95 changes: 95 additions & 0 deletions docs/explanation/install-variables.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
<!-- Generated from docs/orgmode/explanation/install-variables.org by docs/export.el; edit that file. -->

# Where EESSI-extend installs

EESSI-extend chooses the tree EasyBuild installs into when the module loads, from four install variables.
eb-local loads it in two ways: a build sets one variable itself, and `eb-local eb` uses the ones the shell has.
This page says where each variable, and each mix of them, sends an install, why a variable left from an earlier session is a trap, and what eb-local checks.

The results below come from the EESSI-extend module that EESSI's software-layer-scripts generate (commit `169e720f`), loaded under Lmod 8.6.19 on a stand-in EESSI 2026.06 tree for the CPU target `x86_64/amd/zen4`, with `HOME=/home/alice`.

## One install variable

| Set before the load | EESSI-extend installs into | umask, and bit |
|---|---|---|
| `EESSI_USER_INSTALL=DIR` | `DIR/versions/2026.06/software/linux/x86_64/amd/zen4` | 077, sticky |
| `EESSI_PROJECT_INSTALL=DIR` | `DIR/versions/2026.06/software/linux/x86_64/amd/zen4` | 002, setgid |
| `EESSI_SITE_INSTALL` | `$EESSI_SITE_SOFTWARE_PATH`, the site tree in `host_injections` | 022 |
| `EESSI_CVMFS_INSTALL` | `$EESSI_SOFTWARE_PATH`, EESSI's own tree | 022 |
| none | `/home/alice/eessi/versions/2026.06/software/linux/x86_64/amd/zen4` | 077, sticky |

The module tests each variable with `os.getenv(name) ~= nil`, so a variable that is present and empty counts as set.
A named user or project directory must exist, or the module refuses to load; `$HOME/eessi` need not.
The module puts the tree's `modules/all` first on `MODULEPATH`.

## Two at once

| Set before the load | What EESSI-extend does |
|---|---|
| `EESSI_USER_INSTALL` and `EESSI_PROJECT_INSTALL` | loads, installs into the user tree with umask 077 and the sticky bit, and puts both module trees on `MODULEPATH`, the user's first |
| `EESSI_SITE_INSTALL` and a user or project variable | refuses to load |
| `EESSI_CVMFS_INSTALL` and any other | refuses to load |

The first row is the trap.
A `.bashrc` that exports `EESSI_USER_INSTALL`, or `eval "$(eb-local env)"` run earlier for a user prefix, leaves the variable in the shell.
A project install typed by hand there, `export EESSI_PROJECT_INSTALL=/path/to/project/eessi` and then `module load EESSI-extend`, becomes a private user install that the rest of the project cannot read.
Lmod says nothing about it but a second line, `Configuring for use of EESSI_USER_INSTALL under /home/alice/eessi`, after the project one.

## What eb-local does with the variables

A build clears every install variable, sets `EESSI_USER_INSTALL` or `EESSI_PROJECT_INSTALL` to its prefix, and loads EESSI-extend afresh, so what the shell carries does not reach it.
It names each one it cleared:

```text
eb-local: EESSI_USER_INSTALL=/home/alice/eessi is set in this shell; this build clears it, but EESSI-extend or eb run by hand would use it
```

`eb-local eb` uses the variables the shell has, so it refuses two or more, with exit 3, and says what EESSI-extend would do with them:

```text
eb-local: EESSI_USER_INSTALL=/home/alice/eessi and EESSI_PROJECT_INSTALL=/path/to/project/eessi are both set; EESSI-extend would make a private user install (umask 077) under /home/alice/eessi, not a project install under /path/to/project/eessi.
Unset the one an earlier session left (a shell startup file, or eval "$(eb-local env)" for another prefix):
set at most one of EESSI_USER_INSTALL, EESSI_PROJECT_INSTALL, EESSI_SITE_INSTALL, EESSI_CVMFS_INSTALL; with none set, the module installs under $HOME/eessi
```

`eb-local doctor` lists every install variable the shell has, and says which one names your prefix.

## The install path, once the module has loaded

EESSI-extend sets `EASYBUILD_INSTALLPATH` to the tree in the first table.
Once it has loaded, a build and `eb-local eb` stop with exit 2, before `eb` runs, unless that path is in the tree the install variable selects.
For a build that tree is `<prefix>/versions`, which eb-local checked before the build, lists afterwards and cleans of secret values; for `eb-local eb` it is the tree in the first table.
They also stop when `EASYBUILD_INSTALLPATH_SOFTWARE` or `EASYBUILD_INSTALLPATH_MODULES` is set, since EasyBuild would then put the software or the modules somewhere else while `MODULEPATH` names EESSI-extend's tree.

Three things move the install path:

- a site Lmod hook that sets one, which the README's [A site Lmod hook](../index.md#a-site-lmod-hook) warns against;
- a changed EESSI-extend;
- an `EESSI_CVMFS_REPO` with a character such as `-`.

The last is EESSI-extend's own.
It makes a user or project path with `string.gsub(EESSI_SOFTWARE_PATH, EESSI_CVMFS_REPO, DIR)`, and Lua reads the second argument as a pattern, where `-`, `+`, `*`, `?`, `%`, `(`, `)`, `[` and `^` are not themselves.
With a copy of the repository under `/opt/eessi-mirror` the pattern does not match, the path is left as it was, and the install would go into the repository's own tree.
The default, `/cvmfs/software.eessi.io`, has only `.`, which matches itself among the rest.
A build there used to say `installing into /home/alice/eessi (user)` while EasyBuild was given the repository's tree; it now stops:

```text
eb-local: loading EESSI-extend/2026.06-easybuild set the install path to /opt/eessi-mirror/versions/2026.06/software/linux/x86_64/amd/zen4, outside /home/alice/eessi/versions, the tree eb-local checked; nothing was built.
EESSI_USER_INSTALL did not choose it: a site Lmod hook, a changed EESSI-extend, or a character such as - in EESSI_CVMFS_REPO, which EESSI-extend matches as a Lua pattern, did. See "Where EESSI-extend installs" in the README.
```

## A variable set as the module loads

A site Lmod hook that sets an install variable when EESSI-extend loads runs after the module has read the variables, so the load it runs in installs where it would have anyway.
The value stays in the shell after `module unload`, and the next load installs there.
With a hook that sets `EESSI_USER_INSTALL=/home/alice/.local/eessi` when no install variable is set, the first load in a shell installed under `/home/alice/eessi`, and a second one, after `module unload EESSI-extend`, under `/home/alice/.local/eessi`; `MODULEPATH` then held both trees.

eb-local cannot tell a variable a hook left in an earlier load from one set on purpose.
It does see a load that sets or unsets one, and says so:

```text
eb-local: loading EESSI-extend set EESSI_USER_INSTALL=/home/alice/.local/eessi; this install still goes under /home/alice/eessi, but a shell that loads EESSI-extend by hand keeps the change, and its next load installs elsewhere.
A site Lmod hook that sets an install variable acts only on the next load. See "Where EESSI-extend installs" in the README.
```

A site that wants another default prefix sets it before EESSI-extend loads, in a login script, and as `prefix` in eb-local's site file, as the README's [A site Lmod hook](../index.md#a-site-lmod-hook) shows.
136 changes: 136 additions & 0 deletions docs/export.el
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
;;; export.el --- Export docs/orgmode to the MyST pages Sphinx builds -*- lexical-binding: t; -*-

;; The pages under docs/ that have an Org source are written from it:
;; docs/orgmode/PART/NAME.org becomes docs/PART/NAME.md. Edit the Org
;; file and run, from the repository:
;;
;; emacs --batch -l docs/export.el
;;
;; EB_LOCAL_DOCS_OUT names another directory to write into, which is how
;; the tests compare a fresh export with the committed pages. Only Org's
;; own Markdown backend is used, so any Emacs 29 can run this.

;;; Code:

(require 'ox-md)
(require 'subr-x)

(defconst eb-local-docs-root
(file-name-directory (or load-file-name buffer-file-name))
"The docs/ directory.")

(defun eb-local-docs--fence (lang code)
"CODE in a fenced block, tagged LANG when there is one."
(format "```%s\n%s```\n" (or lang "") (org-remove-indentation code)))

(defun eb-local-docs--src-block (src-block _contents info)
(eb-local-docs--fence (org-element-property :language src-block)
(org-export-format-code-default src-block info)))

(defun eb-local-docs--example-block (example-block _contents info)
(eb-local-docs--fence "text" (org-export-format-code-default example-block info)))

(defun eb-local-docs--fixed-width (fixed-width _contents _info)
(eb-local-docs--fence "text" (org-element-property :value fixed-width)))

(defun eb-local-docs--table-cell (_cell contents _info)
(replace-regexp-in-string "|" "\\\\|" (string-trim (or contents ""))))

(defun eb-local-docs--table-row (row _contents info)
"A pipe-table row, and the separator after the header, or after the
first row of a table that has none."
(unless (eq (org-element-property :type row) 'rule)
(let* ((cells (org-element-contents row))
(table (org-export-get-parent-table row))
(header (or (org-export-table-row-ends-header-p row info)
(and (not (org-export-table-has-header-p table info))
(= (org-export-table-row-number row info) 0)))))
(concat "| "
(string-join (mapcar (lambda (cell) (org-export-data cell info)) cells) " | ")
" |\n"
(when header
(concat "|" (string-join (make-list (length cells) "---") "|") "|\n"))))))

(defun eb-local-docs--table (_table contents _info)
"The rows, without the blank lines Org's Markdown puts between elements."
(replace-regexp-in-string "\n\\(?:[ \t]*\n\\)+" "\n" contents))

(defun eb-local-docs--item (item contents info)
"A description item as a MyST definition, anything else as Org's Markdown has it."
(let ((tag (org-element-property :tag item)))
(if (not tag)
(org-md-item item contents info)
(format "%s\n: %s\n" (org-export-data tag info)
(replace-regexp-in-string "\n\\(.\\)" "\n \\1" (string-trim (or contents "")))))))

(defun eb-local-docs--plain-list (plain-list contents info)
"A description list as definitions with a blank line between them."
(if (eq (org-element-property :type plain-list) 'descriptive)
(mapconcat (lambda (item) (org-export-data item info))
(org-element-contents plain-list) "\n")
(org-md-plain-list plain-list contents info)))

(defun eb-local-docs--link (link desc info)
"Org's Markdown link, keeping a #anchor search option on a file link."
(let ((search (org-element-property :search-option link)))
(if (and (string= (org-element-property :type link) "file")
search (string-prefix-p "#" search))
(let ((path (org-element-property :path link)))
(when (string-suffix-p ".org" path)
(setq path (concat (file-name-sans-extension path) ".md")))
(format "[%s](%s%s)" (or desc path) path search))
(org-md-link link desc info))))

(defun eb-local-docs--template (contents info)
"The page: where it comes from, its title, then its contents."
(format "<!-- Generated from docs/orgmode/%s by docs/export.el; edit that file. -->\n\n# %s\n\n%s"
(plist-get info :eb-local-source)
(org-export-data (plist-get info :title) info)
contents))

(defun eb-local-docs--tidy (text _backend _info)
"One blank line between blocks, and a single newline at the end."
(concat (string-trim (replace-regexp-in-string "\n\\(?:[ \t]*\n\\)\\{2,\\}" "\n\n" text)) "\n"))

(org-export-define-derived-backend 'eb-local-myst 'md
:translate-alist '((src-block . eb-local-docs--src-block)
(example-block . eb-local-docs--example-block)
(fixed-width . eb-local-docs--fixed-width)
(table . eb-local-docs--table)
(table-row . eb-local-docs--table-row)
(table-cell . eb-local-docs--table-cell)
(item . eb-local-docs--item)
(plain-list . eb-local-docs--plain-list)
(link . eb-local-docs--link)
(template . eb-local-docs--template))
:filters-alist '((:filter-final-output . eb-local-docs--tidy)))

(defun eb-local-docs-export (out)
"Write each Org page under docs/orgmode as MyST under OUT."
(let ((src (expand-file-name "orgmode" eb-local-docs-root))
(org-md-headline-style 'atx)
(org-md-toplevel-hlevel 2)
(org-export-with-toc nil)
(org-export-with-section-numbers nil)
(org-export-with-author nil)
(org-export-with-date nil)
(org-export-with-sub-superscripts '{})
(org-export-time-stamp-file nil)
(make-backup-files nil))
(dolist (org (directory-files-recursively src "\\.org\\'"))
(let* ((rel (file-relative-name org src))
(md (expand-file-name (concat (file-name-sans-extension rel) ".md") out)))
(make-directory (file-name-directory md) t)
(with-temp-buffer
(insert-file-contents org)
(setq default-directory (file-name-directory org))
(org-mode)
(let ((text (org-export-as 'eb-local-myst nil nil nil
(list :eb-local-source rel))))
(with-temp-file md (insert text))))
(message "%s" md)))))

(when noninteractive
(eb-local-docs-export (or (getenv "EB_LOCAL_DOCS_OUT") eb-local-docs-root)))

;;; export.el ends here
65 changes: 65 additions & 0 deletions docs/howto/scope-secrets.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
<!-- Generated from docs/orgmode/howto/scope-secrets.org by docs/export.el; edit that file. -->

# Give a secret only to the builds that need it

Every build gets every secret the configuration defines, unless `build.secrets` names the ones it gets.
A build reads only the secrets it gets, so one it does not get cannot stop it, and its value never reaches `eb` or anything `eb` runs.
These steps keep a licence and a source token away from the builds that do not use them.
The secrets are defined in `[secrets]`, or as `EB_LOCAL_SECRET_<NAME>` variables, as [Secrets](../index.md#secrets) describes.

## Give no secret by default

Set `build.secrets` to an empty list in your file, beside the secrets:

```toml
# ~/.config/eb-local/config.toml
[build]
secrets = []

[secrets]
LM_LICENSE_FILE = "nimvault:licences/flexlm"
SOURCE_TOKEN = "file:~/.secrets/source-token"
```

`eb-local config` now shows `build.secrets` as `(none)`, and where it was set.
A site file can set it in the same way for everyone; your file then overrides it.

## Name the secrets of a kind of build in a profile

```toml
[profile.licensed]
build.secrets = ["LM_LICENSE_FILE"]
```

```sh
eb-local --profile licensed MATLAB-2026a.eb
```

The profile's list replaces the one above it; it does not add to it.

## Name the secrets of one build

`EB_LOCAL_BUILD_SECRETS` names them, separated by spaces, in place of `build.secrets`; set but empty, it names none:

```sh
EB_LOCAL_BUILD_SECRETS=SOURCE_TOKEN eb-local Private-1.0-GCC-15.2.0.eb
```

The `sbatch` lines that `eb-local --explain` prints for each node type run the build with the environment of the shell that runs them, so export the variable in that shell instead.

## Check what a build gets

```sh
eb-local config # build.secrets, and where it was set
eb-local --explain MATLAB-2026a.eb # which secrets reach eb, and which stay out
eb-local doctor # which secrets can be read, and which would stop a build
```

A name that no secret in force has stops eb-local, which says where the name was set:

```text
eb-local: build.secrets [user /home/alice/.config/eb-local/config.toml:2] names TOKEN, but no secret TOKEN is defined: add it to [secrets], or set EB_LOCAL_SECRET_TOKEN=SOURCE
```

With a site file, `eb-local doctor` says FAIL for a secret that cannot be read only when builds get it; one they do not get is a warning.
Without `build.secrets` anywhere, `doctor` suggests naming the secrets builds need when one of them cannot be read, since every build would stop there.
5 changes: 4 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,15 @@
```{include} ../README.md
:start-after: <!-- docs:start -->
:end-before: <!-- stackhands:begin -->
:relative-docs: docs/
```

```{toctree}
:hidden:
:maxdepth: 1


howto/scope-secrets
reference/refusals
explanation/install-variables
```

Loading
Loading