Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/+literal-leading-star.doc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Code and verbatim text that starts with `*` shows as written: the log rotation page showed `*.log` and `*.txt` as `\*.log` and `\*.txt`. `docs/export.el` writes such text as a plain reStructuredText literal, and `pixi run -e docs docbld` runs `docs/tools/test_export.py`, which exports a page of literals and checks each one.
1 change: 1 addition & 0 deletions changelog.d/+modules-init-one-name.removal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`check`, and every command that loads the site file, refuses a `modules.init` entry with white space, since an entry is one module name. The module wrapper split such an entry at the white space, so `init = ["A B"]` loaded `A` and `B`, and a pasted `module load A` line loaded `module`, `load` and `A`. Write `init = ["A", "B"]`.
1 change: 1 addition & 0 deletions changelog.d/+modules-shell.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`check`, and every command that loads the site file, refuses a `modules.shell` that cannot run the module wrapper. It accepts `sh`, `bash`, `dash`, `ash`, `ksh`, `ksh93`, `mksh` and `yash`, by name or by path. zsh ran the wrapper without splitting `EB_PIPELINE_MODULES`, so the modules of `modules.init` loaded as one name, and csh, tcsh and fish could not parse it. A site file that names zsh and loads one module, which worked, must name one of those shells.
10 changes: 10 additions & 0 deletions docs/examples/site-file.sh
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,16 @@ run 'eb-pipeline check --site pbs.toml'
expect_status 2
expect_line 'ERROR [check] example: scheduler.kind: expected slurm or local, found `pbs`'

sed 's/^init = \["EasyBuild"\]/init = ["module load EasyBuild"]/' site.toml >init.toml
run 'eb-pipeline check --site init.toml'
expect_status 2
expect_line 'ERROR [check] example: modules.init: `module load EasyBuild` has white space; an entry is one module name, so give each module its own entry'

sed '/^init = /a shell = "zsh"' site.toml >zsh.toml
run 'eb-pipeline check --site zsh.toml'
expect_status 2
expect_line 'ERROR [check] example: modules.shell: expected a POSIX shell that runs the module wrapper, one of sh, bash, dash, ash, ksh, ksh93, mksh or yash, by name or by path, found `zsh`'

sed 's/^compress_after_days = 30/compress_after_days = 400/' site.toml >rotate.toml
run 'eb-pipeline check --site rotate.toml'
expect_status 2
Expand Down
6 changes: 6 additions & 0 deletions docs/export.el
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,12 @@
;; x86_64 and EESSI_USER_INSTALL are names, not subscripts; only _{...} and
;; ^{...} are.
(setq org-export-with-sub-superscripts '{})
;; ox-rst writes a leading * of ~code~ and =verbatim= as \*, but a
;; reStructuredText literal takes no backslash escapes: =*.log= showed as
;; \*.log.
(setq org-rst-text-markup-alist
(append '((code . "``%s``") (verbatim . "``%s``"))
org-rst-text-markup-alist))

(defun stackhands-resolve-deferred (tree _backend _info)
"Replace deferred properties in TREE with their values."
Expand Down
20 changes: 11 additions & 9 deletions docs/orgmode/contributing.org
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ pixi run -e docs docbld

The site is in =docs/build/html/=; open =docs/build/html/index.html=.
The build fails on any Sphinx warning, any cross-reference that does not resolve, any rustdoc warning, any public item of the library without a doc comment, any internal link or anchor that does not exist, and, outside the changelog, any name of a real site's machines or paths.
It also fails when =docs/export.el= writes the code or verbatim text of a test page as a different literal, such as =\*.log= for =*.log=.
The first build needs the network, to fetch the other stackhands sites' inventories; later builds reuse them for 30 days.

External links are checked separately, as they need the network:
Expand All @@ -72,15 +73,16 @@ pixi run -e docs linkcheck

** Where things are

| Path | What it holds |
|------------------------------+--------------------------------------------------------------------|
| =docs/orgmode/= | the pages, in Org: edit these |
| =docs/export.el= | the Org to reStructuredText export |
| =docs/source/conf.py= | the Sphinx configuration |
| =docs/source/_ext/= | the documentation template's Sphinx domain for Zig, unused here |
| =docs/tools/check_site.py= | checks internal links, anchors and forbidden names |
| =docs/tools/api-index.html= | the front page of the rustdoc site, which =cargo doc= does not write |
| =docs/examples/= | the scripts that print the output the pages show |
| Path | What it holds |
|-----------------------------+----------------------------------------------------------------------|
| =docs/orgmode/= | the pages, in Org: edit these |
| =docs/export.el= | the Org to reStructuredText export |
| =docs/source/conf.py= | the Sphinx configuration |
| =docs/source/_ext/= | the documentation template's Sphinx domain for Zig, unused here |
| =docs/tools/check_site.py= | checks internal links, anchors and forbidden names |
| =docs/tools/test_export.py= | checks the literals that =docs/export.el= writes |
| =docs/tools/api-index.html= | the front page of the rustdoc site, which =cargo doc= does not write |
| =docs/examples/= | the scripts that print the output the pages show |

The =.rst= files under =docs/source/= and everything under =docs/build/= are generated and not committed.
=pixi run -e docs rustdoc= runs =cargo doc= into =docs/build/cargo/= and copies the result to =docs/build/html/api/=.
Expand Down
16 changes: 9 additions & 7 deletions docs/orgmode/howto/write-a-site-file.org
Original file line number Diff line number Diff line change
Expand Up @@ -148,13 +148,15 @@ Each line is a command that =run= would start, after the phase it belongs to: =s

Each of these makes =check= exit 2; the problems of a site file that parses end with =ERROR [main] -: N problem(s) in the site file=.

| Message | Cause | What to do |
|--------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------+--------------------------------------------------------------------|
| =ERROR [main] -: invalid site file: TOML parse error at line 35, column 1=, then =unknown field `log_rot`, expected one of `scratch`, ...= | The section has no such key, here a misspelt =log_root=. | Correct the key; the message lists the keys of the section. |
| =ERROR [check] example: scheduler.kind: expected slurm or local, found `pbs`= | eb-pipeline launches commands with Slurm's =srun=, or runs them directly. | Set =slurm= or =local=. |
| =ERROR [check] example: logs.rotate.delete_after_days: must be greater than compress_after_days (400), found 365= | A log directory would be deleted before it is compressed. | Make =delete_after_days= the greater. |
| =ERROR [check] example: targets.zen4.build_lists: `zen4.list` not found in .= | A build list of the target is not in the =--lists-dir= directory. | Add the list, or correct its name in =build_lists=. |
| =ERROR [main] -: eb_keep_logs: cannot run `eb-keep-logs --version`: No such file or directory (os error 2)= | =--probe-tools= found no =eb-keep-logs= on =PATH=. | Install it, or name its path in =[site.tools.eb_keep_logs]= =bin=. |
| Message | Cause | What to do |
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------|
| =ERROR [main] -: invalid site file: TOML parse error at line 35, column 1=, then =unknown field `log_rot`, expected one of `scratch`, ...= | The section has no such key, here a misspelt =log_root=. | Correct the key; the message lists the keys of the section. |
| =ERROR [check] example: scheduler.kind: expected slurm or local, found `pbs`= | eb-pipeline launches commands with Slurm's =srun=, or runs them directly. | Set =slurm= or =local=. |
| =ERROR [check] example: modules.init: `module load EasyBuild` has white space; an entry is one module name, so give each module its own entry= | An entry is one module name, and this one holds a whole =module load= line. | Give each module its own entry, here =init = ["EasyBuild"]=. |
| =ERROR [check] example: modules.shell: expected a POSIX shell that runs the module wrapper, one of sh, bash, dash, ash, ksh, ksh93, mksh or yash, by name or by path, found `zsh`= | The module wrapper is a script for a POSIX shell, and zsh would load the modules of =init= as one name. | Name one of those shells, or drop =shell= for the default, =bash=. |
| =ERROR [check] example: logs.rotate.delete_after_days: must be greater than compress_after_days (400), found 365= | A log directory would be deleted before it is compressed. | Make =delete_after_days= the greater. |
| =ERROR [check] example: targets.zen4.build_lists: `zen4.list` not found in .= | A build list of the target is not in the =--lists-dir= directory. | Add the list, or correct its name in =build_lists=. |
| =ERROR [main] -: eb_keep_logs: cannot run `eb-keep-logs --version`: No such file or directory (os error 2)= | =--probe-tools= found no =eb-keep-logs= on =PATH=. | Install it, or name its path in =[site.tools.eb_keep_logs]= =bin=. |

The commands and their output on this page come from =docs/examples/site-file.sh=.

Expand Down
5 changes: 3 additions & 2 deletions docs/orgmode/reference/environment.org
Original file line number Diff line number Diff line change
Expand Up @@ -125,9 +125,10 @@ When =modules.init= is not empty, =modules.unuse_path= is =true= or =modules.she
1. =EB_PIPELINE_SHELL_INIT=: the file it sources first, when set.
2. =EB_PIPELINE_UNUSE_PATH=: when set, it empties =MODULEPATH=.
3. =EB_PIPELINE_MODULES=: it runs =module purge=, then =module load= for each name in it, in order, then the command.
It splits the variable at white space and gives each name to =module load= as written: a name with a pattern in it, such as =GCC/*=, is not matched against file names.
It splits the variable at white space, which =check= refuses in an entry of =modules.init=, and gives each name to =module load= as written: a name with a pattern in it, such as =GCC/*=, is not matched against file names.

The wrapper is a script for a POSIX shell, such as =bash= or =sh=.
The wrapper is a script for a POSIX shell: =check= accepts =sh=, =bash=, =dash=, =ash=, =ksh=, =ksh93=, =mksh= and =yash= as =modules.shell=, by name or by path, and refuses any other.
zsh would load the names of =EB_PIPELINE_MODULES= as one, since it does not split an unquoted variable, and csh, tcsh and fish cannot parse the script.
The admit and rotate commands start without it.

** See also
Expand Down
16 changes: 8 additions & 8 deletions docs/orgmode/reference/site-file.org
Original file line number Diff line number Diff line change
Expand Up @@ -308,14 +308,14 @@ With =--disable-robot= in =extra_args=, a line fetches its own sources only: the

** =[modules]=

| Key | Type | Default | Meaning |
|---------------+-----------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| =init= | array of string | required | The modules to load, in order, after =module purge=, before the shard, prefetch, build, =eb-keep-logs failed= and cache commands; may be empty. |
| =unuse_path= | boolean | =false= | =true= empties =MODULEPATH= after =shell_init= and before =init= loads, so that the first module of =init= needs an absolute path; =check= warns when it has none. |
| =shell= | string | =bash= | The shell that runs the module wrapper, which is a script for a POSIX shell, such as =bash= or =sh=. |
| =shell_init= | string | none | A file to source before =module= is called, for a shell that does not define it. It is sourced without =set -e= and =set -u=; a =module= call that fails still stops the command. |
| =eessi= | table | none | The site layer of EESSI. |
| =package_env= | table of tables | ={}= | Variables per package name, =[modules.package_env.NAME]=, for the prefetch and build commands of the lines of that package. |
| Key | Type | Default | Meaning |
|---------------+-----------------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| =init= | array of string | required | The modules to load, in order, after =module purge=, before the shard, prefetch, build, =eb-keep-logs failed= and cache commands; may be empty. An entry is one module name, so =module load A B= is =["A", "B"]=; =check= refuses an entry with white space. |
| =unuse_path= | boolean | =false= | =true= empties =MODULEPATH= after =shell_init= and before =init= loads, so that the first module of =init= needs an absolute path; =check= warns when it has none. |
| =shell= | string | =bash= | The shell that runs the module wrapper, a script for a POSIX shell, by name or by path: =sh=, =bash=, =dash=, =ash=, =ksh=, =ksh93=, =mksh= or =yash=. =check= refuses any other, such as =zsh=, which does not split the names of =init=. |
| =shell_init= | string | none | A file to source before =module= is called, for a shell that does not define it. It is sourced without =set -e= and =set -u=; a =module= call that fails still stops the command. |
| =eessi= | table | none | The site layer of EESSI. |
| =package_env= | table of tables | ={}= | Variables per package name, =[modules.package_env.NAME]=, for the prefetch and build commands of the lines of that package. |

The module wrapper starts when =init= is not empty, =unuse_path= is =true= or =shell_init= is set; [[file:environment.org][Environment variables]] describes it.

Expand Down
81 changes: 81 additions & 0 deletions docs/tools/test_export.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""Check docs/export.el on a page of its own: code and verbatim text.

python docs/tools/test_export.py

Exports one Org page with docs/export.el, as the pixi task mkrst does but in
a scratch directory, and reads the reStructuredText it writes with docutils.
Each ~code~, =verbatim= and inline source block of the page must come out as
a literal of the same text: a reStructuredText literal takes no backslash
escapes, so one that ox-rst writes as \\*.log shows as \\*.log.

Exit status: 0 when every literal matches, 1 when one does not or the export
fails.
"""

import pathlib
import subprocess
import sys
import tempfile

import docutils.core
import docutils.nodes

EXPORT = pathlib.Path(__file__).resolve().parent.parent / "export.el"

# Each item of the page, in Org, and the literal it must come out as.
CASES = [
("=*.log=", "*.log"),
("~*.txt~", "*.txt"),
("=**/*.org=", "**/*.org"),
("=*=", "*"),
("src_sh{*.log}", "*.log"),
("=GCC/*=", "GCC/*"),
("~a\\b~", "a\\b"),
]


def export(page):
"""The reStructuredText that export.el writes for PAGE, an Org text."""
with tempfile.TemporaryDirectory() as tmp:
root = pathlib.Path(tmp)
(root / "orgmode").mkdir()
(root / "source").mkdir()
(root / "orgmode" / "index.org").write_text(page)
run = subprocess.run(
["emacs", "-Q", "--batch", "--load", str(EXPORT)],
cwd=root,
capture_output=True,
text=True,
)
rst = root / "source" / "index.rst"
if run.returncode != 0 or not rst.is_file():
sys.stdout.write(run.stdout + run.stderr)
raise SystemExit(f"test_export: the export failed with exit status {run.returncode}")
return rst.read_text()


def literals(rst):
"""The text of each literal in RST, in order."""
doc = docutils.core.publish_doctree(rst, settings_overrides={"report_level": 5})
return [node.astext() for node in doc.findall(docutils.nodes.literal)]


def main():
page = "#+TITLE: Literals\n\n" + "".join(f"- {org}\n" for org, _ in CASES)
rst = export(page)
got = literals(rst)
if len(got) != len(CASES):
print(f"test_export: {len(got)} literals, expected {len(CASES)}, in:\n{rst}")
return 1
problems = 0
for (org, want), literal in zip(CASES, got):
if literal != want:
print(f"test_export: {org} comes out as {literal}, expected {want}")
problems += 1
print(f"test_export: {len(CASES)} literals, {problems} problem(s)")
return 1 if problems else 0


if __name__ == "__main__":
sys.exit(main())
Loading
Loading