Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- run: shellcheck -S warning eb-keep-logs tests/test_eb_keep_logs.sh
- run: shellcheck -S warning eb-keep-logs tests/test_eb_keep_logs.sh tests/corpus/run.sh tests/corpus/make-fixtures.sh
- run: bash tests/test_eb_keep_logs.sh
- name: no site names in the tree
run: |
Expand All @@ -35,6 +35,6 @@ jobs:
with:
version: 0.17.1
- run: zig fmt --check build.zig build.zig.zon src
- run: shellcheck -S warning tests/differential.sh
- run: shellcheck -S warning tests/differential.sh tests/test_why.sh
- run: zig build test --summary all
- run: zig build -Drelease=true -Dcommit=${{ github.sha }} && zig-out/bin/eb-keep-logs --version
2 changes: 2 additions & 0 deletions COMPAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ on the same invocations.
| case | shell script | Zig build |
|---|---|---|
| `--version` | `eb-keep-logs 0.1.0` | `eb-keep-logs 0.2.1 (TARGET, MODE, zig VERSION, COMMIT)`; the first two words are unchanged |
| `why` | refused with `eb-keep-logs: why is in the Zig build only`, exit 2 | says why the run's builds failed, from their kept logs, read through gzip when rotated |
| `history` | refused with `eb-keep-logs: history is in the Zig build only`, exit 2 | lists how each kept build of a package went, run by run |
| `--compress-after` or `--delete-after` with no value after it | never returns | refused, exit 2 |
| `EB_KEEP_LOGS_NOW` not empty and not an integer | evaluated as shell arithmetic | refused, exit 2 |
| `rotate`'s lock file cannot be opened, or is a symbolic link or not a regular file | `eb-keep-logs: cannot lock PATH`, exit 2, from a look at the name just before it is opened | `eb-keep-logs: cannot lock PATH: ERROR`, exit 2, from the open itself |
Expand Down
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,20 @@ cp -a /srv/build-logs /tmp/logs-preview
EB_KEEP_LOGS_NOW=$(date -d 2027-09-20 +%s) eb-keep-logs rotate /tmp/logs-preview --compress-after 30 --delete-after 365
```

## Find why a build failed

```bash
eb-keep-logs why --latest --root /srv/build-logs
eb-keep-logs why 20261117_040615 --root /srv/build-logs --target zen4 --json
eb-keep-logs history hello --root /srv/build-logs
```

`why` reads the logs a run kept, as `failed` kept them or `rotate` compressed them, and prints for each failed easyconfig its package and easyconfig, the EasyBuild step it failed in, EasyBuild's message, and the line of its log that shows the error, the first compiler or linker error or else EasyBuild's own error line, with the path, the line number and three lines either side.
For a failure class, feed the kept log to eb-stack.
`history` lists each kept build of a package, run by run, with its outcome and how long EasyBuild took.
`--json` prints either as one line of JSON with `schema_version` 1, for a pipeline's run summary.
Both are in the Zig build only.

## Tests

```bash
Expand Down
10 changes: 9 additions & 1 deletion build.zig
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,17 @@ pub fn build(b: *std.Build) void {
diff.setCwd(b.path("."));
diff.expectExitCode(0);
diff.has_side_effects = true;
// why, which the shell implementation does not have, on the kept logs
// of real EasyBuild failures.
const why = b.addSystemCommand(&.{ "bash", "-c", "TOOL_BIN=\"$0\" exec bash tests/test_why.sh" });
why.addArtifactArg(exe);
why.setCwd(b.path("."));
why.expectExitCode(0);
why.has_side_effects = true;

const test_step = b.step("test", "Run the unit tests, the command-line suite and the differential run");
const test_step = b.step("test", "Run the unit tests, the command-line suites and the differential run");
test_step.dependOn(&b.addRunArtifact(unit).step);
test_step.dependOn(&suite.step);
test_step.dependOn(&diff.step);
test_step.dependOn(&why.step);
}
1 change: 1 addition & 0 deletions changelog.d/+history.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`eb-keep-logs history PACKAGE [--target TARGET] [--root ROOT] [--json]` lists each kept build of a package, oldest run first, with its target, its outcome and how long EasyBuild took, from EasyBuild's output, plain or compressed; `PACKAGE` is an easyconfig's name, its start up to a dash, or a module name. `--json` prints one line with `schema_version` 1. Like `why`, it is in the Zig build only; the shell script refuses it.
1 change: 1 addition & 0 deletions changelog.d/+log-corpus.misc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`tests/fixtures/build-logs/` holds the logs of real EasyBuild 5.4.2 builds on top of EESSI 2025.06 that fail at a checksum, a patch, a missing dependency, configure, a compile, a link, a test and a sanity check, laid out as eb-pipeline lays out its runs, with what `eb-keep-logs failed` kept and four runs compressed by `rotate`. `tests/corpus/` has the easyconfigs, the script that runs them and the one that trims the logs by dropping lines.
1 change: 1 addition & 0 deletions changelog.d/+why-howto.doc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A how-to, Find why a build failed, runs `why` and `history` on the kept logs of real failed builds, and reads `why --json` with `jq`; the front page and the pipeline how-to point to it.
1 change: 1 addition & 0 deletions changelog.d/+why.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json]` says from a run's kept logs why its builds failed: for each failed easyconfig, the package and easyconfig, the EasyBuild step it failed in, EasyBuild's message, and the line of the log that shows the error, with three lines either side, its path and its line number. That line is the first compiler or linker error in the failed step, or else EasyBuild's own error line. It reads a log `rotate` compressed as it is, and a build that kept only its output from that, undoing EasyBuild's wrapping at 80 columns. `--json` prints the same as one line of JSON with `schema_version` 1, for a pipeline's run summary. The shell script refuses `why`, as COMPAT.md lists, and `tests/test_why.sh` checks it on the real failures in `tests/fixtures/build-logs/`.
27 changes: 27 additions & 0 deletions docs/orgmode/contributing.org
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,31 @@ This runs the unit tests, the command-line suite =tests/test_eb_keep_logs.sh= ag
=bash tests/test_eb_keep_logs.sh= runs the suite against the shell script.
A deliberate difference between the two has its row in =COMPAT.md=, which [[file:reference/compatibility.org][Compatibility]] shows, and its label in =tests/differential.sh=.

** The log corpus

=tests/fixtures/build-logs/= is the log tree of real EasyBuild 5.4.2 builds on top of EESSI 2025.06, laid out as eb-pipeline lays out its runs, =RUN/TARGET/NAME.stdout.log=, with what =eb-keep-logs failed= kept beside each failed build.
The builds of GNU hello 2.12.1 in =tests/corpus/cases/= fail as builds do: a checksum of another release, a patch made for an older version, a dependency nobody packaged, a configure that finds no working compiler, a compile error, an undefined reference, a failing test and a sanity check path that is not there; the last run builds it.
=eb-keep-logs rotate= compressed the first four runs.

The builds ran as =alice= on a host named =node1=, so the logs name no real user or host.
To make the tree again, mount EESSI's CernVM-FS repository, put =eb-keep-logs= on =PATH=, and run each case as a run of its own, in UTC as eb-pipeline names its runs:

#+begin_src bash
export TZ=UTC
for c in checksum patch missing-dep configure; do tests/corpus/run.sh sapphirerapids "$c"; done
EB_KEEP_LOGS_NOW=$(($(date +%s) + 86400)) eb-keep-logs rotate ~/build-logs --compress-after 1
for c in compile link; do tests/corpus/run.sh sapphirerapids "$c"; done
export STAMP=$(date +%Y%m%d_%H%M%S)
tests/corpus/run.sh sapphirerapids test sanity
tests/corpus/run.sh generic pass
unset STAMP
tests/corpus/run.sh sapphirerapids pass pass-13
tests/corpus/make-fixtures.sh ~/build-logs tests/fixtures/build-logs "$(command -v eb-keep-logs)"
#+end_src

=make-fixtures.sh= drops lines and never edits one.
Of each kept session log it keeps EasyBuild's warnings and errors and the records of its messages, steps and shell commands, without the output of the commands that succeeded or the middle of a failed command's output longer than 160 lines; EasyBuild's own copies of its logs stay out.

** Build the documentation

The documentation toolchain is pinned in =pixi.lock=: Emacs, Sphinx, lychee, and the tools that read the code.
Expand Down Expand Up @@ -66,6 +91,8 @@ pixi run -e docs linkcheck
| =docs/source/_ext/= | the Sphinx domain for Zig declarations |
| =docs/tools/zig_api.zig= | writes the Zig API pages from the doc comments |
| =docs/tools/check_site.py= | checks internal links, anchors and forbidden names |
| =tests/corpus/= | the builds the log corpus comes from |
| =tests/fixtures/= | the log corpus |

The =.rst= files under =docs/source/= and everything under =docs/source/reference/api/= are generated and not committed.

Expand Down
2 changes: 2 additions & 0 deletions docs/orgmode/howto/call-from-a-pipeline.org
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ eb-keep-logs rotate: 0 files compressed, 0 runs deleted under /home/alice/build-
#+end_example

Each build has =NAME.stdout.log= and its =--tmp-logdir= under =easybuild-logs/=; a failed build also has =NAME.failed.ebout.log= and, when its last command failed, =NAME.failed.command-out.txt=.
=eb-keep-logs why "$LOGDIR"= says from these files why each failed build failed; [[file:find-why-a-build-failed.org][Find why a build failed]] shows how.

** If it fails

Expand All @@ -88,6 +89,7 @@ Each build has =NAME.stdout.log= and its =--tmp-logdir= under =easybuild-logs/=;

** Related

- [[file:find-why-a-build-failed.org][Find why a build failed]]
- [[file:rotate-a-log-tree.org][Rotate a log tree]]
- [[file:../explanation/why-eb-last-log.org][Finding a failed build's log]]
- [[file:../reference/cli.org][The eb-keep-logs command]]
Expand Down
118 changes: 118 additions & 0 deletions docs/orgmode/howto/find-why-a-build-failed.org
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
#+TITLE: Find why a build failed
#+OPTIONS: toc:nil todo:nil num:nil author:nil title:nil
#+STARTUP: content

* Find why a build failed

Use this after a run in which builds failed, to see for each the EasyBuild step it failed in and the line of its log that shows the error without opening the logs, or to see how a package's builds went over the runs you keep.
=eb-keep-logs why= and =eb-keep-logs history= read the logs =eb-keep-logs failed= kept, as they are or as =rotate= compressed them, and change nothing.

** Before you start

- =eb-keep-logs= built with Zig, as =zig build -Drelease=true= in its repository builds it, on your =PATH=: the shell script has no =why= or =history=.
- A log tree, here =~/build-logs=, whose runs are named by their date and time, =YYYYMMDD_HHMMSS=, as [[file:call-from-a-pipeline.org][Call eb-keep-logs from a build pipeline]] makes them.
A run keeps each build's =NAME.stdout.log= and, for a failed build, =NAME.failed.ebout.log=, in its own directory or in a directory per target, as eb-pipeline keeps them.

** Steps

1. Ask about the newest run:

#+begin_src bash
eb-keep-logs why --root ~/build-logs
#+end_src

#+begin_example
/home/alice/build-logs/20261117_042207: 0 failed builds of 2
#+end_example

Without a run, =why= reads the newest, by the date and time in its name; here its two builds succeeded.
2. Name the run whose builds failed, as =ls ~/build-logs= lists it:

#+begin_src bash
eb-keep-logs why 20261117_040615 --root ~/build-logs
#+end_src

#+begin_example
/home/alice/build-logs/20261117_040615: 1 failed build of 1

sapphirerapids/hello-2.12.1-GCCcore-14.2.0 failed in the build step, after 3m06s
package: hello/2.12.1-GCCcore-14.2.0
easyconfig: /home/alice/corpus/cases/compile/hello-2.12.1-GCCcore-14.2.0.eb
EasyBuild: shell command 'make ...' failed with exit code 2 in build step for hello-2.12.1-GCCcore-14.2.0.eb
log: /home/alice/build-logs/20261117_040615/sapphirerapids/hello-2.12.1-GCCcore-14.2.0.failed.ebout.log:34142
34139 | gcc -DLOCALEDIR=\"/home/alice/eessi/versions/2025.06/software/linux/x86_64/intel/sapphirerapids/software/hello/2.12.1-GCCcore-14.2.0/share/locale\" -DHAVE_CONFIG_H -I. -Ilib -I./lib -Isrc -I./src -O2 -ftree-vectorize -march=native -fno-math-errno -MT lib/exitfail.o -MD -MP -MF $depbase.Tpo -c -o lib/exitfail.o lib/exitfail.c &&\
34140 | mv -f $depbase.Tpo $depbase.Po
34141 | src/hello.c: In function 'main':
> 34142 | src/hello.c:149:18: error: 'site_greeting' undeclared (first use in this function); did you mean 'mb_greeting'?
34143 | 149 | greeting_msg = site_greeting;
34144 | | ^~~~~~~~~~~~~
34145 | | mb_greeting
#+end_example

Each failed build gets its target and name, the EasyBuild step it failed in, and how long EasyBuild took; then its package, its easyconfig, EasyBuild's own message, and the line of its log that shows the error, with the line number and three lines either side.
That line is the first compiler or linker error in the failed step, here GCC's; a build that failed without one, such as on a wrong checksum, gets EasyBuild's own error line.
With =--target TARGET=, only the builds in the run's directory =TARGET= are read.
For a failure class, feed the kept log to [[https://eb-stack.rgoswami.me/][eb-stack]]; =why= gives none.
3. To read more of the log than the lines =why= shows, print the lines around the one it gives:

#+begin_src bash
sed -n '34100,34150p' ~/build-logs/20261117_040615/sapphirerapids/hello-2.12.1-GCCcore-14.2.0.failed.ebout.log
#+end_src

A log =rotate= compressed ends in =.gz=; read it with =zcat FILE | sed -n '34100,34150p'=.
4. See how a package's builds went over the runs:

#+begin_src bash
eb-keep-logs history hello --root ~/build-logs
#+end_src

#+begin_example
hello: 11 builds under /home/alice/build-logs: 3 succeeded, 8 failed

RUN TARGET OUTCOME DURATION NAME
20261117_035825 sapphirerapids failed 48s hello-2.12.1-GCCcore-14.2.0
20261117_040032 sapphirerapids failed 41s hello-2.12.1-GCCcore-14.2.0
20261117_040119 sapphirerapids failed - hello-2.12.1-GCCcore-14.2.0
20261117_040311 sapphirerapids failed 2m24s hello-2.12.1-GCCcore-14.2.0
20261117_040615 sapphirerapids failed 3m06s hello-2.12.1-GCCcore-14.2.0
20261117_040928 sapphirerapids failed 2m30s hello-2.12.1-GCCcore-14.2.0
20261117_041222 generic success 2m50s hello-2.12.1-GCCcore-14.2.0
20261117_041222 sapphirerapids failed 2m54s hello-2.12.1-GCCcore-13.3.0
20261117_041222 sapphirerapids failed 2m33s hello-2.12.1-GCCcore-14.2.0
20261117_042207 sapphirerapids success 2m40s hello-2.12.1-GCCcore-13.3.0
20261117_042207 sapphirerapids success 2m52s hello-2.12.1-GCCcore-14.2.0
#+end_example

=hello= takes every easyconfig whose name starts with =hello-=; =hello-2.12.1-GCCcore-13.3.0= takes that one alone.
A duration of =-= is one EasyBuild did not print, as when it stopped before building; ask =why= about that run for the reason.
5. To put the reasons in a script's own summary, read the JSON =--json= prints:

#+begin_src bash
eb-keep-logs why 20261117_041222 --root ~/build-logs --json \
| jq -r '.failures[] | "\(.name): \(.step), \(.log):\(.error.line)"'
#+end_src

#+begin_example
hello-2.12.1-GCCcore-13.3.0: sanitycheck, /home/alice/build-logs/20261117_041222/sapphirerapids/hello-2.12.1-GCCcore-13.3.0.failed.ebout.log:35357
hello-2.12.1-GCCcore-14.2.0: test, /home/alice/build-logs/20261117_041222/sapphirerapids/hello-2.12.1-GCCcore-14.2.0.failed.ebout.log:34323
#+end_example

Its keys keep their meaning while =schema_version= is 1; [[file:../reference/cli.org][The eb-keep-logs command]] lists them.
For an eb-pipeline run, give =--root= the site file's =log_root= and =--target= the target: eb-pipeline keeps each run's logs in =LOG_ROOT/STAMP/TARGET/=.

** If it fails

| Message | Cause | What to do |
|----------------------------------------------+-----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------|
| =eb-keep-logs: why is in the Zig build only= | The =eb-keep-logs= on your =PATH= is the shell script. | Put the Zig build first on =PATH=. |
| =eb-keep-logs: ROOT PATH is not a directory= | =--root= names no directory. | Give =--root= the log tree. |
| =eb-keep-logs: no run under ROOT= | =ROOT= holds no directory named =YYYYMMDD_HHMMSS=. | Give =--root= the directory that holds the runs, not one run. |
| =RUN: 0 failed builds of 0= | The run holds no =NAME.stdout.log= or =NAME.failed.ebout.log=, in its own directory or one below it. | Check that the pipeline writes its logs where =why= reads them. |
| =failed before its first step= | EasyBuild stopped before its first step, as it does for a missing dependency. | Read the =EasyBuild:= line, which gives EasyBuild's message. |
| =log: PATH (no error recorded)= | The log records no failure, or could not be read, as a message on stderr then says. | Read the log itself. |

** Related

- [[file:../reference/cli.org][The eb-keep-logs command]]
- [[file:call-from-a-pipeline.org][Call eb-keep-logs from a build pipeline]]
- [[file:rotate-a-log-tree.org][Rotate a log tree]]
1 change: 1 addition & 0 deletions docs/orgmode/howto/index.org
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ The title names the task.
:maxdepth: 1

call-from-a-pipeline
find-why-a-build-failed
rotate-a-log-tree
mark-licensed-logs
#+end_export
3 changes: 2 additions & 1 deletion docs/orgmode/index.org
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
Keep the logs of EasyBuild builds that a build pipeline would otherwise lose, and keep the pipeline's log tree bounded.
After a failed build, =eb-keep-logs failed= copies the EasyBuild log and the failing command's own output next to the build's output, where the end of the job does not take them, and closes the copies to other users, because a failed compile line of licensed software can print the licence number.
=eb-keep-logs rotate= compresses the logs of old runs and removes the oldest runs.
Each release is one static binary per architecture that needs nothing but =eb=; the shell script it was ported from stays in the repository as the same tool.
=eb-keep-logs why= says from the kept logs why a run's builds failed, and =eb-keep-logs history= how a package's builds went run by run.
Each release is one static binary per architecture that needs nothing but =eb=; the shell script it was ported from stays in the repository as the same tool, without =why= and =history=.

New here?
[[file:tutorials/keep-a-failed-build-log.org][Keep a failed build's logs]] runs an EasyBuild build that fails and ends with its log and the compiler's error kept in a directory of your own.
Expand Down
Loading
Loading