Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions COMPAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,16 @@ on the same invocations.

| | |
|---|---|
| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS]`, `-h`, `--help`, `--version` |
| exit codes | 0 on success and on every `failed` that got past its arguments; 2 for a refused invocation, after `eb-keep-logs: MESSAGE` on stderr; 2 with the usage text on stdout for no arguments |
| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS]`, `-h`, `--help`, `--version` |
| exit codes | 0 on success and on every `failed` that got past its arguments; 2 for a refused invocation, after `eb-keep-logs: MESSAGE` on stderr; 2 with the usage text on stdout for no arguments; 75 when another `rotate` holds ROOT's lock past `--lock-timeout`, after `eb-keep-logs: another rotate holds the lock ROOT/.eb-keep-logs-rotate.lock; gave up after N s (--lock-timeout)` on stderr |
| one rotate at a time | `rotate` holds an exclusive flock(2) on `ROOT/.eb-keep-logs-rotate.lock`, created when missing, from before it reads ROOT until it exits; it waits up to `--lock-timeout` seconds for it, 600 without the option or with an empty value, 0 not at all, 999999999 at most; it opens the file for reading when it may not write it; a lock file that cannot be opened, or a symbolic link or non-regular file at its name, is refused, exit 2; on a file system that cannot lock, `rotate` says so on stderr and rotates without the lock |
| environment | `PATH` (finds `eb`), `EASYBUILD_TMPDIR`, `EB_KEEP_LOGS_NOW`, `EB_PIPELINE_LOG_DIR`, `TZ`, `TZDIR` |
| `eb` | `eb --last-log [--tmp-logdir=TMP_LOGDIR] --output-style=basic`, and the same without `--output-style` when that fails or prints nothing; the last line printed is the log |
| stdout | `Failed build log: LOGDIR/NAME.failed.ebout.log`, `Failed command output: LOGDIR/NAME.failed.command-{out,err}.txt`, `Failed meson test log: LOGDIR/NAME.failed.meson-testlog.txt`, `Failed command output not kept: EASYBUILD_TMPDIR is not set.`, `Failed command output not kept: PATH is not under EASYBUILD_TMPDIR.`, `Failed command output not kept: PATH belongs to another user.`, `No EasyBuild log to keep for NAME.`, `No EasyBuild log to keep for NAME: eb is not on PATH.`, `No EasyBuild log to keep for NAME: eb --last-log exited N.`, `No EasyBuild log to keep for NAME: PATH is another build's log.`, `eb-keep-logs rotate: N files compressed, M runs deleted under ROOT`, with `1 file` and `1 run` for a count of one; LOGDIR in these lines without its trailing slashes |
| files | kept files closed to others (command streams mode 0600), their resolved paths appended to `LOGDIR/.licensed-logs` unless already listed; on a failure, `NAME.stdout.log`, a TMP_LOGDIR below LOGDIR and the regular files directly in it closed to others, a symbolic link left as it is; `rotate` writes `FILE.gz` with the file's mode and updates `.licensed-logs` to the `.gz` path |
| files | kept files closed to others (command streams mode 0600), their resolved paths appended to `LOGDIR/.licensed-logs` unless already listed; on a failure, `NAME.stdout.log`, a TMP_LOGDIR below LOGDIR and the regular files directly in it closed to others, a symbolic link left as it is; `rotate` writes `FILE.gz` with the file's mode and updates `.licensed-logs` to the `.gz` path, writing the new list to a file of its own beside the old one and renaming it over the list, so the list is always one whole version |
| run ages | days from the local midnight of the date in the run's name, as `date -d YYYYMMDD` computes it, including TZ rules, except for the zones under unintended differences; a run whose local midnight does not exist is skipped |
| the run in progress | `rotate` neither compresses nor deletes the run that holds its stdout or stderr, when either is a regular file, or `EB_PIPELINE_LOG_DIR` |
| runs still written into | after taking its lock, `rotate` reads the links `/proc/PID/fd/N` of every process whose descriptors it may read; a run that holds the path a link with its owner's write bit names, also when the path ends in ` (deleted)`, is neither compressed nor deleted; without `/proc`, no run is left alone for this |

## Deliberate differences

Expand All @@ -26,12 +28,15 @@ on the same invocations.
| `--version` | `eb-keep-logs 0.1.0` | `eb-keep-logs 0.2.1 (TARGET, MODE, zig VERSION, COMMIT)`; the first two words are unchanged |
| `--compress-after` or `--delete-after` with no value after it | never returns | refused, exit 2 |
| `EB_KEEP_LOGS_NOW` not empty and not an integer | evaluated as shell arithmetic | refused, exit 2 |
| `rotate`'s lock file cannot be opened, or is a symbolic link or not a regular file | `eb-keep-logs: cannot lock PATH`, exit 2, from a look at the name just before it is opened | `eb-keep-logs: cannot lock PATH: ERROR`, exit 2, from the open itself |
| the file system cannot lock `rotate`'s lock file | flock(1)'s error, then `eb-keep-logs: cannot lock PATH; rotating without the lock`; also when flock(1) is missing | `eb-keep-logs: cannot lock PATH: ERROR; rotating without the lock` |
| a `rotate` waiting for the lock | takes it as soon as it is let go, as flock(1) blocks | tries for it every 20 ms |
| a copy into LOGDIR fails | error from cp, then the file is still listed and reported | error on stderr; the file is neither listed nor reported |
| a copy into an existing LOGDIR file fails part way (a full disk) | cp truncates the file first, so the previous bytes are gone | the previous file is left unchanged; nothing is listed or reported |
| a symbolic link or non-regular file at a destination name in LOGDIR | cp writes through it | refused, reported on stderr |
| `NAME.stdout.log` or TMP_LOGDIR that cannot be closed to others | chmod's error on stderr | `eb-keep-logs: cannot close PATH to others: ERROR` on stderr |
| the failing command's streams | checked by path (`realpath`, `-L`, `stat`), then copied | opened without following a symbolic link at the last component; the same checks run on the open file, and that file is copied |
| `.licensed-logs` after `rotate` | rewritten by every compressed file, as a new file owned by the caller | rewritten once per run when a line changes, keeping its owner (when permitted), mode and access ACL (access control list) entries, replaced atomically |
| `.licensed-logs` after `rotate` | rewritten by every compressed file, as a new file owned by the caller with the list's mode, made by mktemp beside the list and renamed over it; nothing is synced | rewritten once per run when a line changes, keeping its owner (when permitted), mode and access ACL (access control list) entries; written beside the list, synced, renamed over it, and the directory synced, with `eb-keep-logs: cannot sync a .licensed-logs: ERROR` on stderr when that fails |
| a run directory that cannot be removed | counted as deleted | reported on stderr, not counted |
| compression | runs `gzip -9`, `find` and `awk` | in process; the gzip header (name, mtime, flags, OS) matches `gzip -9`, the compressed bytes differ; a file with other hard links or an existing `.gz` is skipped as gzip skips it |
| `eb --last-log` that does not return | waits without a limit | its process group is killed after 300 s, and nothing is kept |
Expand Down
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ install -m 0755 "eb-keep-logs-$v-$arch-linux-musl" ~/.local/bin/eb-keep-logs
```

The shell script `eb-keep-logs` in this repository is the same tool; copy it onto `PATH`.
It needs `bash`, coreutils, GNU `grep` (for `grep -P`), `sed`, `find`, `gzip`, `awk` and, for `failed`, `eb`.
It needs `bash`, coreutils, GNU `grep` (for `grep -P`), `sed`, `find`, `gzip`, `awk`, `flock` from util-linux for `rotate` and, for `failed`, `eb`.
Without `grep -P` it still keeps the log and the command output, but not the meson testlog.

There is no EasyBuild recipe for eb-keep-logs here or in the EasyBuild easyconfigs repository.
Expand Down Expand Up @@ -79,10 +79,15 @@ eb-keep-logs rotate /srv/build-logs --compress-after 30 --delete-after 365
```

`rotate` looks only at run directories named `YYYYMMDD_HHMMSS` directly under the root, and dates each run by its name.
It gzips the `*.log` and `*.txt` files of runs older than `--compress-after` days, keeps each file's mode, and updates the run's `.licensed-logs`.
It gzips the `*.log` and `*.txt` files of runs older than `--compress-after` days, keeps each file's mode, and updates the run's `.licensed-logs`, writing each new list beside the old one and renaming it over it.
It removes whole runs older than `--delete-after` days only when that option is given.
It leaves alone the run it writes into, which is still going: the run that holds its standard output or error, and the run that holds `EB_PIPELINE_LOG_DIR`, the log directory `eb-pipeline` exports, because srun passes the output on over a pipe.
The next `rotate` takes that run.
It also leaves alone every run that a process on the same host has a file in open for writing, as a build that is still going has its log, whichever run started the rotation: it finds them in `/proc/PID/fd`, for the processes whose descriptors it may read.
A later `rotate` takes such a run.

Rotations of one tree take turns: each holds an exclusive flock on `ROOT/.eb-keep-logs-rotate.lock` while it runs.
One that starts while another runs waits for it, up to `--lock-timeout` seconds (600; `0` does not wait), and when the wait runs out it changes nothing and exits 75.
`eb-pipeline`'s `ROOT/.eb-pipeline-rotate.lock` is a separate lock, which `rotate` never takes; a daily job that should wait for eb-pipeline's runs takes that lock around `rotate` with `flock`.

`EB_KEEP_LOGS_NOW`, in seconds since the epoch, sets the time `rotate` dates runs against.
On a copy of the tree, it previews what a later rotation compresses and removes:
Expand Down
1 change: 1 addition & 0 deletions changelog.d/+rotate-atomic-lists.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The shell script writes each new `.licensed-logs` to a file of its own, made by `mktemp` beside the list, and renames it over the list, where it wrote every new list to the fixed name `.licensed-logs.tmp`. Two rotations at once, as where the file system cannot lock, wrote into that one file together and could rename a torn list into place, and a symbolic link at that name was written through and then took the list's place. A temporary file is removed when the rewrite fails. The Zig build, which already wrote and synced a new list beside the old one and renamed it over it, now also syncs the list's directory, as it does for the files `failed` keeps, and says `cannot sync a .licensed-logs: ERROR` when that fails.
1 change: 1 addition & 0 deletions changelog.d/+rotate-live-runs.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`rotate` leaves alone every run that a process on the same host has a file in open for writing, as a build that is still going has its log, not only the run it writes into itself. A build of another run that started before midnight UTC is a day old just after it, so with `--compress-after 1` a rotation from any other run compressed the EasyBuild log the build still wrote into, and the build exited 1 when it went to copy that log into the installation; past `--delete-after` such a run was removed. `rotate` reads the links in `/proc/PID/fd` of the processes whose descriptors it may read, and a later rotation takes the run once its build has ended. A process that only reads a run's files does not hold the run back.
1 change: 1 addition & 0 deletions changelog.d/+rotate-lock.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`rotate` holds an exclusive flock on `ROOT/.eb-keep-logs-rotate.lock` for the whole rotation, so two rotations of one tree no longer run at once. Started together, each compressed part of a run and wrote its `.licensed-logs` back over the other's changes, leaving lists that named files that were gone, and both exited 0. A second `rotate` now waits for the first, up to `--lock-timeout SECONDS` (600; `0` does not wait), then finds the work done; when the wait runs out it changes nothing and exits 75, naming the lock. A lock file that cannot be opened, or a symbolic link or non-regular file at its name, is refused with exit 2; on a file system that cannot lock, `rotate` says so and rotates without the lock. eb-pipeline's `ROOT/.eb-pipeline-rotate.lock` is a separate lock that `rotate` never takes, so the two cannot wait on each other.
1 change: 1 addition & 0 deletions docs/orgmode/explanation/index.org
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@ Nothing here is a step to follow.

why-eb-last-log
local-time
rotate-in-use
#+end_export
Loading
Loading