From fd5fdd467f0fd4583e58a160a495d89e3ee24bc3 Mon Sep 17 00:00:00 2001 From: Rohit Goswami Date: Mon, 16 Nov 2026 23:29:29 +0000 Subject: [PATCH 1/3] fix(rotate): hold a lock on the tree for the whole rotation Two rotations of one tree that ran at the same time each compressed part of a run and wrote its .licensed-logs back with only their own changes, over the other's, so the list named files that were gone, and both exited 0. Started together on a tree of three runs, they left such a list in 189 of 200 rounds with the shell script and in 156 with the Zig build. eb-pipeline keeps its own runs apart with its .eb-pipeline-rotate.lock, but a rotation from cron or by hand still raced with them. rotate now holds an exclusive flock(2) on ROOT/.eb-keep-logs-rotate.lock, created when missing, from before it lists ROOT until it exits. A second rotate waits for it, up to --lock-timeout SECONDS, 600 by default and 0 not at all, and then finds the work done; when the wait runs out it changes nothing and exits 75, EX_TEMPFAIL, naming the lock. The file is opened for reading when it cannot be opened for writing. A lock file that cannot be opened, or a symbolic link or a file that is not a regular one at its name, is refused with exit 2. Where the file system cannot lock, rotate says so and rotates without the lock, as eb-pipeline goes on without its own: a refusal would fail the logs phase of every eb-pipeline run there. The shell script takes the lock with flock(1), the Zig build with flock(2), tried every 20 ms. rotate never takes eb-pipeline's lock, and eb-pipeline takes its own before it runs rotate, so the two are always taken in one order and neither tool can wait for the other for good. A daily job that should wait for eb-pipeline's runs takes that lock around rotate with flock(1), in the same order. The command-line suite holds the lock and checks that rotate refuses at once and after --lock-timeout, takes the lock once it is let go, takes a lock it may only read, and refuses a symbolic link at its name; for the shell script, a flock(1) that fails as on a file system that cannot lock leaves rotate rotating without the lock. It then starts two rotations of one tree together in three rounds: every list stays whole, and one rotation compresses all the logs while the other finds none. The lock checks and both race checks fail on the previous commit. Unit tests cover the lock and the --lock-timeout values; the differential run compares the refusals, an empty value and the exit 75 message. COMPAT.md, the cli reference, the rotate how-to and a new explanation, Rotating a tree in use, describe the lock, its wait, exit 75 and how it sits beside eb-pipeline's lock. --- COMPAT.md | 8 +- README.md | 6 +- changelog.d/+rotate-lock.bugfix.md | 1 + docs/orgmode/explanation/index.org | 1 + docs/orgmode/explanation/rotate-in-use.org | 83 ++++++++++++ docs/orgmode/howto/call-from-a-pipeline.org | 1 + docs/orgmode/howto/rotate-a-log-tree.org | 20 ++- docs/orgmode/reference/cli.org | 19 ++- eb-keep-logs | 40 +++++- src/main.zig | 134 +++++++++++++++++++- tests/differential.sh | 13 +- tests/test_eb_keep_logs.sh | 77 ++++++++++- 12 files changed, 388 insertions(+), 15 deletions(-) create mode 100644 changelog.d/+rotate-lock.bugfix.md create mode 100644 docs/orgmode/explanation/rotate-in-use.org diff --git a/COMPAT.md b/COMPAT.md index c0b83e2..18f8d20 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -9,8 +9,9 @@ on the same invocations. | | | |---|---| -| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS]`, `-h`, `--help`, `--version` | -| exit codes | 0 on success and on every `failed` that got past its arguments; 2 for a refused invocation, after `eb-keep-logs: MESSAGE` on stderr; 2 with the usage text on stdout for no arguments | +| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS]`, `-h`, `--help`, `--version` | +| exit codes | 0 on success and on every `failed` that got past its arguments; 2 for a refused invocation, after `eb-keep-logs: MESSAGE` on stderr; 2 with the usage text on stdout for no arguments; 75 when another `rotate` holds ROOT's lock past `--lock-timeout`, after `eb-keep-logs: another rotate holds the lock ROOT/.eb-keep-logs-rotate.lock; gave up after N s (--lock-timeout)` on stderr | +| one rotate at a time | `rotate` holds an exclusive flock(2) on `ROOT/.eb-keep-logs-rotate.lock`, created when missing, from before it reads ROOT until it exits; it waits up to `--lock-timeout` seconds for it, 600 without the option or with an empty value, 0 not at all, 999999999 at most; it opens the file for reading when it may not write it; a lock file that cannot be opened, or a symbolic link or non-regular file at its name, is refused, exit 2; on a file system that cannot lock, `rotate` says so on stderr and rotates without the lock | | environment | `PATH` (finds `eb`), `EASYBUILD_TMPDIR`, `EB_KEEP_LOGS_NOW`, `EB_PIPELINE_LOG_DIR`, `TZ`, `TZDIR` | | `eb` | `eb --last-log [--tmp-logdir=TMP_LOGDIR] --output-style=basic`, and the same without `--output-style` when that fails or prints nothing; the last line printed is the log | | stdout | `Failed build log: LOGDIR/NAME.failed.ebout.log`, `Failed command output: LOGDIR/NAME.failed.command-{out,err}.txt`, `Failed meson test log: LOGDIR/NAME.failed.meson-testlog.txt`, `Failed command output not kept: EASYBUILD_TMPDIR is not set.`, `Failed command output not kept: PATH is not under EASYBUILD_TMPDIR.`, `Failed command output not kept: PATH belongs to another user.`, `No EasyBuild log to keep for NAME.`, `No EasyBuild log to keep for NAME: eb is not on PATH.`, `No EasyBuild log to keep for NAME: eb --last-log exited N.`, `No EasyBuild log to keep for NAME: PATH is another build's log.`, `eb-keep-logs rotate: N files compressed, M runs deleted under ROOT`, with `1 file` and `1 run` for a count of one; LOGDIR in these lines without its trailing slashes | @@ -26,6 +27,9 @@ on the same invocations. | `--version` | `eb-keep-logs 0.1.0` | `eb-keep-logs 0.2.1 (TARGET, MODE, zig VERSION, COMMIT)`; the first two words are unchanged | | `--compress-after` or `--delete-after` with no value after it | never returns | refused, exit 2 | | `EB_KEEP_LOGS_NOW` not empty and not an integer | evaluated as shell arithmetic | refused, exit 2 | +| `rotate`'s lock file cannot be opened, or is a symbolic link or not a regular file | `eb-keep-logs: cannot lock PATH`, exit 2, from a look at the name just before it is opened | `eb-keep-logs: cannot lock PATH: ERROR`, exit 2, from the open itself | +| the file system cannot lock `rotate`'s lock file | flock(1)'s error, then `eb-keep-logs: cannot lock PATH; rotating without the lock`; also when flock(1) is missing | `eb-keep-logs: cannot lock PATH: ERROR; rotating without the lock` | +| a `rotate` waiting for the lock | takes it as soon as it is let go, as flock(1) blocks | tries for it every 20 ms | | a copy into LOGDIR fails | error from cp, then the file is still listed and reported | error on stderr; the file is neither listed nor reported | | a copy into an existing LOGDIR file fails part way (a full disk) | cp truncates the file first, so the previous bytes are gone | the previous file is left unchanged; nothing is listed or reported | | a symbolic link or non-regular file at a destination name in LOGDIR | cp writes through it | refused, reported on stderr | diff --git a/README.md b/README.md index 226dc98..53c5106 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ install -m 0755 "eb-keep-logs-$v-$arch-linux-musl" ~/.local/bin/eb-keep-logs ``` The shell script `eb-keep-logs` in this repository is the same tool; copy it onto `PATH`. -It needs `bash`, coreutils, GNU `grep` (for `grep -P`), `sed`, `find`, `gzip`, `awk` and, for `failed`, `eb`. +It needs `bash`, coreutils, GNU `grep` (for `grep -P`), `sed`, `find`, `gzip`, `awk`, `flock` from util-linux for `rotate` and, for `failed`, `eb`. Without `grep -P` it still keeps the log and the command output, but not the meson testlog. There is no EasyBuild recipe for eb-keep-logs here or in the EasyBuild easyconfigs repository. @@ -84,6 +84,10 @@ It removes whole runs older than `--delete-after` days only when that option is It leaves alone the run it writes into, which is still going: the run that holds its standard output or error, and the run that holds `EB_PIPELINE_LOG_DIR`, the log directory `eb-pipeline` exports, because srun passes the output on over a pipe. The next `rotate` takes that run. +Rotations of one tree take turns: each holds an exclusive flock on `ROOT/.eb-keep-logs-rotate.lock` while it runs. +One that starts while another runs waits for it, up to `--lock-timeout` seconds (600; `0` does not wait), and when the wait runs out it changes nothing and exits 75. +`eb-pipeline`'s `ROOT/.eb-pipeline-rotate.lock` is a separate lock, which `rotate` never takes; a daily job that should wait for eb-pipeline's runs takes that lock around `rotate` with `flock`. + `EB_KEEP_LOGS_NOW`, in seconds since the epoch, sets the time `rotate` dates runs against. On a copy of the tree, it previews what a later rotation compresses and removes: diff --git a/changelog.d/+rotate-lock.bugfix.md b/changelog.d/+rotate-lock.bugfix.md new file mode 100644 index 0000000..d68e6db --- /dev/null +++ b/changelog.d/+rotate-lock.bugfix.md @@ -0,0 +1 @@ +`rotate` holds an exclusive flock on `ROOT/.eb-keep-logs-rotate.lock` for the whole rotation, so two rotations of one tree no longer run at once. Started together, each compressed part of a run and wrote its `.licensed-logs` back over the other's changes, leaving lists that named files that were gone, and both exited 0. A second `rotate` now waits for the first, up to `--lock-timeout SECONDS` (600; `0` does not wait), then finds the work done; when the wait runs out it changes nothing and exits 75, naming the lock. A lock file that cannot be opened, or a symbolic link or non-regular file at its name, is refused with exit 2; on a file system that cannot lock, `rotate` says so and rotates without the lock. eb-pipeline's `ROOT/.eb-pipeline-rotate.lock` is a separate lock that `rotate` never takes, so the two cannot wait on each other. diff --git a/docs/orgmode/explanation/index.org b/docs/orgmode/explanation/index.org index 5a83a46..af14f30 100644 --- a/docs/orgmode/explanation/index.org +++ b/docs/orgmode/explanation/index.org @@ -13,4 +13,5 @@ Nothing here is a step to follow. why-eb-last-log local-time + rotate-in-use #+end_export diff --git a/docs/orgmode/explanation/rotate-in-use.org b/docs/orgmode/explanation/rotate-in-use.org new file mode 100644 index 0000000..fb22080 --- /dev/null +++ b/docs/orgmode/explanation/rotate-in-use.org @@ -0,0 +1,83 @@ +#+TITLE: Rotating a tree in use +#+OPTIONS: toc:nil todo:nil num:nil author:nil title:nil +#+STARTUP: content + +* Rotating a tree in use + +Why does =eb-keep-logs rotate= take a lock, how does that lock sit beside eb-pipeline's, and why can the two never wait on each other for good? + +** The problem + +Several things start =rotate= on one tree, and they do not take turns by themselves: the end of each pipeline run, a daily job, a person at a shell. +Two rotations of one tree that run at the same time compress the same run between them, each the files the other has not reached yet, and each writes the run's =.licensed-logs= back with only its own changes, over the other's. +The list then names files that are gone, and both rotations exit 0. +Started together on a tree of three runs, two rotations left such a list in most of 200 rounds, with the shell script and with the Zig build. + +** How eb-keep-logs handles it + +Every =rotate= of a tree holds an exclusive lock on one file in it, =ROOT/.eb-keep-logs-rotate.lock=, from before it lists =ROOT= until it exits. +It is a flock(2) lock, which the kernel lets go once the rotation's processes have ended, however they end: a rotation that is killed leaves no lock behind, only the empty file. + +1. A second =rotate= of the tree waits for the lock, up to =--lock-timeout= seconds, 600 by default, and once it has the lock finds the work done. +2. When the wait runs out, it changes nothing and exits 75, =EX_TEMPFAIL= in =sysexits.h=: try again later. + It names the lock it waited for: + + #+begin_example + eb-keep-logs: another rotate holds the lock /home/alice/build-logs/.eb-keep-logs-rotate.lock; gave up after 600 s (--lock-timeout) + #+end_example + +3. =--lock-timeout 0= does not wait, for a caller content to leave the work to the rotation already running. + +=rotate= creates the file when it is missing and leaves it in place. +A symbolic link, or anything but a regular file, at that name is refused with exit status 2, and a user who may read the file but not write it still takes the lock. + +** Beside eb-pipeline's lock + +[[https://github.com/stackhands/eb-pipeline][eb-pipeline]] keeps a lock of its own in the log root, =.eb-pipeline-rotate.lock=, for another question: whether any of its runs still writes logs. +Each of its runs holds that lock shared while it works, and waits for it when another run holds it alone. +At its end, a run rotates only when it can take the lock alone at once, and otherwise leaves the rotation to a run that ends later. +The two locks are separate files, and each tool takes only its own: + +| Lock | Taken by | How | +|-----------------------------+-----------------------------+----------------------------------------------------------------------------------| +| =.eb-pipeline-rotate.lock= | eb-pipeline's runs | shared while a run works; alone, without waiting, around its =eb-keep-logs rotate= | +| =.eb-keep-logs-rotate.lock= | every =eb-keep-logs rotate= | alone, waiting up to =--lock-timeout=, for the whole rotation | + +eb-pipeline takes its lock and then runs =rotate=, which takes the other; nothing takes them in the opposite order, and =rotate= never waits for eb-pipeline's lock. +So no two processes can each hold one of the locks while they wait for the other's, and every wait =rotate= makes ends after =--lock-timeout=. +What remains is a delay: an eb-pipeline run whose =rotate= waits for a rotation from a daily job holds its own lock for as long, and eb-pipeline runs that start meanwhile wait for it. + +A rotation from a daily job that should also leave eb-pipeline's runs alone takes eb-pipeline's lock first, in the same order, with flock(1): + +#+begin_src bash +flock -w 3600 -E 75 ~/build-logs/.eb-pipeline-rotate.lock \ + eb-keep-logs rotate ~/build-logs --compress-after 30 --delete-after 365 +#+end_src + +It waits, up to an hour, for a moment when no eb-pipeline run is working, and exits 75 when none comes; while it rotates, eb-pipeline runs that start wait for it. +The opposite order, a wait for eb-pipeline's lock inside =rotate='s, is the one way to make two processes wait on each other, and then until =--lock-timeout= runs out. + +** What was considered and turned down + +- /Sharing eb-pipeline's lock file./ + An eb-pipeline run holds that lock alone while its own =rotate= runs, so a =rotate= that took it as well would wait for the run that started it. + Telling that hold apart from another's would rest on what the caller passes in the environment, which an older eb-pipeline, a wrapper script or a person does not pass. +- /Waiting without a limit./ + A rotation stalled on a file system would hold up every later one and, through eb-pipeline's lock, the pipeline's runs. +- /A lock elsewhere, under =/tmp= or as a PID file./ + A file in =ROOT= is seen wherever the tree is, and a flock(2) lock cannot outlive its holder as a PID file can. +- /Refusing to rotate where the file system cannot lock./ + eb-pipeline goes on without its own lock there; a refusal would turn that into a failed logs phase in every run. + +** Limits + +- The lock reaches as far as the file system takes flock(2) locks: every process that sees the tree on a local file system or on NFS, where Linux takes them as fcntl(2) locks on the server; only the processes of one host on Lustre mounted with =localflock=. +- On NFS an exclusive lock needs the file open for writing, so a user who may only read the lock file there cannot take it. +- On a file system that cannot lock at all, or cannot lock for that user, =rotate= says so on stderr and rotates without the lock, as eb-pipeline does without its own. +- The lock keeps rotations apart, not other writers: a person, or a tool that does not take it, can still change the tree while =rotate= runs. + +** Further reading + +- [[file:../howto/rotate-a-log-tree.org][Rotate a log tree]], from a pipeline or a daily job. +- [[file:../reference/cli.org][The eb-keep-logs command]]: =--lock-timeout=, exit status 75 and the lock file. +- [[man:flock(2)]], the lock, and [[man:flock(1)]], the command the shell script and the daily job use. diff --git a/docs/orgmode/howto/call-from-a-pipeline.org b/docs/orgmode/howto/call-from-a-pipeline.org index a0071bf..8a788f5 100644 --- a/docs/orgmode/howto/call-from-a-pipeline.org +++ b/docs/orgmode/howto/call-from-a-pipeline.org @@ -52,6 +52,7 @@ The script keeps each failed build's logs next to its output and, at the end of #+end_src =rotate= leaves alone the run that holds its output, so this run stays as it is however long it took, and a later run's =rotate= compresses it once it is old enough. + When another run's =rotate= of the tree is under way, this one waits for it, up to 600 seconds; with =--lock-timeout 0= it leaves the work to that one at once and exits 75. ** Check that it worked diff --git a/docs/orgmode/howto/rotate-a-log-tree.org b/docs/orgmode/howto/rotate-a-log-tree.org index fdbb8b7..bd9342d 100644 --- a/docs/orgmode/howto/rotate-a-log-tree.org +++ b/docs/orgmode/howto/rotate-a-log-tree.org @@ -10,7 +10,7 @@ Use this to keep a directory of build runs from growing without end: compress th ** Before you start - =eb-keep-logs= on your =PATH=; [[file:../tutorials/keep-a-failed-build-log.org][Keep a failed build's logs]] installs it. -- A log tree, here =~/build-logs=, and write access to its runs: =rotate= replaces each log with a =.gz= file beside it, and removes whole runs. +- A log tree, here =~/build-logs=, and write access to it and its runs: =rotate= keeps its lock file in the tree, replaces each log with a =.gz= file beside it, and removes whole runs. ** Steps @@ -44,6 +44,20 @@ Use this to keep a directory of build runs from growing without end: compress th No run is 30 days old yet, so nothing changed. Run the same command at the end of each pipeline run, as the pipeline how-to does, or from a daily job. + Rotations of one tree take turns: one that starts while another runs waits for it, up to =--lock-timeout= seconds, 600 by default, and then finds the work done. +4. To rotate from a daily job, add the command to your crontab with =crontab -e=, its output going to a file outside the tree: + + #+begin_example + 15 3 * * * eb-keep-logs rotate ~/build-logs --compress-after 30 --delete-after 365 >> ~/rotate.log 2>&1 + #+end_example + + When eb-pipeline also rotates the tree, take its lock around the command, so the job waits until no eb-pipeline run is working, up to an hour: + + #+begin_example + 15 3 * * * flock -w 3600 -E 75 ~/build-logs/.eb-pipeline-rotate.lock eb-keep-logs rotate ~/build-logs --compress-after 30 --delete-after 365 >> ~/rotate.log 2>&1 + #+end_example + + Take eb-pipeline's lock outside the command, never inside it; [[file:../explanation/rotate-in-use.org][Rotating a tree in use]] says why. ** Check that it worked @@ -72,10 +86,14 @@ drwxrwxr-x 3 alice alice 4096 Nov 16 17:53 easybuild-logs | =eb-keep-logs: ROOT PATH is not a directory= | The path is wrong, or names a run, not the tree. | Give the directory that holds the runs. | | =eb-keep-logs: cannot compress PATH: TooManyLinks= | The log has other hard links. The shell script prints =gzip='s own message. | Nothing: like =gzip=, =rotate= leaves it as it is. | | =eb-keep-logs: cannot compress PATH: Exists= | =PATH.gz= exists already. The shell script prints =gzip='s own message. | Remove or rename one of the two, and rotate again. | +| =eb-keep-logs: another rotate holds the lock PATH; gave up after 600 s (--lock-timeout)= | Another rotation of the tree ran for longer than =--lock-timeout=; this one changed nothing and exited 75. | Nothing, when the other does the work; else rotate again later, or raise =--lock-timeout=. | +| =eb-keep-logs: cannot lock PATH: ERROR= | The lock file =ROOT/.eb-keep-logs-rotate.lock= cannot be opened, or is a symbolic link or directory; exit status 2. The shell script leaves out =: ERROR=. | Remove what is at that name, or rotate as a user who may write =ROOT=. | +| =eb-keep-logs: cannot lock PATH: ERROR; rotating without the lock= | The file system cannot lock, as Lustre mounted without its =flock= option cannot; =rotate= goes on. | Rotate the tree from one place only, so that rotations cannot overlap. | | A run is not compressed, and there is no message | Its name is not =YYYYMMDD_HHMMSS=, or it holds the output of this =rotate= or =EB_PIPELINE_LOG_DIR=. | Rename it, or leave it to the next =rotate=, after the run has ended. | ** Related - [[file:mark-licensed-logs.org][Mark logs that may carry a licence number]]: =rotate= renames the lines of =.licensed-logs= to the =.gz= files. - [[file:../explanation/local-time.org][Run ages and local time]] +- [[file:../explanation/rotate-in-use.org][Rotating a tree in use]]: the lock, and how it sits beside eb-pipeline's. - [[file:../reference/cli.org][The eb-keep-logs command]] diff --git a/docs/orgmode/reference/cli.org b/docs/orgmode/reference/cli.org index 2504766..0f50d81 100644 --- a/docs/orgmode/reference/cli.org +++ b/docs/orgmode/reference/cli.org @@ -11,7 +11,7 @@ This page describes the Zig build; the shell script =eb-keep-logs= in the reposi #+begin_example eb-keep-logs failed STATUS NAME LOGDIR [TMP_LOGDIR] -eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] +eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS] eb-keep-logs --help | --version #+end_example @@ -31,6 +31,7 @@ eb-keep-logs --help | --version |--------------------------------------------------+----------+----------------------------------------------------------------------------------------------------------------------------------------------| | =--compress-after DAYS=, =--compress-after=DAYS= | =rotate= | Required. Compress the =*.log= and =*.txt= files, at any depth, of runs at least =DAYS= old. =DAYS= is decimal digits. | | =--delete-after DAYS=, =--delete-after=DAYS= | =rotate= | Remove whole runs at least =DAYS= old. =DAYS= must be larger than =--compress-after=. Without it, or with an empty value, no run is removed. | +| =--lock-timeout SECONDS=, =--lock-timeout=SECONDS= | =rotate= | How long to wait for the lock another =rotate= of =ROOT= holds, in seconds, decimal digits; =0= does not wait. Without it, or with an empty value, 600. A value larger than 999999999, which is over 31 years, waits 999999999 seconds. | | =-h=, =--help= | | Print the usage on stdout and exit 0. | | =--version= | | Print the version line on stdout and exit 0. | @@ -55,8 +56,9 @@ eb-keep-logs --help | --version | Status | Meaning | |--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0 | =--help= or =--version=, or a =failed= or =rotate= that accepted its arguments, whatever it could keep, compress or remove: what it could not do is reported, and is no failure. | +| 0 | =--help= or =--version=, a =failed= that accepted its arguments, or a =rotate= that accepted its arguments and did not give up waiting for its lock, whatever it could keep, compress or remove: what it could not do is reported, and is no failure. | | 2 | The invocation was refused, with =eb-keep-logs: MESSAGE= on stderr; or no command was given, and the usage is on stdout. | +| 75 | Another =rotate= held =ROOT='s lock for longer than =--lock-timeout=, and this =rotate= changed nothing; try again later. | | =MESSAGE= | Cause | |-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------| @@ -70,11 +72,18 @@ eb-keep-logs --help | --version | =ROOT ROOT is not a directory= | =ROOT= does not name a directory that can be opened. | | =OPTION needs a number of days= | =--compress-after= is missing, or a value is missing, empty, not decimal digits or too large. | | =--delete-after must be later than --compress-after= | =--delete-after= is not larger than =--compress-after=. | +| =--lock-timeout needs a number of seconds= | The value of =--lock-timeout= is missing, or not decimal digits. | | =EB_KEEP_LOGS_NOW must be seconds since the epoch, not 'VALUE'= | =EB_KEEP_LOGS_NOW= is neither empty nor an integer. | +| =cannot lock ROOT/.eb-keep-logs-rotate.lock: ERROR= | The lock file could not be opened, or is a symbolic link or not a regular file. | | =cannot read ROOT ROOT=, =cannot read ROOT ROOT: ERROR= | =ROOT= could not be listed. | | =cannot read the arguments= | The arguments could not be read into memory. | -=rotate= checks =ROOT= before its options' values, and =failed= checks =STATUS= and then =LOGDIR=. +=rotate= checks =ROOT= before its options' values, then =EB_KEEP_LOGS_NOW=, and then takes the lock; =failed= checks =STATUS= and then =LOGDIR=. +With status 75, =rotate= prints, with =ROOT= without its trailing slashes and =N= the seconds it waited: + +#+begin_example +eb-keep-logs: another rotate holds the lock ROOT/.eb-keep-logs-rotate.lock; gave up after N s (--lock-timeout) +#+end_example ** Files @@ -124,6 +133,7 @@ What =failed= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe | File | Use | |-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| =ROOT/.eb-keep-logs-rotate.lock= | Created empty when missing, and left in place. =rotate= holds an exclusive flock(2) lock on it from before it lists =ROOT= until it exits, and opens it for reading when it may not write it. eb-pipeline's =ROOT/.eb-pipeline-rotate.lock= is another lock, which =rotate= does not take. | | =ROOT/RUN/.../FILE.log=, =FILE.txt= | In a run old enough, replaced by =FILE.gz= with the file's mode, as =gzip -9= does; a symbolic link is not followed, and a file with other hard links or an existing =FILE.gz= is left as it is, with a message on stderr. | | =ROOT/RUN/.../.licensed-logs= | In a run in which files were compressed, a line that names one of them, by its absolute path with symbolic links resolved, is changed to name its =.gz= file. A list that changes keeps its owner when permitted, its mode and its access control list. | | =ROOT/RUN= | Removed with everything in it, when old enough for =--delete-after=, without being compressed first. A symbolic link in it is removed, not followed. Removal stops at the first error. | @@ -140,6 +150,7 @@ What =rotate= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe | =MESSAGE= | Cause | |-----------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------| +| =cannot lock ROOT/.eb-keep-logs-rotate.lock: ERROR; rotating without the lock= | The file system cannot take the lock, or not for this user: =Unsupported= where it takes no locks, =InvalidArgument= on NFS for a lock file open only for reading, =Unexpected= when no lock is available (=ENOLCK=). =rotate= goes on without it. | | =cannot compress PATH: ERROR= | =PATH= is left as it is: it has other hard links (=TooManyLinks=), =PATH.gz= exists (=Exists=), or it could not be read or written. | | =cannot open PATH: ERROR= | A run, or a directory in a run, could not be opened; nothing in it is compressed. | | =cannot read PATH: ERROR= | Listing a run, or a directory in it, failed part way; nothing in it is compressed. | @@ -161,6 +172,6 @@ eb-keep-logs 0.2.1 (x86_64-linux-musl, safe, zig 0.17.1, 5ceaead) - [[file:../tutorials/keep-a-failed-build-log.org][Keep a failed build's logs]] - [[file:../howto/call-from-a-pipeline.org][Call eb-keep-logs from a build pipeline]], [[file:../howto/rotate-a-log-tree.org][Rotate a log tree]] and [[file:../howto/mark-licensed-logs.org][Mark logs that may carry a licence number]] -- [[file:../explanation/why-eb-last-log.org][Finding a failed build's log]] and [[file:../explanation/local-time.org][Run ages and local time]] +- [[file:../explanation/why-eb-last-log.org][Finding a failed build's log]], [[file:../explanation/local-time.org][Run ages and local time]] and [[file:../explanation/rotate-in-use.org][Rotating a tree in use]] - [[file:compatibility.org][Compatibility]], the differences from the shell script. - [[file:api/zig/eb-keep-logs.md][The Zig API]], from the doc comments of the source. diff --git a/eb-keep-logs b/eb-keep-logs index 8f31d5d..9c76de9 100755 --- a/eb-keep-logs +++ b/eb-keep-logs @@ -11,10 +11,13 @@ # EasyBuild's default place is used, unless it is another build's. # NAME.stdout.log and a TMP_LOGDIR below LOGDIR are closed to others too. # eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] +# [--lock-timeout SECONDS] # In the run directories directly under ROOT (named YYYYMMDD_HHMMSS), # gzip the logs of runs older than DAYS; with --delete-after, remove # whole runs older than that. Permissions and .licensed-logs follow. # EB_KEEP_LOGS_NOW, in seconds since the epoch, sets the current time. +# One rotate at a time holds ROOT/.eb-keep-logs-rotate.lock; another +# waits up to SECONDS for it (600; 0 does not wait), then exits 75. # eb-keep-logs --help | --version # # A failed build's log is in EasyBuild's tmpdir on the build node, which goes @@ -25,7 +28,7 @@ set -uo pipefail VERSION=0.1.0 -usage() { sed -n '2,20p' "$0" | sed 's/^# \{0,1\}//'; } +usage() { sed -n '2,23p' "$0" | sed 's/^# \{0,1\}//'; } die() { echo "eb-keep-logs: $*" >&2; exit 2; } @@ -243,14 +246,37 @@ _run_of() { esac } +# Hold an exclusive flock on the lock file LOCK until this rotate exits, +# waiting up to SECONDS for another holder; exit 75 when it does not let go. +# Opened for reading when it cannot be opened for writing, by a user who may +# not write ROOT. On a file system that cannot lock, rotate without it. +_lock_rotate() { + local lock="$1" seconds="$2" fd rc=0 + if [ -L "${lock}" ] || { [ -e "${lock}" ] && [ ! -f "${lock}" ]; }; then + die "cannot lock ${lock}" + fi + { exec {fd}>>"${lock}"; } 2>/dev/null || { exec {fd}<"${lock}"; } 2>/dev/null || die "cannot lock ${lock}" + flock -w "${seconds}" -E 75 "${fd}" || rc=$? + case "${rc}" in + 0) ;; + 75) echo "eb-keep-logs: another rotate holds the lock ${lock}; gave up after ${seconds} s (--lock-timeout)" >&2 + exit 75 ;; + *) echo "eb-keep-logs: cannot lock ${lock}; rotating without the lock" >&2 ;; + esac +} + cmd_rotate() { - local root="" compress_after="" delete_after="" + local root="" compress_after="" delete_after="" lock_timeout="" while [ $# -gt 0 ]; do case "$1" in --compress-after) compress_after="${2:-}"; shift 2 ;; --compress-after=*) compress_after="${1#*=}"; shift ;; --delete-after) delete_after="${2:-}"; shift 2 ;; --delete-after=*) delete_after="${1#*=}"; shift ;; + --lock-timeout) + [ $# -ge 2 ] || die "--lock-timeout needs a number of seconds" + lock_timeout="$2"; shift 2 ;; + --lock-timeout=*) lock_timeout="${1#*=}"; shift ;; -*) die "unknown option $1" ;; *) [ -z "${root}" ] || die "rotate takes one ROOT"; root="$1"; shift ;; esac @@ -262,6 +288,16 @@ cmd_rotate() { [[ "${delete_after}" =~ ^[0-9]+$ ]] || die "--delete-after needs a number of days" [ "${delete_after}" -gt "${compress_after}" ] || die "--delete-after must be later than --compress-after" fi + if [ -n "${lock_timeout}" ]; then + [[ "${lock_timeout}" =~ ^[0-9]+$ ]] || die "--lock-timeout needs a number of seconds" + lock_timeout="${lock_timeout#"${lock_timeout%%[!0]*}"}" + lock_timeout="${lock_timeout:-0}" + # flock(1) cannot set a timer for 10^19 seconds; this is 31 years. + [ "${#lock_timeout}" -le 9 ] || lock_timeout=999999999 + else + lock_timeout=600 + fi + _lock_rotate "$(_joinable "${root}")/.eb-keep-logs-rotate.lock" "${lock_timeout}" local run age file list path compressed=0 deleted=0 own own="$(_own_runs "${root}")/" for run in "$(_joinable "${root}")"/*/; do diff --git a/src/main.zig b/src/main.zig index f7ec9dd..9e7eecb 100644 --- a/src/main.zig +++ b/src/main.zig @@ -31,10 +31,13 @@ const usage_text = \\ EasyBuild's default place is used, unless it is another build's. \\ NAME.stdout.log and a TMP_LOGDIR below LOGDIR are closed to others too. \\ eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] + \\ [--lock-timeout SECONDS] \\ In the run directories directly under ROOT (named YYYYMMDD_HHMMSS), \\ gzip the logs of runs older than DAYS; with --delete-after, remove \\ whole runs older than that. Permissions and .licensed-logs follow. \\ EB_KEEP_LOGS_NOW, in seconds since the epoch, sets the current time. + \\ One rotate at a time holds ROOT/.eb-keep-logs-rotate.lock; another + \\ waits up to SECONDS for it (600; 0 does not wait), then exits 75. \\ eb-keep-logs --help | --version \\ ; @@ -426,6 +429,66 @@ fn parseDays(opt: []const u8, v: ?[]const u8) u64 { return std.fmt.parseInt(u64, s, 10) catch die("{s} needs a number of days", .{opt}); } +/// The lock file every rotate of a tree holds, directly in its ROOT. +const rotate_lock = ".eb-keep-logs-rotate.lock"; +/// How long rotate waits for the lock without --lock-timeout, in seconds. +const default_lock_timeout: u64 = 600; +/// The longest wait --lock-timeout sets, over 31 years: the shell script's +/// flock(1) cannot set a timer for 10^19 seconds. +const max_lock_timeout: u64 = 999_999_999; +/// The exit status when another rotate holds the lock past --lock-timeout, +/// EX_TEMPFAIL: run it again later. +const lock_busy_exit: u8 = 75; + +fn parseSeconds(v: []const u8) u64 { + if (!allDigits(v)) die("--lock-timeout needs a number of seconds", .{}); + return @min(std.fmt.parseInt(u64, v, 10) catch max_lock_timeout, max_lock_timeout); +} + +fn monotonicNs() i128 { + var ts: linux.timespec = undefined; + _ = linux.clock_gettime(.MONOTONIC, &ts); + return @as(i128, ts.sec) * std.time.ns_per_s + ts.nsec; +} + +/// Open `root/.eb-keep-logs-rotate.lock`, created when missing; for reading +/// when it cannot be opened for writing, by a user who may not write ROOT. +/// A symbolic link or a file that is not a regular one is refused. +fn openLock(root: fsx.fd_t) sys.Error!fsx.fd_t { + const fd = sys.openat(root, rotate_lock, .{ + .ACCMODE = .WRONLY, + .APPEND = true, + .CREAT = true, + .NOFOLLOW = true, + .NONBLOCK = true, + }, 0o666) catch |err| switch (err) { + error.AccessDenied, error.ReadOnly => sys.openat(root, rotate_lock, .{ .NOFOLLOW = true, .NONBLOCK = true }, 0) catch return err, + else => return err, + }; + errdefer sys.close(fd); + if (!(try fsx.fstat(fd)).isReg()) return error.InvalidArgument; + return fd; +} + +/// Take an exclusive flock(2) on `fd`, waiting up to `timeout_s` seconds for +/// another holder to let go; 0 tries once. The lock lasts until `fd` is +/// closed, or the process exits. Not the pinned library's +/// `state.lockExclusive`, which gives every failure but EAGAIN as Unexpected. +fn waitLock(fd: fsx.fd_t, timeout_s: u64) (sys.Error || error{TimedOut})!void { + const deadline = monotonicNs() + @as(i128, timeout_s) * std.time.ns_per_s; + while (true) { + const rc = linux.flock(fd, std.posix.LOCK.EX | std.posix.LOCK.NB); + if (linux.errno(rc) != .AGAIN) { + if (try sys.result(rc) != null) return; + continue; + } + const left = deadline - monotonicNs(); + if (left <= 0) return error.TimedOut; + var ts: linux.timespec = .{ .sec = 0, .nsec = @intCast(@min(left, 20 * std.time.ns_per_ms)) }; + _ = linux.nanosleep(&ts, null); + } +} + /// Days since a run directory's name, YYYYMMDD_HHMMSS, at its local /// midnight; null when the name is not a run directory. The name, not the /// mtime: compressing the run's logs moves the mtime of the directory they @@ -546,12 +609,19 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { var root_s: ?[]const u8 = null; var compress_v: ?[]const u8 = null; var delete_v: ?[]const u8 = null; + var lock_v: []const u8 = ""; var have_compress = false; var have_delete = false; var i: usize = 0; while (i < args.len) : (i += 1) { const a = args[i]; - if (std.mem.eql(u8, a, "--compress-after") or std.mem.eql(u8, a, "--delete-after")) { + if (std.mem.eql(u8, a, "--lock-timeout")) { + if (i + 1 >= args.len) die("--lock-timeout needs a number of seconds", .{}); + i += 1; + lock_v = args[i]; + } else if (std.mem.startsWith(u8, a, "--lock-timeout=")) { + lock_v = a["--lock-timeout=".len..]; + } else if (std.mem.eql(u8, a, "--compress-after") or std.mem.eql(u8, a, "--delete-after")) { const v: ?[]const u8 = if (i + 1 < args.len) args[i + 1] else null; if (v == null) die("{s} needs a number of days", .{a}); i += 1; @@ -585,11 +655,25 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { delete_after = parseDays("--delete-after", delete_v); if (delete_after.? <= compress_after) die("--delete-after must be later than --compress-after", .{}); } + const lock_timeout = if (lock_v.len > 0) parseSeconds(lock_v) else default_lock_timeout; const now_s = c.getenv("EB_KEEP_LOGS_NOW") orelse ""; - const now: i64 = if (now_s.len > 0) + const fixed_now: ?i64 = if (now_s.len > 0) std.fmt.parseInt(i64, now_s, 10) catch die("EB_KEEP_LOGS_NOW must be seconds since the epoch, not '{s}'", .{now_s}) - else blk: { + else + null; + const lock = openLock(root) catch |err| die("cannot lock {s}/{s}: {t}", .{ joinable(root_p), rotate_lock, err }); + defer sys.close(lock); + waitLock(lock, lock_timeout) catch |err| switch (err) { + error.TimedOut => { + cli.err(tool ++ ": another rotate holds the lock {s}/{s}; gave up after {d} s (--lock-timeout)\n", .{ joinable(root_p), rotate_lock, lock_timeout }); + return lock_busy_exit; + }, + // The file system cannot lock: Lustre without -o flock, NFS without + // a lock manager, or NFS for a file open only for reading. + else => warn("cannot lock {s}/{s}: {t}; rotating without the lock", .{ joinable(root_p), rotate_lock, err }), + }; + const now: i64 = fixed_now orelse blk: { var ts: linux.timespec = undefined; _ = linux.clock_gettime(.REALTIME, &ts); break :blk ts.sec; @@ -723,6 +807,50 @@ test "a failed copy does not replace a log already kept" { try std.testing.expectEqual(@as(u32, 0o640), got.stat.perm()); } +/// A directory of the test's own under the .zig-cache where `zig build` was +/// started, made when missing; the caller removes it and frees the path. +fn testDir(gpa: Allocator, name: []const u8) ![]u8 { + var cwd_buf: [sys.path_max]u8 = undefined; + const here = try fsx.realpath(fsx.cwd, ".", &cwd_buf); + const base = try std.fmt.allocPrint(gpa, "{s}/.zig-cache/eb-keep-logs-{s}-{d}", .{ here, name, linux.getpid() }); + errdefer gpa.free(base); + try common.atomic.makedirs(base); + return base; +} + +test "the rotate lock: one holder at a time" { + const gpa = std.testing.allocator; + const base = try testDir(gpa, "lock"); + defer gpa.free(base); + defer fsx.deleteTree(gpa, fsx.cwd, base) catch {}; + const dir = try fsx.openDir(fsx.cwd, base, false); + defer sys.close(dir); + + const held = try openLock(dir); + try waitLock(held, 0); + const other = try openLock(dir); + defer sys.close(other); + try std.testing.expectError(error.TimedOut, waitLock(other, 0)); + sys.close(held); + try waitLock(other, 0); + try std.testing.expect((try fsx.statAt(dir, rotate_lock, false)).isReg()); + + try sys.unlinkat(dir, rotate_lock, 0); + try std.testing.expectEqual(linux.E.SUCCESS, linux.errno(linux.symlinkat("elsewhere", dir, rotate_lock))); + try std.testing.expectError(error.SymLink, openLock(dir)); + try sys.unlinkat(dir, rotate_lock, 0); + _ = try sys.call(linux.mkdirat, .{ dir, rotate_lock, 0o755 }); + try std.testing.expectError(error.IsDir, openLock(dir)); +} + +test "parseSeconds" { + try std.testing.expectEqual(@as(u64, 0), parseSeconds("0")); + try std.testing.expectEqual(@as(u64, 7), parseSeconds("007")); + try std.testing.expectEqual(max_lock_timeout, parseSeconds("999999999")); + try std.testing.expectEqual(max_lock_timeout, parseSeconds("1000000000")); + try std.testing.expectEqual(max_lock_timeout, parseSeconds("123456789012345678901234567890")); +} + test "listed" { try std.testing.expect(listed("/l/a.log\n/l/b.log\n", "/l/b.log")); try std.testing.expect(listed("/l/a.log", "/l/a.log")); diff --git a/tests/differential.sh b/tests/differential.sh index 1be5b00..7b75163 100644 --- a/tests/differential.sh +++ b/tests/differential.sh @@ -17,7 +17,7 @@ export PATH="${work}/bin:${PATH}" diffs=0 # Labels whose difference COMPAT.md documents. -deliberate=("no arguments") +deliberate=("no arguments" "rotate, lock is a symbolic link") # The environment both runs get, as env(1) takes it. run_env=(PATH="${PATH}") # compare LABEL ARGS...: run both with the same arguments. @@ -115,6 +115,17 @@ compare "rotate, delete not later" rotate "${work}/logs" --compress-after 5 --de compare "rotate, empty delete" rotate "${work}/logs" --compress-after 5 --delete-after "" compare "rotate, nothing to do" rotate "${work}/logs" --compress-after 5 --delete-after 9 EB_KEEP_LOGS_NOW='' compare "rotate, empty EB_KEEP_LOGS_NOW" rotate "${work}/logs" --compress-after 5 +compare "rotate, bad lock timeout" rotate "${work}/logs" --compress-after 5 --lock-timeout 1.5 +compare "rotate, lock timeout without a value" rotate "${work}/logs" --compress-after 5 --lock-timeout +compare "rotate, empty lock timeout" rotate "${work}/logs" --compress-after 5 --lock-timeout= +exec {held}>>"${work}/logs/.eb-keep-logs-rotate.lock" +flock -n "${held}" +compare "rotate, lock held" rotate "${work}/logs" --compress-after 5 --lock-timeout 0 +compare "rotate, lock held, zeros" rotate "${work}/logs/" --compress-after 5 --lock-timeout=00 +exec {held}>&- +mkdir -p "${work}/symlock" +ln -s "${work}/nowhere" "${work}/symlock/.eb-keep-logs-rotate.lock" +compare "rotate, lock is a symbolic link" rotate "${work}/symlock" --compress-after 5 echo "${diffs} differences" [ "${diffs}" -eq 0 ] diff --git a/tests/test_eb_keep_logs.sh b/tests/test_eb_keep_logs.sh index be475aa..ee5bd76 100755 --- a/tests/test_eb_keep_logs.sh +++ b/tests/test_eb_keep_logs.sh @@ -321,7 +321,7 @@ check "help: EB_KEEP_LOGS_NOW" "$("${tool}" --help | grep -c 'EB_KEEP_LOGS_NOW') # rotate: the run rotate writes into is still going and is left alone. A run # that started before midnight is a day old just after it. next_day=$(date -d '2026-11-14 00:10' +%s) -lsrun() { find "$1" -mindepth 1 -maxdepth 1 -printf '%f\n' | sort | tr '\n' ' '; } +lsrun() { find "$1" -mindepth 1 -maxdepth 1 ! -name .eb-keep-logs-rotate.lock -printf '%f\n' | sort | tr '\n' ' '; } mkrun() { mkdir -p "$1/20261113_235000/site-zen4" "$1/20261110_120000" echo build > "$1/20261113_235000/site-zen4/F.stdout.log" @@ -419,5 +419,80 @@ check "an old mtime on a young run: not compressed" "$(lsrun "${mt_root}/2026111 "${tool}" rotate "${mt_root}" --compress-after 30 --delete-after 40 >/dev/null check "a new mtime on an old run: deleted" "$(lsrun "${mt_root}")" "20261110_120000 " +# rotate: one rotate of a tree at a time, holding ROOT/.eb-keep-logs-rotate.lock +lk_root="${work}/locked" +lock="${lk_root}/.eb-keep-logs-rotate.lock" +mkdir -p "${lk_root}/20261001_120000" +echo old > "${lk_root}/20261001_120000/M.stdout.log" +exec {held}>>"${lock}" +flock -n "${held}" +check "lock held: refused at once" "$("${tool}" rotate "${lk_root}" --compress-after 30 --lock-timeout 0 2>&1; echo "exit $?")" \ + "eb-keep-logs: another rotate holds the lock ${lock}; gave up after 0 s (--lock-timeout) +exit 75" +check "lock held: nothing compressed" "$(lsrun "${lk_root}/20261001_120000")" "M.stdout.log " +start=$(date +%s%N) +"${tool}" rotate "${lk_root}" --compress-after 30 --lock-timeout=1 > /dev/null 2>&1 +check "lock held: refused after --lock-timeout" "$?:$(( $(date +%s%N) - start >= 1000000000 ))" "75:1" +(sleep 1; flock -u "${held}") & +check "lock let go: the waiting rotate rotates" "$("${tool}" rotate "${lk_root}" --compress-after 30 2>&1)" \ + "eb-keep-logs rotate: 1 file compressed, 0 runs deleted under ${lk_root}" +wait +exec {held}>&- +check "the lock stays in ROOT" "$(stat -c '%F' "${lock}")" "regular empty file" +chmod 444 "${lock}" +exec {held}<"${lock}" +flock -n "${held}" +check "a lock this user may only read: still taken" "$("${tool}" rotate "${lk_root}" --compress-after 30 --lock-timeout 0 >/dev/null 2>&1; echo $?)" "75" +exec {held}<&- +check "a lock this user may only read: rotate runs" "$("${tool}" rotate "${lk_root}" --compress-after 30 2>&1)" \ + "eb-keep-logs rotate: 0 files compressed, 0 runs deleted under ${lk_root}" +chmod 644 "${lock}" +"${tool}" rotate "${lk_root}" --compress-after 30 --lock-timeout x 2>/dev/null +check "non-numeric --lock-timeout refused" "$?" "2" +"${tool}" rotate "${lk_root}" --compress-after 30 --lock-timeout 2>/dev/null +check "--lock-timeout without a value refused" "$?" "2" +mkdir -p "${work}/symlock" +ln -s "${work}/nowhere" "${work}/symlock/.eb-keep-logs-rotate.lock" +"${tool}" rotate "${work}/symlock" --compress-after 30 >/dev/null 2>&1 +check "a symbolic link at the lock: refused" "$?" "2" +check "a symbolic link at the lock: not followed" "$(find "${work}" -maxdepth 1 -name nowhere | wc -l)" "0" +# A file system that cannot lock, as flock(1) reports it to the shell script; +# the Zig build calls flock(2), which locks here. +mkdir -p "${work}/nolock-bin" "${work}/nolock/20261001_120000" +printf '#!/bin/sh\necho "flock: 3: Function not implemented" >&2\nexit 65\n' > "${work}/nolock-bin/flock" +chmod +x "${work}/nolock-bin/flock" +echo old > "${work}/nolock/20261001_120000/N.stdout.log" +check "a file system that cannot lock: rotates without the lock" \ + "$(PATH="${work}/nolock-bin:${PATH}" "${tool}" rotate "${work}/nolock" --compress-after 30 2>/dev/null)" \ + "eb-keep-logs rotate: 1 file compressed, 0 runs deleted under ${work}/nolock" + +# rotate: two rotations of one tree started together; one waits for the other +race="${work}/race" +race_lists="" race_out="" +for round in 1 2 3; do + rm -rf "${race}" + for t in a b; do + d="${race}/20261001_120000/${t}" + mkdir -p "${d}" + for i in 1 2 3 4 5 6; do seq 1 20000 > "${d}/n${i}.log"; done + for i in 1 2 3; do echo "$(realpath "${d}")/n${i}.log"; done > "${d}/.licensed-logs" + done + "${tool}" rotate "${race}" --compress-after 30 > "${work}/race1" 2>&1 & + r1=$! + "${tool}" rotate "${race}" --compress-after 30 > "${work}/race2" 2>&1 & + r2=$! + wait "${r1}"; s1=$? + wait "${r2}"; s2=$? + for t in a b; do + d="$(realpath "${race}/20261001_120000/${t}")" + [ "$(cat "${d}/.licensed-logs")" = "$(printf '%s\n' "${d}/n1.log.gz" "${d}/n2.log.gz" "${d}/n3.log.gz")" ] || + race_lists+=" round ${round} ${t}" + done + race_out+="${s1} ${s2} $(cat "${work}/race1" "${work}/race2" | sed 's/ under .*//' | sort | tr '\n' ' ')"$'\n' +done +check "two rotations at once: every list whole" "${race_lists}" "" +check "two rotations at once: one rotates, the other then finds nothing" "$(printf '%s' "${race_out}")" \ + "$(for _ in 1 2 3; do echo "0 0 eb-keep-logs rotate: 0 files compressed, 0 runs deleted eb-keep-logs rotate: 12 files compressed, 0 runs deleted "; done)" + if [ "${fails}" -ne 0 ]; then echo "${fails} failed"; exit 1; fi echo "all passed" From 1ca4c60aba4fd580f8359393e4bf4c832f799aa1 Mon Sep 17 00:00:00 2001 From: Rohit Goswami Date: Mon, 16 Nov 2026 23:41:37 +0000 Subject: [PATCH 2/3] fix(rotate): leave alone the runs that builds still write into rotate passed over only the run it wrote into itself. A run's age counts from the local midnight of the date in its name, so a build of another run that started before midnight UTC was a day old just after it: with --compress-after 1, a rotation from a run started after midnight compressed the EasyBuild log that build still wrote into, and the build exited 1 when it copied the log into the installation. Past --delete-after, such a run was removed while its build ran. After it takes its lock, and before it compresses or deletes anything, rotate now reads the links /proc/PID/fd/N of every process whose descriptors it may read. A link with its owner's write bit names a file open for writing, and the run that file lies in by its path, also once the file is removed, is left whole to a later rotate, as the run rotate writes into is. A file that is only read does not hold a run back. The builds eb-pipeline starts are seen on its own host even when srun runs them elsewhere, as eb-pipeline hands each build's NAME.stdout.log to the command it starts. The shell script finds the links with find -perm -u=w, the Zig build with statx and readlinkat. The command-line suite reproduces the case: a run started at 23:50 UTC whose session log another process holds open for writing, rotated at 00:10 UTC with --compress-after 1 from a run started after midnight. Its files stay as they are while an older run is compressed, and the rotate after the writer ends compresses them. A run that a process only reads is compressed, and a run still written into is not deleted past --delete-after. The checks of the live run and of --delete-after fail on the previous commit, with the shell script and with the Zig build. The suite's listings sort in the C locale, so names that differ in case come out in one order. A unit test covers a file open for writing, one only read and one removed while open; the differential run compares a run past --delete-after that is still written into. COMPAT.md, the cli reference, both how-tos and the explanation of a tree in use describe the runs left alone and the limits of the scan: the same host, the processes rotate may look into, and the path the tree has there. --- COMPAT.md | 1 + README.md | 3 +- changelog.d/+rotate-live-runs.bugfix.md | 1 + docs/orgmode/explanation/rotate-in-use.org | 32 ++++++++- docs/orgmode/howto/call-from-a-pipeline.org | 1 + docs/orgmode/howto/rotate-a-log-tree.org | 5 +- docs/orgmode/reference/cli.org | 4 +- eb-keep-logs | 24 +++++-- src/main.zig | 72 ++++++++++++++++++++- tests/differential.sh | 8 +++ tests/test_eb_keep_logs.sh | 57 +++++++++++++++- 11 files changed, 195 insertions(+), 13 deletions(-) create mode 100644 changelog.d/+rotate-live-runs.bugfix.md diff --git a/COMPAT.md b/COMPAT.md index 18f8d20..2bbdc46 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -18,6 +18,7 @@ on the same invocations. | files | kept files closed to others (command streams mode 0600), their resolved paths appended to `LOGDIR/.licensed-logs` unless already listed; on a failure, `NAME.stdout.log`, a TMP_LOGDIR below LOGDIR and the regular files directly in it closed to others, a symbolic link left as it is; `rotate` writes `FILE.gz` with the file's mode and updates `.licensed-logs` to the `.gz` path | | run ages | days from the local midnight of the date in the run's name, as `date -d YYYYMMDD` computes it, including TZ rules, except for the zones under unintended differences; a run whose local midnight does not exist is skipped | | the run in progress | `rotate` neither compresses nor deletes the run that holds its stdout or stderr, when either is a regular file, or `EB_PIPELINE_LOG_DIR` | +| runs still written into | after taking its lock, `rotate` reads the links `/proc/PID/fd/N` of every process whose descriptors it may read; a run that holds the path a link with its owner's write bit names, also when the path ends in ` (deleted)`, is neither compressed nor deleted; without `/proc`, no run is left alone for this | ## Deliberate differences diff --git a/README.md b/README.md index 53c5106..3609610 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,8 @@ eb-keep-logs rotate /srv/build-logs --compress-after 30 --delete-after 365 It gzips the `*.log` and `*.txt` files of runs older than `--compress-after` days, keeps each file's mode, and updates the run's `.licensed-logs`. It removes whole runs older than `--delete-after` days only when that option is given. It leaves alone the run it writes into, which is still going: the run that holds its standard output or error, and the run that holds `EB_PIPELINE_LOG_DIR`, the log directory `eb-pipeline` exports, because srun passes the output on over a pipe. -The next `rotate` takes that run. +It also leaves alone every run that a process on the same host has a file in open for writing, as a build that is still going has its log, whichever run started the rotation: it finds them in `/proc/PID/fd`, for the processes whose descriptors it may read. +A later `rotate` takes such a run. Rotations of one tree take turns: each holds an exclusive flock on `ROOT/.eb-keep-logs-rotate.lock` while it runs. One that starts while another runs waits for it, up to `--lock-timeout` seconds (600; `0` does not wait), and when the wait runs out it changes nothing and exits 75. diff --git a/changelog.d/+rotate-live-runs.bugfix.md b/changelog.d/+rotate-live-runs.bugfix.md new file mode 100644 index 0000000..6c27f37 --- /dev/null +++ b/changelog.d/+rotate-live-runs.bugfix.md @@ -0,0 +1 @@ +`rotate` leaves alone every run that a process on the same host has a file in open for writing, as a build that is still going has its log, not only the run it writes into itself. A build of another run that started before midnight UTC is a day old just after it, so with `--compress-after 1` a rotation from any other run compressed the EasyBuild log the build still wrote into, and the build exited 1 when it went to copy that log into the installation; past `--delete-after` such a run was removed. `rotate` reads the links in `/proc/PID/fd` of the processes whose descriptors it may read, and a later rotation takes the run once its build has ended. A process that only reads a run's files does not hold the run back. diff --git a/docs/orgmode/explanation/rotate-in-use.org b/docs/orgmode/explanation/rotate-in-use.org index fb22080..952ffc1 100644 --- a/docs/orgmode/explanation/rotate-in-use.org +++ b/docs/orgmode/explanation/rotate-in-use.org @@ -4,7 +4,7 @@ * Rotating a tree in use -Why does =eb-keep-logs rotate= take a lock, how does that lock sit beside eb-pipeline's, and why can the two never wait on each other for good? +Why does =eb-keep-logs rotate= take a lock, how does that lock sit beside eb-pipeline's, why can the two never wait on each other for good, and how does =rotate= keep away from builds that are still going? ** The problem @@ -13,6 +13,10 @@ Two rotations of one tree that run at the same time compress the same run betwee The list then names files that are gone, and both rotations exit 0. Started together on a tree of three runs, two rotations left such a list in most of 200 rounds, with the shell script and with the Zig build. +Builds also write into the tree while it is rotated. +A run's age counts from the local midnight of the date in its name, so a run that started just before midnight is a day old just after it, and =--compress-after 1= takes it while its builds still run. +EasyBuild keeps its log open for writing until a build ends, and then copies it into the installation; a rotation that compressed it in between left the build writing into a removed file, and the copy failed, so a build that had done its work exited 1. + ** How eb-keep-logs handles it Every =rotate= of a tree holds an exclusive lock on one file in it, =ROOT/.eb-keep-logs-rotate.lock=, from before it lists =ROOT= until it exits. @@ -57,6 +61,20 @@ flock -w 3600 -E 75 ~/build-logs/.eb-pipeline-rotate.lock \ It waits, up to an hour, for a moment when no eb-pipeline run is working, and exits 75 when none comes; while it rotates, eb-pipeline runs that start wait for it. The opposite order, a wait for eb-pipeline's lock inside =rotate='s, is the one way to make two processes wait on each other, and then until =--lock-timeout= runs out. +** Builds still going + +Before it compresses or deletes anything, =rotate= looks for the runs that are still written into, and leaves them, whole, to a later rotation: + +- the run that holds the file its standard output or standard error goes to; +- the run that holds =EB_PIPELINE_LOG_DIR=, the log directory eb-pipeline exports, as srun carries the output of an eb-pipeline phase over a pipe; +- every run that a process on this host has a file in open for writing. + +It finds the last in =/proc=: the link =/proc/PID/fd/N= names the file a process has open as descriptor =N=, and has its owner's write bit when the process may write to it. +A file that is only read does not hold a run back, and a file removed while it is still open for writing still does, by the path it had. +Once the build has ended, the next rotation takes its run as it takes any other. + +The builds eb-pipeline starts are seen on the host eb-pipeline runs on even when srun runs them on another node: eb-pipeline opens each build's =NAME.stdout.log= in the run and hands it to the command it starts, which holds it open for writing until the build ends. + ** What was considered and turned down - /Sharing eb-pipeline's lock file./ @@ -68,6 +86,12 @@ The opposite order, a wait for eb-pipeline's lock inside =rotate='s, is the one A file in =ROOT= is seen wherever the tree is, and a flock(2) lock cannot outlive its holder as a PID file can. - /Refusing to rotate where the file system cannot lock./ eb-pipeline goes on without its own lock there; a refusal would turn that into a failed logs phase in every run. +- /Judging a run by the times its files changed./ + A build can write nothing into its run for hours, through one long compile step, so a quiet run is no sign that its builds have ended. +- /A lock or a marker file that each build holds./ + It would reach builds on other hosts, but only the builds that take it, and EasyBuild takes none; for eb-pipeline's runs, its own lock around a daily job does that already. +- /Matching open files by device and inode, not by path./ + It would also see a process that has the tree mounted at another path, as in a container, at the cost of a walk through every run before it is rotated; the builds eb-pipeline starts are seen by path already, through the output file it holds on its own host. ** Limits @@ -75,9 +99,13 @@ The opposite order, a wait for eb-pipeline's lock inside =rotate='s, is the one - On NFS an exclusive lock needs the file open for writing, so a user who may only read the lock file there cannot take it. - On a file system that cannot lock at all, or cannot lock for that user, =rotate= says so on stderr and rotates without the lock, as eb-pipeline does without its own. - The lock keeps rotations apart, not other writers: a person, or a tool that does not take it, can still change the tree while =rotate= runs. +- =rotate= sees the writers on its own host, and of those only the processes whose descriptors it may read in =/proc=: the user's own, except one running a set-user-ID program, and every process for root. +- A process that has the tree at another path, as in a container that mounts it elsewhere, is not seen, nor one that opens a file in an old run after =rotate= has looked. +- A build on another host is seen only through a process on this one that writes into its run, as eb-pipeline's output file is; a daily job on another host than eb-pipeline's takes eb-pipeline's lock around =rotate=, as above, to wait for its runs. ** Further reading - [[file:../howto/rotate-a-log-tree.org][Rotate a log tree]], from a pipeline or a daily job. -- [[file:../reference/cli.org][The eb-keep-logs command]]: =--lock-timeout=, exit status 75 and the lock file. +- [[file:../reference/cli.org][The eb-keep-logs command]]: =--lock-timeout=, exit status 75, the lock file and the runs left alone. - [[man:flock(2)]], the lock, and [[man:flock(1)]], the command the shell script and the daily job use. +- [[man:proc(5)]], for the links in =/proc/PID/fd=. diff --git a/docs/orgmode/howto/call-from-a-pipeline.org b/docs/orgmode/howto/call-from-a-pipeline.org index 8a788f5..57aaa0a 100644 --- a/docs/orgmode/howto/call-from-a-pipeline.org +++ b/docs/orgmode/howto/call-from-a-pipeline.org @@ -52,6 +52,7 @@ The script keeps each failed build's logs next to its output and, at the end of #+end_src =rotate= leaves alone the run that holds its output, so this run stays as it is however long it took, and a later run's =rotate= compresses it once it is old enough. + It leaves alone as well the runs of other jobs whose builds still write their logs, when they run on the same host. When another run's =rotate= of the tree is under way, this one waits for it, up to 600 seconds; with =--lock-timeout 0= it leaves the work to that one at once and exits 75. ** Check that it worked diff --git a/docs/orgmode/howto/rotate-a-log-tree.org b/docs/orgmode/howto/rotate-a-log-tree.org index bd9342d..ae38045 100644 --- a/docs/orgmode/howto/rotate-a-log-tree.org +++ b/docs/orgmode/howto/rotate-a-log-tree.org @@ -45,6 +45,7 @@ Use this to keep a directory of build runs from growing without end: compress th No run is 30 days old yet, so nothing changed. Run the same command at the end of each pipeline run, as the pipeline how-to does, or from a daily job. Rotations of one tree take turns: one that starts while another runs waits for it, up to =--lock-timeout= seconds, 600 by default, and then finds the work done. + A run that a build still writes into is left, whatever its age, for a rotation after the build has ended. 4. To rotate from a daily job, add the command to your crontab with =crontab -e=, its output going to a file outside the tree: #+begin_example @@ -89,11 +90,11 @@ drwxrwxr-x 3 alice alice 4096 Nov 16 17:53 easybuild-logs | =eb-keep-logs: another rotate holds the lock PATH; gave up after 600 s (--lock-timeout)= | Another rotation of the tree ran for longer than =--lock-timeout=; this one changed nothing and exited 75. | Nothing, when the other does the work; else rotate again later, or raise =--lock-timeout=. | | =eb-keep-logs: cannot lock PATH: ERROR= | The lock file =ROOT/.eb-keep-logs-rotate.lock= cannot be opened, or is a symbolic link or directory; exit status 2. The shell script leaves out =: ERROR=. | Remove what is at that name, or rotate as a user who may write =ROOT=. | | =eb-keep-logs: cannot lock PATH: ERROR; rotating without the lock= | The file system cannot lock, as Lustre mounted without its =flock= option cannot; =rotate= goes on. | Rotate the tree from one place only, so that rotations cannot overlap. | -| A run is not compressed, and there is no message | Its name is not =YYYYMMDD_HHMMSS=, or it holds the output of this =rotate= or =EB_PIPELINE_LOG_DIR=. | Rename it, or leave it to the next =rotate=, after the run has ended. | +| A run is not compressed or removed, and there is no message | Its name is not =YYYYMMDD_HHMMSS=, or it holds the output of this =rotate= or =EB_PIPELINE_LOG_DIR=, or a process on this host has a file in it open for writing, as a build that is still going does. | Rename it, or leave it to a later =rotate=, after the run has ended; =lsof +D ~/build-logs/RUN= shows a process that still has a file in it open. | ** Related - [[file:mark-licensed-logs.org][Mark logs that may carry a licence number]]: =rotate= renames the lines of =.licensed-logs= to the =.gz= files. - [[file:../explanation/local-time.org][Run ages and local time]] -- [[file:../explanation/rotate-in-use.org][Rotating a tree in use]]: the lock, and how it sits beside eb-pipeline's. +- [[file:../explanation/rotate-in-use.org][Rotating a tree in use]]: the lock, how it sits beside eb-pipeline's, and builds still going. - [[file:../reference/cli.org][The eb-keep-logs command]] diff --git a/docs/orgmode/reference/cli.org b/docs/orgmode/reference/cli.org index 0f50d81..9184608 100644 --- a/docs/orgmode/reference/cli.org +++ b/docs/orgmode/reference/cli.org @@ -137,8 +137,10 @@ What =failed= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe | =ROOT/RUN/.../FILE.log=, =FILE.txt= | In a run old enough, replaced by =FILE.gz= with the file's mode, as =gzip -9= does; a symbolic link is not followed, and a file with other hard links or an existing =FILE.gz= is left as it is, with a message on stderr. | | =ROOT/RUN/.../.licensed-logs= | In a run in which files were compressed, a line that names one of them, by its absolute path with symbolic links resolved, is changed to name its =.gz= file. A list that changes keeps its owner when permitted, its mode and its access control list. | | =ROOT/RUN= | Removed with everything in it, when old enough for =--delete-after=, without being compressed first. A symbolic link in it is removed, not followed. Removal stops at the first error. | +| =/proc/PID/fd/N= | Read, for every process whose descriptors =rotate= may read, after it has the lock: a link with its owner's write bit names a file the process has open for writing, and the run that file is in is left alone. Without =/proc=, no run is left alone for it. | -A run that holds the file =rotate='s standard output or standard error goes to, or =EB_PIPELINE_LOG_DIR=, is left alone; the next =rotate= takes it. +A run that holds the file =rotate='s standard output or standard error goes to, or =EB_PIPELINE_LOG_DIR=, or a file another process has open for writing, as a build that is still going has its log, is left alone, neither compressed nor deleted; a later =rotate= takes it. +The path is what counts: the path the link in =/proc= names, or for a file removed since, that path with =" (deleted)"= after it, has to start with =ROOT='s, with symbolic links resolved. On stdout =rotate= prints one line, with =ROOT= as given: #+begin_example diff --git a/eb-keep-logs b/eb-keep-logs index 9c76de9..bad1d39 100755 --- a/eb-keep-logs +++ b/eb-keep-logs @@ -246,6 +246,21 @@ _run_of() { esac } +# "/NAME" for each run directly under ROOT that a process has a file in open +# for writing, of the processes whose descriptors this one may read in /proc: +# a build that is still going, whichever run it writes into. A link +# /proc/PID/fd/N has its owner's write bit when N is open for writing; for a +# file removed since, the link names the path it had, with " (deleted)" added. +_written_runs() { + local root target + root=$(realpath -e -- "$1") || return 0 + root="${root%/}" + while IFS= read -r -d '' target; do + _run_of "${root}" "${target}" + done < <(find /proc/[0-9]*/fd -mindepth 1 -maxdepth 1 -type l -perm -u=w -printf '%l\0' 2>/dev/null) + return 0 +} + # Hold an exclusive flock on the lock file LOCK until this rotate exits, # waiting up to SECONDS for another holder; exit 75 when it does not let go. # Opened for reading when it cannot be opened for writing, by a user who may @@ -298,14 +313,15 @@ cmd_rotate() { lock_timeout=600 fi _lock_rotate "$(_joinable "${root}")/.eb-keep-logs-rotate.lock" "${lock_timeout}" - local run age file list path compressed=0 deleted=0 own - own="$(_own_runs "${root}")/" + local run age file list path compressed=0 deleted=0 live + live="$(_own_runs "${root}")$(_written_runs "${root}")/" for run in "$(_joinable "${root}")"/*/; do run="${run%/}" [ -d "${run}" ] && [ ! -L "${run}" ] || continue age=$(_run_age_days "${run}") || continue - # The next rotate takes the run this one writes into. - case "${own}" in *"/${run##*/}/"*) continue ;; esac + # A later rotate takes a run that is still written into, by this + # rotate or by a build that is still going. + case "${live}" in *"/${run##*/}/"*) continue ;; esac if [ -n "${delete_after}" ] && [ "${age}" -ge "${delete_after}" ]; then rm -rf -- "${run}" deleted=$((deleted + 1)) diff --git a/src/main.zig b/src/main.zig index 9e7eecb..c019fde 100644 --- a/src/main.zig +++ b/src/main.zig @@ -605,6 +605,44 @@ fn contains(names: []const []const u8, name: []const u8) bool { return false; } +/// The runs directly under `root` that a process in `procfs` has a file in +/// open for writing, of the processes whose descriptors this one may read: a +/// build that is still going, whichever run it writes into. A link PID/fd/N +/// has its owner's write bit when N is open for writing; for a file removed +/// since, the link names the path it had, with " (deleted)" added. +fn writtenRuns(gpa: Allocator, arena: Allocator, procfs: []const u8, root: fsx.fd_t) []const []const u8 { + var names: std.ArrayList([]const u8) = .empty; + var rb: [sys.path_max]u8 = undefined; + const real = fsx.realpathFd(root, &rb) catch return &.{}; + const here = arena.dupe(u8, if (std.mem.eql(u8, real, "/")) "" else real) catch return &.{}; + const pd = fsx.openDir(fsx.cwd, procfs, false) catch return &.{}; + defer sys.close(pd); + var it = fsx.DirIter.init(gpa, pd) catch return &.{}; + defer it.deinit(gpa); + while (it.next() catch null) |p| { + if (!allDigits(p.name)) continue; + var fb: [32]u8 = undefined; + const fd_dir = std.fmt.bufPrint(&fb, "{s}/fd", .{p.name}) catch continue; + const fds = fsx.openDir(pd, fd_dir, false) catch continue; + defer sys.close(fds); + var fit = fsx.DirIter.init(gpa, fds) catch continue; + defer fit.deinit(gpa); + while (fit.next() catch null) |f| { + const st = fsx.statAt(fds, f.name, false) catch continue; + if (st.mode & 0o200 == 0) continue; + addRunOf(arena, &names, here, readLinkAt(fds, f.name, &rb) catch continue); + } + } + return names.items; +} + +/// The target of the symbolic link `name` in `dir`. +fn readLinkAt(dir: fsx.fd_t, name: []const u8, out: *[sys.path_max]u8) sys.Error![]const u8 { + var zb: [sys.path_max:0]u8 = undefined; + const n = try sys.call(linux.readlinkat, .{ dir, (try sys.z(&zb, name)).ptr, out, out.len }); + return out[0..n]; +} + fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { var root_s: ?[]const u8 = null; var compress_v: ?[]const u8 = null; @@ -701,12 +739,14 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { }.lt); const own = ownRuns(c, arena, root); + const written = writtenRuns(c.gpa, arena, "/proc", root); var compressed: usize = 0; var deleted: usize = 0; for (runs.items) |run| { const age = runAgeDays(&zone, run, now) orelse continue; - // The next rotate takes the run this one writes into. - if (contains(own, run)) continue; + // A later rotate takes a run that is still written into, by this + // rotate or by a build that is still going. + if (contains(own, run) or contains(written, run)) continue; if (delete_after) |d| if (age >= @as(i64, @intCast(@min(d, std.math.maxInt(i64))))) { fsx.deleteTree(c.gpa, root, run) catch |err| { warn("cannot delete {s}/{s}: {t}", .{ joinable(root_p), run, err }); @@ -843,6 +883,34 @@ test "the rotate lock: one holder at a time" { try std.testing.expectError(error.IsDir, openLock(dir)); } +test "writtenRuns: the runs a process has a file in open for writing" { + const gpa = std.testing.allocator; + const base = try testDir(gpa, "written"); + defer gpa.free(base); + defer fsx.deleteTree(gpa, fsx.cwd, base) catch {}; + const root = try fsx.openDir(fsx.cwd, base, false); + defer sys.close(root); + for ([_][:0]const u8{ "20261113_235000", "20261001_120000", "20261002_120000" }) |run| + _ = try sys.call(linux.mkdirat, .{ root, run.ptr, 0o755 }); + + const writing = try sys.openat(root, "20261113_235000/build.log", .{ .ACCMODE = .WRONLY, .APPEND = true, .CREAT = true }, 0o644); + defer sys.close(writing); + const reading = try sys.openat(root, "20261001_120000/old.log", .{ .CREAT = true }, 0o644); + defer sys.close(reading); + const removed = try sys.openat(root, "20261002_120000/gone.log", .{ .ACCMODE = .RDWR, .CREAT = true }, 0o644); + defer sys.close(removed); + try sys.unlinkat(root, "20261002_120000/gone.log", 0); + + var arena_state = std.heap.ArenaAllocator.init(gpa); + defer arena_state.deinit(); + const arena = arena_state.allocator(); + const names = writtenRuns(gpa, arena, "/proc", root); + try std.testing.expect(contains(names, "20261113_235000")); + try std.testing.expect(!contains(names, "20261001_120000")); + try std.testing.expect(contains(names, "20261002_120000")); + try std.testing.expectEqual(@as(usize, 0), writtenRuns(gpa, arena, "/nonexistent", root).len); +} + test "parseSeconds" { try std.testing.expectEqual(@as(u64, 0), parseSeconds("0")); try std.testing.expectEqual(@as(u64, 7), parseSeconds("007")); diff --git a/tests/differential.sh b/tests/differential.sh index 7b75163..d165af2 100644 --- a/tests/differential.sh +++ b/tests/differential.sh @@ -126,6 +126,14 @@ exec {held}>&- mkdir -p "${work}/symlock" ln -s "${work}/nowhere" "${work}/symlock/.eb-keep-logs-rotate.lock" compare "rotate, lock is a symbolic link" rotate "${work}/symlock" --compress-after 5 +# The one run is past --delete-after, and another process writes into it. +mkdir -p "${work}/live/20261001_120000" +echo build > "${work}/live/20261001_120000/B.stdout.log" +(exec 3>>"${work}/live/20261001_120000/B.stdout.log" && : > "${work}/held" && exec sleep 60) & +holder=$! +until [ -e "${work}/held" ]; do sleep 0.05; done +compare "rotate, a run still written into" rotate "${work}/live" --compress-after 5 --delete-after 9 +kill "${holder}"; wait "${holder}" 2>/dev/null echo "${diffs} differences" [ "${diffs}" -eq 0 ] diff --git a/tests/test_eb_keep_logs.sh b/tests/test_eb_keep_logs.sh index ee5bd76..e948d53 100755 --- a/tests/test_eb_keep_logs.sh +++ b/tests/test_eb_keep_logs.sh @@ -321,7 +321,7 @@ check "help: EB_KEEP_LOGS_NOW" "$("${tool}" --help | grep -c 'EB_KEEP_LOGS_NOW') # rotate: the run rotate writes into is still going and is left alone. A run # that started before midnight is a day old just after it. next_day=$(date -d '2026-11-14 00:10' +%s) -lsrun() { find "$1" -mindepth 1 -maxdepth 1 ! -name .eb-keep-logs-rotate.lock -printf '%f\n' | sort | tr '\n' ' '; } +lsrun() { find "$1" -mindepth 1 -maxdepth 1 ! -name .eb-keep-logs-rotate.lock -printf '%f\n' | LC_ALL=C sort | tr '\n' ' '; } mkrun() { mkdir -p "$1/20261113_235000/site-zen4" "$1/20261110_120000" echo build > "$1/20261113_235000/site-zen4/F.stdout.log" @@ -353,6 +353,61 @@ check "own run through a pipe: not compressed" "$(lsrun "${log_dir}")" "F.stdout EB_KEEP_LOGS_NOW=${next_day} "${tool}" rotate "${piped}" --compress-after 1 > /dev/null 2>&1 check "a finished run is compressed" "$(lsrun "${log_dir}")" "F.stdout.log.gz rotate.log.gz " +# rotate: a build of another run, started before midnight UTC, is still +# going: EasyBuild holds its session log open for writing, and copies it to +# the installation at the end. rotate runs from a run started after midnight. +# hold MODE FILE: a process that holds FILE open, for writing with >> and for +# reading with <, until it is killed; its PID is in ${holder}. +hold() { + rm -f "${work}/held" + case "$1" in + '>>') (exec 3>>"$2" && : > "${work}/held" && exec sleep 60) & ;; + '<') (exec 3<"$2" && : > "${work}/held" && exec sleep 60) & ;; + esac + holder=$! + until [ -e "${work}/held" ]; do sleep 0.05; done +} +utc_next_day=$(TZ=UTC date -d '2026-11-14 00:10' +%s) +live="${work}/live" +live_dir="${live}/20261113_235000/site-zen3/easybuild-logs/Foo-1.0" +mkdir -p "${live_dir}" "${live}/20261114_000500/site-zen4" "${live}/20261001_120000" +echo started > "${live_dir}/easybuild-Foo-1.0-20261113.235001.log" +echo build > "${live}/20261113_235000/site-zen3/Foo-1.0.stdout.log" +echo older > "${live}/20261001_120000/G.stdout.log" +hold '>>' "${live_dir}/easybuild-Foo-1.0-20261113.235001.log" +log="${live}/20261114_000500/site-zen4/rotate.log" +TZ=UTC EB_KEEP_LOGS_NOW=${utc_next_day} "${tool}" rotate "${live}" --compress-after 1 > "${log}" 2>&1 +check "a build still going in another run: its log left alone" "$(lsrun "${live_dir}")" "easybuild-Foo-1.0-20261113.235001.log " +check "a build still going in another run: nothing in its run compressed" \ + "$(lsrun "${live}/20261113_235000/site-zen3")" "Foo-1.0.stdout.log easybuild-logs " +check "a build still going in another run: an older run compressed" "$(lsrun "${live}/20261001_120000")" "G.stdout.log.gz " +check "a build still going in another run: rotate's summary" "$(cat "${log}")" \ + "eb-keep-logs rotate: 1 file compressed, 0 runs deleted under ${live}" +kill "${holder}"; wait "${holder}" 2>/dev/null +TZ=UTC EB_KEEP_LOGS_NOW=${utc_next_day} "${tool}" rotate "${live}" --compress-after 1 > /dev/null 2>&1 +check "a build that has ended: its run compressed" "$(lsrun "${live_dir}")$(lsrun "${live}/20261113_235000/site-zen3")" \ + "easybuild-Foo-1.0-20261113.235001.log.gz Foo-1.0.stdout.log.gz easybuild-logs " + +# A process that only reads a run's logs does not hold the run back. +read_root="${work}/read" +mkdir -p "${read_root}/20261113_235000" +echo build > "${read_root}/20261113_235000/R.stdout.log" +hold '<' "${read_root}/20261113_235000/R.stdout.log" +TZ=UTC EB_KEEP_LOGS_NOW=${utc_next_day} "${tool}" rotate "${read_root}" --compress-after 1 > /dev/null 2>&1 +check "a run a process only reads: compressed" "$(lsrun "${read_root}/20261113_235000")" "R.stdout.log.gz " +kill "${holder}"; wait "${holder}" 2>/dev/null + +# Nor is a run that is still written into deleted past --delete-after. +gone_live="${work}/gone-live" +mkdir -p "${gone_live}/20261001_120000" "${gone_live}/20261002_120000" +echo hung > "${gone_live}/20261001_120000/H.stdout.log" +echo ended > "${gone_live}/20261002_120000/I.stdout.log" +hold '>>' "${gone_live}/20261001_120000/H.stdout.log" +TZ=UTC EB_KEEP_LOGS_NOW=${utc_next_day} "${tool}" rotate "${gone_live}" --compress-after 1 --delete-after 30 > /dev/null 2>&1 +check "a run still written into past --delete-after: kept as it is" "$(lsrun "${gone_live}")$(lsrun "${gone_live}/20261001_120000")" \ + "20261001_120000 H.stdout.log " +kill "${holder}"; wait "${holder}" 2>/dev/null + # rotate: an empty EB_KEEP_LOGS_NOW is the current time, as an unset one is now_root="${work}/now" old_run="$(TZ=UTC date -d '40 days ago' +%Y%m%d)_000000" From b5700876cb435f495e37bc2e45afb1cda555caa4 Mon Sep 17 00:00:00 2001 From: Rohit Goswami Date: Mon, 16 Nov 2026 23:49:18 +0000 Subject: [PATCH 3/3] fix(rotate): replace each .licensed-logs whole, through a file of its own The shell script wrote every new .licensed-logs to the fixed name .licensed-logs.tmp beside it and renamed that over the list. Two rotations at once wrote into that one file together, and the one that renamed it first could put a torn list in place: with a flock(1) that fails as on a file system that cannot lock, two shell rotations at once left 73 of 120 lists with a line that names none of the run's logs, or a log named twice or not at all. A symbolic link at the fixed name was written through, and then renamed over the list in its place. The shell script now makes the new list with mktemp in the list's directory, keeps the list's mode on it and renames it over the list with mv -f; a failed rewrite removes it. In the same 120 lists, none was torn; 49 named a log by the name it had before the other rotation compressed it, which the lock prevents. The Zig build already wrote the new list beside the old one with O_EXCL, synced it and renamed it over the list; it now also syncs the list's directory, as copyDurable does for the files failed keeps, and says "cannot sync a .licensed-logs: ERROR" on stderr when that fails. The shell script, which syncs nothing anywhere, still syncs nothing. The command-line suite checks that no temporary file is left beside a list, that a symbolic link at the old fixed name is not written through and the list stays a regular file with its mode, that a list its owner may not write is still replaced with its mode, and, for the shell script with flock(1) failing, that two rotations at once in five rounds leave every list whole. The link checks and the check without the lock fail with the shell script of the previous commit. COMPAT.md, the cli reference, the explanation of a tree in use and the README say how a list is replaced, and that a line added to a list while rotate rewrites it goes with the old list. --- COMPAT.md | 4 +- README.md | 2 +- changelog.d/+rotate-atomic-lists.bugfix.md | 1 + docs/orgmode/explanation/rotate-in-use.org | 8 ++++ docs/orgmode/reference/cli.org | 3 +- eb-keep-logs | 10 +++-- src/main.zig | 9 +++- tests/test_eb_keep_logs.sh | 48 ++++++++++++++++++++++ 8 files changed, 76 insertions(+), 9 deletions(-) create mode 100644 changelog.d/+rotate-atomic-lists.bugfix.md diff --git a/COMPAT.md b/COMPAT.md index 2bbdc46..e25911f 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -15,7 +15,7 @@ on the same invocations. | environment | `PATH` (finds `eb`), `EASYBUILD_TMPDIR`, `EB_KEEP_LOGS_NOW`, `EB_PIPELINE_LOG_DIR`, `TZ`, `TZDIR` | | `eb` | `eb --last-log [--tmp-logdir=TMP_LOGDIR] --output-style=basic`, and the same without `--output-style` when that fails or prints nothing; the last line printed is the log | | stdout | `Failed build log: LOGDIR/NAME.failed.ebout.log`, `Failed command output: LOGDIR/NAME.failed.command-{out,err}.txt`, `Failed meson test log: LOGDIR/NAME.failed.meson-testlog.txt`, `Failed command output not kept: EASYBUILD_TMPDIR is not set.`, `Failed command output not kept: PATH is not under EASYBUILD_TMPDIR.`, `Failed command output not kept: PATH belongs to another user.`, `No EasyBuild log to keep for NAME.`, `No EasyBuild log to keep for NAME: eb is not on PATH.`, `No EasyBuild log to keep for NAME: eb --last-log exited N.`, `No EasyBuild log to keep for NAME: PATH is another build's log.`, `eb-keep-logs rotate: N files compressed, M runs deleted under ROOT`, with `1 file` and `1 run` for a count of one; LOGDIR in these lines without its trailing slashes | -| files | kept files closed to others (command streams mode 0600), their resolved paths appended to `LOGDIR/.licensed-logs` unless already listed; on a failure, `NAME.stdout.log`, a TMP_LOGDIR below LOGDIR and the regular files directly in it closed to others, a symbolic link left as it is; `rotate` writes `FILE.gz` with the file's mode and updates `.licensed-logs` to the `.gz` path | +| files | kept files closed to others (command streams mode 0600), their resolved paths appended to `LOGDIR/.licensed-logs` unless already listed; on a failure, `NAME.stdout.log`, a TMP_LOGDIR below LOGDIR and the regular files directly in it closed to others, a symbolic link left as it is; `rotate` writes `FILE.gz` with the file's mode and updates `.licensed-logs` to the `.gz` path, writing the new list to a file of its own beside the old one and renaming it over the list, so the list is always one whole version | | run ages | days from the local midnight of the date in the run's name, as `date -d YYYYMMDD` computes it, including TZ rules, except for the zones under unintended differences; a run whose local midnight does not exist is skipped | | the run in progress | `rotate` neither compresses nor deletes the run that holds its stdout or stderr, when either is a regular file, or `EB_PIPELINE_LOG_DIR` | | runs still written into | after taking its lock, `rotate` reads the links `/proc/PID/fd/N` of every process whose descriptors it may read; a run that holds the path a link with its owner's write bit names, also when the path ends in ` (deleted)`, is neither compressed nor deleted; without `/proc`, no run is left alone for this | @@ -36,7 +36,7 @@ on the same invocations. | a symbolic link or non-regular file at a destination name in LOGDIR | cp writes through it | refused, reported on stderr | | `NAME.stdout.log` or TMP_LOGDIR that cannot be closed to others | chmod's error on stderr | `eb-keep-logs: cannot close PATH to others: ERROR` on stderr | | the failing command's streams | checked by path (`realpath`, `-L`, `stat`), then copied | opened without following a symbolic link at the last component; the same checks run on the open file, and that file is copied | -| `.licensed-logs` after `rotate` | rewritten by every compressed file, as a new file owned by the caller | rewritten once per run when a line changes, keeping its owner (when permitted), mode and access ACL (access control list) entries, replaced atomically | +| `.licensed-logs` after `rotate` | rewritten by every compressed file, as a new file owned by the caller with the list's mode, made by mktemp beside the list and renamed over it; nothing is synced | rewritten once per run when a line changes, keeping its owner (when permitted), mode and access ACL (access control list) entries; written beside the list, synced, renamed over it, and the directory synced, with `eb-keep-logs: cannot sync a .licensed-logs: ERROR` on stderr when that fails | | a run directory that cannot be removed | counted as deleted | reported on stderr, not counted | | compression | runs `gzip -9`, `find` and `awk` | in process; the gzip header (name, mtime, flags, OS) matches `gzip -9`, the compressed bytes differ; a file with other hard links or an existing `.gz` is skipped as gzip skips it | | `eb --last-log` that does not return | waits without a limit | its process group is killed after 300 s, and nothing is kept | diff --git a/README.md b/README.md index 3609610..936fd80 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,7 @@ eb-keep-logs rotate /srv/build-logs --compress-after 30 --delete-after 365 ``` `rotate` looks only at run directories named `YYYYMMDD_HHMMSS` directly under the root, and dates each run by its name. -It gzips the `*.log` and `*.txt` files of runs older than `--compress-after` days, keeps each file's mode, and updates the run's `.licensed-logs`. +It gzips the `*.log` and `*.txt` files of runs older than `--compress-after` days, keeps each file's mode, and updates the run's `.licensed-logs`, writing each new list beside the old one and renaming it over it. It removes whole runs older than `--delete-after` days only when that option is given. It leaves alone the run it writes into, which is still going: the run that holds its standard output or error, and the run that holds `EB_PIPELINE_LOG_DIR`, the log directory `eb-pipeline` exports, because srun passes the output on over a pipe. It also leaves alone every run that a process on the same host has a file in open for writing, as a build that is still going has its log, whichever run started the rotation: it finds them in `/proc/PID/fd`, for the processes whose descriptors it may read. diff --git a/changelog.d/+rotate-atomic-lists.bugfix.md b/changelog.d/+rotate-atomic-lists.bugfix.md new file mode 100644 index 0000000..6717100 --- /dev/null +++ b/changelog.d/+rotate-atomic-lists.bugfix.md @@ -0,0 +1 @@ +The shell script writes each new `.licensed-logs` to a file of its own, made by `mktemp` beside the list, and renames it over the list, where it wrote every new list to the fixed name `.licensed-logs.tmp`. Two rotations at once, as where the file system cannot lock, wrote into that one file together and could rename a torn list into place, and a symbolic link at that name was written through and then took the list's place. A temporary file is removed when the rewrite fails. The Zig build, which already wrote and synced a new list beside the old one and renamed it over it, now also syncs the list's directory, as it does for the files `failed` keeps, and says `cannot sync a .licensed-logs: ERROR` when that fails. diff --git a/docs/orgmode/explanation/rotate-in-use.org b/docs/orgmode/explanation/rotate-in-use.org index 952ffc1..db7bc97 100644 --- a/docs/orgmode/explanation/rotate-in-use.org +++ b/docs/orgmode/explanation/rotate-in-use.org @@ -75,6 +75,13 @@ Once the build has ended, the next rotation takes its run as it takes any other. The builds eb-pipeline starts are seen on the host eb-pipeline runs on even when srun runs them on another node: eb-pipeline opens each build's =NAME.stdout.log= in the run and hands it to the command it starts, which holds it open for writing until the build ends. +** Lists replaced whole + +=rotate= changes a run's =.licensed-logs= by writing the new list to a file of its own beside the old one and renaming that file over the list, which the kernel does in one step. +A reader finds the old list or the new one, never part of one, and so does a rotation that overlaps another where the file system cannot lock. +The Zig build also syncs the new list before the rename and its directory after it, as it does for the files =failed= keeps, so the change outlasts a crash; the shell script syncs nothing. +Two rotations that do overlap can still leave a list that names a log by the name it had before the other compressed it, which the lock is there to prevent. + ** What was considered and turned down - /Sharing eb-pipeline's lock file./ @@ -102,6 +109,7 @@ The builds eb-pipeline starts are seen on the host eb-pipeline runs on even when - =rotate= sees the writers on its own host, and of those only the processes whose descriptors it may read in =/proc=: the user's own, except one running a set-user-ID program, and every process for root. - A process that has the tree at another path, as in a container that mounts it elsewhere, is not seen, nor one that opens a file in an old run after =rotate= has looked. - A build on another host is seen only through a process on this one that writes into its run, as eb-pipeline's output file is; a daily job on another host than eb-pipeline's takes eb-pipeline's lock around =rotate=, as above, to wait for its runs. +- A line added to a =.licensed-logs= while =rotate= rewrites that list goes with the old list; add lines between rotations. ** Further reading diff --git a/docs/orgmode/reference/cli.org b/docs/orgmode/reference/cli.org index 9184608..c94e774 100644 --- a/docs/orgmode/reference/cli.org +++ b/docs/orgmode/reference/cli.org @@ -135,7 +135,7 @@ What =failed= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe |-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | =ROOT/.eb-keep-logs-rotate.lock= | Created empty when missing, and left in place. =rotate= holds an exclusive flock(2) lock on it from before it lists =ROOT= until it exits, and opens it for reading when it may not write it. eb-pipeline's =ROOT/.eb-pipeline-rotate.lock= is another lock, which =rotate= does not take. | | =ROOT/RUN/.../FILE.log=, =FILE.txt= | In a run old enough, replaced by =FILE.gz= with the file's mode, as =gzip -9= does; a symbolic link is not followed, and a file with other hard links or an existing =FILE.gz= is left as it is, with a message on stderr. | -| =ROOT/RUN/.../.licensed-logs= | In a run in which files were compressed, a line that names one of them, by its absolute path with symbolic links resolved, is changed to name its =.gz= file. A list that changes keeps its owner when permitted, its mode and its access control list. | +| =ROOT/RUN/.../.licensed-logs= | In a run in which files were compressed, a line that names one of them, by its absolute path with symbolic links resolved, is changed to name its =.gz= file. A list that changes keeps its owner when permitted, its mode and its access control list. It is replaced whole: the new list is written to a file beside it, synced and renamed over it, and the directory is synced, so a reader finds the old list or the new one, never part of one. | | =ROOT/RUN= | Removed with everything in it, when old enough for =--delete-after=, without being compressed first. A symbolic link in it is removed, not followed. Removal stops at the first error. | | =/proc/PID/fd/N= | Read, for every process whose descriptors =rotate= may read, after it has the lock: a link with its owner's write bit names a file the process has open for writing, and the run that file is in is left alone. Without =/proc=, no run is left alone for it. | @@ -159,6 +159,7 @@ What =rotate= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe | =cannot resolve a run directory: ERROR= | The absolute path of a run, or of a directory in it, could not be found; nothing in it is compressed. | | =cannot read a .licensed-logs: ERROR= | A list could not be read, or is larger than 64 MiB; it is left as it is. | | =cannot update a .licensed-logs: ERROR= | A list could not be replaced; it is left as it is. | +| =cannot sync a .licensed-logs: ERROR= | A list was replaced, but its directory could not be synced; after a crash the old list may be back. | | =cannot delete ROOT/RUN: ERROR= | The run could not be removed completely; what was removed stays removed. | *** --version diff --git a/eb-keep-logs b/eb-keep-logs index bad1d39..b83312e 100755 --- a/eb-keep-logs +++ b/eb-keep-logs @@ -313,7 +313,7 @@ cmd_rotate() { lock_timeout=600 fi _lock_rotate "$(_joinable "${root}")/.eb-keep-logs-rotate.lock" "${lock_timeout}" - local run age file list path compressed=0 deleted=0 live + local run age file list tmp path compressed=0 deleted=0 live live="$(_own_runs "${root}")$(_written_runs "${root}")/" for run in "$(_joinable "${root}")"/*/; do run="${run%/}" @@ -334,9 +334,13 @@ cmd_rotate() { gzip -9 -- "${file}" || continue compressed=$((compressed + 1)) # A .licensed-logs in the run names the file by path; follow it. + # The new list is written beside the old one, under a name of its + # own, and renamed over it, so a reader sees one list or the other. while IFS= read -r -d '' list; do - awk -v p="${path}" '$0 == p { $0 = p ".gz" } { print }' "${list}" > "${list}.tmp" && - chmod --reference="${list}" "${list}.tmp" && mv -- "${list}.tmp" "${list}" + tmp=$(mktemp -- "${list%/*}/.licensed-logs.XXXXXXXX") || continue + { awk -v p="${path}" '$0 == p { $0 = p ".gz" } { print }' "${list}" > "${tmp}" && + chmod --reference="${list}" -- "${tmp}" && mv -f -- "${tmp}" "${list}"; } || + rm -f -- "${tmp}" done < <(find "${run}" -name .licensed-logs -type f -print0) done < <(find "${run}" -type f \( -name '*.log' -o -name '*.txt' \) -print0) done diff --git a/src/main.zig b/src/main.zig index c019fde..c12865b 100644 --- a/src/main.zig +++ b/src/main.zig @@ -546,7 +546,9 @@ fn compressTree(c: Ctx, arena: Allocator, dir: fsx.fd_t, renames: *Renames, coun } /// Rewrite every .licensed-logs under `dir` so it names the compressed -/// files. +/// files. Each new list is written beside the old one, synced, renamed over +/// it and its directory synced, as copyDurable does, so a reader or a crash +/// finds one list or the other. fn followLists(c: Ctx, dir: fsx.fd_t, renames: *const Renames) void { var it = fsx.DirIter.init(c.gpa, dir) catch return; defer it.deinit(c.gpa); @@ -566,8 +568,11 @@ fn followLists(c: Ctx, dir: fsx.fd_t, renames: *const Renames) void { defer c.gpa.free(text); const new = (logs.renameInList(c.gpa, text, renames) catch continue) orelse continue; defer c.gpa.free(new); - fsx.replaceContents(c.gpa, dir, e.name, new) catch |err| + fsx.replaceContents(c.gpa, dir, e.name, new) catch |err| { warn("cannot update a {s}: {t}", .{ logs.licensed_list, err }); + continue; + }; + sys.fsync(dir) catch |err| warn("cannot sync a {s}: {t}", .{ logs.licensed_list, err }); } } } diff --git a/tests/test_eb_keep_logs.sh b/tests/test_eb_keep_logs.sh index e948d53..4ac67c4 100755 --- a/tests/test_eb_keep_logs.sh +++ b/tests/test_eb_keep_logs.sh @@ -298,6 +298,29 @@ check "old run compressed" "$(ls "${root}/20261001_120000/site-zen4" | sort | tr check "compressed content intact" "$(zcat "${root}/20261001_120000/site-zen4/A.stdout.log.gz")" "old" check "closed log stays closed" "$(stat -c '%A' "${root}/20261001_120000/site-zen4/B.failed.ebout.log.gz" | cut -c8-10)" "---" check ".licensed-logs follows" "$(cat "${root}/20261001_120000/site-zen4/.licensed-logs")" "$(realpath "${root}/20261001_120000/site-zen4")/B.failed.ebout.log.gz" +check ".licensed-logs follows: no temporary file left" "$(find "${root}/20261001_120000" -name '.licensed-logs?*' | wc -l)" "0" +# The new list is written to a file of its own beside the old one, not +# through a symbolic link at the fixed name the shell script used to write to. +planted="${work}/planted/20261001_120000/site-zen4" +mkdir -p "${planted}" +echo log > "${planted}/P.stdout.log" +realpath "${planted}/P.stdout.log" > "${planted}/.licensed-logs" +chmod 640 "${planted}/.licensed-logs" +echo victim > "${work}/victim" +ln -s "${work}/victim" "${planted}/.licensed-logs.tmp" +"${tool}" rotate "${work}/planted" --compress-after 30 >/dev/null 2>&1 +check "a link beside a list: not written through" "$(cat "${work}/victim")" "victim" +check "a link beside a list: the list replaced" "$(stat -c '%F %a' "${planted}/.licensed-logs"): $(cat "${planted}/.licensed-logs")" \ + "regular file 640: $(realpath "${planted}")/P.stdout.log.gz" +# A list its owner may not write is still replaced, and keeps its mode. +ro="${work}/ro/20261001_120000" +mkdir -p "${ro}" +echo log > "${ro}/Q.stdout.log" +realpath "${ro}/Q.stdout.log" > "${ro}/.licensed-logs" +chmod 444 "${ro}/.licensed-logs" +"${tool}" rotate "${work}/ro" --compress-after 30 >/dev/null 2>&1 +check "a list its owner may not write: replaced, mode kept" \ + "$(stat -c '%a' "${ro}/.licensed-logs"): $(cat "${ro}/.licensed-logs")" "444: $(realpath "${ro}")/Q.stdout.log.gz" check "recent run untouched" "$(ls "${root}/20261110_120000")" "C.stdout.log" check "non-run directory untouched" "$(ls "${root}/notarun")" "E.log" check "no deletion without --delete-after" "$(ls -d "${root}/20260801_090000" 2>/dev/null | wc -l | tr -d ' ')" "1" @@ -548,6 +571,31 @@ done check "two rotations at once: every list whole" "${race_lists}" "" check "two rotations at once: one rotates, the other then finds nothing" "$(printf '%s' "${race_out}")" \ "$(for _ in 1 2 3; do echo "0 0 eb-keep-logs rotate: 0 files compressed, 0 runs deleted eb-keep-logs rotate: 12 files compressed, 0 runs deleted "; done)" +# Without the lock, as the shell script rotates where the file system cannot +# lock, two rotations at once can each list a file by the name it had when +# they read the list; but each list is one of the lists they wrote, whole: +# every line names one of the run's logs, each log once. +race_torn="" +for round in 1 2 3 4 5; do + rm -rf "${race}" + for t in a b; do + d="${race}/20261001_120000/${t}" + mkdir -p "${d}" + for i in 1 2 3 4 5 6; do seq 1 20000 > "${d}/n${i}.log"; done + for i in 1 2 3; do echo "$(realpath "${d}")/n${i}.log"; done > "${d}/.licensed-logs" + done + PATH="${work}/nolock-bin:${PATH}" "${tool}" rotate "${race}" --compress-after 30 > /dev/null 2>&1 & + r1=$! + PATH="${work}/nolock-bin:${PATH}" "${tool}" rotate "${race}" --compress-after 30 > /dev/null 2>&1 & + r2=$! + wait "${r1}" "${r2}" + for t in a b; do + d="$(realpath "${race}/20261001_120000/${t}")" + [ "$(sed 's/\.gz$//' "${d}/.licensed-logs")" = "$(printf '%s\n' "${d}/n1.log" "${d}/n2.log" "${d}/n3.log")" ] || + race_torn+=" round ${round} ${t}" + done +done +check "two rotations without the lock: every list whole" "${race_torn}" "" if [ "${fails}" -ne 0 ]; then echo "${fails} failed"; exit 1; fi echo "all passed"