Skip to content

fix(rotate): hold a lock on the tree for the whole - #19

Merged
HaoZeke merged 3 commits into
mainfrom
fix/rotate-lock
Nov 17, 2026
Merged

HaoZeke merged 3 commits into
mainfrom
fix/rotate-lock

Conversation

@HaoZeke

@HaoZeke HaoZeke commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

rotate holds an exclusive flock on ROOT/.eb-keep-logs-rotate.lock for the whole rotation, with --lock-timeout (600 seconds by default; 0 fails at once). When the wait runs out it changes nothing and exits 75. The lock is separate from eb-pipeline's ROOT/.eb-pipeline-rotate.lock. rotate never takes that one, and eb-pipeline takes its own before it runs rotate, so the two are always taken in one order and cannot wait on each other. A cron job that should wait for eb-pipeline's runs takes eb-pipeline's lock around rotate with flock(1), in the same order.

After it takes the lock, rotate leaves alone every run that a process on the same host has a file in open for writing, by reading the links in /proc/PID/fd. That covers a build of another target still writing its log. A process that only reads a run leaves that run free to rotate.

The shell script writes each new .licensed-logs with mktemp beside the list and renames it over the list. The Zig build, which already wrote a new list with O_EXCL and renamed it, now also syncs the list's directory.

On 0.2.1, two rotations started together damaged the lists in 191/200 rounds with the shell script and in 148/200 with the Zig build, and both exited 0. At this tip, 0/200. The cross-target live run went from 50/50 broken to 0/50.

The live-run scan is of the same host only (/proc). On a file system that cannot lock, rotate warns and rotates without the lock.

Two rotations of one tree that ran at the same time each compressed
part of a run and wrote its .licensed-logs back with only their own
changes, over the other's, so the list named files that were gone, and
both exited 0. Started together on a tree of three runs, they left such
a list in 189 of 200 rounds with the shell script and in 156 with the
Zig build. eb-pipeline keeps its own runs apart with its
.eb-pipeline-rotate.lock, but a rotation from cron or by hand still
raced with them.

rotate now holds an exclusive flock(2) on ROOT/.eb-keep-logs-rotate.lock,
created when missing, from before it lists ROOT until it exits. A
second rotate waits for it, up to --lock-timeout SECONDS, 600 by
default and 0 not at all, and then finds the work done; when the wait
runs out it changes nothing and exits 75, EX_TEMPFAIL, naming the lock.
The file is opened for reading when it cannot be opened for writing. A
lock file that cannot be opened, or a symbolic link or a file that is
not a regular one at its name, is refused with exit 2. Where the file
system cannot lock, rotate says so and rotates without the lock, as
eb-pipeline goes on without its own: a refusal would fail the logs
phase of every eb-pipeline run there. The shell script takes the lock
with flock(1), the Zig build with flock(2), tried every 20 ms.

rotate never takes eb-pipeline's lock, and eb-pipeline takes its own
before it runs rotate, so the two are always taken in one order and
neither tool can wait for the other for good. A daily job that should
wait for eb-pipeline's runs takes that lock around rotate with
flock(1), in the same order.

The command-line suite holds the lock and checks that rotate refuses
at once and after --lock-timeout, takes the lock once it is let go,
takes a lock it may only read, and refuses a symbolic link at its name;
for the shell script, a flock(1) that fails as on a file system that
cannot lock leaves rotate rotating without the lock. It then starts two
rotations of one tree together in three rounds: every list stays whole,
and one rotation compresses all the logs while the other finds none.
The lock checks and both race checks fail on the previous commit. Unit
tests cover the lock and the --lock-timeout values; the differential
run compares the refusals, an empty value and the exit 75 message.

COMPAT.md, the cli reference, the rotate how-to and a new explanation,
Rotating a tree in use, describe the lock, its wait, exit 75 and how it
sits beside eb-pipeline's lock.
rotate passed over only the run it wrote into itself. A run's age
counts from the local midnight of the date in its name, so a build of
another run that started before midnight UTC was a day old just after
it: with --compress-after 1, a rotation from a run started after
midnight compressed the EasyBuild log that build still wrote into, and
the build exited 1 when it copied the log into the installation. Past
--delete-after, such a run was removed while its build ran.

After it takes its lock, and before it compresses or deletes anything,
rotate now reads the links /proc/PID/fd/N of every process whose
descriptors it may read. A link with its owner's write bit names a
file open for writing, and the run that file lies in by its path, also
once the file is removed, is left whole to a later rotate, as the run
rotate writes into is. A file that is only read does not hold a run
back. The builds eb-pipeline starts are seen on its own host even when
srun runs them elsewhere, as eb-pipeline hands each build's
NAME.stdout.log to the command it starts. The shell script finds the
links with find -perm -u=w, the Zig build with statx and readlinkat.

The command-line suite reproduces the case: a run started at 23:50 UTC
whose session log another process holds open for writing, rotated at
00:10 UTC with --compress-after 1 from a run started after midnight.
Its files stay as they are while an older run is compressed, and the
rotate after the writer ends compresses them. A run that a process
only reads is compressed, and a run still written into is not deleted
past --delete-after. The checks of the live run and of --delete-after
fail on the previous commit, with the shell script and with the Zig
build. The suite's listings sort in the C locale, so names that differ
in case come out in one order. A unit test covers a file open for
writing, one only read and one removed while open; the differential
run compares a run past --delete-after that is still written into.

COMPAT.md, the cli reference, both how-tos and the explanation of a
tree in use describe the runs left alone and the limits of the scan:
the same host, the processes rotate may look into, and the path the
tree has there.
… own

The shell script wrote every new .licensed-logs to the fixed name
.licensed-logs.tmp beside it and renamed that over the list. Two
rotations at once wrote into that one file together, and the one that
renamed it first could put a torn list in place: with a flock(1) that
fails as on a file system that cannot lock, two shell rotations at once
left 73 of 120 lists with a line that names none of the run's logs, or
a log named twice or not at all. A symbolic link at the fixed name was
written through, and then renamed over the list in its place.

The shell script now makes the new list with mktemp in the list's
directory, keeps the list's mode on it and renames it over the list
with mv -f; a failed rewrite removes it. In the same 120 lists, none
was torn; 49 named a log by the name it had before the other rotation
compressed it, which the lock prevents. The Zig build already wrote
the new list beside the old one with O_EXCL, synced it and renamed it
over the list; it now also syncs the list's directory, as copyDurable
does for the files failed keeps, and says "cannot sync a
.licensed-logs: ERROR" on stderr when that fails. The shell script,
which syncs nothing anywhere, still syncs nothing.

The command-line suite checks that no temporary file is left beside a
list, that a symbolic link at the old fixed name is not written
through and the list stays a regular file with its mode, that a list
its owner may not write is still replaced with its mode, and, for the
shell script with flock(1) failing, that two rotations at once in five
rounds leave every list whole. The link checks and the check without
the lock fail with the shell script of the previous commit.

COMPAT.md, the cli reference, the explanation of a tree in use and the
README say how a list is replaced, and that a line added to a list
while rotate rewrites it goes with the old list.
@HaoZeke
HaoZeke merged commit c31f6ee into main Nov 17, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant