Repository navigation
fix(rotate): hold a lock on the tree for the whole - #19
Merged
Merged
Conversation
Two rotations of one tree that ran at the same time each compressed part of a run and wrote its .licensed-logs back with only their own changes, over the other's, so the list named files that were gone, and both exited 0. Started together on a tree of three runs, they left such a list in 189 of 200 rounds with the shell script and in 156 with the Zig build. eb-pipeline keeps its own runs apart with its .eb-pipeline-rotate.lock, but a rotation from cron or by hand still raced with them. rotate now holds an exclusive flock(2) on ROOT/.eb-keep-logs-rotate.lock, created when missing, from before it lists ROOT until it exits. A second rotate waits for it, up to --lock-timeout SECONDS, 600 by default and 0 not at all, and then finds the work done; when the wait runs out it changes nothing and exits 75, EX_TEMPFAIL, naming the lock. The file is opened for reading when it cannot be opened for writing. A lock file that cannot be opened, or a symbolic link or a file that is not a regular one at its name, is refused with exit 2. Where the file system cannot lock, rotate says so and rotates without the lock, as eb-pipeline goes on without its own: a refusal would fail the logs phase of every eb-pipeline run there. The shell script takes the lock with flock(1), the Zig build with flock(2), tried every 20 ms. rotate never takes eb-pipeline's lock, and eb-pipeline takes its own before it runs rotate, so the two are always taken in one order and neither tool can wait for the other for good. A daily job that should wait for eb-pipeline's runs takes that lock around rotate with flock(1), in the same order. The command-line suite holds the lock and checks that rotate refuses at once and after --lock-timeout, takes the lock once it is let go, takes a lock it may only read, and refuses a symbolic link at its name; for the shell script, a flock(1) that fails as on a file system that cannot lock leaves rotate rotating without the lock. It then starts two rotations of one tree together in three rounds: every list stays whole, and one rotation compresses all the logs while the other finds none. The lock checks and both race checks fail on the previous commit. Unit tests cover the lock and the --lock-timeout values; the differential run compares the refusals, an empty value and the exit 75 message. COMPAT.md, the cli reference, the rotate how-to and a new explanation, Rotating a tree in use, describe the lock, its wait, exit 75 and how it sits beside eb-pipeline's lock.
rotate passed over only the run it wrote into itself. A run's age counts from the local midnight of the date in its name, so a build of another run that started before midnight UTC was a day old just after it: with --compress-after 1, a rotation from a run started after midnight compressed the EasyBuild log that build still wrote into, and the build exited 1 when it copied the log into the installation. Past --delete-after, such a run was removed while its build ran. After it takes its lock, and before it compresses or deletes anything, rotate now reads the links /proc/PID/fd/N of every process whose descriptors it may read. A link with its owner's write bit names a file open for writing, and the run that file lies in by its path, also once the file is removed, is left whole to a later rotate, as the run rotate writes into is. A file that is only read does not hold a run back. The builds eb-pipeline starts are seen on its own host even when srun runs them elsewhere, as eb-pipeline hands each build's NAME.stdout.log to the command it starts. The shell script finds the links with find -perm -u=w, the Zig build with statx and readlinkat. The command-line suite reproduces the case: a run started at 23:50 UTC whose session log another process holds open for writing, rotated at 00:10 UTC with --compress-after 1 from a run started after midnight. Its files stay as they are while an older run is compressed, and the rotate after the writer ends compresses them. A run that a process only reads is compressed, and a run still written into is not deleted past --delete-after. The checks of the live run and of --delete-after fail on the previous commit, with the shell script and with the Zig build. The suite's listings sort in the C locale, so names that differ in case come out in one order. A unit test covers a file open for writing, one only read and one removed while open; the differential run compares a run past --delete-after that is still written into. COMPAT.md, the cli reference, both how-tos and the explanation of a tree in use describe the runs left alone and the limits of the scan: the same host, the processes rotate may look into, and the path the tree has there.
… own The shell script wrote every new .licensed-logs to the fixed name .licensed-logs.tmp beside it and renamed that over the list. Two rotations at once wrote into that one file together, and the one that renamed it first could put a torn list in place: with a flock(1) that fails as on a file system that cannot lock, two shell rotations at once left 73 of 120 lists with a line that names none of the run's logs, or a log named twice or not at all. A symbolic link at the fixed name was written through, and then renamed over the list in its place. The shell script now makes the new list with mktemp in the list's directory, keeps the list's mode on it and renames it over the list with mv -f; a failed rewrite removes it. In the same 120 lists, none was torn; 49 named a log by the name it had before the other rotation compressed it, which the lock prevents. The Zig build already wrote the new list beside the old one with O_EXCL, synced it and renamed it over the list; it now also syncs the list's directory, as copyDurable does for the files failed keeps, and says "cannot sync a .licensed-logs: ERROR" on stderr when that fails. The shell script, which syncs nothing anywhere, still syncs nothing. The command-line suite checks that no temporary file is left beside a list, that a symbolic link at the old fixed name is not written through and the list stays a regular file with its mode, that a list its owner may not write is still replaced with its mode, and, for the shell script with flock(1) failing, that two rotations at once in five rounds leave every list whole. The link checks and the check without the lock fail with the shell script of the previous commit. COMPAT.md, the cli reference, the explanation of a tree in use and the README say how a list is replaced, and that a line added to a list while rotate rewrites it goes with the old list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rotateholds an exclusive flock onROOT/.eb-keep-logs-rotate.lockfor the whole rotation, with--lock-timeout(600 seconds by default; 0 fails at once). When the wait runs out it changes nothing and exits 75. The lock is separate from eb-pipeline'sROOT/.eb-pipeline-rotate.lock.rotatenever takes that one, and eb-pipeline takes its own before it runsrotate, so the two are always taken in one order and cannot wait on each other. A cron job that should wait for eb-pipeline's runs takes eb-pipeline's lock aroundrotatewith flock(1), in the same order.After it takes the lock,
rotateleaves alone every run that a process on the same host has a file in open for writing, by reading the links in/proc/PID/fd. That covers a build of another target still writing its log. A process that only reads a run leaves that run free to rotate.The shell script writes each new
.licensed-logswithmktempbeside the list and renames it over the list. The Zig build, which already wrote a new list withO_EXCLand renamed it, now also syncs the list's directory.On 0.2.1, two rotations started together damaged the lists in 191/200 rounds with the shell script and in 148/200 with the Zig build, and both exited 0. At this tip, 0/200. The cross-target live run went from 50/50 broken to 0/50.
The live-run scan is of the same host only (
/proc). On a file system that cannot lock,rotatewarns and rotates without the lock.