Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion COMPAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ on the same invocations.
| | |
|---|---|
| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS]`, `-h`, `--help`, `--version` |
| usage | the same text on stdout for `-h`, `--help` and no arguments |
| exit codes | 0 on success and on every `failed` that got past its arguments; 2 for a refused invocation, after `eb-keep-logs: MESSAGE` on stderr; 2 with the usage text on stdout for no arguments; 75 when another `rotate` holds ROOT's lock past `--lock-timeout`, after `eb-keep-logs: another rotate holds the lock ROOT/.eb-keep-logs-rotate.lock; gave up after N s (--lock-timeout)` on stderr |
| one rotate at a time | `rotate` holds an exclusive flock(2) on `ROOT/.eb-keep-logs-rotate.lock`, created when missing, from before it reads ROOT until it exits; it waits up to `--lock-timeout` seconds for it, 600 without the option or with an empty value, 0 not at all, 999999999 at most; it opens the file for reading when it may not write it; a lock file that cannot be opened, or a symbolic link or non-regular file at its name, is refused, exit 2; on a file system that cannot lock, `rotate` says so on stderr and rotates without the lock |
| environment | `PATH` (finds `eb`), `EASYBUILD_TMPDIR`, `EB_KEEP_LOGS_NOW`, `EB_PIPELINE_LOG_DIR`, `TZ`, `TZDIR` |
Expand All @@ -24,7 +25,6 @@ on the same invocations.

| case | shell script | Zig build |
|---|---|---|
| no arguments, `--help` | prints a trailing half sentence from the script's header | stops after the `--version` line |
| `--version` | `eb-keep-logs 0.1.0` | `eb-keep-logs 0.2.1 (TARGET, MODE, zig VERSION, COMMIT)`; the first two words are unchanged |
| `--compress-after` or `--delete-after` with no value after it | never returns | refused, exit 2 |
| `EB_KEEP_LOGS_NOW` not empty and not an integer | evaluated as shell arithmetic | refused, exit 2 |
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,8 @@ eb-keep-logs rotate /srv/build-logs --compress-after 30 --delete-after 365
```

`rotate` looks only at run directories named `YYYYMMDD_HHMMSS` directly under the root, and dates each run by its name.
It gzips the `*.log` and `*.txt` files of runs older than `--compress-after` days, keeps each file's mode, and updates the run's `.licensed-logs`, writing each new list beside the old one and renaming it over it.
It removes whole runs older than `--delete-after` days only when that option is given.
It gzips the `*.log` and `*.txt` files of runs at least `--compress-after` days old, keeps each file's mode, and updates the run's `.licensed-logs`, writing each new list beside the old one and renaming it over it.
It removes whole runs at least `--delete-after` days old only when that option is given.
It leaves alone the run it writes into, which is still going: the run that holds its standard output or error, and the run that holds `EB_PIPELINE_LOG_DIR`, the log directory `eb-pipeline` exports, because srun passes the output on over a pipe.
It also leaves alone every run that a process on the same host has a file in open for writing, as a build that is still going has its log, whichever run started the rotation: it finds them in `/proc/PID/fd`, for the processes whose descriptors it may read.
A later `rotate` takes such a run.
Expand Down
1 change: 1 addition & 0 deletions changelog.d/+help-reference.doc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The command-line reference shows the usage that `-h`, `--help` and no arguments print, which the suite checks against both builds, and its synopsis and exit statuses name `-h`. README.md says that `rotate` compresses and removes the runs at least DAYS old, as the usage and the reference do.
1 change: 1 addition & 0 deletions changelog.d/+help-text.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`--help`, `-h` and no arguments print the whole usage, the same in both builds: the shell script no longer ends it with a blank line and half a sentence of its own header. It lists every option, `-h` and the `--OPTION=VALUE` forms included, with what each of `rotate`'s options defaults to (600 seconds for `--lock-timeout`), the environment variables the commands read and the exit statuses 0, 2 and 75, and it says that `rotate` takes the runs at least DAYS old, not the runs older than DAYS. COMPAT.md lists the usage as kept, no longer as a deliberate difference.
65 changes: 63 additions & 2 deletions docs/orgmode/reference/cli.org
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ This page describes the Zig build; the shell script =eb-keep-logs= in the reposi
#+begin_example
eb-keep-logs failed STATUS NAME LOGDIR [TMP_LOGDIR]
eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS]
eb-keep-logs --help | --version
eb-keep-logs -h | --help
eb-keep-logs --version
#+end_example

** Arguments
Expand Down Expand Up @@ -56,7 +57,7 @@ eb-keep-logs --help | --version

| Status | Meaning |
|--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0 | =--help= or =--version=, a =failed= that accepted its arguments, or a =rotate= that accepted its arguments and did not give up waiting for its lock, whatever it could keep, compress or remove: what it could not do is reported, and is no failure. |
| 0 | =-h=, =--help= or =--version=, a =failed= that accepted its arguments, or a =rotate= that accepted its arguments and did not give up waiting for its lock, whatever it could keep, compress or remove: what it could not do is reported, and is no failure. |
| 2 | The invocation was refused, with =eb-keep-logs: MESSAGE= on stderr; or no command was given, and the usage is on stdout. |
| 75 | Another =rotate= held =ROOT='s lock for longer than =--lock-timeout=, and this =rotate= changed nothing; try again later. |

Expand Down Expand Up @@ -162,6 +163,66 @@ What =rotate= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe
| =cannot sync a .licensed-logs: ERROR= | A list was replaced, but its directory could not be synced; after a crash the old list may be back. |
| =cannot delete ROOT/RUN: ERROR= | The run could not be removed completely; what was removed stays removed. |

*** --help

The usage =-h=, =--help= and no arguments print on stdout, the same from the shell script:

#+begin_example
eb-keep-logs: keep the logs of EasyBuild builds that a build pipeline would
otherwise lose, and keep the pipeline's log tree bounded.

eb-keep-logs failed STATUS NAME LOGDIR [TMP_LOGDIR]
After `eb` returned STATUS for easyconfig NAME: when STATUS is not 0,
copy the EasyBuild session log `eb --last-log` names, the failing
command's output and a meson testlog next to LOGDIR/NAME.stdout.log,
closed to others and listed in LOGDIR/.licensed-logs. TMP_LOGDIR is
the --tmp-logdir eb ran with; without it, the newest log in
EasyBuild's default place is used, unless it is another build's.
NAME.stdout.log and a TMP_LOGDIR below LOGDIR are closed to others too.
eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS]
[--lock-timeout SECONDS]
In the run directories directly under ROOT (named YYYYMMDD_HHMMSS),
gzip the *.log and *.txt files of old runs and remove older runs
whole, leaving alone a run that is still written into. A run's age is
in days from the local midnight of the date in its name. Permissions
and .licensed-logs follow. One rotate of ROOT runs at a time, holding
ROOT/.eb-keep-logs-rotate.lock. Options and ROOT come in any order.
--compress-after DAYS, --compress-after=DAYS
Gzip the files of runs at least DAYS old. Required; there is no
default.
--delete-after DAYS, --delete-after=DAYS
Remove whole runs at least DAYS old; DAYS must be larger than
that of --compress-after. Default: none; no run is removed. An
empty value is the default.
--lock-timeout SECONDS, --lock-timeout=SECONDS
While another rotate holds the lock, wait up to SECONDS for it,
then give up and exit 75. Default: 600; 0 does not wait. An empty
value is the default.
eb-keep-logs -h | --help
Print this help on stdout and exit 0.
eb-keep-logs --version
Print the version line on stdout and exit 0.

Environment:
PATH Where failed looks up eb.
EASYBUILD_TMPDIR failed copies a command's output only from below it;
unset or empty, it copies none.
EB_KEEP_LOGS_NOW The time rotate takes as now, in seconds since the
epoch; unset or empty, the current time.
EB_PIPELINE_LOG_DIR rotate leaves alone the run that holds this path.
TZ, TZDIR The time zone of the local midnights, and the
directory of zone files.

Exit status:
0 -h, --help or --version, or a failed or rotate that accepted its
arguments, whatever it could keep, compress or remove: what it could
not do is reported on stderr.
2 The invocation was refused, with eb-keep-logs: MESSAGE on stderr; or
no command was given, and this help is on stdout.
75 rotate waited --lock-timeout seconds for the lock another rotate
held, then gave up and changed nothing; run it again later.
#+end_example

*** --version

The version line follows the [[site:zig-common:reference/cli-conventions][command-line conventions]] of the stackhands Zig tools.
Expand Down
78 changes: 59 additions & 19 deletions eb-keep-logs
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,6 @@
# eb-keep-logs: keep the logs of EasyBuild builds that a build pipeline would
# otherwise lose, and keep the pipeline's log tree bounded.
#
# eb-keep-logs failed STATUS NAME LOGDIR [TMP_LOGDIR]
# After `eb` returned STATUS for easyconfig NAME: when STATUS is not 0,
# copy the EasyBuild session log `eb --last-log` names, the failing
# command's output and a meson testlog next to LOGDIR/NAME.stdout.log,
# closed to others and listed in LOGDIR/.licensed-logs. TMP_LOGDIR is
# the --tmp-logdir eb ran with; without it, the newest log in
# EasyBuild's default place is used, unless it is another build's.
# NAME.stdout.log and a TMP_LOGDIR below LOGDIR are closed to others too.
# eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS]
# [--lock-timeout SECONDS]
# In the run directories directly under ROOT (named YYYYMMDD_HHMMSS),
# gzip the logs of runs older than DAYS; with --delete-after, remove
# whole runs older than that. Permissions and .licensed-logs follow.
# EB_KEEP_LOGS_NOW, in seconds since the epoch, sets the current time.
# One rotate at a time holds ROOT/.eb-keep-logs-rotate.lock; another
# waits up to SECONDS for it (600; 0 does not wait), then exits 75.
# eb-keep-logs --help | --version
#
# A failed build's log is in EasyBuild's tmpdir on the build node, which goes
# with the job; and EasyBuild wraps the "Results of the build can be found in
# the log file(s)" line, so parsing stdout for the path finds nothing.
Expand All @@ -28,7 +10,65 @@ set -uo pipefail

VERSION=0.1.0

usage() { sed -n '2,23p' "$0" | sed 's/^# \{0,1\}//'; }
# src/main.zig's usage_text holds the same text: tests/differential.sh
# compares the two.
usage() {
cat <<'EOF'
eb-keep-logs: keep the logs of EasyBuild builds that a build pipeline would
otherwise lose, and keep the pipeline's log tree bounded.

eb-keep-logs failed STATUS NAME LOGDIR [TMP_LOGDIR]
After `eb` returned STATUS for easyconfig NAME: when STATUS is not 0,
copy the EasyBuild session log `eb --last-log` names, the failing
command's output and a meson testlog next to LOGDIR/NAME.stdout.log,
closed to others and listed in LOGDIR/.licensed-logs. TMP_LOGDIR is
the --tmp-logdir eb ran with; without it, the newest log in
EasyBuild's default place is used, unless it is another build's.
NAME.stdout.log and a TMP_LOGDIR below LOGDIR are closed to others too.
eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS]
[--lock-timeout SECONDS]
In the run directories directly under ROOT (named YYYYMMDD_HHMMSS),
gzip the *.log and *.txt files of old runs and remove older runs
whole, leaving alone a run that is still written into. A run's age is
in days from the local midnight of the date in its name. Permissions
and .licensed-logs follow. One rotate of ROOT runs at a time, holding
ROOT/.eb-keep-logs-rotate.lock. Options and ROOT come in any order.
--compress-after DAYS, --compress-after=DAYS
Gzip the files of runs at least DAYS old. Required; there is no
default.
--delete-after DAYS, --delete-after=DAYS
Remove whole runs at least DAYS old; DAYS must be larger than
that of --compress-after. Default: none; no run is removed. An
empty value is the default.
--lock-timeout SECONDS, --lock-timeout=SECONDS
While another rotate holds the lock, wait up to SECONDS for it,
then give up and exit 75. Default: 600; 0 does not wait. An empty
value is the default.
eb-keep-logs -h | --help
Print this help on stdout and exit 0.
eb-keep-logs --version
Print the version line on stdout and exit 0.

Environment:
PATH Where failed looks up eb.
EASYBUILD_TMPDIR failed copies a command's output only from below it;
unset or empty, it copies none.
EB_KEEP_LOGS_NOW The time rotate takes as now, in seconds since the
epoch; unset or empty, the current time.
EB_PIPELINE_LOG_DIR rotate leaves alone the run that holds this path.
TZ, TZDIR The time zone of the local midnights, and the
directory of zone files.

Exit status:
0 -h, --help or --version, or a failed or rotate that accepted its
arguments, whatever it could keep, compress or remove: what it could
not do is reported on stderr.
2 The invocation was refused, with eb-keep-logs: MESSAGE on stderr; or
no command was given, and this help is on stdout.
75 rotate waited --lock-timeout seconds for the lock another rotate
held, then gave up and changed nothing; run it again later.
EOF
}

die() { echo "eb-keep-logs: $*" >&2; exit 2; }

Expand Down
55 changes: 49 additions & 6 deletions src/main.zig
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ const tool = "eb-keep-logs";
/// build.zig.zon names the same one, and a release changes both.
pub const version = "0.2.1";

/// What `-h`, `--help` and no arguments print. The shell script's `usage`
/// prints the same text: tests/differential.sh compares the two.
const usage_text =
\\eb-keep-logs: keep the logs of EasyBuild builds that a build pipeline would
\\otherwise lose, and keep the pipeline's log tree bounded.
Expand All @@ -33,12 +35,45 @@ const usage_text =
\\ eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS]
\\ [--lock-timeout SECONDS]
\\ In the run directories directly under ROOT (named YYYYMMDD_HHMMSS),
\\ gzip the logs of runs older than DAYS; with --delete-after, remove
\\ whole runs older than that. Permissions and .licensed-logs follow.
\\ EB_KEEP_LOGS_NOW, in seconds since the epoch, sets the current time.
\\ One rotate at a time holds ROOT/.eb-keep-logs-rotate.lock; another
\\ waits up to SECONDS for it (600; 0 does not wait), then exits 75.
\\ eb-keep-logs --help | --version
\\ gzip the *.log and *.txt files of old runs and remove older runs
\\ whole, leaving alone a run that is still written into. A run's age is
\\ in days from the local midnight of the date in its name. Permissions
\\ and .licensed-logs follow. One rotate of ROOT runs at a time, holding
\\ ROOT/.eb-keep-logs-rotate.lock. Options and ROOT come in any order.
\\ --compress-after DAYS, --compress-after=DAYS
\\ Gzip the files of runs at least DAYS old. Required; there is no
\\ default.
\\ --delete-after DAYS, --delete-after=DAYS
\\ Remove whole runs at least DAYS old; DAYS must be larger than
\\ that of --compress-after. Default: none; no run is removed. An
\\ empty value is the default.
\\ --lock-timeout SECONDS, --lock-timeout=SECONDS
\\ While another rotate holds the lock, wait up to SECONDS for it,
\\ then give up and exit 75. Default: 600; 0 does not wait. An empty
\\ value is the default.
\\ eb-keep-logs -h | --help
\\ Print this help on stdout and exit 0.
\\ eb-keep-logs --version
\\ Print the version line on stdout and exit 0.
\\
\\Environment:
\\ PATH Where failed looks up eb.
\\ EASYBUILD_TMPDIR failed copies a command's output only from below it;
\\ unset or empty, it copies none.
\\ EB_KEEP_LOGS_NOW The time rotate takes as now, in seconds since the
\\ epoch; unset or empty, the current time.
\\ EB_PIPELINE_LOG_DIR rotate leaves alone the run that holds this path.
\\ TZ, TZDIR The time zone of the local midnights, and the
\\ directory of zone files.
\\
\\Exit status:
\\ 0 -h, --help or --version, or a failed or rotate that accepted its
\\ arguments, whatever it could keep, compress or remove: what it could
\\ not do is reported on stderr.
\\ 2 The invocation was refused, with eb-keep-logs: MESSAGE on stderr; or
\\ no command was given, and this help is on stdout.
\\ 75 rotate waited --lock-timeout seconds for the lock another rotate
\\ held, then gave up and changed nothing; run it again later.
\\
;

Expand Down Expand Up @@ -924,6 +959,14 @@ test "parseSeconds" {
try std.testing.expectEqual(max_lock_timeout, parseSeconds("123456789012345678901234567890"));
}

test "usage_text gives the lock timeout's default and the exit statuses" {
const fmt = std.fmt.comptimePrint;
try std.testing.expect(std.mem.indexOf(u8, usage_text, fmt("Default: {d};", .{default_lock_timeout})) != null);
try std.testing.expect(std.mem.indexOf(u8, usage_text, fmt("give up and exit {d}.", .{lock_busy_exit})) != null);
try std.testing.expect(std.mem.indexOf(u8, usage_text, fmt("\n {d} The invocation was refused", .{cli.usage_exit})) != null);
try std.testing.expect(std.mem.indexOf(u8, usage_text, fmt("\n {d} rotate waited --lock-timeout", .{lock_busy_exit})) != null);
}

test "listed" {
try std.testing.expect(listed("/l/a.log\n/l/b.log\n", "/l/b.log"));
try std.testing.expect(listed("/l/a.log", "/l/a.log"));
Expand Down
Loading
Loading