diff --git a/COMPAT.md b/COMPAT.md index 7857039..337e7ec 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -9,7 +9,7 @@ on the same invocations. | | | |---|---| -| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS]`, `-h`, `--help`, `--version` | +| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS] [--dry-run] [--lock-timeout SECONDS]`, `-h`, `--help`, `--version` | | usage | the same text on stdout for `-h`, `--help` and no arguments | | exit codes | 0 on success and on every `failed` that got past its arguments; 2 for a refused invocation, after `eb-keep-logs: MESSAGE` on stderr; 2 with the usage text on stdout for no arguments; 75 when another `rotate` holds ROOT's lock past `--lock-timeout`, after `eb-keep-logs: another rotate holds the lock ROOT/.eb-keep-logs-rotate.lock; gave up after N s (--lock-timeout)` on stderr | | one rotate at a time | `rotate` holds an exclusive flock(2) on `ROOT/.eb-keep-logs-rotate.lock`, created when missing, from before it reads ROOT until it exits; it waits up to `--lock-timeout` seconds for it, 600 without the option or with an empty value, 0 not at all, 999999999 at most; it opens the file for reading when it may not write it; a lock file that cannot be opened, or a symbolic link or non-regular file at its name, is refused, exit 2; on a file system that cannot lock, `rotate` says so on stderr and rotates without the lock | @@ -20,6 +20,7 @@ on the same invocations. | run ages | days from the local midnight of the date in the run's name, as `date -d YYYYMMDD` computes it, including TZ rules, except for the zones under unintended differences; a run whose local midnight does not exist is skipped | | the run in progress | `rotate` neither compresses nor deletes the run that holds its stdout or stderr, when either is a regular file, or `EB_PIPELINE_LOG_DIR` | | runs still written into | after taking its lock, `rotate` reads the links `/proc/PID/fd/N` of every process whose descriptors it may read; a run that holds the path a link with its owner's write bit names, also when the path ends in ` (deleted)`, is neither compressed nor deleted; without `/proc`, no run is left alone for this | +| rotation preview | `rotate --dry-run` prints each `compress PATH` and `delete PATH` action followed by the counts, changes nothing, creates no lock file, emits no syslog or journal event and writes no metrics; because it takes no lock, the output is a snapshot | ## Deliberate differences diff --git a/README.md b/README.md index f7f9cb1..225c4c7 100644 --- a/README.md +++ b/README.md @@ -91,12 +91,12 @@ Rotations of one tree take turns: each holds an exclusive flock on `ROOT/.eb-kee One that starts while another runs waits for it, up to `--lock-timeout` seconds (600; `0` does not wait), and when the wait runs out it changes nothing and exits 75. `eb-pipeline`'s `ROOT/.eb-pipeline-rotate.lock` is a separate lock, which `rotate` never takes; a daily job that should wait for eb-pipeline's runs takes that lock around `rotate` with `flock`. -`EB_KEEP_LOGS_NOW`, in seconds since the epoch, sets the time `rotate` dates runs against. -On a copy of the tree, it previews what a later rotation compresses and removes: +`--dry-run` prints each file that would be compressed and run that would be deleted, but changes nothing, creates no lock and emits no syslog or journal event. +`EB_KEEP_LOGS_NOW`, in seconds since the epoch, sets the time `rotate` dates runs against, so the two together preview a later rotation: ```bash -cp -a /srv/build-logs /tmp/logs-preview -EB_KEEP_LOGS_NOW=$(date -d 2027-09-20 +%s) eb-keep-logs rotate /tmp/logs-preview --compress-after 30 --delete-after 365 +EB_KEEP_LOGS_NOW=$(date -d 2027-09-20 +%s) \ + eb-keep-logs rotate /srv/build-logs --compress-after 30 --delete-after 365 --dry-run ``` ## Find why a build failed diff --git a/changelog.d/+rotate-dry-run.feature.md b/changelog.d/+rotate-dry-run.feature.md new file mode 100644 index 0000000..41ede21 --- /dev/null +++ b/changelog.d/+rotate-dry-run.feature.md @@ -0,0 +1 @@ +`eb-keep-logs rotate --dry-run` lists the log files it would compress and run directories it would delete without changing the tree, creating its lock file, emitting an event, or writing metrics. diff --git a/docs/orgmode/howto/rotate-a-log-tree.org b/docs/orgmode/howto/rotate-a-log-tree.org index ae38045..e3da9de 100644 --- a/docs/orgmode/howto/rotate-a-log-tree.org +++ b/docs/orgmode/howto/rotate-a-log-tree.org @@ -18,20 +18,23 @@ Use this to keep a directory of build runs from growing without end: compress th A run's age is the number of whole days from the local midnight of the date in its name. =--compress-after DAYS= compresses the =*.log= and =*.txt= files of runs at least =DAYS= old, at any depth. =--delete-after DAYS= removes runs at least =DAYS= old, and must be larger; without it, no run is removed. -2. Try the ages on a copy of the tree, at a later date. - =EB_KEEP_LOGS_NOW= sets the time =rotate= takes as now, in seconds since the epoch: +2. Preview the rotation. + =--dry-run= lists every file it would compress and run it would delete, but changes nothing, creates no lock file, emits no syslog or journal event, and writes no metrics. + =EB_KEEP_LOGS_NOW= sets the time =rotate= takes as now, in seconds since the epoch, so this also previews the policy at a later date: #+begin_src bash - cp -a ~/build-logs /tmp/build-logs-preview EB_KEEP_LOGS_NOW=$(date -d 2027-01-01 +%s) \ - eb-keep-logs rotate /tmp/build-logs-preview --compress-after 30 --delete-after 365 + eb-keep-logs rotate ~/build-logs --compress-after 30 --delete-after 365 --dry-run #+end_src #+begin_example - eb-keep-logs rotate: 5 files compressed, 0 runs deleted under /tmp/build-logs-preview + compress /home/alice/build-logs/20261116_175300/Example-1.0.failed.ebout.log + compress /home/alice/build-logs/20261116_175300/Example-1.0.stdout.log + eb-keep-logs rotate dry-run: 2 files would be compressed, 0 runs would be deleted under /home/alice/build-logs #+end_example On 2027-01-01 the run is 46 days old: at least 30, less than 365. + The preview is a snapshot: it deliberately takes no rotation lock, so preview again when the tree may have changed before the real rotation. 3. Rotate the tree: #+begin_src bash @@ -66,7 +69,7 @@ The summary line counts the files =rotate= compressed and the runs it removed. In a compressed run, each log is a =.gz= file with the log's mode: #+begin_src bash -ls -l /tmp/build-logs-preview/*/ +ls -l ~/build-logs/*/ #+end_src #+begin_example @@ -83,7 +86,6 @@ drwxrwxr-x 3 alice alice 4096 Nov 16 17:53 easybuild-logs |--------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------| | =eb-keep-logs: --compress-after needs a number of days= | The option is missing, or its value is not a whole number, like =30d=. | Give a number of days: =--compress-after 30=. | | =eb-keep-logs: --delete-after must be later than --compress-after= | =--delete-after= is not larger than =--compress-after=. | Raise =--delete-after=, or leave it out to remove nothing. | -| =eb-keep-logs: unknown option --dry-run= | =rotate= has no dry run. | Try the ages on a copy, as in step 2. | | =eb-keep-logs: ROOT PATH is not a directory= | The path is wrong, or names a run, not the tree. | Give the directory that holds the runs. | | =eb-keep-logs: cannot compress PATH: TooManyLinks= | The log has other hard links. The shell script prints =gzip='s own message. | Nothing: like =gzip=, =rotate= leaves it as it is. | | =eb-keep-logs: cannot compress PATH: Exists= | =PATH.gz= exists already. The shell script prints =gzip='s own message. | Remove or rename one of the two, and rotate again. | diff --git a/docs/orgmode/reference/cli.org b/docs/orgmode/reference/cli.org index 2f70359..69dd84c 100644 --- a/docs/orgmode/reference/cli.org +++ b/docs/orgmode/reference/cli.org @@ -11,7 +11,7 @@ This page describes the Zig build; the shell script =eb-keep-logs= in the reposi #+begin_example eb-keep-logs failed [--syslog TARGET] [--metrics-file PATH] STATUS NAME LOGDIR [TMP_LOGDIR] -eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS] [--syslog TARGET] [--metrics-file PATH] +eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--dry-run] [--lock-timeout SECONDS] [--syslog TARGET] [--metrics-file PATH] eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json] eb-keep-logs history PACKAGE [--target TARGET] [--root ROOT] [--json] eb-keep-logs loads --module NAME [--json] FILE @@ -43,9 +43,10 @@ eb-keep-logs --version |--------------------------------------------------+----------+----------------------------------------------------------------------------------------------------------------------------------------------| | =--compress-after DAYS=, =--compress-after=DAYS= | =rotate= | Required. Compress the =*.log= and =*.txt= files, at any depth, of runs at least =DAYS= old. =DAYS= is decimal digits. | | =--delete-after DAYS=, =--delete-after=DAYS= | =rotate= | Remove whole runs at least =DAYS= old. =DAYS= must be larger than =--compress-after=. Without it, or with an empty value, no run is removed. | +| =--dry-run= | =rotate= | Print =compress PATH= for each log file that would be compressed and =delete PATH= for each run that would be removed, followed by a summary; change nothing, create no lock file, emit no syslog or journal event, and write no metrics. | | =--lock-timeout SECONDS=, =--lock-timeout=SECONDS= | =rotate= | How long to wait for the lock another =rotate= of =ROOT= holds, in seconds, decimal digits; =0= does not wait. Without it, or with an empty value, 600. A value larger than 999999999, which is over 31 years, waits 999999999 seconds. | | =--syslog TARGET=, =--syslog=TARGET= | =failed=, =rotate= | Emit an RFC 5424 or native =systemd-journald= event to =TARGET= (=journald=, =syslog=, =unix:///path=, =journal+unix:///path=, =udp://host:port=, or =tcp://host:port=); overrides =EB_KEEP_LOGS_SYSLOG=. An empty value disables emission. | -| =--metrics-file PATH=, =--metrics-file=PATH= | =failed=, =rotate= | Atomically update the Prometheus text exposition at =PATH=. Without it, no metrics file is read or written. | +| =--metrics-file PATH=, =--metrics-file=PATH= | =failed=, =rotate= | Atomically update the Prometheus text exposition at =PATH=, except on =rotate --dry-run=, which writes no metrics. Without it, no metrics file is read or written. | | =--latest= | =why= | Read the newest run under =--root=, by its name; with =--target=, the newest that has that target. The default without =RUN=. | | =--target TARGET=, =--target=TARGET= | =why=, =history= | Read only the builds in the run's directory =TARGET=; for =history=, in each run's, a run without one passed over. Without it, those in the run's own directory and then in each directory in it, by name. | | =--root ROOT=, =--root=ROOT= | =why=, =history= | The directory that holds the runs, directories named =YYYYMMDD_HHMMSS=, not symbolic links. Without it, the current directory. | @@ -74,6 +75,7 @@ eb-keep-logs --version | =--version= | | Print the version line on stdout and exit 0. | - A run's age is the number of whole days from the local midnight of the date in its name; [[file:../explanation/local-time.org][Run ages and local time]] gives the details. +- =--dry-run= reads the tree and the processes writing into it without taking the rotation lock, so its output is a snapshot, not a reservation. It also skips syslog emission and =--metrics-file= updates. - =rotate='s options and =ROOT= come in any order; the value of an option is the next argument, whatever it is. An option given twice takes its last value. - =why='s options and =RUN=, and =history='s and =PACKAGE=, come in any order, and an option given twice takes its last value; the value of an option is the next argument, whatever it is, and an empty one is refused. @@ -97,7 +99,7 @@ eb-keep-logs --version | Status | Meaning | |--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0 | =-h=, =--help= or =--version=, a =failed=, =why= or =history= that accepted its arguments, or a =rotate= that accepted its arguments and did not give up waiting for its lock, whatever it could keep, compress, remove or read: what it could not do is reported, and is no failure. A =why= exits 0 whether or not the run's builds failed, and a =history= whether or not it found a build of the package. | +| 0 | =-h=, =--help= or =--version=, a =failed=, =why= or =history= that accepted its arguments, or a =rotate= that accepted its arguments and did not give up waiting for its lock, whatever it could keep, compress, remove, preview or read: what it could not do is reported, and is no failure. A =why= exits 0 whether or not the run's builds failed, and a =history= whether or not it found a build of the package. | | 2 | The invocation was refused, with =eb-keep-logs: MESSAGE= on stderr; or no command was given, and the usage is on stdout. | | 75 | Another =rotate= held =ROOT='s lock for longer than =--lock-timeout=, and this =rotate= changed nothing; try again later. | @@ -191,6 +193,8 @@ What =failed= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe | =ROOT/RUN= | Removed with everything in it, when old enough for =--delete-after=, without being compressed first. A symbolic link in it is removed, not followed. Removal stops at the first error. | | =/proc/PID/fd/N= | Read, for every process whose descriptors =rotate= may read, after it has the lock: a link with its owner's write bit names a file the process has open for writing, and the run that file is in is left alone. Without =/proc=, no run is left alone for it. | +With =--dry-run= these paths are only read; =ROOT/.eb-keep-logs-rotate.lock= is neither opened nor created, and =--metrics-file= neither reads nor writes any metrics or lock file. + A run that holds the file =rotate='s standard output or standard error goes to, or =EB_PIPELINE_LOG_DIR=, or a file another process has open for writing, as a build that is still going has its log, is left alone, neither compressed nor deleted; a later =rotate= takes it. The path is what counts: the path the link in =/proc= names, or for a file removed since, that path with =" (deleted)"= after it, has to start with =ROOT='s, with symbolic links resolved. On stdout =rotate= prints one line, with =ROOT= as given: @@ -217,6 +221,7 @@ What =rotate= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe *** Metrics With =--metrics-file PATH=, =failed= and =rotate= update one Prometheus text exposition file. +When =--dry-run= is given to =rotate=, it writes no metrics and leaves any existing metrics file and its sidecar lock untouched. The file is written to a temporary file in =PATH='s directory, synced, and renamed over =PATH= while a sidecar lock is held. An existing valid file supplies the counters and timestamp that the other command owns. An invalid file is left unchanged and reported on stderr. @@ -225,7 +230,7 @@ An invalid file is left unchanged and reported on stderr. |--------+------+-------| | =eb_keep_logs_logs_kept_total= | counter | Successful log-copy operations by =failed=, including a retry that copies the same destination again. | | =eb_keep_logs_bytes_kept_total= | counter | Bytes in those successfully written copies. | -| =eb_keep_logs_last_prune_timestamp_seconds= | gauge | Actual Unix timestamp when =rotate= most recently completed, including a rotation that found nothing to change. It is chosen under the metrics lock, never moves backwards when rotations complete out of order, is independent of =EB_KEEP_LOGS_NOW=, and is zero until one completes. | +| =eb_keep_logs_last_prune_timestamp_seconds= | gauge | Actual Unix timestamp when =rotate= most recently completed (excluding =--dry-run=), including a rotation that found nothing to change. It is chosen under the metrics lock, never moves backwards when rotations complete out of order, is independent of =EB_KEEP_LOGS_NOW=, and is zero until one completes. | Metrics are values this tool observes while keeping and rotating logs. They do not describe dependency gaps, rebuild closures, failure classes, or other stack analysis. @@ -540,8 +545,8 @@ otherwise lose, and keep the pipeline's log tree bounded. Atomically update Prometheus counters for kept logs and bytes. Default: none. eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] - [--lock-timeout SECONDS] [--syslog TARGET] - [--metrics-file PATH] + [--dry-run] [--lock-timeout SECONDS] + [--syslog TARGET] [--metrics-file PATH] In the run directories directly under ROOT (named YYYYMMDD_HHMMSS), gzip the *.log and *.txt files of old runs and remove older runs whole, leaving alone a run that is still written into. A run's age is @@ -555,6 +560,10 @@ otherwise lose, and keep the pipeline's log tree bounded. Remove whole runs at least DAYS old; DAYS must be larger than that of --compress-after. Default: none; no run is removed. An empty value is the default. + --dry-run + Print each file that would be compressed and run that would be + deleted, but change nothing, create no lock, emit no event and + write no metrics. --lock-timeout SECONDS, --lock-timeout=SECONDS While another rotate holds the lock, wait up to SECONDS for it, then give up and exit 75. Default: 600; 0 does not wait. An empty @@ -564,7 +573,8 @@ otherwise lose, and keep the pipeline's log tree bounded. compressed and deleted counts. Default: EB_KEEP_LOGS_SYSLOG, else none. --metrics-file PATH, --metrics-file=PATH - Atomically record when this rotation completed. Default: none. + Atomically record when this rotation completed, unless --dry-run + is given. Default: none. eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json] Say why the builds of a run failed: for each failed easyconfig, its package and easyconfig, the EasyBuild step it failed in, EasyBuild's diff --git a/eb-keep-logs b/eb-keep-logs index 5dd6d89..a2ab89c 100755 --- a/eb-keep-logs +++ b/eb-keep-logs @@ -32,8 +32,8 @@ otherwise lose, and keep the pipeline's log tree bounded. Atomically update Prometheus counters for kept logs and bytes. Default: none. eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] - [--lock-timeout SECONDS] [--syslog TARGET] - [--metrics-file PATH] + [--dry-run] [--lock-timeout SECONDS] + [--syslog TARGET] [--metrics-file PATH] In the run directories directly under ROOT (named YYYYMMDD_HHMMSS), gzip the *.log and *.txt files of old runs and remove older runs whole, leaving alone a run that is still written into. A run's age is @@ -47,6 +47,10 @@ otherwise lose, and keep the pipeline's log tree bounded. Remove whole runs at least DAYS old; DAYS must be larger than that of --compress-after. Default: none; no run is removed. An empty value is the default. + --dry-run + Print each file that would be compressed and run that would be + deleted, but change nothing, create no lock, emit no event and + write no metrics. --lock-timeout SECONDS, --lock-timeout=SECONDS While another rotate holds the lock, wait up to SECONDS for it, then give up and exit 75. Default: 600; 0 does not wait. An empty @@ -56,7 +60,8 @@ otherwise lose, and keep the pipeline's log tree bounded. compressed and deleted counts. Default: EB_KEEP_LOGS_SYSLOG, else none. --metrics-file PATH, --metrics-file=PATH - Atomically record when this rotation completed. Default: none. + Atomically record when this rotation completed, unless --dry-run + is given. Default: none. eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json] Say why the builds of a run failed: for each failed easyconfig, its package and easyconfig, the EasyBuild step it failed in, EasyBuild's @@ -911,13 +916,14 @@ _lock_rotate() { } cmd_rotate() { - local root="" compress_after="" delete_after="" lock_timeout="" syslog_opt="" have_syslog=0 metrics_file="" have_metrics=0 + local root="" compress_after="" delete_after="" lock_timeout="" syslog_opt="" have_syslog=0 dry_run=0 metrics_file="" have_metrics=0 while [ $# -gt 0 ]; do case "$1" in --compress-after) compress_after="${2:-}"; shift 2 ;; --compress-after=*) compress_after="${1#*=}"; shift ;; --delete-after) delete_after="${2:-}"; shift 2 ;; --delete-after=*) delete_after="${1#*=}"; shift ;; + --dry-run) dry_run=1; shift ;; --lock-timeout) [ $# -ge 2 ] || die "--lock-timeout needs a number of seconds" lock_timeout="$2"; shift 2 ;; @@ -960,7 +966,7 @@ cmd_rotate() { _resolve_syslog "${EB_KEEP_LOGS_SYSLOG}" syslog_target="${SYSLOG_TARGET}" fi - _lock_rotate "$(_joinable "${root}")/.eb-keep-logs-rotate.lock" "${lock_timeout}" + [ "${dry_run}" -eq 1 ] || _lock_rotate "$(_joinable "${root}")/.eb-keep-logs-rotate.lock" "${lock_timeout}" local run age file list tmp path compressed=0 deleted=0 live live="$(_own_runs "${root}")$(_written_runs "${root}")/" for run in "$(_joinable "${root}")"/*/; do @@ -971,11 +977,23 @@ cmd_rotate() { # rotate or by a build that is still going. case "${live}" in *"/${run##*/}/"*) continue ;; esac if [ -n "${delete_after}" ] && [ "${age}" -ge "${delete_after}" ]; then + if [ "${dry_run}" -eq 1 ]; then + echo "delete ${run}" + deleted=$((deleted + 1)) + continue + fi rm -rf -- "${run}" deleted=$((deleted + 1)) continue fi [ "${age}" -ge "${compress_after}" ] || continue + if [ "${dry_run}" -eq 1 ]; then + while IFS= read -r -d '' file; do + echo "compress $(realpath "${file%/*}")/${file##*/}" + compressed=$((compressed + 1)) + done < <(find "${run}" -type f \( -name '*.log' -o -name '*.txt' \) -print0 | sort -z) + continue + fi while IFS= read -r -d '' file; do path="$(realpath "${file%/*}")/${file##*/}" # gzip keeps the file's mode, so a closed log stays closed. @@ -993,8 +1011,13 @@ cmd_rotate() { done < <(find "${run}" -type f \( -name '*.log' -o -name '*.txt' \) -print0) done local summary - summary="eb-keep-logs rotate: $(_count "${compressed}" file) compressed, $(_count "${deleted}" run) deleted under ${root}" + if [ "${dry_run}" -eq 1 ]; then + summary="eb-keep-logs rotate dry-run: $(_count "${compressed}" file) would be compressed, $(_count "${deleted}" run) would be deleted under ${root}" + else + summary="eb-keep-logs rotate: $(_count "${compressed}" file) compressed, $(_count "${deleted}" run) deleted under ${root}" + fi echo "${summary}" + [ "${dry_run}" -eq 1 ] && return 0 _emit_syslog "${syslog_target}" 6 "EB_ROTATE_DONE" "${summary}" \ "P:cmd=rotate" "P:root=${root}" "P:compressed=${compressed}" "P:deleted=${deleted}" \ "J:EB_KEEP_CMD=rotate" "J:EB_ROTATE_ROOT=${root}" "J:EB_ROTATE_COMPRESSED=${compressed}" "J:EB_ROTATE_DELETED=${deleted}" diff --git a/src/main.zig b/src/main.zig index faa883d..9a861ef 100644 --- a/src/main.zig +++ b/src/main.zig @@ -49,8 +49,8 @@ const usage_text = \\ Atomically update Prometheus counters for kept logs and bytes. \\ Default: none. \\ eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] - \\ [--lock-timeout SECONDS] [--syslog TARGET] - \\ [--metrics-file PATH] + \\ [--dry-run] [--lock-timeout SECONDS] + \\ [--syslog TARGET] [--metrics-file PATH] \\ In the run directories directly under ROOT (named YYYYMMDD_HHMMSS), \\ gzip the *.log and *.txt files of old runs and remove older runs \\ whole, leaving alone a run that is still written into. A run's age is @@ -64,6 +64,10 @@ const usage_text = \\ Remove whole runs at least DAYS old; DAYS must be larger than \\ that of --compress-after. Default: none; no run is removed. An \\ empty value is the default. + \\ --dry-run + \\ Print each file that would be compressed and run that would be + \\ deleted, but change nothing, create no lock, emit no event and + \\ write no metrics. \\ --lock-timeout SECONDS, --lock-timeout=SECONDS \\ While another rotate holds the lock, wait up to SECONDS for it, \\ then give up and exit 75. Default: 600; 0 does not wait. An empty @@ -73,7 +77,8 @@ const usage_text = \\ compressed and deleted counts. Default: EB_KEEP_LOGS_SYSLOG, \\ else none. \\ --metrics-file PATH, --metrics-file=PATH - \\ Atomically record when this rotation completed. Default: none. + \\ Atomically record when this rotation completed, unless --dry-run + \\ is given. Default: none. \\ eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json] \\ Say why the builds of a run failed: for each failed easyconfig, its \\ package and easyconfig, the EasyBuild step it failed in, EasyBuild's @@ -1001,6 +1006,45 @@ fn compressTree(c: Ctx, arena: Allocator, dir: fsx.fd_t, renames: *Renames, coun } } +/// Print every *.log and *.txt regular file under `dir` that a rotation would +/// try to compress, without following symbolic links or changing anything. +fn previewTree(c: Ctx, arena: Allocator, dir: fsx.fd_t, count: *usize) void { + var rb: [sys.path_max]u8 = undefined; + const here = arena.dupe(u8, fsx.realpathFd(dir, &rb) catch |err| return warn("cannot resolve a run directory: {t}", .{err})) catch return; + var it = fsx.DirIter.init(c.gpa, dir) catch return; + defer it.deinit(c.gpa); + var files: std.ArrayList([]const u8) = .empty; + var subdirs: std.ArrayList([]const u8) = .empty; + while (it.next() catch |err| return warn("cannot read {s}: {t}", .{ here, err })) |e| { + const t = fsx.entryType(dir, e) catch continue; + const n = arena.dupe(u8, e.name) catch return; + if (t == fsx.DT.DIR) { + subdirs.append(arena, n) catch return; + } else if (t == fsx.DT.REG and (std.mem.endsWith(u8, n, ".log") or std.mem.endsWith(u8, n, ".txt"))) { + files.append(arena, n) catch return; + } + } + const less = struct { + fn lt(_: void, a: []const u8, b: []const u8) bool { + return std.mem.lessThan(u8, a, b); + } + }.lt; + std.mem.sort([]const u8, files.items, {}, less); + std.mem.sort([]const u8, subdirs.items, {}, less); + for (files.items) |n| { + cli.out("compress {s}/{s}\n", .{ here, n }); + count.* += 1; + } + for (subdirs.items) |n| { + const sub = fsx.openDir(dir, n, false) catch |err| { + warn("cannot open {s}/{s}: {t}", .{ here, n, err }); + continue; + }; + defer sys.close(sub); + previewTree(c, arena, sub, count); + } +} + /// Rewrite every .licensed-logs under `dir` so it names the compressed /// files. Each new list is written beside the old one, synced, renamed over /// it and its directory synced, as copyDurable does, so a reader or a crash @@ -1111,6 +1155,7 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { var lock_v: []const u8 = ""; var syslog_v: ?[]const u8 = null; var metrics_file: ?[]const u8 = null; + var dry_run = false; var have_compress = false; var have_delete = false; var i: usize = 0; @@ -1134,6 +1179,8 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { metrics_file = args[i]; } else if (std.mem.startsWith(u8, a, "--metrics-file=")) { metrics_file = a["--metrics-file=".len..]; + } else if (std.mem.eql(u8, a, "--dry-run")) { + dry_run = true; } else if (std.mem.eql(u8, a, "--compress-after") or std.mem.eql(u8, a, "--delete-after")) { const v: ?[]const u8 = if (i + 1 < args.len) args[i + 1] else null; if (v == null) die("{s} needs a number of days", .{a}); @@ -1177,17 +1224,21 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { std.fmt.parseInt(i64, now_s, 10) catch die("EB_KEEP_LOGS_NOW must be seconds since the epoch, not '{s}'", .{now_s}) else null; - const lock = openLock(root) catch |err| die("cannot lock {s}/{s}: {t}", .{ joinable(root_p), rotate_lock, err }); - defer sys.close(lock); - waitLock(lock, lock_timeout) catch |err| switch (err) { - error.TimedOut => { - cli.err(tool ++ ": another rotate holds the lock {s}/{s}; gave up after {d} s (--lock-timeout)\n", .{ joinable(root_p), rotate_lock, lock_timeout }); - return lock_busy_exit; - }, - // The file system cannot lock: Lustre without -o flock, NFS without - // a lock manager, or NFS for a file open only for reading. - else => warn("cannot lock {s}/{s}: {t}; rotating without the lock", .{ joinable(root_p), rotate_lock, err }), - }; + var lock: ?fsx.fd_t = null; + if (!dry_run) { + lock = openLock(root) catch |err| die("cannot lock {s}/{s}: {t}", .{ joinable(root_p), rotate_lock, err }); + waitLock(lock.?, lock_timeout) catch |err| switch (err) { + error.TimedOut => { + sys.close(lock.?); + cli.err(tool ++ ": another rotate holds the lock {s}/{s}; gave up after {d} s (--lock-timeout)\n", .{ joinable(root_p), rotate_lock, lock_timeout }); + return lock_busy_exit; + }, + // The file system cannot lock: Lustre without -o flock, NFS without + // a lock manager, or NFS for a file open only for reading. + else => warn("cannot lock {s}/{s}: {t}; rotating without the lock", .{ joinable(root_p), rotate_lock, err }), + }; + } + defer if (lock) |fd| sys.close(fd); const now: i64 = fixed_now orelse realtimeSeconds(); var zone = localtime.load(c.gpa, c.getenv("TZ"), c.getenv("TZDIR")); defer zone.deinit(); @@ -1221,6 +1272,11 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { // rotate or by a build that is still going. if (contains(own, run) or contains(written, run)) continue; if (delete_after) |d| if (age >= @as(i64, @intCast(@min(d, std.math.maxInt(i64))))) { + if (dry_run) { + cli.out("delete {s}/{s}\n", .{ joinable(root_p), run }); + deleted += 1; + continue; + } fsx.deleteTree(c.gpa, root, run) catch |err| { warn("cannot delete {s}/{s}: {t}", .{ joinable(root_p), run, err }); continue; @@ -1234,19 +1290,28 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { continue; }; defer sys.close(dir); + if (dry_run) { + previewTree(c, arena, dir, &compressed); + continue; + } var renames: Renames = .empty; var n: usize = 0; compressTree(c, arena, dir, &renames, &n); compressed += n; if (renames.count() > 0) followLists(c, dir, &renames); } - const summary = arena.print("eb-keep-logs rotate: {d} file{s} compressed, {d} run{s} deleted under {s}", .{ compressed, plural(compressed), deleted, plural(deleted), root_p }) catch "eb-keep-logs rotate"; + const summary = if (dry_run) + arena.print("eb-keep-logs rotate dry-run: {d} file{s} would be compressed, {d} run{s} would be deleted under {s}", .{ compressed, plural(compressed), deleted, plural(deleted), root_p }) catch "eb-keep-logs rotate dry-run" + else + arena.print("eb-keep-logs rotate: {d} file{s} compressed, {d} run{s} deleted under {s}", .{ compressed, plural(compressed), deleted, plural(deleted), root_p }) catch "eb-keep-logs rotate"; cli.out("{s}\n", .{summary}); - emitRotateSyslog(c, syslog_target, root_p, compressed, deleted, summary); - // Run ages use the snapshot above. The metric chooses its completion time - // while holding the metrics lock, after all rotation work and event - // emission have completed. - updateMetrics(c, metrics_file, 0, 0, true); + if (!dry_run) { + emitRotateSyslog(c, syslog_target, root_p, compressed, deleted, summary); + // Run ages use the snapshot above. The metric chooses its completion time + // while holding the metrics lock, after all rotation work and event + // emission have completed. + updateMetrics(c, metrics_file, 0, 0, true); + } return 0; } diff --git a/tests/differential.sh b/tests/differential.sh index 1e01035..9c2066b 100644 --- a/tests/differential.sh +++ b/tests/differential.sh @@ -133,6 +133,17 @@ compare "rotate, bad syslog target" rotate "${work}/logs" --compress-after 5 --s EB_KEEP_LOGS_SYSLOG_PEN=0 compare "failed, bad syslog pen" failed 1 Foo "${work}/logs" EB_KEEP_LOGS_SYSLOG_PEN=abc compare "rotate, bad syslog pen" rotate "${work}/logs" --compress-after 5 EB_KEEP_LOGS_NOW='' compare "rotate, empty EB_KEEP_LOGS_NOW" rotate "${work}/logs" --compress-after 5 +mkdir -p "${work}/preview/20261001_120000" +echo old > "${work}/preview/20261001_120000/Old.stdout.log" +run_env=(PATH="${PATH}" EB_KEEP_LOGS_NOW="$(date -d 2026-11-13 +%s)") +compare "rotate, dry run" rotate "${work}/preview" --compress-after 5 --delete-after 90 --dry-run +compare "rotate, dry run with metrics-file" rotate "${work}/preview" --compress-after 5 --delete-after 90 --dry-run --metrics-file "${work}/dry.prom" +if [ -e "${work}/dry.prom" ] || [ -e "${work}/.dry.prom.lock" ]; then + echo "DIFF rotate, dry run with metrics-file: wrote metrics or lock" >&2 + diffs=$((diffs + 1)) +fi +compare "rotate, dry run with empty metrics-file" rotate "${work}/preview" --compress-after 5 --dry-run --metrics-file "" +run_env=(PATH="${PATH}") compare "rotate, bad lock timeout" rotate "${work}/logs" --compress-after 5 --lock-timeout 1.5 compare "rotate, lock timeout without a value" rotate "${work}/logs" --compress-after 5 --lock-timeout compare "rotate, empty lock timeout" rotate "${work}/logs" --compress-after 5 --lock-timeout= diff --git a/tests/test_eb_keep_logs.sh b/tests/test_eb_keep_logs.sh index d82a546..88c5614 100755 --- a/tests/test_eb_keep_logs.sh +++ b/tests/test_eb_keep_logs.sh @@ -293,6 +293,19 @@ echo other > "${root}/notarun/E.log" EB_KEEP_LOGS_NOW=$(date -d 2026-11-13 +%s) export EB_KEEP_LOGS_NOW +check "dry run: lists files and runs without changing them" \ + "$("${tool}" rotate "${root}" --compress-after 30 --delete-after 90 --dry-run)" \ + "delete ${root}/20260801_090000 +compress $(realpath "${root}/20261001_120000/site-zen4")/A.stdout.log +compress $(realpath "${root}/20261001_120000/site-zen4")/B.failed.ebout.log +eb-keep-logs rotate dry-run: 2 files would be compressed, 1 run would be deleted under ${root}" +check "dry run: files and runs stay in place" \ + "$(find "${root}" -name '*.gz' -o -name 'D.stdout.log' | LC_ALL=C sort | tr '\n' ' ')" \ + "${root}/20260801_090000/D.stdout.log " +lock_missing=1 +test -e "${root}/.eb-keep-logs-rotate.lock" && lock_missing=0 +check "dry run: creates no lock" "$lock_missing" "1" + "${tool}" rotate "${root}" --compress-after 30 >/dev/null check "old run compressed" "$(ls "${root}/20261001_120000/site-zen4" | sort | tr '\n' ' ')" "A.stdout.log.gz B.failed.ebout.log.gz " check "compressed content intact" "$(zcat "${root}/20261001_120000/site-zen4/A.stdout.log.gz")" "old" diff --git a/tests/test_metrics.sh b/tests/test_metrics.sh index 63bf3d7..c07bce4 100755 --- a/tests/test_metrics.sh +++ b/tests/test_metrics.sh @@ -128,6 +128,27 @@ overflow=$(PATH="${work}/bin:${PATH}" "${tool}" failed --metrics-file "${metrics case "${overflow}" in *"counter overflow"*|*"Overflow"*) ;; *) exit 1 ;; esac [ "$(sha256sum "${metrics}")" = "${before}" ] +# A dry-run rotation writes no metrics, creates no sidecar lock, and leaves +# an existing exposition untouched even when --metrics-file is given. +dry_root="${work}/dry-root" +mkdir -p "${dry_root}/20261001_120000" +printf 'old\n' > "${dry_root}/20261001_120000/Old.stdout.log" +dry_new="${work}/collector/dry-new.prom" +dry_out=$(EB_KEEP_LOGS_NOW=$(date -d 2026-11-13 +%s) "${tool}" rotate "${dry_root}" --compress-after 5 --dry-run --metrics-file "${dry_new}") || exit 1 +case "${dry_out}" in *"eb-keep-logs rotate dry-run: 1 file would be compressed, 0 runs would be deleted under ${dry_root}"*) ;; *) exit 1 ;; esac +[ ! -e "${dry_new}" ] || exit 1 +[ ! -e "${work}/collector/.dry-new.prom.lock" ] || exit 1 +[ ! -e "${dry_root}/.eb-keep-logs-rotate.lock" ] || exit 1 +[ -f "${dry_root}/20261001_120000/Old.stdout.log" ] || exit 1 + +dry_existing="${work}/collector/dry-existing.prom" +cp "${metrics}" "${dry_existing}" +before=$(sha256sum "${dry_existing}") +EB_KEEP_LOGS_NOW=$(date -d 2026-11-13 +%s) "${tool}" rotate "${dry_root}" --compress-after 5 --dry-run --metrics-file="${dry_existing}" >/dev/null || exit 1 +[ "$(sha256sum "${dry_existing}")" = "${before}" ] || exit 1 +[ ! -e "${work}/collector/.dry-existing.prom.lock" ] || exit 1 +[ ! -e "${dry_root}/.eb-keep-logs-rotate.lock" ] || exit 1 + # No temporary output remains after any successful replacement. [ "$(find "${work}/collector" -maxdepth 1 -name '*.tmp' -print | wc -l)" -eq 0 ] echo "metrics tests passed"