From 69563e91c31ab0d7ebecd0ba52a2fff34ab3722d Mon Sep 17 00:00:00 2001 From: Rohit Goswami Date: Wed, 18 Nov 2026 00:30:52 +0000 Subject: [PATCH] feat: preview log rotation Rotate gained an opt-in dry-run mode in both the shell script and the Zig binary. It lists the files that would be compressed and the runs that would be deleted, without changing the tree, creating its lock file, emitting an event, or writing metrics when a metrics file is given. --- COMPAT.md | 3 +- README.md | 8 +- changelog.d/+rotate-dry-run.feature.md | 1 + docs/orgmode/howto/rotate-a-log-tree.org | 16 ++-- docs/orgmode/reference/cli.org | 24 ++++-- eb-keep-logs | 35 ++++++-- src/main.zig | 105 ++++++++++++++++++----- tests/differential.sh | 11 +++ tests/test_eb_keep_logs.sh | 13 +++ tests/test_metrics.sh | 21 +++++ 10 files changed, 192 insertions(+), 45 deletions(-) create mode 100644 changelog.d/+rotate-dry-run.feature.md diff --git a/COMPAT.md b/COMPAT.md index 7857039..337e7ec 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -9,7 +9,7 @@ on the same invocations. | | | |---|---| -| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS]`, `-h`, `--help`, `--version` | +| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS] [--dry-run] [--lock-timeout SECONDS]`, `-h`, `--help`, `--version` | | usage | the same text on stdout for `-h`, `--help` and no arguments | | exit codes | 0 on success and on every `failed` that got past its arguments; 2 for a refused invocation, after `eb-keep-logs: MESSAGE` on stderr; 2 with the usage text on stdout for no arguments; 75 when another `rotate` holds ROOT's lock past `--lock-timeout`, after `eb-keep-logs: another rotate holds the lock ROOT/.eb-keep-logs-rotate.lock; gave up after N s (--lock-timeout)` on stderr | | one rotate at a time | `rotate` holds an exclusive flock(2) on `ROOT/.eb-keep-logs-rotate.lock`, created when missing, from before it reads ROOT until it exits; it waits up to `--lock-timeout` seconds for it, 600 without the option or with an empty value, 0 not at all, 999999999 at most; it opens the file for reading when it may not write it; a lock file that cannot be opened, or a symbolic link or non-regular file at its name, is refused, exit 2; on a file system that cannot lock, `rotate` says so on stderr and rotates without the lock | @@ -20,6 +20,7 @@ on the same invocations. | run ages | days from the local midnight of the date in the run's name, as `date -d YYYYMMDD` computes it, including TZ rules, except for the zones under unintended differences; a run whose local midnight does not exist is skipped | | the run in progress | `rotate` neither compresses nor deletes the run that holds its stdout or stderr, when either is a regular file, or `EB_PIPELINE_LOG_DIR` | | runs still written into | after taking its lock, `rotate` reads the links `/proc/PID/fd/N` of every process whose descriptors it may read; a run that holds the path a link with its owner's write bit names, also when the path ends in ` (deleted)`, is neither compressed nor deleted; without `/proc`, no run is left alone for this | +| rotation preview | `rotate --dry-run` prints each `compress PATH` and `delete PATH` action followed by the counts, changes nothing, creates no lock file, emits no syslog or journal event and writes no metrics; because it takes no lock, the output is a snapshot | ## Deliberate differences diff --git a/README.md b/README.md index f7f9cb1..225c4c7 100644 --- a/README.md +++ b/README.md @@ -91,12 +91,12 @@ Rotations of one tree take turns: each holds an exclusive flock on `ROOT/.eb-kee One that starts while another runs waits for it, up to `--lock-timeout` seconds (600; `0` does not wait), and when the wait runs out it changes nothing and exits 75. `eb-pipeline`'s `ROOT/.eb-pipeline-rotate.lock` is a separate lock, which `rotate` never takes; a daily job that should wait for eb-pipeline's runs takes that lock around `rotate` with `flock`. -`EB_KEEP_LOGS_NOW`, in seconds since the epoch, sets the time `rotate` dates runs against. -On a copy of the tree, it previews what a later rotation compresses and removes: +`--dry-run` prints each file that would be compressed and run that would be deleted, but changes nothing, creates no lock and emits no syslog or journal event. +`EB_KEEP_LOGS_NOW`, in seconds since the epoch, sets the time `rotate` dates runs against, so the two together preview a later rotation: ```bash -cp -a /srv/build-logs /tmp/logs-preview -EB_KEEP_LOGS_NOW=$(date -d 2027-09-20 +%s) eb-keep-logs rotate /tmp/logs-preview --compress-after 30 --delete-after 365 +EB_KEEP_LOGS_NOW=$(date -d 2027-09-20 +%s) \ + eb-keep-logs rotate /srv/build-logs --compress-after 30 --delete-after 365 --dry-run ``` ## Find why a build failed diff --git a/changelog.d/+rotate-dry-run.feature.md b/changelog.d/+rotate-dry-run.feature.md new file mode 100644 index 0000000..41ede21 --- /dev/null +++ b/changelog.d/+rotate-dry-run.feature.md @@ -0,0 +1 @@ +`eb-keep-logs rotate --dry-run` lists the log files it would compress and run directories it would delete without changing the tree, creating its lock file, emitting an event, or writing metrics. diff --git a/docs/orgmode/howto/rotate-a-log-tree.org b/docs/orgmode/howto/rotate-a-log-tree.org index ae38045..e3da9de 100644 --- a/docs/orgmode/howto/rotate-a-log-tree.org +++ b/docs/orgmode/howto/rotate-a-log-tree.org @@ -18,20 +18,23 @@ Use this to keep a directory of build runs from growing without end: compress th A run's age is the number of whole days from the local midnight of the date in its name. =--compress-after DAYS= compresses the =*.log= and =*.txt= files of runs at least =DAYS= old, at any depth. =--delete-after DAYS= removes runs at least =DAYS= old, and must be larger; without it, no run is removed. -2. Try the ages on a copy of the tree, at a later date. - =EB_KEEP_LOGS_NOW= sets the time =rotate= takes as now, in seconds since the epoch: +2. Preview the rotation. + =--dry-run= lists every file it would compress and run it would delete, but changes nothing, creates no lock file, emits no syslog or journal event, and writes no metrics. + =EB_KEEP_LOGS_NOW= sets the time =rotate= takes as now, in seconds since the epoch, so this also previews the policy at a later date: #+begin_src bash - cp -a ~/build-logs /tmp/build-logs-preview EB_KEEP_LOGS_NOW=$(date -d 2027-01-01 +%s) \ - eb-keep-logs rotate /tmp/build-logs-preview --compress-after 30 --delete-after 365 + eb-keep-logs rotate ~/build-logs --compress-after 30 --delete-after 365 --dry-run #+end_src #+begin_example - eb-keep-logs rotate: 5 files compressed, 0 runs deleted under /tmp/build-logs-preview + compress /home/alice/build-logs/20261116_175300/Example-1.0.failed.ebout.log + compress /home/alice/build-logs/20261116_175300/Example-1.0.stdout.log + eb-keep-logs rotate dry-run: 2 files would be compressed, 0 runs would be deleted under /home/alice/build-logs #+end_example On 2027-01-01 the run is 46 days old: at least 30, less than 365. + The preview is a snapshot: it deliberately takes no rotation lock, so preview again when the tree may have changed before the real rotation. 3. Rotate the tree: #+begin_src bash @@ -66,7 +69,7 @@ The summary line counts the files =rotate= compressed and the runs it removed. In a compressed run, each log is a =.gz= file with the log's mode: #+begin_src bash -ls -l /tmp/build-logs-preview/*/ +ls -l ~/build-logs/*/ #+end_src #+begin_example @@ -83,7 +86,6 @@ drwxrwxr-x 3 alice alice 4096 Nov 16 17:53 easybuild-logs |--------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------| | =eb-keep-logs: --compress-after needs a number of days= | The option is missing, or its value is not a whole number, like =30d=. | Give a number of days: =--compress-after 30=. | | =eb-keep-logs: --delete-after must be later than --compress-after= | =--delete-after= is not larger than =--compress-after=. | Raise =--delete-after=, or leave it out to remove nothing. | -| =eb-keep-logs: unknown option --dry-run= | =rotate= has no dry run. | Try the ages on a copy, as in step 2. | | =eb-keep-logs: ROOT PATH is not a directory= | The path is wrong, or names a run, not the tree. | Give the directory that holds the runs. | | =eb-keep-logs: cannot compress PATH: TooManyLinks= | The log has other hard links. The shell script prints =gzip='s own message. | Nothing: like =gzip=, =rotate= leaves it as it is. | | =eb-keep-logs: cannot compress PATH: Exists= | =PATH.gz= exists already. The shell script prints =gzip='s own message. | Remove or rename one of the two, and rotate again. | diff --git a/docs/orgmode/reference/cli.org b/docs/orgmode/reference/cli.org index 2f70359..69dd84c 100644 --- a/docs/orgmode/reference/cli.org +++ b/docs/orgmode/reference/cli.org @@ -11,7 +11,7 @@ This page describes the Zig build; the shell script =eb-keep-logs= in the reposi #+begin_example eb-keep-logs failed [--syslog TARGET] [--metrics-file PATH] STATUS NAME LOGDIR [TMP_LOGDIR] -eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS] [--syslog TARGET] [--metrics-file PATH] +eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--dry-run] [--lock-timeout SECONDS] [--syslog TARGET] [--metrics-file PATH] eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json] eb-keep-logs history PACKAGE [--target TARGET] [--root ROOT] [--json] eb-keep-logs loads --module NAME [--json] FILE @@ -43,9 +43,10 @@ eb-keep-logs --version |--------------------------------------------------+----------+----------------------------------------------------------------------------------------------------------------------------------------------| | =--compress-after DAYS=, =--compress-after=DAYS= | =rotate= | Required. Compress the =*.log= and =*.txt= files, at any depth, of runs at least =DAYS= old. =DAYS= is decimal digits. | | =--delete-after DAYS=, =--delete-after=DAYS= | =rotate= | Remove whole runs at least =DAYS= old. =DAYS= must be larger than =--compress-after=. Without it, or with an empty value, no run is removed. | +| =--dry-run= | =rotate= | Print =compress PATH= for each log file that would be compressed and =delete PATH= for each run that would be removed, followed by a summary; change nothing, create no lock file, emit no syslog or journal event, and write no metrics. | | =--lock-timeout SECONDS=, =--lock-timeout=SECONDS= | =rotate= | How long to wait for the lock another =rotate= of =ROOT= holds, in seconds, decimal digits; =0= does not wait. Without it, or with an empty value, 600. A value larger than 999999999, which is over 31 years, waits 999999999 seconds. | | =--syslog TARGET=, =--syslog=TARGET= | =failed=, =rotate= | Emit an RFC 5424 or native =systemd-journald= event to =TARGET= (=journald=, =syslog=, =unix:///path=, =journal+unix:///path=, =udp://host:port=, or =tcp://host:port=); overrides =EB_KEEP_LOGS_SYSLOG=. An empty value disables emission. | -| =--metrics-file PATH=, =--metrics-file=PATH= | =failed=, =rotate= | Atomically update the Prometheus text exposition at =PATH=. Without it, no metrics file is read or written. | +| =--metrics-file PATH=, =--metrics-file=PATH= | =failed=, =rotate= | Atomically update the Prometheus text exposition at =PATH=, except on =rotate --dry-run=, which writes no metrics. Without it, no metrics file is read or written. | | =--latest= | =why= | Read the newest run under =--root=, by its name; with =--target=, the newest that has that target. The default without =RUN=. | | =--target TARGET=, =--target=TARGET= | =why=, =history= | Read only the builds in the run's directory =TARGET=; for =history=, in each run's, a run without one passed over. Without it, those in the run's own directory and then in each directory in it, by name. | | =--root ROOT=, =--root=ROOT= | =why=, =history= | The directory that holds the runs, directories named =YYYYMMDD_HHMMSS=, not symbolic links. Without it, the current directory. | @@ -74,6 +75,7 @@ eb-keep-logs --version | =--version= | | Print the version line on stdout and exit 0. | - A run's age is the number of whole days from the local midnight of the date in its name; [[file:../explanation/local-time.org][Run ages and local time]] gives the details. +- =--dry-run= reads the tree and the processes writing into it without taking the rotation lock, so its output is a snapshot, not a reservation. It also skips syslog emission and =--metrics-file= updates. - =rotate='s options and =ROOT= come in any order; the value of an option is the next argument, whatever it is. An option given twice takes its last value. - =why='s options and =RUN=, and =history='s and =PACKAGE=, come in any order, and an option given twice takes its last value; the value of an option is the next argument, whatever it is, and an empty one is refused. @@ -97,7 +99,7 @@ eb-keep-logs --version | Status | Meaning | |--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0 | =-h=, =--help= or =--version=, a =failed=, =why= or =history= that accepted its arguments, or a =rotate= that accepted its arguments and did not give up waiting for its lock, whatever it could keep, compress, remove or read: what it could not do is reported, and is no failure. A =why= exits 0 whether or not the run's builds failed, and a =history= whether or not it found a build of the package. | +| 0 | =-h=, =--help= or =--version=, a =failed=, =why= or =history= that accepted its arguments, or a =rotate= that accepted its arguments and did not give up waiting for its lock, whatever it could keep, compress, remove, preview or read: what it could not do is reported, and is no failure. A =why= exits 0 whether or not the run's builds failed, and a =history= whether or not it found a build of the package. | | 2 | The invocation was refused, with =eb-keep-logs: MESSAGE= on stderr; or no command was given, and the usage is on stdout. | | 75 | Another =rotate= held =ROOT='s lock for longer than =--lock-timeout=, and this =rotate= changed nothing; try again later. | @@ -191,6 +193,8 @@ What =failed= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe | =ROOT/RUN= | Removed with everything in it, when old enough for =--delete-after=, without being compressed first. A symbolic link in it is removed, not followed. Removal stops at the first error. | | =/proc/PID/fd/N= | Read, for every process whose descriptors =rotate= may read, after it has the lock: a link with its owner's write bit names a file the process has open for writing, and the run that file is in is left alone. Without =/proc=, no run is left alone for it. | +With =--dry-run= these paths are only read; =ROOT/.eb-keep-logs-rotate.lock= is neither opened nor created, and =--metrics-file= neither reads nor writes any metrics or lock file. + A run that holds the file =rotate='s standard output or standard error goes to, or =EB_PIPELINE_LOG_DIR=, or a file another process has open for writing, as a build that is still going has its log, is left alone, neither compressed nor deleted; a later =rotate= takes it. The path is what counts: the path the link in =/proc= names, or for a file removed since, that path with =" (deleted)"= after it, has to start with =ROOT='s, with symbolic links resolved. On stdout =rotate= prints one line, with =ROOT= as given: @@ -217,6 +221,7 @@ What =rotate= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe *** Metrics With =--metrics-file PATH=, =failed= and =rotate= update one Prometheus text exposition file. +When =--dry-run= is given to =rotate=, it writes no metrics and leaves any existing metrics file and its sidecar lock untouched. The file is written to a temporary file in =PATH='s directory, synced, and renamed over =PATH= while a sidecar lock is held. An existing valid file supplies the counters and timestamp that the other command owns. An invalid file is left unchanged and reported on stderr. @@ -225,7 +230,7 @@ An invalid file is left unchanged and reported on stderr. |--------+------+-------| | =eb_keep_logs_logs_kept_total= | counter | Successful log-copy operations by =failed=, including a retry that copies the same destination again. | | =eb_keep_logs_bytes_kept_total= | counter | Bytes in those successfully written copies. | -| =eb_keep_logs_last_prune_timestamp_seconds= | gauge | Actual Unix timestamp when =rotate= most recently completed, including a rotation that found nothing to change. It is chosen under the metrics lock, never moves backwards when rotations complete out of order, is independent of =EB_KEEP_LOGS_NOW=, and is zero until one completes. | +| =eb_keep_logs_last_prune_timestamp_seconds= | gauge | Actual Unix timestamp when =rotate= most recently completed (excluding =--dry-run=), including a rotation that found nothing to change. It is chosen under the metrics lock, never moves backwards when rotations complete out of order, is independent of =EB_KEEP_LOGS_NOW=, and is zero until one completes. | Metrics are values this tool observes while keeping and rotating logs. They do not describe dependency gaps, rebuild closures, failure classes, or other stack analysis. @@ -540,8 +545,8 @@ otherwise lose, and keep the pipeline's log tree bounded. Atomically update Prometheus counters for kept logs and bytes. Default: none. eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] - [--lock-timeout SECONDS] [--syslog TARGET] - [--metrics-file PATH] + [--dry-run] [--lock-timeout SECONDS] + [--syslog TARGET] [--metrics-file PATH] In the run directories directly under ROOT (named YYYYMMDD_HHMMSS), gzip the *.log and *.txt files of old runs and remove older runs whole, leaving alone a run that is still written into. A run's age is @@ -555,6 +560,10 @@ otherwise lose, and keep the pipeline's log tree bounded. Remove whole runs at least DAYS old; DAYS must be larger than that of --compress-after. Default: none; no run is removed. An empty value is the default. + --dry-run + Print each file that would be compressed and run that would be + deleted, but change nothing, create no lock, emit no event and + write no metrics. --lock-timeout SECONDS, --lock-timeout=SECONDS While another rotate holds the lock, wait up to SECONDS for it, then give up and exit 75. Default: 600; 0 does not wait. An empty @@ -564,7 +573,8 @@ otherwise lose, and keep the pipeline's log tree bounded. compressed and deleted counts. Default: EB_KEEP_LOGS_SYSLOG, else none. --metrics-file PATH, --metrics-file=PATH - Atomically record when this rotation completed. Default: none. + Atomically record when this rotation completed, unless --dry-run + is given. Default: none. eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json] Say why the builds of a run failed: for each failed easyconfig, its package and easyconfig, the EasyBuild step it failed in, EasyBuild's diff --git a/eb-keep-logs b/eb-keep-logs index 5dd6d89..a2ab89c 100755 --- a/eb-keep-logs +++ b/eb-keep-logs @@ -32,8 +32,8 @@ otherwise lose, and keep the pipeline's log tree bounded. Atomically update Prometheus counters for kept logs and bytes. Default: none. eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] - [--lock-timeout SECONDS] [--syslog TARGET] - [--metrics-file PATH] + [--dry-run] [--lock-timeout SECONDS] + [--syslog TARGET] [--metrics-file PATH] In the run directories directly under ROOT (named YYYYMMDD_HHMMSS), gzip the *.log and *.txt files of old runs and remove older runs whole, leaving alone a run that is still written into. A run's age is @@ -47,6 +47,10 @@ otherwise lose, and keep the pipeline's log tree bounded. Remove whole runs at least DAYS old; DAYS must be larger than that of --compress-after. Default: none; no run is removed. An empty value is the default. + --dry-run + Print each file that would be compressed and run that would be + deleted, but change nothing, create no lock, emit no event and + write no metrics. --lock-timeout SECONDS, --lock-timeout=SECONDS While another rotate holds the lock, wait up to SECONDS for it, then give up and exit 75. Default: 600; 0 does not wait. An empty @@ -56,7 +60,8 @@ otherwise lose, and keep the pipeline's log tree bounded. compressed and deleted counts. Default: EB_KEEP_LOGS_SYSLOG, else none. --metrics-file PATH, --metrics-file=PATH - Atomically record when this rotation completed. Default: none. + Atomically record when this rotation completed, unless --dry-run + is given. Default: none. eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json] Say why the builds of a run failed: for each failed easyconfig, its package and easyconfig, the EasyBuild step it failed in, EasyBuild's @@ -911,13 +916,14 @@ _lock_rotate() { } cmd_rotate() { - local root="" compress_after="" delete_after="" lock_timeout="" syslog_opt="" have_syslog=0 metrics_file="" have_metrics=0 + local root="" compress_after="" delete_after="" lock_timeout="" syslog_opt="" have_syslog=0 dry_run=0 metrics_file="" have_metrics=0 while [ $# -gt 0 ]; do case "$1" in --compress-after) compress_after="${2:-}"; shift 2 ;; --compress-after=*) compress_after="${1#*=}"; shift ;; --delete-after) delete_after="${2:-}"; shift 2 ;; --delete-after=*) delete_after="${1#*=}"; shift ;; + --dry-run) dry_run=1; shift ;; --lock-timeout) [ $# -ge 2 ] || die "--lock-timeout needs a number of seconds" lock_timeout="$2"; shift 2 ;; @@ -960,7 +966,7 @@ cmd_rotate() { _resolve_syslog "${EB_KEEP_LOGS_SYSLOG}" syslog_target="${SYSLOG_TARGET}" fi - _lock_rotate "$(_joinable "${root}")/.eb-keep-logs-rotate.lock" "${lock_timeout}" + [ "${dry_run}" -eq 1 ] || _lock_rotate "$(_joinable "${root}")/.eb-keep-logs-rotate.lock" "${lock_timeout}" local run age file list tmp path compressed=0 deleted=0 live live="$(_own_runs "${root}")$(_written_runs "${root}")/" for run in "$(_joinable "${root}")"/*/; do @@ -971,11 +977,23 @@ cmd_rotate() { # rotate or by a build that is still going. case "${live}" in *"/${run##*/}/"*) continue ;; esac if [ -n "${delete_after}" ] && [ "${age}" -ge "${delete_after}" ]; then + if [ "${dry_run}" -eq 1 ]; then + echo "delete ${run}" + deleted=$((deleted + 1)) + continue + fi rm -rf -- "${run}" deleted=$((deleted + 1)) continue fi [ "${age}" -ge "${compress_after}" ] || continue + if [ "${dry_run}" -eq 1 ]; then + while IFS= read -r -d '' file; do + echo "compress $(realpath "${file%/*}")/${file##*/}" + compressed=$((compressed + 1)) + done < <(find "${run}" -type f \( -name '*.log' -o -name '*.txt' \) -print0 | sort -z) + continue + fi while IFS= read -r -d '' file; do path="$(realpath "${file%/*}")/${file##*/}" # gzip keeps the file's mode, so a closed log stays closed. @@ -993,8 +1011,13 @@ cmd_rotate() { done < <(find "${run}" -type f \( -name '*.log' -o -name '*.txt' \) -print0) done local summary - summary="eb-keep-logs rotate: $(_count "${compressed}" file) compressed, $(_count "${deleted}" run) deleted under ${root}" + if [ "${dry_run}" -eq 1 ]; then + summary="eb-keep-logs rotate dry-run: $(_count "${compressed}" file) would be compressed, $(_count "${deleted}" run) would be deleted under ${root}" + else + summary="eb-keep-logs rotate: $(_count "${compressed}" file) compressed, $(_count "${deleted}" run) deleted under ${root}" + fi echo "${summary}" + [ "${dry_run}" -eq 1 ] && return 0 _emit_syslog "${syslog_target}" 6 "EB_ROTATE_DONE" "${summary}" \ "P:cmd=rotate" "P:root=${root}" "P:compressed=${compressed}" "P:deleted=${deleted}" \ "J:EB_KEEP_CMD=rotate" "J:EB_ROTATE_ROOT=${root}" "J:EB_ROTATE_COMPRESSED=${compressed}" "J:EB_ROTATE_DELETED=${deleted}" diff --git a/src/main.zig b/src/main.zig index faa883d..9a861ef 100644 --- a/src/main.zig +++ b/src/main.zig @@ -49,8 +49,8 @@ const usage_text = \\ Atomically update Prometheus counters for kept logs and bytes. \\ Default: none. \\ eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] - \\ [--lock-timeout SECONDS] [--syslog TARGET] - \\ [--metrics-file PATH] + \\ [--dry-run] [--lock-timeout SECONDS] + \\ [--syslog TARGET] [--metrics-file PATH] \\ In the run directories directly under ROOT (named YYYYMMDD_HHMMSS), \\ gzip the *.log and *.txt files of old runs and remove older runs \\ whole, leaving alone a run that is still written into. A run's age is @@ -64,6 +64,10 @@ const usage_text = \\ Remove whole runs at least DAYS old; DAYS must be larger than \\ that of --compress-after. Default: none; no run is removed. An \\ empty value is the default. + \\ --dry-run + \\ Print each file that would be compressed and run that would be + \\ deleted, but change nothing, create no lock, emit no event and + \\ write no metrics. \\ --lock-timeout SECONDS, --lock-timeout=SECONDS \\ While another rotate holds the lock, wait up to SECONDS for it, \\ then give up and exit 75. Default: 600; 0 does not wait. An empty @@ -73,7 +77,8 @@ const usage_text = \\ compressed and deleted counts. Default: EB_KEEP_LOGS_SYSLOG, \\ else none. \\ --metrics-file PATH, --metrics-file=PATH - \\ Atomically record when this rotation completed. Default: none. + \\ Atomically record when this rotation completed, unless --dry-run + \\ is given. Default: none. \\ eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json] \\ Say why the builds of a run failed: for each failed easyconfig, its \\ package and easyconfig, the EasyBuild step it failed in, EasyBuild's @@ -1001,6 +1006,45 @@ fn compressTree(c: Ctx, arena: Allocator, dir: fsx.fd_t, renames: *Renames, coun } } +/// Print every *.log and *.txt regular file under `dir` that a rotation would +/// try to compress, without following symbolic links or changing anything. +fn previewTree(c: Ctx, arena: Allocator, dir: fsx.fd_t, count: *usize) void { + var rb: [sys.path_max]u8 = undefined; + const here = arena.dupe(u8, fsx.realpathFd(dir, &rb) catch |err| return warn("cannot resolve a run directory: {t}", .{err})) catch return; + var it = fsx.DirIter.init(c.gpa, dir) catch return; + defer it.deinit(c.gpa); + var files: std.ArrayList([]const u8) = .empty; + var subdirs: std.ArrayList([]const u8) = .empty; + while (it.next() catch |err| return warn("cannot read {s}: {t}", .{ here, err })) |e| { + const t = fsx.entryType(dir, e) catch continue; + const n = arena.dupe(u8, e.name) catch return; + if (t == fsx.DT.DIR) { + subdirs.append(arena, n) catch return; + } else if (t == fsx.DT.REG and (std.mem.endsWith(u8, n, ".log") or std.mem.endsWith(u8, n, ".txt"))) { + files.append(arena, n) catch return; + } + } + const less = struct { + fn lt(_: void, a: []const u8, b: []const u8) bool { + return std.mem.lessThan(u8, a, b); + } + }.lt; + std.mem.sort([]const u8, files.items, {}, less); + std.mem.sort([]const u8, subdirs.items, {}, less); + for (files.items) |n| { + cli.out("compress {s}/{s}\n", .{ here, n }); + count.* += 1; + } + for (subdirs.items) |n| { + const sub = fsx.openDir(dir, n, false) catch |err| { + warn("cannot open {s}/{s}: {t}", .{ here, n, err }); + continue; + }; + defer sys.close(sub); + previewTree(c, arena, sub, count); + } +} + /// Rewrite every .licensed-logs under `dir` so it names the compressed /// files. Each new list is written beside the old one, synced, renamed over /// it and its directory synced, as copyDurable does, so a reader or a crash @@ -1111,6 +1155,7 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { var lock_v: []const u8 = ""; var syslog_v: ?[]const u8 = null; var metrics_file: ?[]const u8 = null; + var dry_run = false; var have_compress = false; var have_delete = false; var i: usize = 0; @@ -1134,6 +1179,8 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { metrics_file = args[i]; } else if (std.mem.startsWith(u8, a, "--metrics-file=")) { metrics_file = a["--metrics-file=".len..]; + } else if (std.mem.eql(u8, a, "--dry-run")) { + dry_run = true; } else if (std.mem.eql(u8, a, "--compress-after") or std.mem.eql(u8, a, "--delete-after")) { const v: ?[]const u8 = if (i + 1 < args.len) args[i + 1] else null; if (v == null) die("{s} needs a number of days", .{a}); @@ -1177,17 +1224,21 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { std.fmt.parseInt(i64, now_s, 10) catch die("EB_KEEP_LOGS_NOW must be seconds since the epoch, not '{s}'", .{now_s}) else null; - const lock = openLock(root) catch |err| die("cannot lock {s}/{s}: {t}", .{ joinable(root_p), rotate_lock, err }); - defer sys.close(lock); - waitLock(lock, lock_timeout) catch |err| switch (err) { - error.TimedOut => { - cli.err(tool ++ ": another rotate holds the lock {s}/{s}; gave up after {d} s (--lock-timeout)\n", .{ joinable(root_p), rotate_lock, lock_timeout }); - return lock_busy_exit; - }, - // The file system cannot lock: Lustre without -o flock, NFS without - // a lock manager, or NFS for a file open only for reading. - else => warn("cannot lock {s}/{s}: {t}; rotating without the lock", .{ joinable(root_p), rotate_lock, err }), - }; + var lock: ?fsx.fd_t = null; + if (!dry_run) { + lock = openLock(root) catch |err| die("cannot lock {s}/{s}: {t}", .{ joinable(root_p), rotate_lock, err }); + waitLock(lock.?, lock_timeout) catch |err| switch (err) { + error.TimedOut => { + sys.close(lock.?); + cli.err(tool ++ ": another rotate holds the lock {s}/{s}; gave up after {d} s (--lock-timeout)\n", .{ joinable(root_p), rotate_lock, lock_timeout }); + return lock_busy_exit; + }, + // The file system cannot lock: Lustre without -o flock, NFS without + // a lock manager, or NFS for a file open only for reading. + else => warn("cannot lock {s}/{s}: {t}; rotating without the lock", .{ joinable(root_p), rotate_lock, err }), + }; + } + defer if (lock) |fd| sys.close(fd); const now: i64 = fixed_now orelse realtimeSeconds(); var zone = localtime.load(c.gpa, c.getenv("TZ"), c.getenv("TZDIR")); defer zone.deinit(); @@ -1221,6 +1272,11 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { // rotate or by a build that is still going. if (contains(own, run) or contains(written, run)) continue; if (delete_after) |d| if (age >= @as(i64, @intCast(@min(d, std.math.maxInt(i64))))) { + if (dry_run) { + cli.out("delete {s}/{s}\n", .{ joinable(root_p), run }); + deleted += 1; + continue; + } fsx.deleteTree(c.gpa, root, run) catch |err| { warn("cannot delete {s}/{s}: {t}", .{ joinable(root_p), run, err }); continue; @@ -1234,19 +1290,28 @@ fn cmdRotate(c: Ctx, args: []const [:0]const u8) u8 { continue; }; defer sys.close(dir); + if (dry_run) { + previewTree(c, arena, dir, &compressed); + continue; + } var renames: Renames = .empty; var n: usize = 0; compressTree(c, arena, dir, &renames, &n); compressed += n; if (renames.count() > 0) followLists(c, dir, &renames); } - const summary = arena.print("eb-keep-logs rotate: {d} file{s} compressed, {d} run{s} deleted under {s}", .{ compressed, plural(compressed), deleted, plural(deleted), root_p }) catch "eb-keep-logs rotate"; + const summary = if (dry_run) + arena.print("eb-keep-logs rotate dry-run: {d} file{s} would be compressed, {d} run{s} would be deleted under {s}", .{ compressed, plural(compressed), deleted, plural(deleted), root_p }) catch "eb-keep-logs rotate dry-run" + else + arena.print("eb-keep-logs rotate: {d} file{s} compressed, {d} run{s} deleted under {s}", .{ compressed, plural(compressed), deleted, plural(deleted), root_p }) catch "eb-keep-logs rotate"; cli.out("{s}\n", .{summary}); - emitRotateSyslog(c, syslog_target, root_p, compressed, deleted, summary); - // Run ages use the snapshot above. The metric chooses its completion time - // while holding the metrics lock, after all rotation work and event - // emission have completed. - updateMetrics(c, metrics_file, 0, 0, true); + if (!dry_run) { + emitRotateSyslog(c, syslog_target, root_p, compressed, deleted, summary); + // Run ages use the snapshot above. The metric chooses its completion time + // while holding the metrics lock, after all rotation work and event + // emission have completed. + updateMetrics(c, metrics_file, 0, 0, true); + } return 0; } diff --git a/tests/differential.sh b/tests/differential.sh index 1e01035..9c2066b 100644 --- a/tests/differential.sh +++ b/tests/differential.sh @@ -133,6 +133,17 @@ compare "rotate, bad syslog target" rotate "${work}/logs" --compress-after 5 --s EB_KEEP_LOGS_SYSLOG_PEN=0 compare "failed, bad syslog pen" failed 1 Foo "${work}/logs" EB_KEEP_LOGS_SYSLOG_PEN=abc compare "rotate, bad syslog pen" rotate "${work}/logs" --compress-after 5 EB_KEEP_LOGS_NOW='' compare "rotate, empty EB_KEEP_LOGS_NOW" rotate "${work}/logs" --compress-after 5 +mkdir -p "${work}/preview/20261001_120000" +echo old > "${work}/preview/20261001_120000/Old.stdout.log" +run_env=(PATH="${PATH}" EB_KEEP_LOGS_NOW="$(date -d 2026-11-13 +%s)") +compare "rotate, dry run" rotate "${work}/preview" --compress-after 5 --delete-after 90 --dry-run +compare "rotate, dry run with metrics-file" rotate "${work}/preview" --compress-after 5 --delete-after 90 --dry-run --metrics-file "${work}/dry.prom" +if [ -e "${work}/dry.prom" ] || [ -e "${work}/.dry.prom.lock" ]; then + echo "DIFF rotate, dry run with metrics-file: wrote metrics or lock" >&2 + diffs=$((diffs + 1)) +fi +compare "rotate, dry run with empty metrics-file" rotate "${work}/preview" --compress-after 5 --dry-run --metrics-file "" +run_env=(PATH="${PATH}") compare "rotate, bad lock timeout" rotate "${work}/logs" --compress-after 5 --lock-timeout 1.5 compare "rotate, lock timeout without a value" rotate "${work}/logs" --compress-after 5 --lock-timeout compare "rotate, empty lock timeout" rotate "${work}/logs" --compress-after 5 --lock-timeout= diff --git a/tests/test_eb_keep_logs.sh b/tests/test_eb_keep_logs.sh index d82a546..88c5614 100755 --- a/tests/test_eb_keep_logs.sh +++ b/tests/test_eb_keep_logs.sh @@ -293,6 +293,19 @@ echo other > "${root}/notarun/E.log" EB_KEEP_LOGS_NOW=$(date -d 2026-11-13 +%s) export EB_KEEP_LOGS_NOW +check "dry run: lists files and runs without changing them" \ + "$("${tool}" rotate "${root}" --compress-after 30 --delete-after 90 --dry-run)" \ + "delete ${root}/20260801_090000 +compress $(realpath "${root}/20261001_120000/site-zen4")/A.stdout.log +compress $(realpath "${root}/20261001_120000/site-zen4")/B.failed.ebout.log +eb-keep-logs rotate dry-run: 2 files would be compressed, 1 run would be deleted under ${root}" +check "dry run: files and runs stay in place" \ + "$(find "${root}" -name '*.gz' -o -name 'D.stdout.log' | LC_ALL=C sort | tr '\n' ' ')" \ + "${root}/20260801_090000/D.stdout.log " +lock_missing=1 +test -e "${root}/.eb-keep-logs-rotate.lock" && lock_missing=0 +check "dry run: creates no lock" "$lock_missing" "1" + "${tool}" rotate "${root}" --compress-after 30 >/dev/null check "old run compressed" "$(ls "${root}/20261001_120000/site-zen4" | sort | tr '\n' ' ')" "A.stdout.log.gz B.failed.ebout.log.gz " check "compressed content intact" "$(zcat "${root}/20261001_120000/site-zen4/A.stdout.log.gz")" "old" diff --git a/tests/test_metrics.sh b/tests/test_metrics.sh index 63bf3d7..c07bce4 100755 --- a/tests/test_metrics.sh +++ b/tests/test_metrics.sh @@ -128,6 +128,27 @@ overflow=$(PATH="${work}/bin:${PATH}" "${tool}" failed --metrics-file "${metrics case "${overflow}" in *"counter overflow"*|*"Overflow"*) ;; *) exit 1 ;; esac [ "$(sha256sum "${metrics}")" = "${before}" ] +# A dry-run rotation writes no metrics, creates no sidecar lock, and leaves +# an existing exposition untouched even when --metrics-file is given. +dry_root="${work}/dry-root" +mkdir -p "${dry_root}/20261001_120000" +printf 'old\n' > "${dry_root}/20261001_120000/Old.stdout.log" +dry_new="${work}/collector/dry-new.prom" +dry_out=$(EB_KEEP_LOGS_NOW=$(date -d 2026-11-13 +%s) "${tool}" rotate "${dry_root}" --compress-after 5 --dry-run --metrics-file "${dry_new}") || exit 1 +case "${dry_out}" in *"eb-keep-logs rotate dry-run: 1 file would be compressed, 0 runs would be deleted under ${dry_root}"*) ;; *) exit 1 ;; esac +[ ! -e "${dry_new}" ] || exit 1 +[ ! -e "${work}/collector/.dry-new.prom.lock" ] || exit 1 +[ ! -e "${dry_root}/.eb-keep-logs-rotate.lock" ] || exit 1 +[ -f "${dry_root}/20261001_120000/Old.stdout.log" ] || exit 1 + +dry_existing="${work}/collector/dry-existing.prom" +cp "${metrics}" "${dry_existing}" +before=$(sha256sum "${dry_existing}") +EB_KEEP_LOGS_NOW=$(date -d 2026-11-13 +%s) "${tool}" rotate "${dry_root}" --compress-after 5 --dry-run --metrics-file="${dry_existing}" >/dev/null || exit 1 +[ "$(sha256sum "${dry_existing}")" = "${before}" ] || exit 1 +[ ! -e "${work}/collector/.dry-existing.prom.lock" ] || exit 1 +[ ! -e "${dry_root}/.eb-keep-logs-rotate.lock" ] || exit 1 + # No temporary output remains after any successful replacement. [ "$(find "${work}/collector" -maxdepth 1 -name '*.tmp' -print | wc -l)" -eq 0 ] echo "metrics tests passed"