Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion COMPAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on the same invocations.

| | |
|---|---|
| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS]`, `-h`, `--help`, `--version` |
| commands | `failed STATUS NAME LOGDIR [TMP_LOGDIR]`, `rotate ROOT --compress-after DAYS [--delete-after DAYS] [--dry-run] [--lock-timeout SECONDS]`, `-h`, `--help`, `--version` |
| usage | the same text on stdout for `-h`, `--help` and no arguments |
| exit codes | 0 on success and on every `failed` that got past its arguments; 2 for a refused invocation, after `eb-keep-logs: MESSAGE` on stderr; 2 with the usage text on stdout for no arguments; 75 when another `rotate` holds ROOT's lock past `--lock-timeout`, after `eb-keep-logs: another rotate holds the lock ROOT/.eb-keep-logs-rotate.lock; gave up after N s (--lock-timeout)` on stderr |
| one rotate at a time | `rotate` holds an exclusive flock(2) on `ROOT/.eb-keep-logs-rotate.lock`, created when missing, from before it reads ROOT until it exits; it waits up to `--lock-timeout` seconds for it, 600 without the option or with an empty value, 0 not at all, 999999999 at most; it opens the file for reading when it may not write it; a lock file that cannot be opened, or a symbolic link or non-regular file at its name, is refused, exit 2; on a file system that cannot lock, `rotate` says so on stderr and rotates without the lock |
Expand All @@ -20,6 +20,7 @@ on the same invocations.
| run ages | days from the local midnight of the date in the run's name, as `date -d YYYYMMDD` computes it, including TZ rules, except for the zones under unintended differences; a run whose local midnight does not exist is skipped |
| the run in progress | `rotate` neither compresses nor deletes the run that holds its stdout or stderr, when either is a regular file, or `EB_PIPELINE_LOG_DIR` |
| runs still written into | after taking its lock, `rotate` reads the links `/proc/PID/fd/N` of every process whose descriptors it may read; a run that holds the path a link with its owner's write bit names, also when the path ends in ` (deleted)`, is neither compressed nor deleted; without `/proc`, no run is left alone for this |
| rotation preview | `rotate --dry-run` prints each `compress PATH` and `delete PATH` action followed by the counts, changes nothing, creates no lock file, emits no syslog or journal event and writes no metrics; because it takes no lock, the output is a snapshot |

## Deliberate differences

Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,12 +91,12 @@ Rotations of one tree take turns: each holds an exclusive flock on `ROOT/.eb-kee
One that starts while another runs waits for it, up to `--lock-timeout` seconds (600; `0` does not wait), and when the wait runs out it changes nothing and exits 75.
`eb-pipeline`'s `ROOT/.eb-pipeline-rotate.lock` is a separate lock, which `rotate` never takes; a daily job that should wait for eb-pipeline's runs takes that lock around `rotate` with `flock`.

`EB_KEEP_LOGS_NOW`, in seconds since the epoch, sets the time `rotate` dates runs against.
On a copy of the tree, it previews what a later rotation compresses and removes:
`--dry-run` prints each file that would be compressed and run that would be deleted, but changes nothing, creates no lock and emits no syslog or journal event.
`EB_KEEP_LOGS_NOW`, in seconds since the epoch, sets the time `rotate` dates runs against, so the two together preview a later rotation:

```bash
cp -a /srv/build-logs /tmp/logs-preview
EB_KEEP_LOGS_NOW=$(date -d 2027-09-20 +%s) eb-keep-logs rotate /tmp/logs-preview --compress-after 30 --delete-after 365
EB_KEEP_LOGS_NOW=$(date -d 2027-09-20 +%s) \
eb-keep-logs rotate /srv/build-logs --compress-after 30 --delete-after 365 --dry-run
```

## Find why a build failed
Expand Down
1 change: 1 addition & 0 deletions changelog.d/+rotate-dry-run.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`eb-keep-logs rotate --dry-run` lists the log files it would compress and run directories it would delete without changing the tree, creating its lock file, emitting an event, or writing metrics.
16 changes: 9 additions & 7 deletions docs/orgmode/howto/rotate-a-log-tree.org
Original file line number Diff line number Diff line change
Expand Up @@ -18,20 +18,23 @@ Use this to keep a directory of build runs from growing without end: compress th
A run's age is the number of whole days from the local midnight of the date in its name.
=--compress-after DAYS= compresses the =*.log= and =*.txt= files of runs at least =DAYS= old, at any depth.
=--delete-after DAYS= removes runs at least =DAYS= old, and must be larger; without it, no run is removed.
2. Try the ages on a copy of the tree, at a later date.
=EB_KEEP_LOGS_NOW= sets the time =rotate= takes as now, in seconds since the epoch:
2. Preview the rotation.
=--dry-run= lists every file it would compress and run it would delete, but changes nothing, creates no lock file, emits no syslog or journal event, and writes no metrics.
=EB_KEEP_LOGS_NOW= sets the time =rotate= takes as now, in seconds since the epoch, so this also previews the policy at a later date:

#+begin_src bash
cp -a ~/build-logs /tmp/build-logs-preview
EB_KEEP_LOGS_NOW=$(date -d 2027-01-01 +%s) \
eb-keep-logs rotate /tmp/build-logs-preview --compress-after 30 --delete-after 365
eb-keep-logs rotate ~/build-logs --compress-after 30 --delete-after 365 --dry-run
#+end_src

#+begin_example
eb-keep-logs rotate: 5 files compressed, 0 runs deleted under /tmp/build-logs-preview
compress /home/alice/build-logs/20261116_175300/Example-1.0.failed.ebout.log
compress /home/alice/build-logs/20261116_175300/Example-1.0.stdout.log
eb-keep-logs rotate dry-run: 2 files would be compressed, 0 runs would be deleted under /home/alice/build-logs
#+end_example

On 2027-01-01 the run is 46 days old: at least 30, less than 365.
The preview is a snapshot: it deliberately takes no rotation lock, so preview again when the tree may have changed before the real rotation.
3. Rotate the tree:

#+begin_src bash
Expand Down Expand Up @@ -66,7 +69,7 @@ The summary line counts the files =rotate= compressed and the runs it removed.
In a compressed run, each log is a =.gz= file with the log's mode:

#+begin_src bash
ls -l /tmp/build-logs-preview/*/
ls -l ~/build-logs/*/
#+end_src

#+begin_example
Expand All @@ -83,7 +86,6 @@ drwxrwxr-x 3 alice alice 4096 Nov 16 17:53 easybuild-logs
|--------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------|
| =eb-keep-logs: --compress-after needs a number of days= | The option is missing, or its value is not a whole number, like =30d=. | Give a number of days: =--compress-after 30=. |
| =eb-keep-logs: --delete-after must be later than --compress-after= | =--delete-after= is not larger than =--compress-after=. | Raise =--delete-after=, or leave it out to remove nothing. |
| =eb-keep-logs: unknown option --dry-run= | =rotate= has no dry run. | Try the ages on a copy, as in step 2. |
| =eb-keep-logs: ROOT PATH is not a directory= | The path is wrong, or names a run, not the tree. | Give the directory that holds the runs. |
| =eb-keep-logs: cannot compress PATH: TooManyLinks= | The log has other hard links. The shell script prints =gzip='s own message. | Nothing: like =gzip=, =rotate= leaves it as it is. |
| =eb-keep-logs: cannot compress PATH: Exists= | =PATH.gz= exists already. The shell script prints =gzip='s own message. | Remove or rename one of the two, and rotate again. |
Expand Down
24 changes: 17 additions & 7 deletions docs/orgmode/reference/cli.org
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ This page describes the Zig build; the shell script =eb-keep-logs= in the reposi

#+begin_example
eb-keep-logs failed [--syslog TARGET] [--metrics-file PATH] STATUS NAME LOGDIR [TMP_LOGDIR]
eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS] [--syslog TARGET] [--metrics-file PATH]
eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--dry-run] [--lock-timeout SECONDS] [--syslog TARGET] [--metrics-file PATH]
eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json]
eb-keep-logs history PACKAGE [--target TARGET] [--root ROOT] [--json]
eb-keep-logs loads --module NAME [--json] FILE
Expand Down Expand Up @@ -43,9 +43,10 @@ eb-keep-logs --version
|--------------------------------------------------+----------+----------------------------------------------------------------------------------------------------------------------------------------------|
| =--compress-after DAYS=, =--compress-after=DAYS= | =rotate= | Required. Compress the =*.log= and =*.txt= files, at any depth, of runs at least =DAYS= old. =DAYS= is decimal digits. |
| =--delete-after DAYS=, =--delete-after=DAYS= | =rotate= | Remove whole runs at least =DAYS= old. =DAYS= must be larger than =--compress-after=. Without it, or with an empty value, no run is removed. |
| =--dry-run= | =rotate= | Print =compress PATH= for each log file that would be compressed and =delete PATH= for each run that would be removed, followed by a summary; change nothing, create no lock file, emit no syslog or journal event, and write no metrics. |
| =--lock-timeout SECONDS=, =--lock-timeout=SECONDS= | =rotate= | How long to wait for the lock another =rotate= of =ROOT= holds, in seconds, decimal digits; =0= does not wait. Without it, or with an empty value, 600. A value larger than 999999999, which is over 31 years, waits 999999999 seconds. |
| =--syslog TARGET=, =--syslog=TARGET= | =failed=, =rotate= | Emit an RFC 5424 or native =systemd-journald= event to =TARGET= (=journald=, =syslog=, =unix:///path=, =journal+unix:///path=, =udp://host:port=, or =tcp://host:port=); overrides =EB_KEEP_LOGS_SYSLOG=. An empty value disables emission. |
| =--metrics-file PATH=, =--metrics-file=PATH= | =failed=, =rotate= | Atomically update the Prometheus text exposition at =PATH=. Without it, no metrics file is read or written. |
| =--metrics-file PATH=, =--metrics-file=PATH= | =failed=, =rotate= | Atomically update the Prometheus text exposition at =PATH=, except on =rotate --dry-run=, which writes no metrics. Without it, no metrics file is read or written. |
| =--latest= | =why= | Read the newest run under =--root=, by its name; with =--target=, the newest that has that target. The default without =RUN=. |
| =--target TARGET=, =--target=TARGET= | =why=, =history= | Read only the builds in the run's directory =TARGET=; for =history=, in each run's, a run without one passed over. Without it, those in the run's own directory and then in each directory in it, by name. |
| =--root ROOT=, =--root=ROOT= | =why=, =history= | The directory that holds the runs, directories named =YYYYMMDD_HHMMSS=, not symbolic links. Without it, the current directory. |
Expand Down Expand Up @@ -74,6 +75,7 @@ eb-keep-logs --version
| =--version= | | Print the version line on stdout and exit 0. |

- A run's age is the number of whole days from the local midnight of the date in its name; [[file:../explanation/local-time.org][Run ages and local time]] gives the details.
- =--dry-run= reads the tree and the processes writing into it without taking the rotation lock, so its output is a snapshot, not a reservation. It also skips syslog emission and =--metrics-file= updates.
- =rotate='s options and =ROOT= come in any order; the value of an option is the next argument, whatever it is.
An option given twice takes its last value.
- =why='s options and =RUN=, and =history='s and =PACKAGE=, come in any order, and an option given twice takes its last value; the value of an option is the next argument, whatever it is, and an empty one is refused.
Expand All @@ -97,7 +99,7 @@ eb-keep-logs --version

| Status | Meaning |
|--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0 | =-h=, =--help= or =--version=, a =failed=, =why= or =history= that accepted its arguments, or a =rotate= that accepted its arguments and did not give up waiting for its lock, whatever it could keep, compress, remove or read: what it could not do is reported, and is no failure. A =why= exits 0 whether or not the run's builds failed, and a =history= whether or not it found a build of the package. |
| 0 | =-h=, =--help= or =--version=, a =failed=, =why= or =history= that accepted its arguments, or a =rotate= that accepted its arguments and did not give up waiting for its lock, whatever it could keep, compress, remove, preview or read: what it could not do is reported, and is no failure. A =why= exits 0 whether or not the run's builds failed, and a =history= whether or not it found a build of the package. |
| 2 | The invocation was refused, with =eb-keep-logs: MESSAGE= on stderr; or no command was given, and the usage is on stdout. |
| 75 | Another =rotate= held =ROOT='s lock for longer than =--lock-timeout=, and this =rotate= changed nothing; try again later. |

Expand Down Expand Up @@ -191,6 +193,8 @@ What =failed= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe
| =ROOT/RUN= | Removed with everything in it, when old enough for =--delete-after=, without being compressed first. A symbolic link in it is removed, not followed. Removal stops at the first error. |
| =/proc/PID/fd/N= | Read, for every process whose descriptors =rotate= may read, after it has the lock: a link with its owner's write bit names a file the process has open for writing, and the run that file is in is left alone. Without =/proc=, no run is left alone for it. |

With =--dry-run= these paths are only read; =ROOT/.eb-keep-logs-rotate.lock= is neither opened nor created, and =--metrics-file= neither reads nor writes any metrics or lock file.

A run that holds the file =rotate='s standard output or standard error goes to, or =EB_PIPELINE_LOG_DIR=, or a file another process has open for writing, as a build that is still going has its log, is left alone, neither compressed nor deleted; a later =rotate= takes it.
The path is what counts: the path the link in =/proc= names, or for a file removed since, that path with =" (deleted)"= after it, has to start with =ROOT='s, with symbolic links resolved.
On stdout =rotate= prints one line, with =ROOT= as given:
Expand All @@ -217,6 +221,7 @@ What =rotate= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe
*** Metrics

With =--metrics-file PATH=, =failed= and =rotate= update one Prometheus text exposition file.
When =--dry-run= is given to =rotate=, it writes no metrics and leaves any existing metrics file and its sidecar lock untouched.
The file is written to a temporary file in =PATH='s directory, synced, and renamed over =PATH= while a sidecar lock is held.
An existing valid file supplies the counters and timestamp that the other command owns.
An invalid file is left unchanged and reported on stderr.
Expand All @@ -225,7 +230,7 @@ An invalid file is left unchanged and reported on stderr.
|--------+------+-------|
| =eb_keep_logs_logs_kept_total= | counter | Successful log-copy operations by =failed=, including a retry that copies the same destination again. |
| =eb_keep_logs_bytes_kept_total= | counter | Bytes in those successfully written copies. |
| =eb_keep_logs_last_prune_timestamp_seconds= | gauge | Actual Unix timestamp when =rotate= most recently completed, including a rotation that found nothing to change. It is chosen under the metrics lock, never moves backwards when rotations complete out of order, is independent of =EB_KEEP_LOGS_NOW=, and is zero until one completes. |
| =eb_keep_logs_last_prune_timestamp_seconds= | gauge | Actual Unix timestamp when =rotate= most recently completed (excluding =--dry-run=), including a rotation that found nothing to change. It is chosen under the metrics lock, never moves backwards when rotations complete out of order, is independent of =EB_KEEP_LOGS_NOW=, and is zero until one completes. |

Metrics are values this tool observes while keeping and rotating logs.
They do not describe dependency gaps, rebuild closures, failure classes, or other stack analysis.
Expand Down Expand Up @@ -540,8 +545,8 @@ otherwise lose, and keep the pipeline's log tree bounded.
Atomically update Prometheus counters for kept logs and bytes.
Default: none.
eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS]
[--lock-timeout SECONDS] [--syslog TARGET]
[--metrics-file PATH]
[--dry-run] [--lock-timeout SECONDS]
[--syslog TARGET] [--metrics-file PATH]
In the run directories directly under ROOT (named YYYYMMDD_HHMMSS),
gzip the *.log and *.txt files of old runs and remove older runs
whole, leaving alone a run that is still written into. A run's age is
Expand All @@ -555,6 +560,10 @@ otherwise lose, and keep the pipeline's log tree bounded.
Remove whole runs at least DAYS old; DAYS must be larger than
that of --compress-after. Default: none; no run is removed. An
empty value is the default.
--dry-run
Print each file that would be compressed and run that would be
deleted, but change nothing, create no lock, emit no event and
write no metrics.
--lock-timeout SECONDS, --lock-timeout=SECONDS
While another rotate holds the lock, wait up to SECONDS for it,
then give up and exit 75. Default: 600; 0 does not wait. An empty
Expand All @@ -564,7 +573,8 @@ otherwise lose, and keep the pipeline's log tree bounded.
compressed and deleted counts. Default: EB_KEEP_LOGS_SYSLOG,
else none.
--metrics-file PATH, --metrics-file=PATH
Atomically record when this rotation completed. Default: none.
Atomically record when this rotation completed, unless --dry-run
is given. Default: none.
eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json]
Say why the builds of a run failed: for each failed easyconfig, its
package and easyconfig, the EasyBuild step it failed in, EasyBuild's
Expand Down
Loading
Loading