Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions COMPAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ on the same invocations.
| `--version` | `eb-keep-logs 0.1.0` | `eb-keep-logs 0.2.1 (TARGET, MODE, zig VERSION, COMMIT)`; the first two words are unchanged |
| `why` | refused with `eb-keep-logs: why is in the Zig build only`, exit 2 | says why the run's builds failed, from their kept logs, read through gzip when rotated |
| `history` | refused with `eb-keep-logs: history is in the Zig build only`, exit 2 | lists how each kept build of a package went, run by run |
| `failures` | refused with `eb-keep-logs: failures is in the Zig build only`, exit 2 | lists failed builds across runs, filtered by recipe or date |
| `loads` | refused with `eb-keep-logs: loads is in the Zig build only`, exit 2 | prints who loaded a module, from Lmod's ModuleUsageTracking lines |
| `runs` | refused with `eb-keep-logs: runs is in the Zig build only`, exit 2 | prints who ran an executable, from XALT run records |
| `used` | refused with `eb-keep-logs: used is in the Zig build only`, exit 2 | lists paths under a prefix in XALT and Lmod records |
Expand Down
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,13 +105,15 @@ EB_KEEP_LOGS_NOW=$(date -d 2027-09-20 +%s) \
eb-keep-logs why --latest --root /srv/build-logs
eb-keep-logs why 20261117_040615 --root /srv/build-logs --target zen4 --json
eb-keep-logs history hello --root /srv/build-logs
eb-keep-logs failures --package hello --since 2026-11-17 --root /srv/build-logs
```

`why` reads the logs a run kept, as `failed` kept them or `rotate` compressed them, and prints for each failed easyconfig its package and easyconfig, the EasyBuild step it failed in, EasyBuild's message, and the line of its log that shows the error, the first compiler or linker error or else EasyBuild's own error line, with the path, the line number and three lines either side.
For a failure class, feed the kept log to eb-stack.
`history` lists each kept build of a package, run by run, with its outcome and how long EasyBuild took.
`--json` prints either as one line of JSON with `schema_version` 1, for a pipeline's run summary.
Both are in the Zig build only.
`history` lists each kept build of a package, run by run, with its outcome and how long EasyBuild took, and accepts `--since`, `--until` and `--failed`.
`failures` lists failed builds across runs with their failed EasyBuild step and duration, filtered by `--package`, `--target`, `--since` and `--until`; `why` accepts `--package`, `--since` and `--until` as well.
`--json` prints any of these as one line of JSON with `schema_version` 1, for a pipeline's run summary.
All three are in the Zig build only.

## Emit syslog and journald events

Expand Down
6 changes: 6 additions & 0 deletions build.zig
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,11 @@ pub fn build(b: *std.Build) void {
stable_output_test.setCwd(b.path("."));
stable_output_test.expectExitCode(0);
stable_output_test.has_side_effects = true;
const failures_test = b.addSystemCommand(&.{ "bash", "-c", "TOOL_BIN=\"$0\" exec bash tests/test_failures.sh" });
failures_test.addArtifactArg(exe);
failures_test.setCwd(b.path("."));
failures_test.expectExitCode(0);
failures_test.has_side_effects = true;
const metrics_test = b.addSystemCommand(&.{ "bash", "-c", "TOOL_BIN=\"$0\" exec bash tests/test_metrics.sh" });
metrics_test.addArtifactArg(exe);
metrics_test.setCwd(b.path("."));
Expand All @@ -133,5 +138,6 @@ pub fn build(b: *std.Build) void {
test_step.dependOn(&test_bundle_test.step);
test_step.dependOn(&lifecycle_test.step);
test_step.dependOn(&stable_output_test.step);
test_step.dependOn(&failures_test.step);
test_step.dependOn(&metrics_test.step);
}
1 change: 1 addition & 0 deletions changelog.d/+filter-failed-runs.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`eb-keep-logs failures` and opt-in `--package`, `--since`, `--until` and `--failed` filters on `why` and `history` filter failed builds across `YYYYMMDD_HHMMSS` runs by recipe or date, with schema-validated `--json` output and unchanged default command outputs.
24 changes: 23 additions & 1 deletion docs/orgmode/howto/find-why-a-build-failed.org
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,29 @@ Use this after a run in which builds failed, to see for each the EasyBuild step

=hello= takes every easyconfig whose name starts with =hello-=; =hello-2.12.1-GCCcore-13.3.0= takes that one alone.
A duration of =-= is one EasyBuild did not print, as when it stopped before building; ask =why= about that run for the reason.
5. To put the reasons in a script's own summary, read the JSON =--json= prints:
Pass =--since DATE=, =--until DATE= and =--failed= to limit =history= to failed builds in a date window.
5. List failed builds across runs, filtered by recipe or date:

#+begin_src bash
eb-keep-logs failures --root ~/build-logs --package hello --since 2026-11-17
#+end_src

#+begin_example
/home/alice/build-logs: 8 failed builds in 7 runs

RUN TARGET STEP DURATION NAME
20261117_035825 sapphirerapids fetch 48s hello-2.12.1-GCCcore-14.2.0
20261117_040032 sapphirerapids patch 41s hello-2.12.1-GCCcore-14.2.0
20261117_040119 sapphirerapids - - hello-2.12.1-GCCcore-14.2.0
20261117_040311 sapphirerapids configure 2m24s hello-2.12.1-GCCcore-14.2.0
20261117_040615 sapphirerapids build 3m06s hello-2.12.1-GCCcore-14.2.0
20261117_040928 sapphirerapids build 2m30s hello-2.12.1-GCCcore-14.2.0
20261117_041222 sapphirerapids sanitycheck 2m54s hello-2.12.1-GCCcore-13.3.0
20261117_041222 sapphirerapids test 2m33s hello-2.12.1-GCCcore-14.2.0
#+end_example

Leave out =--package= to list failures across every recipe in the date range, or pass =--package PACKAGE= to =why= to inspect one recipe within a run.
6. To put the reasons in a script's own summary, read the JSON =--json= prints:

#+begin_src bash
eb-keep-logs why 20261117_041222 --root ~/build-logs --json \
Expand Down
87 changes: 76 additions & 11 deletions docs/orgmode/reference/cli.org
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,17 @@

* The eb-keep-logs command

=eb-keep-logs= keeps the logs of a failed EasyBuild build, compresses and removes the old runs of a log tree, says from the kept logs why a run's builds failed, lists how a package's builds went run by run, reads who loaded a module from Lmod's usage log, reads who ran an executable from an XALT run record, lists paths under an install prefix from those records, classifies installed software by its recorded use, reads what an EasyBuild dry-run or missing-modules report still has to build, writes a flaky-test evidence bundle, and compares such bundles across sites.
This page describes the Zig build; the shell script =eb-keep-logs= in the repository behaves the same, except as [[file:compatibility.org][Compatibility]] lists: it has no =why=, =history=, =loads=, =runs=, =used=, =usage= or =pending=.
=eb-keep-logs= keeps the logs of a failed EasyBuild build, compresses and removes the old runs of a log tree, says from the kept logs why a run's builds failed, lists how a package's builds went run by run, lists failed builds across runs filtered by recipe or date, reads who loaded a module from Lmod's usage log, reads who ran an executable from an XALT run record, lists paths under an install prefix from those records, classifies installed software by its recorded use, reads what an EasyBuild dry-run or missing-modules report still has to build, writes a flaky-test evidence bundle, and compares such bundles across sites.
This page describes the Zig build; the shell script =eb-keep-logs= in the repository behaves the same, except as [[file:compatibility.org][Compatibility]] lists: it has no =why=, =history=, =failures=, =loads=, =runs=, =used=, =usage= or =pending=.

** Synopsis

#+begin_example
eb-keep-logs failed [--syslog TARGET] [--metrics-file PATH] STATUS NAME LOGDIR [TMP_LOGDIR]
eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--dry-run] [--lock-timeout SECONDS] [--syslog TARGET] [--metrics-file PATH]
eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json]
eb-keep-logs history PACKAGE [--target TARGET] [--root ROOT] [--json]
eb-keep-logs why [RUN | --latest] [--target TARGET] [--package PACKAGE] [--since DATE] [--until DATE] [--root ROOT] [--json]
eb-keep-logs history PACKAGE [--target TARGET] [--since DATE] [--until DATE] [--failed] [--root ROOT] [--json]
eb-keep-logs failures [--package PACKAGE] [--target TARGET] [--since DATE] [--until DATE] [--root ROOT] [--json]
eb-keep-logs loads --module NAME [--json] FILE
eb-keep-logs runs --exec PATH [--json] FILE
eb-keep-logs used --prefix PREFIX [--json] [--require-none] FILE...
Expand Down Expand Up @@ -48,9 +49,13 @@ eb-keep-logs --version
| =--syslog TARGET=, =--syslog=TARGET= | =failed=, =rotate= | Emit an RFC 5424 or native =systemd-journald= event to =TARGET= (=journald=, =syslog=, =unix:///path=, =journal+unix:///path=, =udp://host:port=, or =tcp://host:port=); overrides =EB_KEEP_LOGS_SYSLOG=. An empty value disables emission. |
| =--metrics-file PATH=, =--metrics-file=PATH= | =failed=, =rotate= | Atomically update the Prometheus text exposition at =PATH=, except on =rotate --dry-run=, which writes no metrics. Without it, no metrics file is read or written. |
| =--latest= | =why= | Read the newest run under =--root=, by its name; with =--target=, the newest that has that target. The default without =RUN=. |
| =--target TARGET=, =--target=TARGET= | =why=, =history= | Read only the builds in the run's directory =TARGET=; for =history=, in each run's, a run without one passed over. Without it, those in the run's own directory and then in each directory in it, by name. |
| =--root ROOT=, =--root=ROOT= | =why=, =history= | The directory that holds the runs, directories named =YYYYMMDD_HHMMSS=, not symbolic links. Without it, the current directory. |
| =--json= | =why=, =history=, =loads=, =runs=, =pending= | Print one line of JSON instead of the report; its keys are under [[file:json-schemas.org][JSON schemas]] and below. |
| =--target TARGET=, =--target=TARGET= | =why=, =history=, =failures= | Read only the builds in the run's directory =TARGET=; for =history= and =failures=, in each run's, a run without one passed over. Without it, those in the run's own directory and then in each directory in it, by name. |
| =--package PACKAGE=, =--package=PACKAGE= | =why=, =failures= | Read only builds whose easyconfig name matches =PACKAGE= (prefix up to a dash, or =NAME/VERSION=). Default: all builds. |
| =--since DATE=, =--since=DATE= | =why=, =history=, =failures= | Bound candidate runs to those at or after =DATE= (=YYYY-MM-DD=, =YYYYMMDD=, or =YYYYMMDD_HHMMSS=); for =why=, applies when picking =--latest=. |
| =--until DATE=, =--until=DATE= | =why=, =history=, =failures= | Bound candidate runs to those at or before =DATE= (=YYYY-MM-DD=, =YYYYMMDD=, or =YYYYMMDD_HHMMSS=); for =why=, applies when picking =--latest=. |
| =--failed= | =history= | Print only failed builds. |
| =--root ROOT=, =--root=ROOT= | =why=, =history=, =failures= | The directory that holds the runs, directories named =YYYYMMDD_HHMMSS=, not symbolic links. Without it, the current directory. |
| =--json= | =why=, =history=, =failures=, =loads=, =runs=, =pending= | Print one line of JSON instead of the report; its keys are under [[file:json-schemas.org][JSON schemas]] and below. |
| =--software DIR=, =--software=DIR= | =usage= | Required. Read installed software from the direct =Name/Version= directories below =DIR=. |
| =--modules DIR=, =--modules=DIR= | =usage= | Required. Count an Lmod record only when its module-file path is =DIR= or below it. |
| =--since EPOCH=, =--since=EPOCH= | =usage= | Count records at or after the non-negative epoch time; leave out records with no numeric time. |
Expand Down Expand Up @@ -105,7 +110,7 @@ eb-keep-logs --version

| =MESSAGE= | Cause |
|-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------|
| =unknown command 'WORD' (failed, rotate, why, history, loads, runs, used, usage, pending, test-bundle, test-compare, --help)= | The first argument is none of the commands. |
| =unknown command 'WORD' (failed, rotate, why, history, failures, loads, runs, used, usage, pending, test-bundle, test-compare, lifecycle, --help)= | The first argument is none of the commands. |
| =failed needs STATUS NAME LOGDIR [TMP_LOGDIR]= | =failed= got fewer than three arguments. |
| =STATUS must be a number, not 'STATUS'= | =STATUS= is empty or holds something other than digits. |
| =LOGDIR LOGDIR is not a directory= | =LOGDIR= does not name a directory that can be opened. |
Expand Down Expand Up @@ -347,6 +352,22 @@ With =--json=, =history= prints one line of JSON instead, its keys in this order

What =history= cannot read it reports on stderr as =why= does, a run that cannot be opened with =cannot open PATH: ERROR= too, and goes on with the rest; a build whose output could not be read is still listed.

*** failures

=failures= only reads, scanning the runs under =ROOT= oldest first and filtering by =--since=, =--until=, =--target= and =--package=.
For each failed build it reads the kept session log (or =NAME.stdout.log= when no session log exists) to report the failed EasyBuild step and duration:

#+begin_example
tests/fixtures/build-logs: 8 failed builds in 7 runs

RUN TARGET STEP DURATION NAME
20261117_035825 sapphirerapids fetch 48s hello-2.12.1-GCCcore-14.2.0
20261117_040032 sapphirerapids patch 41s hello-2.12.1-GCCcore-14.2.0
#+end_example

With no matching failure, it prints =ROOT: 0 failed builds in 0 runs=.
With =--json=, =failures= prints one line of JSON with =schema_version= 1, =kind= ="failures"=, =root=, =package=, =target=, =since=, =until=, =counts= (={"runs": N, "failed": N}=), and =failures= (each item with =run=, =target=, =name=, =package=, =easyconfig=, =step=, =message=, =duration_s=, and =log=).

** loads

=loads --module NAME [--json] FILE= prints who loaded =NAME=, from Lmod's ModuleUsageTracking lines in =FILE=.
Expand Down Expand Up @@ -575,7 +596,8 @@ otherwise lose, and keep the pipeline's log tree bounded.
--metrics-file PATH, --metrics-file=PATH
Atomically record when this rotation completed, unless --dry-run
is given. Default: none.
eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json]
eb-keep-logs why [RUN | --latest] [--target TARGET] [--package PACKAGE]
[--since DATE] [--until DATE] [--root ROOT] [--json]
Say why the builds of a run failed: for each failed easyconfig, its
package and easyconfig, the EasyBuild step it failed in, EasyBuild's
message, and the line of its kept log that shows the error: the first
Expand All @@ -591,12 +613,24 @@ otherwise lose, and keep the pipeline's log tree bounded.
--target TARGET, --target=TARGET
Only the builds in the run's directory TARGET. Default: those in
the run's own directory and in every directory in it.
--package PACKAGE, --package=PACKAGE
Only builds matching PACKAGE (an easyconfig name without .eb,
a prefix up to a dash, or NAME/VERSION). Default: all builds.
--since DATE, --since=DATE
Only consider runs at or after DATE when picking --latest.
DATE is YYYY-MM-DD, YYYYMMDD, or YYYYMMDD_HHMMSS.
Default: none.
--until DATE, --until=DATE
Only consider runs at or before DATE when picking --latest.
DATE is YYYY-MM-DD, YYYYMMDD, or YYYYMMDD_HHMMSS.
Default: none.
--root ROOT, --root=ROOT
The directory that holds the runs. Default: the current
directory.
--json
Print one line of JSON instead, with schema_version 1.
eb-keep-logs history PACKAGE [--target TARGET] [--root ROOT] [--json]
eb-keep-logs history PACKAGE [--target TARGET] [--since DATE]
[--until DATE] [--failed] [--root ROOT] [--json]
Say how each kept build of PACKAGE went, oldest run first: success,
failed or unknown, and how long EasyBuild took, from its output.
PACKAGE is an easyconfig's name or its start up to a dash, as hello
Expand All @@ -606,6 +640,37 @@ otherwise lose, and keep the pipeline's log tree bounded.
--target TARGET, --target=TARGET
Only the builds in each run's directory TARGET. Default: those in
each run's own directory and in every directory in it.
--since DATE, --since=DATE
Only read runs at or after DATE (YYYY-MM-DD, YYYYMMDD, or
YYYYMMDD_HHMMSS). Default: none.
--until DATE, --until=DATE
Only read runs at or before DATE (YYYY-MM-DD, YYYYMMDD, or
YYYYMMDD_HHMMSS). Default: none.
--failed
Only list failed builds.
--root ROOT, --root=ROOT
The directory that holds the runs. Default: the current
directory.
--json
Print one line of JSON instead, with schema_version 1.
eb-keep-logs failures [--package PACKAGE] [--target TARGET]
[--since DATE] [--until DATE] [--root ROOT] [--json]
List failed builds across runs under --root, oldest run first.
Each row gives the run, target, failed step, duration and build
name. Zig build only. Options come in any order.
--package PACKAGE, --package=PACKAGE
Only failed builds matching PACKAGE (an easyconfig name
without .eb, a prefix up to a dash, or NAME/VERSION). Default:
all packages.
--target TARGET, --target=TARGET
Only the builds in each run's directory TARGET. Default: those
in each run's own directory and in every directory in it.
--since DATE, --since=DATE
Only read runs at or after DATE (YYYY-MM-DD, YYYYMMDD, or
YYYYMMDD_HHMMSS). Default: none.
--until DATE, --until=DATE
Only read runs at or before DATE (YYYY-MM-DD, YYYYMMDD, or
YYYYMMDD_HHMMSS). Default: none.
--root ROOT, --root=ROOT
The directory that holds the runs. Default: the current
directory.
Expand Down Expand Up @@ -803,7 +868,7 @@ Environment:
directory of zone files.

Exit status:
0 -h, --help or --version, or a failed, rotate, why, history, loads, runs, used, usage, pending, test-bundle, test-compare or lifecycle that
0 -h, --help or --version, or a failed, rotate, why, history, failures, loads, runs, used, usage, pending, test-bundle, test-compare or lifecycle that
accepted its arguments, whatever it could keep, compress, remove or
read: what it could not do is reported on stderr. used with
--require-none also exits 2 when any path was listed.
Expand Down
Loading
Loading