Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,15 @@ jobs:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- run: shellcheck -S warning eb-keep-logs tests/test_eb_keep_logs.sh tests/corpus/run.sh tests/corpus/make-fixtures.sh
- name: Install promtool 3.7.3
run: |
curl -fsSLO https://github.com/prometheus/prometheus/releases/download/v3.7.3/prometheus-3.7.3.linux-amd64.tar.gz
echo 'fc9e5da126817438cf2820d9a7206c75e3122802ba8d20add3a6219a59ca6913 prometheus-3.7.3.linux-amd64.tar.gz' | sha256sum -c -
tar -xzf prometheus-3.7.3.linux-amd64.tar.gz
sudo install -m 0755 prometheus-3.7.3.linux-amd64/promtool /usr/local/bin/promtool
- run: shellcheck -S warning eb-keep-logs tests/test_eb_keep_logs.sh tests/test_metrics.sh tests/corpus/run.sh tests/corpus/make-fixtures.sh
- run: bash tests/test_eb_keep_logs.sh
- run: bash tests/test_metrics.sh
- name: no site names in the tree
run: |
! git grep -nIiE 'snellius|snel-|surf\.nl|/gpfs/|/projects/0|rg\.terra|scratch-shared|scratch-node' -- . ':!.github/workflows/ci.yml'
Expand All @@ -34,8 +41,20 @@ jobs:
- uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2.2.1
with:
version: 0.17.1
- name: Install promtool 3.7.3
run: |
case '${{ runner.arch }}' in
X64) arch=amd64; sha=fc9e5da126817438cf2820d9a7206c75e3122802ba8d20add3a6219a59ca6913 ;;
ARM64) arch=arm64; sha=6af448fa4f3640eb9bd51ce71575689a1ef79843b1cf08ef1fcc7ff9a1928bfd ;;
*) echo 'unsupported runner architecture: ${{ runner.arch }}' >&2; exit 1 ;;
esac
archive="prometheus-3.7.3.linux-${arch}.tar.gz"
curl -fsSLO "https://github.com/prometheus/prometheus/releases/download/v3.7.3/${archive}"
echo "${sha} ${archive}" | sha256sum -c -
tar -xzf "${archive}"
sudo install -m 0755 "prometheus-3.7.3.linux-${arch}/promtool" /usr/local/bin/promtool
- run: zig fmt --check build.zig build.zig.zon src
- run: shellcheck -S warning tests/differential.sh tests/test_why.sh tests/test_json_schemas.sh
- run: shellcheck -S warning tests/differential.sh tests/test_metrics.sh tests/test_why.sh tests/test_json_schemas.sh
- run: pip install jsonschema
- run: zig build test --summary all
- run: zig build -Drelease=true -Dcommit=${{ github.sha }} && zig-out/bin/eb-keep-logs --version
12 changes: 12 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,18 @@ jobs:
- uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2.2.1
with:
version: 0.17.1
- name: Install promtool 3.7.3
run: |
case '${{ matrix.arch }}' in
x86_64) prom_arch=amd64; sha=fc9e5da126817438cf2820d9a7206c75e3122802ba8d20add3a6219a59ca6913 ;;
aarch64) prom_arch=arm64; sha=6af448fa4f3640eb9bd51ce71575689a1ef79843b1cf08ef1fcc7ff9a1928bfd ;;
esac
archive="prometheus-3.7.3.linux-${prom_arch}.tar.gz"
curl -fsSLO "https://github.com/prometheus/prometheus/releases/download/v3.7.3/${archive}"
echo "${sha} ${archive}" | sha256sum -c -
tar -xzf "${archive}"
sudo install -m 0755 "prometheus-3.7.3.linux-${prom_arch}/promtool" /usr/local/bin/promtool
- run: pip install jsonschema
- run: zig build test --summary all
- name: build
run: |
Expand Down
6 changes: 6 additions & 0 deletions build.zig
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,11 @@ pub fn build(b: *std.Build) void {
stable_output_test.setCwd(b.path("."));
stable_output_test.expectExitCode(0);
stable_output_test.has_side_effects = true;
const metrics_test = b.addSystemCommand(&.{ "bash", "-c", "TOOL_BIN=\"$0\" exec bash tests/test_metrics.sh" });
metrics_test.addArtifactArg(exe);
metrics_test.setCwd(b.path("."));
metrics_test.expectExitCode(0);
metrics_test.has_side_effects = true;

const test_step = b.step("test", "Run the unit tests, the command-line suites and the differential run");
test_step.dependOn(&b.addRunArtifact(unit).step);
Expand All @@ -128,4 +133,5 @@ pub fn build(b: *std.Build) void {
test_step.dependOn(&test_bundle_test.step);
test_step.dependOn(&lifecycle_test.step);
test_step.dependOn(&stable_output_test.step);
test_step.dependOn(&metrics_test.step);
}
1 change: 1 addition & 0 deletions changelog.d/+prometheus-metrics.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add opt-in atomic Prometheus textfile metrics for kept logs, kept bytes, and the last completed rotation.
3 changes: 2 additions & 1 deletion docs/orgmode/explanation/local-time.org
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ The Zig build has to give the same ages without =date=: it is one static binary
4. Where a change to daylight saving time skips midnight, the clock jumping from 00:00 to 01:00, that date's midnight does not exist.
=date -d= refuses such a date, and =rotate= skips the run, as the shell script did.

=EB_KEEP_LOGS_NOW= replaces now with a time of your choosing, in seconds since the epoch; nothing else changes.
=EB_KEEP_LOGS_NOW= replaces now for these age calculations with a time of your choosing, in seconds since the epoch.
It does not replace the actual completion time written to a metrics file.

** What was considered and turned down

Expand Down
79 changes: 79 additions & 0 deletions docs/orgmode/howto/export-prometheus-metrics.org
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
#+TITLE: Export metrics to Prometheus
#+OPTIONS: toc:nil todo:nil num:nil author:nil title:nil
#+STARTUP: content

* Export metrics to Prometheus

Use the node_exporter textfile collector when =eb-keep-logs= and node_exporter run on the same machine.
The metrics output is opt-in, so existing commands write no metrics file.

** Choose the collector directory

Start node_exporter with a textfile collector directory:

#+begin_src sh
node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
#+end_src

For a systemd service, add the option to the service's command line and restart the service.
As =root=, create the collector directory with the build account as its owner and node_exporter's group as its group:

#+begin_src sh
install -d -o build-account -g node-exporter -m 0750 \
/var/lib/node_exporter/textfile_collector
#+end_src

Replace =build-account= with the account that keeps and rotates the logs, and =node-exporter= with the group of the node_exporter service account.
The owner can create the sidecar lock and temporary file, rename the temporary file atomically, and remove stale temporary files.
The service group can traverse and read the directory.
Do not create an empty =.prom= file: the first command that uses =--metrics-file= creates a complete exposition with mode 0644.

** Record kept logs

Pass the same metrics path to each =failed= invocation:

#+begin_src sh
eb-keep-logs failed \
--metrics-file /var/lib/node_exporter/textfile_collector/eb-keep-logs.prom \
"${status}" "${easyconfig_name}" "${log_dir}" "${tmp_log_dir}"
#+end_src

Each successfully copied log increments =eb_keep_logs_logs_kept_total=.
Its written size increments =eb_keep_logs_bytes_kept_total=.
A retry that copies a log again is another keep operation and increments both counters again.

** Record the last rotation

Pass the path to =rotate= too:

#+begin_src sh
eb-keep-logs rotate /srv/build-logs \
--compress-after 30 --delete-after 90 \
--metrics-file /var/lib/node_exporter/textfile_collector/eb-keep-logs.prom
#+end_src

After rotation completes, =eb_keep_logs_last_prune_timestamp_seconds= is the actual completion Unix timestamp, even when =EB_KEEP_LOGS_NOW= set a different time for run-age calculations.
The time is chosen while holding the metrics lock, and an older completion never replaces a newer one when rotations finish concurrently.
The counters remain unchanged.
A rotation that gives up waiting for its lock does not update the timestamp.

The tool locks a sidecar file, writes a temporary file in the collector directory, and renames it over the metrics file.
The collector therefore reads either the previous complete exposition or the new one.

** Check the result

Validate the Prometheus text exposition:

#+begin_src sh
promtool check metrics < /var/lib/node_exporter/textfile_collector/eb-keep-logs.prom
#+end_src

Then inspect node_exporter's metrics endpoint:

#+begin_src sh
curl -fsS http://127.0.0.1:9100/metrics |
grep '^eb_keep_logs_'
#+end_src

All three names start with =eb_keep_logs_=.
When =--metrics-file= is omitted, none of them is written.
1 change: 1 addition & 0 deletions docs/orgmode/howto/index.org
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,5 @@ The title names the task.
rotate-a-log-tree
mark-licensed-logs
emit-syslog-events
export-prometheus-metrics
#+end_export
32 changes: 28 additions & 4 deletions docs/orgmode/reference/cli.org
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ This page describes the Zig build; the shell script =eb-keep-logs= in the reposi
** Synopsis

#+begin_example
eb-keep-logs failed [--syslog TARGET] STATUS NAME LOGDIR [TMP_LOGDIR]
eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS] [--syslog TARGET]
eb-keep-logs failed [--syslog TARGET] [--metrics-file PATH] STATUS NAME LOGDIR [TMP_LOGDIR]
eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS] [--lock-timeout SECONDS] [--syslog TARGET] [--metrics-file PATH]
eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json]
eb-keep-logs history PACKAGE [--target TARGET] [--root ROOT] [--json]
eb-keep-logs loads --module NAME [--json] FILE
Expand Down Expand Up @@ -45,6 +45,7 @@ eb-keep-logs --version
| =--delete-after DAYS=, =--delete-after=DAYS= | =rotate= | Remove whole runs at least =DAYS= old. =DAYS= must be larger than =--compress-after=. Without it, or with an empty value, no run is removed. |
| =--lock-timeout SECONDS=, =--lock-timeout=SECONDS= | =rotate= | How long to wait for the lock another =rotate= of =ROOT= holds, in seconds, decimal digits; =0= does not wait. Without it, or with an empty value, 600. A value larger than 999999999, which is over 31 years, waits 999999999 seconds. |
| =--syslog TARGET=, =--syslog=TARGET= | =failed=, =rotate= | Emit an RFC 5424 or native =systemd-journald= event to =TARGET= (=journald=, =syslog=, =unix:///path=, =journal+unix:///path=, =udp://host:port=, or =tcp://host:port=); overrides =EB_KEEP_LOGS_SYSLOG=. An empty value disables emission. |
| =--metrics-file PATH=, =--metrics-file=PATH= | =failed=, =rotate= | Atomically update the Prometheus text exposition at =PATH=. Without it, no metrics file is read or written. |
| =--latest= | =why= | Read the newest run under =--root=, by its name; with =--target=, the newest that has that target. The default without =RUN=. |
| =--target TARGET=, =--target=TARGET= | =why=, =history= | Read only the builds in the run's directory =TARGET=; for =history=, in each run's, a run without one passed over. Without it, those in the run's own directory and then in each directory in it, by name. |
| =--root ROOT=, =--root=ROOT= | =why=, =history= | The directory that holds the runs, directories named =YYYYMMDD_HHMMSS=, not symbolic links. Without it, the current directory. |
Expand Down Expand Up @@ -77,7 +78,7 @@ eb-keep-logs --version
An option given twice takes its last value.
- =why='s options and =RUN=, and =history='s and =PACKAGE=, come in any order, and an option given twice takes its last value; the value of an option is the next argument, whatever it is, and an empty one is refused.
- =-h=, =--help= and =--version= count only in the place of the command.
- =failed= takes no options.
- =failed='s options and positional arguments may appear in any order.

** Environment

Expand Down Expand Up @@ -213,6 +214,23 @@ What =rotate= cannot do it reports on stderr as =eb-keep-logs: MESSAGE=, and goe
| =cannot sync a .licensed-logs: ERROR= | A list was replaced, but its directory could not be synced; after a crash the old list may be back. |
| =cannot delete ROOT/RUN: ERROR= | The run could not be removed completely; what was removed stays removed. |

*** Metrics

With =--metrics-file PATH=, =failed= and =rotate= update one Prometheus text exposition file.
The file is written to a temporary file in =PATH='s directory, synced, and renamed over =PATH= while a sidecar lock is held.
An existing valid file supplies the counters and timestamp that the other command owns.
An invalid file is left unchanged and reported on stderr.

| Metric | Type | Value |
|--------+------+-------|
| =eb_keep_logs_logs_kept_total= | counter | Successful log-copy operations by =failed=, including a retry that copies the same destination again. |
| =eb_keep_logs_bytes_kept_total= | counter | Bytes in those successfully written copies. |
| =eb_keep_logs_last_prune_timestamp_seconds= | gauge | Actual Unix timestamp when =rotate= most recently completed, including a rotation that found nothing to change. It is chosen under the metrics lock, never moves backwards when rotations complete out of order, is independent of =EB_KEEP_LOGS_NOW=, and is zero until one completes. |

Metrics are values this tool observes while keeping and rotating logs.
They do not describe dependency gaps, rebuild closures, failure classes, or other stack analysis.
See [[file:../howto/export-prometheus-metrics.org][Export metrics to Prometheus]] for the node_exporter textfile collector setup.

*** why

=why= only reads.
Expand Down Expand Up @@ -507,7 +525,7 @@ The usage =-h=, =--help= and no arguments print on stdout, the same from the she
eb-keep-logs: keep the logs of EasyBuild builds that a build pipeline would
otherwise lose, and keep the pipeline's log tree bounded.

eb-keep-logs failed [--syslog TARGET] STATUS NAME LOGDIR [TMP_LOGDIR]
eb-keep-logs failed [--syslog TARGET] [--metrics-file PATH] STATUS NAME LOGDIR [TMP_LOGDIR]
After `eb` returned STATUS for easyconfig NAME: when STATUS is not 0,
copy the EasyBuild session log `eb --last-log` names, the failing
command's output and a meson testlog next to LOGDIR/NAME.stdout.log,
Expand All @@ -518,8 +536,12 @@ otherwise lose, and keep the pipeline's log tree bounded.
--syslog TARGET, --syslog=TARGET
Emit an RFC 5424 or native systemd-journald event when logs are
kept or none are found. Default: EB_KEEP_LOGS_SYSLOG, else none.
--metrics-file PATH, --metrics-file=PATH
Atomically update Prometheus counters for kept logs and bytes.
Default: none.
eb-keep-logs rotate ROOT --compress-after DAYS [--delete-after DAYS]
[--lock-timeout SECONDS] [--syslog TARGET]
[--metrics-file PATH]
In the run directories directly under ROOT (named YYYYMMDD_HHMMSS),
gzip the *.log and *.txt files of old runs and remove older runs
whole, leaving alone a run that is still written into. A run's age is
Expand All @@ -541,6 +563,8 @@ otherwise lose, and keep the pipeline's log tree bounded.
Emit an RFC 5424 or native systemd-journald event with the
compressed and deleted counts. Default: EB_KEEP_LOGS_SYSLOG,
else none.
--metrics-file PATH, --metrics-file=PATH
Atomically record when this rotation completed. Default: none.
eb-keep-logs why [RUN | --latest] [--target TARGET] [--root ROOT] [--json]
Say why the builds of a run failed: for each failed easyconfig, its
package and easyconfig, the EasyBuild step it failed in, EasyBuild's
Expand Down
Loading
Loading