Skip to content

About

Site-neutral EasyBuild pipeline driver: one TOML site file plus a build list becomes an ordered set of scheduler-launched phases

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

eb-pipeline

eb-pipeline

A site-neutral EasyBuild pipeline driver. One per-site Tom's Obvious Minimal Language (TOML) file describes targets, scheduler allocations, paths, admission, modules, caches, log handling and licences. eb-pipeline turns that file and a build list into an ordered set of subprocesses, so a continuous integration (CI) system (Jenkins, GitLab CI, a cron job) becomes a thin caller that passes a list and a target and reads an exit code and a result file.

It starts eb-stack for stack shard, stack admit and rebuild-set, eb-keep-logs for failed and rotate, and eb-staged-rebuild for run, and never links them in. Nothing about a site lives in this repository: the templates carry placeholders only, and a test rejects any template value that looks site-specific.

Documentation

https://stackhands.github.io/eb-pipeline/, built from the Org pages in docs/orgmode/:

pixi run -e docs docbld

The HTML is written to docs/build/html. Contributing says how to write a page, run the tests and record a change.

Install

Rust 1.90 or newer, in a checkout:

cargo install --locked --path . --root ~/.local
eb-pipeline --version

That install puts six commands in ~/.local/bin:

for cmd in eb-pipeline eb-retry-list eb-shard-weights eb-test-map eb-junit eb-cvmfs; do
  command -v "$cmd" && "$cmd" --help >/dev/null
done

Releases after 0.3.0 also ship all six commands (0.3.0 ships eb-pipeline only):

for cmd in eb-pipeline eb-retry-list eb-shard-weights eb-test-map eb-junit eb-cvmfs; do
  command -v "$cmd" && "$cmd" --help >/dev/null
done

Install eb-pipeline has the release binaries and what to do when the build fails.

Licence

MIT. See LICENSE.

Part of stackhands

eb-pipeline is one of the stackhands tools for building, installing and maintaining an HPC software stack with EasyBuild on top of EESSI.

Tool Job Docs
eb-stack Turn conda-forge, Spack, PyPI, CRAN and Cargo recipes into easyconfigs; solve, shard and plan an EasyBuild build list docs
eb-staged-rebuild Rebuild an installation beside the live one and swap it in, for a stack outside CernVM-FS that jobs read during the rebuild docs
eessi-arch-guard Reload EESSI in a Slurm task that inherited a stack loaded for another CPU or GPU, as srun from a login shell does docs
eb-local Build on EESSI-extend inside a Slurm job's node-local disk, compile your own code with RPATH, list and uninstall per CPU target docs
eb-keep-logs Keep a failed EasyBuild command's own output, which --tmp-logdir does not; rotate a pipeline's log tree docs
eb-pipeline Drive a site's EasyBuild builds from one site file, through eb-stack, eb-staged-rebuild and eb-keep-logs docs
rfc9162 Append-only Merkle logs hashed as RFC 9162, with tiled storage and signed checkpoints docs
zig-common The shared Zig library under the Zig tools: system calls, checked files, child processes docs

About

Site-neutral EasyBuild pipeline driver: one TOML site file plus a build list becomes an ordered set of scheduler-launched phases

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages