A site-neutral EasyBuild pipeline driver. One per-site Tom's Obvious Minimal
Language (TOML) file describes targets, scheduler allocations, paths,
admission, modules, caches, log handling and licences. eb-pipeline turns that
file and a build list into an ordered set of subprocesses, so a continuous
integration (CI) system (Jenkins, GitLab CI, a cron job) becomes a thin caller
that passes a list and a target and reads an exit code and a result file.
It starts eb-stack for stack shard,
stack admit and rebuild-set,
eb-keep-logs for failed and
rotate, and
eb-staged-rebuild for run,
and never links them in. Nothing about a site lives in this repository: the
templates carry placeholders only, and a test rejects any template value that
looks site-specific.
https://stackhands.github.io/eb-pipeline/, built from the Org pages in
docs/orgmode/:
- Tutorial: Run your first build list.
- How-to guides: install eb-pipeline, write a site file for your cluster, build a list on every target from continuous integration, build in a directory that each job has on its node, let users load what a target installs on top of EESSI, rotate the log tree, rebuild installed software through eb-staged-rebuild, build some lines of a list on some targets only, build what eb-stack plans, follow runs through their summaries, rebuild what an EESSI change reaches.
- Explanation: why one site file drives the build, how a run goes through its phases, why the rotation takes two locks, what the targets of a site file share.
- Reference: the eb-pipeline command, the site file, the plan and result files, the run summary, the watch state and plan, the log tree and its locks, the site file templates, environment variables, exit status, and the API of the crate, from rustdoc.
pixi run -e docs docbldThe HTML is written to docs/build/html.
Contributing says how to write a page, run
the tests and record a change.
Rust 1.90 or newer, in a checkout:
cargo install --locked --path . --root ~/.local
eb-pipeline --versionThat install puts six commands in ~/.local/bin:
for cmd in eb-pipeline eb-retry-list eb-shard-weights eb-test-map eb-junit eb-cvmfs; do
command -v "$cmd" && "$cmd" --help >/dev/null
doneReleases after 0.3.0 also ship all six commands (0.3.0 ships eb-pipeline only):
for cmd in eb-pipeline eb-retry-list eb-shard-weights eb-test-map eb-junit eb-cvmfs; do
command -v "$cmd" && "$cmd" --help >/dev/null
doneInstall eb-pipeline has the release binaries and what to do when the build fails.
MIT. See LICENSE.
eb-pipeline is one of the stackhands tools for building, installing and maintaining an HPC software stack with EasyBuild on top of EESSI.
| Tool | Job | Docs |
|---|---|---|
| eb-stack | Turn conda-forge, Spack, PyPI, CRAN and Cargo recipes into easyconfigs; solve, shard and plan an EasyBuild build list | docs |
| eb-staged-rebuild | Rebuild an installation beside the live one and swap it in, for a stack outside CernVM-FS that jobs read during the rebuild | docs |
| eessi-arch-guard | Reload EESSI in a Slurm task that inherited a stack loaded for another CPU or GPU, as srun from a login shell does | docs |
| eb-local | Build on EESSI-extend inside a Slurm job's node-local disk, compile your own code with RPATH, list and uninstall per CPU target | docs |
| eb-keep-logs | Keep a failed EasyBuild command's own output, which --tmp-logdir does not; rotate a pipeline's log tree | docs |
| eb-pipeline | Drive a site's EasyBuild builds from one site file, through eb-stack, eb-staged-rebuild and eb-keep-logs | docs |
| rfc9162 | Append-only Merkle logs hashed as RFC 9162, with tiled storage and signed checkpoints | docs |
| zig-common | The shared Zig library under the Zig tools: system calls, checked files, child processes | docs |