eb-staged-rebuild rebuilds an existing EasyBuild installation in a versioned generation directory, then exchanges that complete tree with the stable software/<name>/<version> directory and publishes the module file with a same-directory replace. The previous tree stays on disk as software/<name>/.<version>.<run-id>.
EasyBuild --rebuild and --force empty an existing install before the new install has passed its sanity checks. This program sends EasyBuild at <state-root>/generations/<run-id> and exchanges the live directory only after that generation is complete.
On EESSI's CernVM-FS repositories you do not need it: the publish transaction is the atomic swap. EESSI's rebuild hook (eb_hooks.py, pre_fetch_hook_remove_install_dir_for_rebuilds) warns that a rebuild over a live install can pick up its old files (easybuild-framework#5362), and removes the old install early only when EESSI_FORCE_EARLY_REMOVAL_OF_INSTALL_DIR is set, which EESSI's install script does for /cvmfs targets.
It is for a stack on NFS, Lustre or GPFS that jobs read while it is rebuilt, where there is no transaction.
EasyBuild's own --bwrap builds the new tree at a temporary path with the real path bound over it, and writes bwrap_info.json there; moving the tree into place is left to the caller. With --backend bwrap, the first one auto tries, this program runs that build and does the move: it exchanges the finished tree with the live one with renameat2(RENAME_EXCHANGE), or with three ordinary renames that restore both names on failure where the filesystem refuses rename flags, publishes the module file with a same-directory replace, and keeps the previous tree for jobs that still have it open. installpath and userns are for hosts without bubblewrap and rewrite the generation path in each file before the exchange.
Documentation: https://stackhands.github.io/eb-staged-rebuild/, built from the Org pages in docs/orgmode/: a tutorial, how-to guides, explanation, and a reference, including the API generated from the Python and Zig code.
pixi run -e docs docbldThe HTML is written to docs/build/html. docs/orgmode/contributing.org says how to write a page.
Release bundles now include bash and zsh completion scripts with a man page produced from _parser, using pinned shtab and argparse-manpage in CI. Fish output is excluded because the generator toolchain does not provide fish completion support.
Linux only. There are two builds of the same command-line tool; their outputs agree byte for byte except where COMPAT.md lists a deliberate difference.
The static binary, one file per architecture, from the latest release (v1.0.0):
v=1.0.0 arch=x86_64 # or aarch64
curl -fsSLO "https://github.com/stackhands/eb-staged-rebuild/releases/download/v${v}/eb-staged-rebuild-${v}-${arch}-linux-musl"
curl -fsSLO "https://github.com/stackhands/eb-staged-rebuild/releases/download/v${v}/SHA256SUMS"
sha256sum --check --ignore-missing SHA256SUMS
install -m 0755 "eb-staged-rebuild-${v}-${arch}-linux-musl" ~/.local/bin/eb-staged-rebuild
eb-staged-rebuild --versiongh attestation verify FILE --repo stackhands/eb-staged-rebuild checks the build provenance. To build it yourself, Zig 0.17.1: zig build -Drelease=true.
The Python package, Python 3.11 or newer, no third-party dependencies:
python3 -m pip install .
eb-staged-rebuild --versionRebuild Example/1.0-GCCcore-15.2.0, which EasyBuild installed under /opt/site, from the directory that holds its easyconfig, and look at the run:
eb-staged-rebuild probe --path /opt/site/software
eb-staged-rebuild run --software /opt/site/software --modules /opt/site/modules \
--run-id example-1 -- Example-1.0-GCCcore-15.2.0.eb
eb-staged-rebuild status --state-root /opt/site/.eb-staged-rebuild example-1probe reports the sandboxes the machine has and whether the filesystem exchanges two directories in one call. run builds into /opt/site/.eb-staged-rebuild/generations/example-1, in the default state root for that software tree, and swaps the new tree in for software/Example/1.0-GCCcore-15.2.0, then its module file. Arguments after -- go to eb. rollback with the run id swaps the previous tree back. The how-to guide covers the dry run and the output of each step, and the guides after it cover failed runs, rollback and gc.
Sites that keep a filesystem reserve can add --min-free-bytes BYTES to run; without it, the command does not check free space.
A change adds a news fragment, changelog.d/+<name>.<type>.md, with type feature, bugfix, doc, removal or misc. towncrier build --draft --version X.Y.Z prints the release notes the fragments make. A release writes them into CHANGELOG.md, as RELEASING.md describes.
MIT, see LICENSE. Author: Rohit Goswami <rgoswami@ieee.org>.
eb-staged-rebuild is one of the stackhands tools for building, installing and maintaining an HPC software stack with EasyBuild on top of EESSI.
| Tool | Job | Docs |
|---|---|---|
| eb-stack | Turn conda-forge, Spack, PyPI, CRAN and Cargo recipes into easyconfigs; solve, shard and plan an EasyBuild build list | docs |
| eb-staged-rebuild | Rebuild an installation beside the live one and swap it in, for a stack outside CernVM-FS that jobs read during the rebuild | docs |
| eessi-arch-guard | Reload EESSI in a Slurm task that inherited a stack loaded for another CPU or GPU, as srun from a login shell does | docs |
| eb-local | Build on EESSI-extend inside a Slurm job's node-local disk, compile your own code with RPATH, list and uninstall per CPU target | docs |
| eb-keep-logs | Keep a failed EasyBuild command's own output, which --tmp-logdir does not; rotate a pipeline's log tree | docs |
| eb-pipeline | Drive a site's EasyBuild builds from one site file, through eb-stack, eb-staged-rebuild and eb-keep-logs | docs |
| rfc9162 | Append-only Merkle logs hashed as RFC 9162, with tiled storage and signed checkpoints | docs |
| zig-common | The shared Zig library under the Zig tools: system calls, checked files, child processes | docs |