Improvements to snakemake containerization by cailmdaley · Pull Request #309 · CosmoStat/sp_validation

cailmdaley · 2026-08-24T22:45:38Z

This PR began by addressing Martin's snakemake run issues (#300-#304), and evolved into a redesign of how the container is used. CosmoSIS + the standard library have also been packaged into the image.

The container model

Snakemake wraps every job in the container itself. The candide profile sets software-deployment-method: apptainer plus the bind list; rules are plain shell:/script: with no hand-assembled apptainer exec anywhere. A few rules that call host toolchains (CosmoCov, ImageMagick) keep container: None with a docstring saying why.

Everyone runs their own image, managed by a small CLI. spv-container (new console script, stdlib-only) owns a per-user image at ~/.cache/sp_validation/:

spv-container pull — fetch the CI image to your canonical path (atomic swap), print its revision;
spv-container status — which image layer is live, its built-from revision vs your checkout HEAD (in-sync / behind / diverged);
spv-container exec <cmd> — one-off commands inside the image, standard binds;
spv-container sandbox + exec --writable — an opt-in writable unpack for mid-analysis pip install; pull + sandbox --force resets to clean. A failed rebuild never destroys a working sandbox.

Resolution order everywhere (workflow, drivers, CLI): --config container= override → sandbox → your SIF → the registry tag. CI publishes an image for every branch, so --config container=docker://ghcr.io/cosmostat/sp_validation:<branch> tests a branch's own stack.

The launched checkout's sp_validation is what runs. The image is the frozen dependency stack; common.configure() prepends your checkout's src/ to the container's PYTHONPATH (as image_sims always did), so script: files and import sp_validation come from the same commit. Opt out with --config checkout_pythonpath=false to reproduce from the image alone.

CosmoSIS and the standard library

CosmoSIS was an undeclared, user-supplied dependency of the inference step (#303's hand-patched ~/.local). Now: cosmosis>=3.25 is a real dependency built into the image (MPIFC pinned absolutely or the sampler Makefiles silently skip their MPI targets), and cosmosis-standard-library ships too, pinned to Sacha's fork at sachaguer/cosmosis-standard-library@b26fa7f — the version the current pipelines actually use (tau-statistics, sample_S8, z-dependent linear alignment). Built via cosmosis-configure && make -C shear (the only compiled modules our templates touch); the templates' hardcoded personal paths are replaced by the in-image %(CSL_DIR)s. Updating the fork toward upstream (373 commits behind) is future work and Sacha's call.

Run-report fixes

n23 mount fail #304 — /automnt prefixes dropped everywhere; those paths don't exist on the disk-owning node, so jobs died before writing a log.
snakemake version #302 — snakemake runs on the host as a pinned uv tool; nothing in the image can shadow it.
cs_util version #301 — no code change; the image rebuild against uv.lock is the fix.
Restored cosmocov_process.py (deleted in Clean up cosmo_inference folder #236; its bare exit() had recorded success on non-positive-definite matrices), fixed script: path resolution under module: composition, deleted the never-used unblinding ceremony, parked the never-runnable xi_highres with its MPI reasoning preserved.

Verified

test_container_smoke submits a real SLURM job through the committed profile and asserts APPTAINER_CONTAINER is set — the job demonstrably ran inside the image. Passing.
Sandbox mechanics exercised on candide: unprivileged --sandbox build, writable installs persisting into read-only job mounts, atomic --force rebuild.
Clean dry-run DAGs (bmodes, cosmo_val, image_sims, container_smoke); snakemake --lint clean; Planck18 reproduced through the profile.
The CSL image build is validated by this branch's CI run.

Closes #300
Closes #302
Closes #303
Closes #304

cailmdaley

I want to take another pass at this. for Cosmosis, I think the README shouldn't advise the version. We should just be pinning the version, no, in pyproject? I think as long as this doesn't break the... I suppose the problem is we've added custom Cosmosis modules. Anyway, I feel like we haven't quite gotten to the fix on number 303. We should just verify that if we're on Cosmosis 3.1.6.1, this is fixed.

I would rather not get into the switching everything to MPI as long as the SMP works for our case. Or, I mean, do we even need the SMP to work for our case? Anyway, I would like this number 303 to get pushed a little further. it's not sufficient to say "closed issue upstream."

cailmdaley · 2026-08-26T01:39:02Z

 )

-from snakemake.script import snakemake  # noqa: E402
+# `snakemake` is injected as a module global by Snakemake's `script:`


let's not replace a broken line with a four-line comment that doesn't make sense without the original line

cailmdaley · 2026-08-26T01:41:30Z

-  # Apptainer bind mounts.  /automnt is required when repos/data are
-  # automounted (candide gotcha); harmless otherwise.  [operational]
-  binds: /n17data,/n09data,/home,/automnt
+  # Bind mounts are no longer set per-run: each rule below carries a plain


same comment as above about comments..

cailmdaley · 2026-08-26T01:42:17Z

-# top-level container is needed here.
+# No single image covers this chain -- the ShapePipe stages (pipeline, merge)
+# and the sp_validation stages (manifest, extract, calibrate, m_bias) run in
+# two different images (config["image_sims"]["sif"] / ["sif_pipeline"]), so


i see no reason not to use the sp_validation container for both stages, since it's built on top of shapepipe

cailmdaley · 2026-08-26T01:43:47Z

-#     ``software-deployment-method: apptainer`` / ``apptainer-args`` -- those
-#     would wrap a *second*, redundant container around jobs that already run
-#     inside one.
+# Two explicit exceptions still set ``container: None`` and own an inline


why is the profile talking about specific exceptions? clean up these comments please! throughout!

cailmdaley · 2026-08-26T01:44:55Z

@@ -0,0 +1,19 @@
+# Container smoke test -- validates the base container contract every


we can test this, but maybe properly as a test, rather than in the main workflow?

cailmdaley · 2026-08-26T01:52:04Z

+
+    Pure host-side introspection (``git -C``, a plain-text scan of the SIFs'
+    OCI labels, PyYAML) -- no sp_validation/ShapePipe import, so it stays a
+    ``run:`` block with no container.  Snakemake never containerizes ``run:``


why are we working so hard to make this a run block? also, i worry this whole rule is a bit hacky

cailmdaley · 2026-08-26T01:52:22Z

        }

-        os.makedirs(os.path.dirname(output.results), exist_ok=True)
+        os.makedirs(params.results_dir, exist_ok=True)


doesn't snakelike create necessary directories?

cailmdaley · 2026-08-26T01:52:51Z

+    running the estimator against the config ``im_mbias_config`` assembled
+    (manifest's shear/branch facts, science knobs, provenance) -- the one
+    sp_validation-stage compute call in the chain, so it is the one place a
+    real ``container:``/``shell:`` split (rather than a ``run:`` block's


too much fixation on this in comments

cailmdaley · 2026-08-26T01:53:26Z

 from cv_runner import _unbuffer_streams, make_cv, touch_sentinels
-from snakemake.script import snakemake

+# `snakemake` is injected as a module global by Snakemake's `script:` preamble


same comment 15 times across the codebase??

cailmdaley · 2026-08-26T01:55:53Z

+# so it is correct no matter how a rule file that references it was reached --
+# standalone (`-s workflow/Snakefile`) or composed via `module:` from a paper
+# Snakefile (each paper Snakefile also derives its own path to workflow/ the
+# same way, from `workflow.basedir`, under the *different* name WORKFLOW_DIR


this is really needed? why doesn't shell: python scripts/script_a.py allow you to import scripts/script_b.py in script a? doesn't the script's directory get imported to python path?

Rules: drop the /automnt prefix from the hardcoded paths in xi_highres (twopoint.smk) and covariance_glass_mock (covariance.smk). /automnt/nXXdataN does not exist on the node that owns that disk, so a job landing there fails immediately, before any log is written. Every canonical path in common.py already uses the plain /nXXdataN form. Docs: workflow/README.md gains a note on the /automnt trap and one on host ~/.local shadowing the container's pinned Snakemake; cosmo_inference/README.md recommends CosmoSIS --mpi over the fragile upstream --smp process pool (cosmosis#170) and cosmosis >= 3.16.1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU

candide profile now owns the container: software-deployment-method: apptainer + apptainer-args carries the bind mounts (matching the app bash-function binds in the top-level UNIONS CLAUDE.md), replacing the old rationale for leaving containerization to each rule. Rewrites the profile's doc comment to describe the new model and its two documented exceptions (xi_highres MPI, covariance_cosmocov host toolchain). Adds a container_smoke rule (workflow/rules/container_smoke.smk + scripts/container_smoke.py) as a cheap end-to-end check of the profile-driven container path (editable sp_validation import, numpy + OMP_NUM_THREADS, git provenance) via `script:`, wired unconditionally into workflow/Snakefile. Reconciles image_sims/Snakefile's container: None comment: it now documents that the two-image (SIF/SIF_PIPELINE) chain is a per-rule container: choice in image_sims.smk, still wrapped by the profile's apptainer deployment -- not a rule-owned apptainer exec call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU

Strip every rule's explicit apptainer exec wrapper (_EXEC_PREFIX/EXEC/ EXEC_PIPELINE) from image_sims.smk. Each compute rule now carries a plain per-rule container: SIF / container: SIF_PIPELINE directive; Snakemake wraps the shell: command via the profile's software-deployment-method: apptainer + apptainer-args (set A). PYTHONPATH/PSF_DICT/OMP_NUM_THREADS injection and the SLURM_* env strip move from apptainer --env/-u flags to plain shell VAR=value / env -u syntax at the front of each shell: string (_ENV_PREFIX) -- identical effect, no apptainer-specific mechanism, works the same whether or not the command is container-wrapped. im_mbias split into im_mbias_config (run:, host-side git/provenance introspection + yaml write -- Snakemake never containerizes run: regardless of container:, so this must stay a driver-side step) and im_mbias (shell:, container: SIF, runs the actual m-bias compute). This was the one rule whose apptainer call lived inside a run: block's trailing shell() -- splitting it out is what makes container: apply to it at all. binds: dropped from image_sims config/schema -- it's now the profile's apptainer-args (one bind list for the whole workflow), not a per-run-config value. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU

… rules + docs) Completes the pivot for the rules outside image_sims: xi_highres and covariance_cosmocov keep their container: None + inline apptainer exec / host-toolchain call (multi-node MPI and a host-compiled binary respectively, both genuinely incompatible with Snakemake's own container wrapping), now documented in their rule docstrings as deliberate exceptions rather than leftovers. No other rule outside image_sims called apptainer directly. While touching these files, retired the stale /pure_eb/ absolute paths left from the old repo layout: added workflow.common.WORKFLOW_SCRIPTS (Path(__file__)-based, correct under both standalone and module-composed runs) for the handful of shell: rules that call a workflow script directly, and reused the existing COSMO_INFERENCE constant elsewhere. Removed the run_cosmo_val rule in twopoint.smk, dead since cosmo_val.smk decomposed it into per-diagnostic rules (its own docstring says so) and still pointing at a stale path plus a nonsensical host .local PYTHONPATH injection. Flagged (not fixed) covariance_process: it calls cosmo_inference/scripts/ cosmocov_process.py, deleted in the #236 cleanup and never restored, so the rule fails on the default covariance target -- pre-existing, unrelated to this pivot. Rewrote workflow/README.md and cosmo_inference/README.md to the new model: snakemake is a thin host-side tool pinned via `uv tool install snakemake snakemake-executor-plugin-slurm`, run directly on the host, never from inside an apptainer shell; the candide profile's software-deployment-method puts each job in the container instead. Added a short pointer from the top-level README's dev-shell instructions to workflow/README.md so the two don't get conflated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU

container_smoke failed for real (SLURM jobs 839818/839819) with ModuleNotFoundError: snakemake.iocontainers -- the container image had its own snakemake==9.16.3 pip-installed directly (leftover from the old apptainer-shell-then-snakemake-inside pattern this pivot retires), shadowing the host-mounted 9.23.1 orchestrator that script: bind-mounts in and sys.path.extends (appended, not prepended). Removed it and its snakemake-executor-plugin-slurm/-slurm-jobstep/-interface-* family from the image (verified Required-by: none outside the family itself). Separately, apptainer-args never actually isolated host tooling: the image's own /.singularity.d/env/50-bashrc.sh unconditionally sourced the host ~/.bashrc for every apptainer action, not just an interactive `apptainer shell` -- so a host dotfile (asdf init) ran on every exec too, pushing host PATH entries (~/.local/bin) ahead of the image's own /usr/local/bin. A bare `python` in any shell:/script: rule was silently running the host's interpreter, invisibly, surviving --cleanenv. Gated the bashrc sourcing on APPTAINER_COMMAND=shell (set by apptainer itself before these scripts run). Fixing both surfaced a third, previously-masked bug: every script: rule (19 files) imports `from snakemake.script import snakemake`, which is IDE-hint-only in this snakemake version -- snakemake.script exposes no such runtime attribute (only the Snakemake class), and the preamble that actually gets pickled in already provides `snakemake` as a plain global before the rest of the file executes. Removed the broken import repo-wide; the object resolves via normal global lookup exactly as before, including inside the functions/branches a few scripts defer it into. Verified end to end: container_smoke now completes for real through SLURM (jobid 839822, python 3.12.12, sp_validation editable install resolved, correct HEAD commit read from inside the job) with no apptainer exec left in any rule. Dry-run coverage for every rule that owns an edited script (masks_only, cv_weights, and friends) shows clean DAGs. The container-image invariants (no in-image snakemake, exec/run must not source host dotfiles) aren't reproducible from this repo -- the sandbox at /n17data/cdaley/containers/containers has no tracked build recipe -- so they're now documented in workflow/README.md for the next rebuild. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU

…ript docs presentation_pte_cosebis was the one rule left with an inline apptainer exec in its shell: block (container: None override); every sibling presentation_* rule already relies on the module-level container default. Drop the override and the raw call so it's wrapped like the rest. Also update the three sweep-script docstrings (run_xi_sweep, run_cosebis_ptes_sweep, run_cl_sweep) whose example invocations still showed the retired apptainer-exec-then-python pattern, to match the plain `python script.py ...` convention already used by their sibling CLI scripts.

Per-node smoke tests showed default env passthrough makes container python resolution nondeterministic (host ~/.local shadowing — the #302 mechanism). --cleanenv makes every job's environment container-defined. Verified: container_smoke green through the profile with it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU

calculate_pure_eb now returns one results dict per tomographic bin pair (#297); the pure-E/B integration test still indexed the flat mode keys. Unwrap the non-tomographic "tomo_bin_all_tomo_bin_all" entry and correct the docstring that still advertised the flat return. glass_mock's map path imports cosmology.compat.camb, which is absent in the image, so the xfail's raises=AttributeError no longer matched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QndBZicN3QvyZG4XDDPmGs

Remove the 'snakemake is injected' comment repeated in 19 script: files (one note in workflow/README.md instead), trim the candide profile header to the operational lessons, and cut re-narrations of the container model in Snakefile/common.py/README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

…script Collapse sif/sif_pipeline to the single sp_validation image (it ships shapepipe; must be rebuilt from uv.lock — the current 2026-07-04 image predates the lock and its numpy 2.5 breaks numba/ngmix). Remove the env -u SLURM_* prefix: shapepipe#744 gates mpi4py on OMPI/PMI vars, and --cleanenv strips the host env anyway (verified in-container). Convert im_mbias_config from a run: block to script:, drop the redundant os.makedirs, and trim pivot re-narration from comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

Deleted in the #236 cleanup with no replacement; restored from history to workflow/scripts/ (the rule is its only caller) and converted the rule from shell: to script:. Fixes on the way: bare exit() on a non-PD matrix returned 0 (Snakemake saw success) — now sys.exit(1); eigvalsh for the symmetric matrix; Agg backend; plot dpi 2000 -> 300. Verified round-trip on synthetic input in the container. Drop the NOTE and stale comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

The rule asserted nothing and put a non-scientific artifact in every paper's results/. Now: tests/data/container_smoke/{Snakefile,script} driven by test_container_smoke.py (@slow, skipped off-cluster), which submits one tiny SLURM job through the committed candide profile and asserts APPTAINER_CONTAINER is set (the job really ran in the image), the editable install resolved, seed-42 eigh values match, and git works inside the container. OMP_NUM_THREADS is recorded, not asserted — unset is the profile's designed state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

Replace the hand-rolled WORKFLOW_SCRIPTS constant with Snakemake's first-class mechanism, which the docs specifically prescribe because manual path construction breaks under module composition. The script goes in input: (not params:, which would cause spurious reruns), so it also becomes an honest dependency. Fixes a live bug on the way: papers/bmodes unblinding_ceremony called 'python workflow/scripts/unblinding_ceremony.py', which from the paper workdir resolves into the paper's own scripts/ dir -- stale since 091bba8 and only detectable at run time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

script: resolves relative to the .smk defining it, module composition included, so it defeats the paper-workdir trap without making scripts into input files -- and matches what the other 20+ rules already do. Only xi_highres keeps source_path, where script: is structurally impossible (snakemake would wrap the whole mpiexec line in one container). unblinding_ceremony was already written for script: -- its _config_from_snakemake was dead code because the rule invoked it via shell:, so it silently ran _config_from_cli, which re-derives paths from constants that no longer exist (a cosmo_val dir deleted from the referenced checkout, and a _PROJECT_ROOT off by one since the script moved to papers/bmodes/scripts/). The rule declared 6 inputs and 9 params while passing 2 on the command line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

Never used, and silently broken: the rule invoked the script via shell:, so the script's snakemake branch was dead code and its CLI branch re-derived paths from constants that no longer exist. Nothing imported it and no rule consumed its output. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

CI builds ghcr.io/cosmostat/sp_validation on every push from uv.lock; the hand-built SIFs it replaces were stale in ways that only failed at run time (no shapepipe.modules in one, numpy 2.5 breaking numba in the other). Every call site -- the Snakefiles, image_sims, the MPI rule's own apptainer exec, the paper shell drivers, interactive use -- now names one file, refreshed deliberately (see workflow/README.md). Overridable with --config container=<path or docker:// URI>. im_mbias_config read the OCI revision by opening the configured sif path; it now reads APPTAINER_CONTAINER, since the image a job actually ran in may be overridden and current.sif is a moving target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

CosmoSIS was an undeclared, user-supplied dependency of the inference step -- which is why #303 was hand-patched in someone's ~/.local. It pip-installs into the image against the base gfortran/GSL/cfitsio in ~2 min, so declare it. MPIFC must be set at build time or the sampler Makefiles silently skip the MPI targets and --mpi fails at load; chains must run under MPI because the upstream --smp pool is still broken at 3.25.2. cosmology 2022.10.9 was vestigial: the cosmology.compat.camb adapter comes from cosmology-compat-camb via glass[examples]. Relocking drops it and nothing else. UV_PYTHON pins uv to the image's own interpreter, since $HOME is bind-mounted and uv would otherwise pick a host CPython carrying none of the stack. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

Snakemake pulls docker://ghcr.io/cosmostat/sp_validation:develop into a shared apptainer prefix on first use and never again, so no digest or path is written down. One constant, CONTAINER_URI in workflow/common.py, is the single source of truth; host-side callers that need a concrete file (the paper shell drivers, interactive use) derive it via workflow/scripts/container_path.py. xi_highres is parked as a comment block: it has never been runnable -- its shell is a bare 'python run_2pcf_highres.py' while the script requires --cat-config and --out -- and parking it leaves covariance_cosmocov as the workflow's only container exception. The MPI reasoning is preserved in the block for whoever revives it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

The xfail asked for a compatible glass+cosmology pair verified in a fresh image. glass 2026.2 with cosmology-compat-camb is that pair: the map path runs end to end (11 shells, 66 spectra, monotonic kappa accumulation), so the marker now only hides regressions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

cosmosis's bundled MultiNest links -llapack and the base image ships only the runtime liblapack.so.3 with no dev symlink, so the build died at 'cannot find -llapack'. It passed on candide only because that sandbox had liblapack-dev installed at some point. MPIFC takes the absolute path: /opt/ompi/bin is not always on PATH, and a miss silently drops the MPI sampler libraries while the install still reports success. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

uv pip honours UV_PYTHON over VIRTUAL_ENV, so naming the system interpreter sent the editable install of sp_validation there instead of /app/.venv -- the image built fine and then failed its own import smoke test. The venv's own python satisfies the original intent (uv can't wander onto a host CPython from the bind-mounted $HOME) while keeping uv pip pointed at the venv. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA

Snakemake's `script:` directive already executes the checkout's script files, while `import sp_validation` resolved to the image's baked copy -- the two halves of one commit, split. `common.inject_checkout_pythonpath()` prepends the checkout's src/ to APPTAINERENV_PYTHONPATH (preserving any user-set value), so the image supplies the frozen dependency stack and the launched tree supplies sp_validation. This is what the image-sims chain has always done for both repos (`_ENV_PREFIX` in rules/image_sims.smk); the main workflow now matches it. Opt out with `--config checkout_pythonpath=false` to reproduce from the image alone. The flag is parsed tolerantly because `--config k=false` can arrive as the string "false". Also drops the hardcoded candide OpenMPI path from workflow/Snakefile: a machine path does not belong in generic workflow code, and it moves to the candide profile in a following commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… layer workflow/profiles/default carries the container model and nothing else, so the workflow runs off candide with `--profile workflow/profiles/default -j N`. Snakemake cannot compose profiles (one --profile, no `inherits:`), so the small machine-independent set -- software-deployment-method, rerun-triggers, latency-wait -- is duplicated verbatim in both files, marked GENERIC and cross-referenced. That is the least-magic arrangement available. apptainer-args and apptainer-prefix stay out of the shared block: every machine has its own disks and image cache. candide's apptainer-args now carries `--env LD_LIBRARY_PATH=/softs/openmpi/...`, previously an os.environ line in workflow/Snakefile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The Moriond talk-figure rules called `python` with no container, so they ran against whatever interpreter the driver happened to have. Let them inherit the module-level `container:` like every other rule. The two ImageMagick `convert` rules keep `container: None`: `convert` is a host tool, absent from the image. Same for covariance_cosmocov, whose docstring says so directly rather than pointing at a list elsewhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

`image_sims: {sif: ...}` was a required structural key, so every run config repeated the image path — a second place for it to drift from what the rest of the workflow runs. Default it to null and resolve it through the same code path as every other entry point; a run config still overrides it to name its own image or a branch tag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The workflow ran out of one shared image directory on candide (/n17data/cdaley/containers/snakemake-sif) with a hand-maintained current.sif symlink pointing at whatever Snakemake last autopulled. That only worked for one person: refreshing the image or repointing the symlink needed write access to another user's directory, and a refresh moved the ground under everyone at once. Everyone now runs their own image file at one canonical per-user path, ~/.cache/sp_validation/sp_validation.sif (SPV_CONTAINER overrides it), owned by a small CLI: spv-container pull # fetch the tag there, atomically spv-container status # revision label vs. this checkout's HEAD spv-container exec <cmd...> # one-off run inside it, candide binds applied sp_validation.container is stdlib-only on purpose: it runs on the host, outside the container, so it must import without the scientific stack — and it works straight from a checkout (`python3 src/sp_validation/container.py status`) with nothing installed. It also holds CONTAINER_URI, which workflow/common.py loads from this checkout by file path, so the CLI and the workflow can never name different images. `container:` now resolves to that local .sif when it exists and to the registry tag otherwise (Snakemake accepts either, and autopulls the tag into .snakemake/singularity). `--config container=...` still overrides both. The candide profile drops apptainer-prefix accordingly, and common.configure() warns — once, never fatally — when the local image predates the checkout. Also drops workflow/scripts/container_path.py, which existed to locate the shared cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Rewrites the container sections of workflow/README.md, CLAUDE.md, CONTRIBUTING.md and README.md for the per-user model: one image per person at ~/.cache/sp_validation/sp_validation.sif, `spv-container` to fill and inspect it, and how `container:` resolves to it. The shared-prefix machinery is gone -- current.sif bootstrap, the atomic-mv refresh recipe, the group-writable TODO. Trims the commentary while there. The profile pair says "change one, change the other" once instead of shouting it in three places; off-candide gets a paragraph rather than parallel billing, since candide is where everyone runs; and the enumeration of container exceptions is dropped in favour of the docstring on each rule that opts out. Also repoints the two paper Snakefiles, which resolve `container:` themselves, at common.resolve_container -- and replaces the obsolete `apptainer build --sandbox` recipe in README.md and installation.rst with `apptainer pull`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The pristine SIF is read-only, which is what you want almost always -- but it lost the one real advantage of the old hand-built sandbox workflow: `pip install` mid-analysis, when you need a package the image does not carry yet and a CI rebuild is too slow a loop to think in. `spv-container sandbox` unpacks the image into a writable directory at ~/.cache/sp_validation/sandbox/, and `spv-container exec --writable` runs against it so installs persist. Opt-in: nothing builds one for you. Resolution order is now one thing, shared by the CLI, the run_*.sh drivers and the workflow's `container:` -- sandbox if it exists, else SIF if it exists, else the registry tag. Snakemake execs a sandbox directory as happily as a .sif, so a package installed into the sandbox is there for workflow jobs too, with no further wiring. `resolve_image()` in sp_validation.container is the single implementation; common.resolve_container defers to it. The build stages into a sibling directory and swaps it in, as `pull` does, for a sharper reason than pull has: a half-written .sif fails loudly, but a half-unpacked sandbox is still a *directory*, so resolution would elect it and every job would silently run a broken tree. Building before removing also means a `--force` rebuild that fails -- a typo in --source, a network blip -- leaves the sandbox you already had intact, instead of deleting a working environment on the way to not replacing it. The cost of a sandbox is that what runs is no longer fully described by a revision label, so the divergence is made visible rather than left silent: `status` names which layer is live and says the revision only describes what the sandbox was built from (falling back to the SIF's label, marked as inferred, when the sandbox carries none), and the workflow prints one line at launch when a sandbox is in play. `spv-container pull && spv-container sandbox --force` resets. Verified on candide (apptainer 1.5.3): unprivileged `build --sandbox` works through user namespaces with no fakeroot and no subuid mapping; `exec --writable` persists writes while plain `exec` gets a read-only filesystem; `inspect --labels` still reports the source image's OCI labels from a sandbox directory; `--fix-perms` at build time is what keeps the tree removable afterwards (without it apptainer leaves directories that defeat `rm -rf`, which would strand `--force`); and a failed `--force` rebuild leaves the existing sandbox and its contents untouched, with no staging directory left behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The cosmo_inference .ini templates pointed COSMOSIS_DIR at two different people's home directories (/home/guerrini/... and a scratch path of Lisa's), so running the inference pipeline meant either being one of them or editing the templates by hand. CosmoSIS itself already ships in the image via the `workflow` extra; only the Standard Library — the tree of module files the pipelines name — was missing. Clone and build it at /opt/cosmosis-standard-library, pinned to Sacha Guerrini's fork at b26fa7ff. That fork is 4 commits ahead of upstream and 373 behind; the four are what the UNIONS pipelines need (tau statistics, sample_S8, two z-dependent linear-alignment modules). Carrying them onto current upstream is future work, noted in cosmo_inference/README.md. The templates now read COSMOSIS_DIR from %(CSL_DIR)s, which the image sets — CosmoSIS reads environment variables into an ini's [DEFAULT] section, which is how the existing %(SCRATCH)s references already work. Off-image, export CSL_DIR and the same templates work unchanged. The build follows CSL's documented procedure for a pip-installed cosmosis (`source cosmosis-configure && make`), but targets `shear/` rather than the top-level `make`: the top level also descends into likelihood/, building the Planck, WMAP and ACT likelihoods, which no UNIONS pipeline uses. Of the modules our templates do name, all are pure Python except two under shear/ — `limber`, which project_2d.py links, and cl_to_xi_nicaea's nicaea_interface.so. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The container model, the checkout-PYTHONPATH default, the profile GENERIC mirroring and the CSL_DIR resolution were each explained in three to six places. Give every concept one home -- workflow/README.md for the user-facing story, the docstring of the thing itself for mechanism -- and leave pointers elsewhere. Drop comments narrating what the code used to do; git holds that. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The container model landed the same few lines in several places; fold each into one home. * the four run_*.sh sweep drivers resolved this user's image (and repeated the bind list) inline -- now one sourced papers/bmodes/scripts/container_env.sh, resolving exactly as sp_validation/container.py does (sandbox first, then SPV_CONTAINER/XDG_CACHE_HOME, binds from SPV_APPTAINER_BINDS) * container.py: one _require_apptainer() instead of three copies of the PATH guard, and compare_revision's merge-base calls go through _git * common.py: resolve_container takes the override value, so image_sims.smk no longer wraps IMSIM["sif"] in a synthetic config dict; drop the CONTAINER_URI / local_sif / local_sandbox re-exports, which have no callers * cosmocov_process.py: only Snakemake runs it, so drop the argv entry point and its main() indirection, matching im_mbias_config.py * xip_xim.py: one catalog() builder for the tomographic and non-tomographic paths instead of two near-identical treecorr.Catalog blocks Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ed README section The four papers/bmodes sweep drivers each repeated the worktree path, the derived script/source dirs, and a full `apptainer exec ... /usr/local/bin/python` invocation (six sites). container_env.sh now owns all of it and exposes `spv_python` / `sweep_versions`; argv is byte-identical. workflow/README.md explained `--config container=` twice, once for a local .sif and once for a branch tag. One subsection now covers both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ithout running make Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc

… unset paths The test -f artifact guard is what keeps a no-op build loud; -u had become the thing breaking the build instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc

The container/workflow work is orthogonal to the tomography branch it was accidentally based on. Restore develop's pure_eb docstring, test_cosmo_val call shape, and glass_mock xfail; keep develop's glass==2025.1 pinned set (cosmology 2022.10.9 is load-bearing there, not vestigial) and relock. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143zcEsfSWr13AfSroMteEC

README leads with the four-line install (clone, symlink onto PATH, pull, exec-check); container.py gets a shebang + exec bit so the symlink is a real CLI with no packaging. installation.rst carries the depth (subcommands, per-user model, sandbox, raw apptainer/docker); CONTRIBUTING and workflow/README point at the same symlink step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143zcEsfSWr13AfSroMteEC

cailmdaley · 2026-08-27T13:33:05Z

@sachaguer and @martinkilbinger please take a look at this! snakemake + the container should work much better now. one thing i'm ambivalent about is whether we should prefer snakemake script: directives (which means snakemake is baked into the scripts and they're much less portable) or shell: python script.py, which gives up a lot of the simplicity/expressivity of passing parameters and config to the scripts. i went with the former, curious what you think

cailmdaley changed the title ~~Fixes from the snakemake inference run report (#300)~~ Snakemake run fixes + profile-driven container execution (#300) Aug 25, 2026

cailmdaley commented Aug 26, 2026

View reviewed changes

cailmdaley changed the title ~~Snakemake run fixes + profile-driven container execution (#300)~~ Run the workflow in the CI-published container (#300) Aug 26, 2026

cailmdaley changed the title ~~Run the workflow in the CI-published container (#300)~~ Improvements to snakemake containerization Aug 27, 2026

cailmdaley and others added 26 commits August 27, 2026 11:08

cailmdaley and others added 11 commits August 27, 2026 11:09

image: actually build CSL — cosmosis-configure exits 0 under set -u w…

ba2fd7c

…ithout running make Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc

image: point limber's Makefile at Debian's GSL (GSL_INC/GSL_LIB)

10bcf28

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc

cailmdaley force-pushed the fix/300-cluster-snakemake-run-fixes branch from 7faa8cd to 06ec1f3 Compare August 27, 2026 09:25

cailmdaley changed the base branch from feature/sp_validation-extend-to-tomography to develop August 27, 2026 09:25

cailmdaley requested review from martinkilbinger and sachaguer August 27, 2026 13:29

cailmdaley marked this pull request as ready for review August 27, 2026 13:29

Navigation Menu

Uh oh!

Improvements to snakemake containerization - #309

Improvements to snakemake containerization#309
cailmdaley wants to merge 37 commits into
developfrom
fix/300-cluster-snakemake-run-fixes

cailmdaley commented Aug 24, 2026 •

edited

Loading

Uh oh!

cailmdaley left a comment

Uh oh!

cailmdaley Aug 26, 2026

Uh oh!

cailmdaley Aug 26, 2026

Uh oh!

cailmdaley Aug 26, 2026

Uh oh!

cailmdaley Aug 26, 2026

Uh oh!

cailmdaley Aug 26, 2026

Uh oh!

cailmdaley Aug 26, 2026

Uh oh!

cailmdaley Aug 26, 2026

Uh oh!

cailmdaley Aug 26, 2026

Uh oh!

cailmdaley Aug 26, 2026

Uh oh!

cailmdaley Aug 26, 2026

Uh oh!

cailmdaley commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		@@ -0,0 +1,19 @@
		# Container smoke test -- validates the base container contract every

Uh oh!

Conversation

cailmdaley commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The container model

CosmoSIS and the standard library

Run-report fixes

Verified

Uh oh!

cailmdaley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cailmdaley commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cailmdaley commented Aug 24, 2026 •

edited

Loading