From f1210c2187aeaa88c45a821ebb8e92896f187fe9 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Tue, 25 Aug 2026 00:45:19 +0200 Subject: [PATCH 01/37] Fixes from the snakemake inference run report (#300) Rules: drop the /automnt prefix from the hardcoded paths in xi_highres (twopoint.smk) and covariance_glass_mock (covariance.smk). /automnt/nXXdataN does not exist on the node that owns that disk, so a job landing there fails immediately, before any log is written. Every canonical path in common.py already uses the plain /nXXdataN form. Docs: workflow/README.md gains a note on the /automnt trap and one on host ~/.local shadowing the container's pinned Snakemake; cosmo_inference/README.md recommends CosmoSIS --mpi over the fragile upstream --smp process pool (cosmosis#170) and cosmosis >= 3.16.1. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU --- cosmo_inference/README.md | 9 +++++++++ workflow/README.md | 19 +++++++++++++++++++ workflow/rules/covariance.smk | 14 +++++++------- workflow/rules/twopoint.smk | 2 +- 4 files changed, 36 insertions(+), 8 deletions(-) diff --git a/cosmo_inference/README.md b/cosmo_inference/README.md index 5d753010..6b460375 100644 --- a/cosmo_inference/README.md +++ b/cosmo_inference/README.md @@ -6,6 +6,15 @@ This folder contains the files neccessary to run the cosmological inference pipe ### Requirements To run the pipeline, one would need to have installed [CosmoSIS](https://cosmosis.readthedocs.io/en/latest/). To sample the PSF leakage parameters, the fork of [cosmosis-standard-library](https://github.com/sachaguer/cosmosis-standard-library/) of Sacha Guerrini has to be used. +Run CosmoSIS with `--mpi`, not `--smp`. The `--smp` process pool is fragile and +barely maintained upstream: its `bcast`, `gather` and `allreduce` methods all +return a `self.data` attribute that is never set, so a run can crash right after +sampling finishes (upstream issue +[cosmosis#170](https://github.com/cosmosis-developers/cosmosis/issues/170) — the +`allreduce` crash was fixed in cosmosis 3.16.1, the rest is still open). Use +cosmosis 3.16.1 or newer, and prefer `--mpi`, which is what the upstream +maintainer recommends. + ### To Run The inference pipeline is now orchestrated through Python. Run the main Snakemake workflow from the parent directory: diff --git a/workflow/README.md b/workflow/README.md index f20ffe8f..96262a6a 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -57,3 +57,22 @@ that same `apptainer exec` line, since the slurm executor's `--export=ALL` propagates the driver's env, not a profile flag). Per-rule `mem_mb` / `runtime` stay on the rules. Off-cluster, drop `--profile` and add `-j N`. See the profile's own comments for the full rationale. + +### Never write `/automnt/nXXdataN` in a path + +Use the plain form `/nXXdataN/...` in every rule, config, and invocation +directory. `/automnt/nXXdataN` works only from a node that does *not* own that +disk. On the owning node the disk is mounted directly at `/nXXdataN` and there +is no `/automnt/nXXdataN` entry at all, so a job that lands there dies about one +second after the allocation starts, before any log file is written. This is why +`n17` is in the profile's exclude list. Every canonical path in `common.py` +already uses the plain form; keep new paths the same. + +### Snakemake version: host or container, not both + +Apptainer passes your `PATH` and mounts your `$HOME` into the container, so a +host-side `pip install --user snakemake` in `~/.local` can shadow the container's +own pinned Snakemake. Mixing the two versions in one session produces confusing +errors. Inside the container, check that `which snakemake` gives +`/app/.venv/bin/snakemake`. If you drive the workflow from the host instead, use +the same Snakemake version that `uv.lock` pins. diff --git a/workflow/rules/covariance.smk b/workflow/rules/covariance.smk index 49e40201..8a30709f 100644 --- a/workflow/rules/covariance.smk +++ b/workflow/rules/covariance.smk @@ -203,13 +203,13 @@ rule covariance_glass_mock: seed=range(config["glass_mocks"]["seed_range"][0], config["glass_mocks"]["seed_range"][1] + 1), ), output: - xi_covariance="/automnt/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/xi_covariance.npy", - cl_covariance="/automnt/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/cl_covariance.npy", - combined_covariance="/automnt/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/combined_covariance.npy", - correlation_plot="/automnt/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/combined_correlation.png", - xi_mean="/automnt/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/xi_mean.npy", - cl_mean="/automnt/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/cl_mean.npy", - combined_mean="/automnt/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/combined_mean.npy", + xi_covariance="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/xi_covariance.npy", + cl_covariance="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/cl_covariance.npy", + combined_covariance="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/combined_covariance.npy", + correlation_plot="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/combined_correlation.png", + xi_mean="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/xi_mean.npy", + cl_mean="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/cl_mean.npy", + combined_mean="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/combined_mean.npy", script: "../scripts/compute_glass_mock_covariance.py" diff --git a/workflow/rules/twopoint.smk b/workflow/rules/twopoint.smk index 22c09db2..313b175f 100644 --- a/workflow/rules/twopoint.smk +++ b/workflow/rules/twopoint.smk @@ -45,7 +45,7 @@ rule xi_highres: "--bind /home,/n09data,/n17data,/n23data1,/softs " "--env LD_LIBRARY_PATH=/softs/openmpi/5.0.5-slurm-CentOS8/lib " "/n17data/cdaley/containers/containers " - "python /automnt/n17data/cdaley/unions/pure_eb/code/sp_validation/workflow/scripts/run_2pcf_highres.py" + "python /n17data/cdaley/unions/pure_eb/code/sp_validation/workflow/scripts/run_2pcf_highres.py" rule run_cosmo_val: From 1fe0e1e7fd70ee1f5a4debddca8d848cbbd54c67 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Tue, 25 Aug 2026 01:35:17 +0200 Subject: [PATCH 02/37] workflow: profile-driven apptainer containerization (set A) candide profile now owns the container: software-deployment-method: apptainer + apptainer-args carries the bind mounts (matching the app bash-function binds in the top-level UNIONS CLAUDE.md), replacing the old rationale for leaving containerization to each rule. Rewrites the profile's doc comment to describe the new model and its two documented exceptions (xi_highres MPI, covariance_cosmocov host toolchain). Adds a container_smoke rule (workflow/rules/container_smoke.smk + scripts/container_smoke.py) as a cheap end-to-end check of the profile-driven container path (editable sp_validation import, numpy + OMP_NUM_THREADS, git provenance) via `script:`, wired unconditionally into workflow/Snakefile. Reconciles image_sims/Snakefile's container: None comment: it now documents that the two-image (SIF/SIF_PIPELINE) chain is a per-rule container: choice in image_sims.smk, still wrapped by the profile's apptainer deployment -- not a rule-owned apptainer exec call. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU --- workflow/Snakefile | 6 +++ workflow/image_sims/Snakefile | 12 ++++- workflow/profiles/candide/config.yaml | 55 ++++++++++++------- workflow/rules/container_smoke.smk | 19 +++++++ workflow/scripts/container_smoke.py | 76 +++++++++++++++++++++++++++ 5 files changed, 147 insertions(+), 21 deletions(-) create mode 100644 workflow/rules/container_smoke.smk create mode 100644 workflow/scripts/container_smoke.py diff --git a/workflow/Snakefile b/workflow/Snakefile index 7d91db81..09cd38e3 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -35,6 +35,12 @@ from common import * wildcard_constraints: **WILDCARD_CONSTRAINTS +# Container smoke test — validates the profile-driven container contract +# (executor: slurm + software-deployment-method: apptainer + apptainer-args) +# that every rule below inherits. No config gate: it needs nothing from a +# run config, so it is always available as a target. +include: "rules/container_smoke.smk" + # Compute rules (infrastructure — raw outputs, no evidence.json) include: "rules/twopoint.smk" include: "rules/covariance.smk" diff --git a/workflow/image_sims/Snakefile b/workflow/image_sims/Snakefile index d7a89065..113c0270 100644 --- a/workflow/image_sims/Snakefile +++ b/workflow/image_sims/Snakefile @@ -33,8 +33,16 @@ there under ``if "image_sims" in config``. configfile: "workflow/image_sims/config.yaml" -# The image-sims rules own their container invocation explicitly, so no -# top-level container is needed here. +# No single image covers this chain -- the ShapePipe stages (pipeline, merge) +# and the sp_validation stages (manifest, extract, calibrate, m_bias) run in +# two different images (config["image_sims"]["sif"] / ["sif_pipeline"]), so +# there is no correct *default* to set here. Each rule in +# workflow/rules/image_sims.smk instead declares its own ``container: SIF`` / +# ``container: SIF_PIPELINE``; Snakemake wraps every job through the +# profile's ``software-deployment-method: apptainer`` exactly as it does for +# the single-image rules elsewhere -- no rule shells out to ``apptainer exec`` +# itself. ``container: None`` here just means "no module-level fallback", +# not "unwrapped": every included rule sets its own. container: None diff --git a/workflow/profiles/candide/config.yaml b/workflow/profiles/candide/config.yaml index 0316b909..c9dceca6 100644 --- a/workflow/profiles/candide/config.yaml +++ b/workflow/profiles/candide/config.yaml @@ -6,28 +6,36 @@ # -s workflow/image_sims/Snakefile \ # --configfile # -# and Snakemake owns all scheduling -- it fans out one SLURM job per branch x -# tile and drives them against the cluster, MPI-free. Everything here is -# cluster policy (executor, account, partition, node excludes, per-job -# defaults); it carries no science and no workflow logic. +# and Snakemake owns everything: scheduling (one SLURM job per branch x tile, +# fanned out against the cluster, MPI-free) *and* the container -- every rule +# runs through Snakemake's own container wrapping (``container:`` on the rule +# or an inherited module-level default; see workflow/Snakefile), never a +# rule's own ``apptainer exec`` shell call. This file is the single home for +# that: ``software-deployment-method: apptainer`` turns the wrapping on, +# ``apptainer-args`` carries the bind mounts every rule needs. Rules stay +# plain ``shell:``/``script:`` commands with no container knowledge at all. # -# What is deliberately NOT here: +# The orchestrator itself (``snakemake``) is a thin host-side tool, pinned via +# ``uv tool install`` (see workflow/README.md) -- it is never run from inside +# an ``apptainer shell``. It spawns the apptainer wrapping per job; there is +# exactly one container per job, never a nested one. # -# * Container / apptainer settings. The image-sims rules set -# ``container: None`` and own their ``apptainer exec`` call through the -# shared ``EXEC`` prefix (one image for every stage, with PYTHONPATH / -# PSF_DICT / OMP_NUM_THREADS injected there). So no -# ``software-deployment-method: apptainer`` / ``apptainer-args`` -- those -# would wrap a *second*, redundant container around jobs that already run -# inside one. +# Two explicit exceptions still set ``container: None`` and own an inline +# ``apptainer exec``/host-toolchain call directly in their rule: ``xi_highres`` +# (multi-node MPI -- Snakemake's wrapping puts the whole shell command in one +# container, which cannot spawn per-rank containers across nodes) and +# ``covariance_cosmocov`` (calls a host-compiled Fortran/C binary via +# environment-modules, not a Python entry point). Both are documented at their +# rule definition in workflow/rules/. # -# * OMP_NUM_THREADS. It is pinned to 1 on the ``apptainer exec`` line in -# workflow/rules/image_sims.smk, not here. The slurm executor submits with -# ``--export=ALL``, which propagates the *driver's* ambient environment; a -# profile only sets CLI flags, never the driver's own env, so an -# ``OMP_NUM_THREADS`` set here would silently depend on the operator having -# exported it by hand. Injecting it at the container boundary puts it where -# the compute runs, committed and independent of the launching shell. +# What is still deliberately NOT here: +# +# * OMP_NUM_THREADS. The slurm executor submits with ``--export=ALL``, +# which propagates the *driver's* ambient environment; a profile only sets +# CLI flags, never the driver's own env, so an ``OMP_NUM_THREADS`` set here +# would silently depend on the operator having exported it by hand. Rules +# that need it pinned set it themselves (``envvars:`` / ``params:``), which +# keeps it committed and independent of the launching shell. # # * Per-rule resources (mem_mb, runtime). Those live on each rule in the # .smk; the ``default-resources`` below are only the floor for rules that @@ -35,6 +43,15 @@ executor: slurm +# Every rule runs inside the sp_validation container -- the top-level +# ``container:`` in workflow/Snakefile / workflow/image_sims/Snakefile (or a +# rule's own override, e.g. the image-sims SIF/SIF_PIPELINE pair) names the +# image; this just turns Snakemake's wrapping on and supplies the binds. Kept +# in sync with the ``app`` bash function's raw ``apptainer exec`` in the +# top-level UNIONS CLAUDE.md -- update both together. +software-deployment-method: apptainer +apptainer-args: "--bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data" + # Cluster policy applied to every job unless a rule overrides it. The excludes # are the flaky/no-internet candide nodes (n17 mount issues, n09 no internet, # n36); ``slurm_extra`` is passed verbatim onto the sbatch line by the executor diff --git a/workflow/rules/container_smoke.smk b/workflow/rules/container_smoke.smk new file mode 100644 index 00000000..eef00730 --- /dev/null +++ b/workflow/rules/container_smoke.smk @@ -0,0 +1,19 @@ +# Container smoke test -- validates the base container contract every +# composed workflow inherits: profile executor (slurm) + software-deployment +# -method (apptainer) + apptainer-args (binds), no rule-level `container:` +# override and no `apptainer exec` in the shell. See +# workflow/scripts/container_smoke.py for what it actually checks. +# +# Run standalone before trusting the pivot on real compute: +# +# snakemake --profile workflow/profiles/candide -s workflow/Snakefile \ +# --configfile container_smoke + + +rule container_smoke: + output: + "results/container_smoke.yaml", + resources: + runtime=5, + script: + "../scripts/container_smoke.py" diff --git a/workflow/scripts/container_smoke.py b/workflow/scripts/container_smoke.py new file mode 100644 index 00000000..a1276d33 --- /dev/null +++ b/workflow/scripts/container_smoke.py @@ -0,0 +1,76 @@ +"""Rule container_smoke: exercise the containerized-SLURM path end to end. + +Cheap sanity check for the profile-driven-container pivot -- same executor +(slurm), same software-deployment-method (apptainer), same apptainer-args +binds, same container image every real rule uses. No rule-level `container:` +or `apptainer exec` anywhere here; Snakemake wraps the job entirely from the +profile. Three things it proves, each written to the output YAML: + + * the editable ``sp_validation`` install resolves on the container's + PYTHONPATH (import provenance: file + version, not just import success); + * the numeric stack works and honours threading env (numpy eigh on a small + fixed matrix, plus OMP_NUM_THREADS as seen inside the job); + * which commit of this checkout is running (git rev-parse from inside the + container -- proves /home is bound and usable, not just readable). + +Run it directly with `snakemake ... container_smoke` before trusting the +pivot on real compute. +""" + +import os +import platform +import subprocess + +import numpy as np +import yaml +from snakemake.script import snakemake + +# --- editable install resolves inside the container ------------------------ +import sp_validation + +sp_validation_info = { + "version": getattr(sp_validation, "__version__", "unknown"), + "file": sp_validation.__file__, +} + +# --- numeric stack + threading ----------------------------------------- +rng = np.random.default_rng(seed=42) +a = rng.standard_normal((8, 8)) +symmetric = a + a.T +eigenvalues = np.linalg.eigh(symmetric)[0] + +numeric_info = { + "numpy_version": np.__version__, + "eigenvalues": [float(v) for v in eigenvalues], + "omp_num_threads": os.environ.get("OMP_NUM_THREADS", "unset"), +} + +# --- provenance: what commit is actually running in the container --------- +repo_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +try: + commit = subprocess.run( + ["git", "-C", repo_dir, "rev-parse", "HEAD"], + capture_output=True, + text=True, + check=True, + ).stdout.strip() +except (subprocess.CalledProcessError, FileNotFoundError) as exc: + commit = f"unavailable ({exc})" + +provenance = { + "repo_dir": repo_dir, + "commit": commit, + "hostname": platform.node(), + "python": platform.python_version(), +} + +with open(snakemake.output[0], "w") as f: + yaml.safe_dump( + { + "sp_validation": sp_validation_info, + "numeric": numeric_info, + "provenance": provenance, + }, + f, + sort_keys=False, + ) From 2033f584fa61679f81ef1ed3ad22f59a2060d6b4 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Tue, 25 Aug 2026 01:37:15 +0200 Subject: [PATCH 03/37] workflow: profile-driven apptainer containerization (set B: image_sims) Strip every rule's explicit apptainer exec wrapper (_EXEC_PREFIX/EXEC/ EXEC_PIPELINE) from image_sims.smk. Each compute rule now carries a plain per-rule container: SIF / container: SIF_PIPELINE directive; Snakemake wraps the shell: command via the profile's software-deployment-method: apptainer + apptainer-args (set A). PYTHONPATH/PSF_DICT/OMP_NUM_THREADS injection and the SLURM_* env strip move from apptainer --env/-u flags to plain shell VAR=value / env -u syntax at the front of each shell: string (_ENV_PREFIX) -- identical effect, no apptainer-specific mechanism, works the same whether or not the command is container-wrapped. im_mbias split into im_mbias_config (run:, host-side git/provenance introspection + yaml write -- Snakemake never containerizes run: regardless of container:, so this must stay a driver-side step) and im_mbias (shell:, container: SIF, runs the actual m-bias compute). This was the one rule whose apptainer call lived inside a run: block's trailing shell() -- splitting it out is what makes container: apply to it at all. binds: dropped from image_sims config/schema -- it's now the profile's apptainer-args (one bind list for the whole workflow), not a per-run-config value. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU --- workflow/image_sims/config.yaml | 9 +- workflow/rules/image_sims.smk | 153 +++++++++++++++++++------------- 2 files changed, 99 insertions(+), 63 deletions(-) diff --git a/workflow/image_sims/config.yaml b/workflow/image_sims/config.yaml index 8f719c68..28fb6277 100644 --- a/workflow/image_sims/config.yaml +++ b/workflow/image_sims/config.yaml @@ -31,9 +31,12 @@ image_sims: # shadows pure-Python code only, never binary deps. sif: /n17data/cdaley/containers/sp_validation_im_sims.sif # extract/calibrate/m-bias sif_pipeline: /n17data/cdaley/containers/shapepipe_im_sims-runtime.sif # pipeline/merge - # Apptainer bind mounts. /automnt is required when repos/data are - # automounted (candide gotcha); harmless otherwise. [operational] - binds: /n17data,/n09data,/home,/automnt + # Bind mounts are no longer set per-run: each rule below carries a plain + # ``container: sif``/``container: sif_pipeline`` directive, and Snakemake + # wraps its ``shell:`` command in that image via the driving profile's + # ``apptainer-args`` (workflow/profiles/candide/config.yaml) -- one bind + # list for every containerized rule in the whole workflow, not a value + # threaded through image_sims config. # --- repositories ----------------------------------------------------- # Bound into the image; both repos' src go on PYTHONPATH so this branch's diff --git a/workflow/rules/image_sims.smk b/workflow/rules/image_sims.smk index a4eb92d0..d1fcf120 100644 --- a/workflow/rules/image_sims.smk +++ b/workflow/rules/image_sims.smk @@ -25,10 +25,13 @@ same split the gate766 baseline ran: ``calibrate`` (-> cut cat) and ``m_bias`` (-> ``m_bias_results.yaml``) run in ``sif`` (the sp_validation image). -Every rule sets ``container: None`` and calls ``apptainer exec`` explicitly -through a shared prefix template (``EXEC_PIPELINE`` / ``EXEC`` -- identical -env injections, different image), because the images are not the workflow's -top-level container. Everything is parameterised under +Every compute rule declares its own ``container: SIF`` or +``container: SIF_PIPELINE`` (a plain per-rule Snakemake directive -- no rule +shells out to ``apptainer`` itself); Snakemake wraps the rule's ``shell:`` +command in the right image via the driving profile's +``software-deployment-method: apptainer`` + ``apptainer-args`` (binds live +there now, not in this file). Two images, because the images are not the +workflow's top-level container. Everything is parameterised under ``config["image_sims"]`` -- the two ``sif`` keys, repository roots, data roots, the PSF dictionary, the explicit ``tile_ids`` list and the sim/calibration knobs -- so a fresh user drives it from config alone, with no hard-coded clone @@ -81,7 +84,6 @@ _DEPRECATED_KEYS = { # Operational keys: default (visibly) in the workflow config.yaml; the .smk # reads them bare, so config.yaml is their one home. _OPERATIONAL_KEYS = { - "binds", "sims_type", "branches", "shape", @@ -131,10 +133,13 @@ if _missing_structural: # --- containers ----------------------------------------------------------- # Two images (see module docstring): the ShapePipe image for the pipeline and # merge stages, the sp_validation image for everything downstream. Collapse -# back to one image once sp_validation's env is lock-managed. +# back to one image once sp_validation's env is lock-managed. Each compute +# rule below carries its own ``container: SIF`` / ``container: SIF_PIPELINE`` +# directive; the bind mounts these images need are the driving profile's +# ``apptainer-args`` (workflow/profiles/candide/config.yaml), not a value read +# from this config -- there is no per-rule bind string left to own. SIF = IMSIM["sif"] # sp_validation stages SIF_PIPELINE = IMSIM["sif_pipeline"] # ShapePipe stages -BINDS = IMSIM["binds"] # --- repositories (bound into the image; branch code overrides) ----------- SHAPEPIPE_REPO = IMSIM["shapepipe_repo"] @@ -181,10 +186,14 @@ CALIBRATE = IMSIM["calibrate_script"] # m-bias is *this branch's* extracted core, injected on PYTHONPATH. COMPUTE_M_BIAS = f"{SPV_REPO}/scripts/compute_m_bias_image_sims.py" -# --- container exec prefixes ---------------------------------------------- -# One prefix *shape* for every stage -- two instances, one per image. Three -# env injections make the on-disk branch -# code and the sim PSF win over the image's baked copies: +# --- in-command env prefix ------------------------------------------------- +# One prefix *shape* for every stage, now expressed as plain shell +# ``VAR=value`` / ``env -u`` syntax at the front of each rule's ``shell:`` +# string rather than as ``apptainer exec --env``/``-u`` flags -- Snakemake +# wraps the whole shell string inside the container (see each rule's +# ``container:``), so setting/unsetting the vars as the first shell tokens +# lands them exactly where the ``--env``/``-u`` flags used to, with no +# apptainer-specific mechanism required. Three things it does: # # * PYTHONPATH prepends BOTH repos' ``src`` (ShapePipe first, then # sp_validation), so Python resolves the worktree build before @@ -196,33 +205,25 @@ COMPUTE_M_BIAS = f"{SPV_REPO}/scripts/compute_m_bias_image_sims.py" # shadowed by PYTHONPATH. # * PSF_DICT points the fake_psf module (PSF_DICT_PATH = $PSF_DICT, expanded # via getexpanded) at this run's PSF dictionary. +# * The SLURM env vars are stripped (``env -u ...``) so that when the +# ShapePipe pipeline stage's OpenMPI initialises inside the image it does +# not try to attach to the host SLURM launcher (cf. apptainer_noslurm.sh). +# The strip is harmless for the pure-Python sp_validation stages, so one +# prefix serves all. # -# The SLURM env vars are stripped (``env -u ...``) so that when the ShapePipe -# pipeline stage's OpenMPI initialises inside the image it does not try to -# attach to the host SLURM launcher (cf. apptainer_noslurm.sh). The strip is -# harmless for the pure-Python sp_validation stages, so one prefix serves all. -# -# ``OMP_NUM_THREADS=1`` is injected here, at the ``apptainer exec`` call, and -# not left to the SLURM profile. The chain is MPI-free: Snakemake fans out one -# job per branch x tile and each job's parallelism is ShapePipe's own internal -# multiprocessing (``-N n_smp``), so the OpenMP/BLAS thread pool inside the -# container must be pinned to 1 to avoid oversubscription. The SLURM profile -# cannot pin it reliably: the slurm executor submits with ``--export=ALL``, -# which propagates the *driver's* ambient environment -- but a Snakemake -# profile only sets CLI flags, never the driver's own env, so an -# ``OMP_NUM_THREADS`` there would depend on the operator having exported it by -# hand (the implicit, uncommitted state the "one run command" is meant to -# retire). Injecting it on the ``apptainer exec`` line puts it where the -# compute actually runs -- inside the container, independent of the driver's -# env -- the same lever this prefix already uses for PYTHONPATH/PSF_DICT. -_EXEC_PREFIX = ( +# ``OMP_NUM_THREADS=1`` rides the same prefix, and not the SLURM profile, for +# the same reason as before: the chain is MPI-free (Snakemake fans out one job +# per branch x tile; parallelism inside a job is ShapePipe's own +# ``-N n_smp``), so the OpenMP/BLAS thread pool must be pinned to 1 to avoid +# oversubscription, and the slurm executor's ``--export=ALL`` only propagates +# the *driver's* ambient environment, not a value a profile could pin. Setting +# it here, inside the command every rule actually runs, keeps it committed and +# independent of both the driver's env and which container wraps the job. +_ENV_PREFIX = ( + f"PYTHONPATH={SHAPEPIPE_REPO}/src:{SPV_REPO}/src " + f"PSF_DICT={PSF_DICT} OMP_NUM_THREADS=1 " "env -u SLURM_JOBID -u SLURM_JOB_ID -u SLURM_PROCID " - f"apptainer exec --bind {BINDS} " - f"--env PYTHONPATH={SHAPEPIPE_REPO}/src:{SPV_REPO}/src " - f"--env PSF_DICT={PSF_DICT} --env OMP_NUM_THREADS=1 " ) -EXEC = _EXEC_PREFIX + SIF # sp_validation stages -EXEC_PIPELINE = _EXEC_PREFIX + SIF_PIPELINE # ShapePipe stages JOB_MASK = sum([1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]) @@ -300,8 +301,10 @@ rule im_manifest: input_sims_base=INPUT_SIMS_BASE, sims_type=SIMS_TYPE, num=NUM, + container: + SIF shell: - "{EXEC} python {BUILD_MANIFEST} " + "{_ENV_PREFIX} python {BUILD_MANIFEST} " "--input-sims-base {params.input_sims_base} " "--sims-type {params.sims_type} --num {params.num} " "{params.branch_args} -o {output.manifest}" @@ -375,9 +378,11 @@ rule im_pipeline: resources: mem_mb=16000, runtime=720, + container: + SIF_PIPELINE shell: "cd {params.run_dir} && " - "{EXEC_PIPELINE} bash {RUN_JOB} " + "{_ENV_PREFIX} bash {RUN_JOB} " "-e {wildcards.tile} -t image_sims -j {JOB_MASK} " "-p {params.psf} -N {params.n_smp}" @@ -397,9 +402,11 @@ rule im_merge: cat=f"{GRIDS_BASE}/{{sim}}/final_cat_{{sim}}.hdf5", params: run_dir=lambda wc: f"{GRIDS_BASE}/{wc.sim}", + container: + SIF_PIPELINE shell: "cd {params.run_dir} && " - "{EXEC_PIPELINE} python {CREATE_FINAL_CAT} " + "{_ENV_PREFIX} python {CREATE_FINAL_CAT} " "-I -m final_cat_{wildcards.sim}.hdf5 -i .. " "-p cfis/final_cat.param -P {wildcards.sim} " "-o n_tiles_final.txt -v" @@ -418,8 +425,10 @@ rule im_extract: cat=f"{GRIDS_BASE}/{{sim}}/shape_catalog_comprehensive_{SHAPE}.fits", params: run_dir=lambda wc: f"{GRIDS_BASE}/{wc.sim}", + container: + SIF shell: - "cd {params.run_dir} && {EXEC} python {EXTRACT_INFO}" + "cd {params.run_dir} && {_ENV_PREFIX} python {EXTRACT_INFO}" rule im_calibrate: @@ -436,20 +445,26 @@ rule im_calibrate: cat=f"{GRIDS_BASE}/{{sim}}/shape_catalog_cut_{SHAPE}.fits", params: run_dir=lambda wc: f"{GRIDS_BASE}/{wc.sim}", + container: + SIF shell: "cd {params.run_dir} && " - "{EXEC} python {CALIBRATE} -s calibrate" - - -rule im_mbias: - """Multiplicative/additive shear bias from the calibrated grids. - - Produces the workflow's headline artifact, ``m_bias_results.yaml``. The - injected shear (``shear_amplitude`` and the branch map) comes from - ``manifest.yaml`` alone -- no literal amplitude here or in config.yaml. The - generated ``m_bias_config.yaml`` carries the manifest's ``branches`` and - ``pairs``, so the estimator's sim list and pairing are the campaign's, not a - hard-coded default. + "{_ENV_PREFIX} python {CALIBRATE} -s calibrate" + + +rule im_mbias_config: + """Assemble ``m_bias_config.yaml`` -- the manifest's shear/branch facts, + this run's science knobs, and git/container provenance -- ahead of the + m-bias compute step. + + Pure host-side introspection (``git -C``, a plain-text scan of the SIFs' + OCI labels, PyYAML) -- no sp_validation/ShapePipe import, so it stays a + ``run:`` block with no container. Snakemake never containerizes ``run:`` + regardless of a rule's ``container:`` directive, which is exactly why this + step is split out of the compute rule below rather than left as a + ``run:`` block that shells out to ``{EXEC}`` at the end: a ``run:`` rule + can't carry the container the *compute* actually needs, but a ``shell:`` + rule can. """ input: manifest=MANIFEST, @@ -457,9 +472,8 @@ rule im_mbias: f"{GRIDS_BASE}/{{sim}}/shape_catalog_cut_{SHAPE}.fits", sim=SIMS ), output: - results=f"{GRIDS_BASE}/results/m_bias_results.yaml", - params: cfg=f"{GRIDS_BASE}/results/m_bias_config.yaml", + params: grids_base=GRIDS_BASE, num=NUM, cat_name=f"shape_catalog_cut_{SHAPE}.fits", @@ -467,6 +481,8 @@ rule im_mbias: sif_pipeline=SIF_PIPELINE, shapepipe_repo=SHAPEPIPE_REPO, sp_validation_repo=SPV_REPO, + results_dir=f"{GRIDS_BASE}/results", + results=f"{GRIDS_BASE}/results/m_bias_results.yaml", # Science knobs, read bare from the run config (no default here). match_radius_deg=IMSIM["match_radius_deg"], w_cols=IMSIM["w_cols"], @@ -536,7 +552,7 @@ rule im_mbias: }, } - os.makedirs(os.path.dirname(output.results), exist_ok=True) + os.makedirs(params.results_dir, exist_ok=True) # Emit *every* key the estimator requires -- pair_match and # bootstrap_seed included. Requiring a key without emitting it would # be a KeyError at run time, so the generated config is the complete @@ -557,12 +573,29 @@ rule im_mbias: "pair_match": params.pair_match, "n_bootstrap": params.n_bootstrap, "bootstrap_seed": params.bootstrap_seed, - "results_dir": os.path.dirname(output.results), - "output_path": output.results, + "results_dir": params.results_dir, + "output_path": params.results, "provenance": provenance, } - with open(params.cfg, "w") as fh: + with open(output.cfg, "w") as fh: yaml.safe_dump(mbias_cfg, fh) - shell( - "{EXEC} python {COMPUTE_M_BIAS} -c {params.cfg} -v" - ) + + +rule im_mbias: + """Multiplicative/additive shear bias from the calibrated grids. + + Produces the workflow's headline artifact, ``m_bias_results.yaml``, by + running the estimator against the config ``im_mbias_config`` assembled + (manifest's shear/branch facts, science knobs, provenance) -- the one + sp_validation-stage compute call in the chain, so it is the one place a + real ``container:``/``shell:`` split (rather than a ``run:`` block's + trailing ``shell()``) is required to containerize it at all. + """ + input: + cfg=f"{GRIDS_BASE}/results/m_bias_config.yaml", + output: + results=f"{GRIDS_BASE}/results/m_bias_results.yaml", + container: + SIF + shell: + "{_ENV_PREFIX} python {COMPUTE_M_BIAS} -c {input.cfg} -v" From 99ca91c26da19370db56aa22fdaa950f91d1ee61 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Tue, 25 Aug 2026 01:51:30 +0200 Subject: [PATCH 04/37] workflow: profile-driven apptainer containerization (set C: remaining rules + docs) Completes the pivot for the rules outside image_sims: xi_highres and covariance_cosmocov keep their container: None + inline apptainer exec / host-toolchain call (multi-node MPI and a host-compiled binary respectively, both genuinely incompatible with Snakemake's own container wrapping), now documented in their rule docstrings as deliberate exceptions rather than leftovers. No other rule outside image_sims called apptainer directly. While touching these files, retired the stale /pure_eb/ absolute paths left from the old repo layout: added workflow.common.WORKFLOW_SCRIPTS (Path(__file__)-based, correct under both standalone and module-composed runs) for the handful of shell: rules that call a workflow script directly, and reused the existing COSMO_INFERENCE constant elsewhere. Removed the run_cosmo_val rule in twopoint.smk, dead since cosmo_val.smk decomposed it into per-diagnostic rules (its own docstring says so) and still pointing at a stale path plus a nonsensical host .local PYTHONPATH injection. Flagged (not fixed) covariance_process: it calls cosmo_inference/scripts/ cosmocov_process.py, deleted in the #236 cleanup and never restored, so the rule fails on the default covariance target -- pre-existing, unrelated to this pivot. Rewrote workflow/README.md and cosmo_inference/README.md to the new model: snakemake is a thin host-side tool pinned via `uv tool install snakemake snakemake-executor-plugin-slurm`, run directly on the host, never from inside an apptainer shell; the candide profile's software-deployment-method puts each job in the container instead. Added a short pointer from the top-level README's dev-shell instructions to workflow/README.md so the two don't get conflated. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU --- README.md | 5 ++++ cosmo_inference/README.md | 13 ++++++++-- workflow/README.md | 49 +++++++++++++++++++++++++---------- workflow/common.py | 13 ++++++++++ workflow/rules/covariance.smk | 40 +++++++++++++++++++++------- workflow/rules/glass_mock.smk | 2 +- workflow/rules/twopoint.smk | 34 ++++++++++-------------- 7 files changed, 111 insertions(+), 45 deletions(-) diff --git a/README.md b/README.md index e14a2071..5da899f3 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,11 @@ docker run --rm -it ghcr.io/cosmostat/sp_validation:develop python -c "import sp We do not currently build images for Apple Silicon/arm64; however the amd64 images should work on these systems, albeit with reduced performance. +This shell is for interactive development and debugging. To run the analysis +workflow (`workflow/`), do not enter this shell — see +[`workflow/README.md`](workflow/README.md): Snakemake runs on the host, and +the profile puts each job in the container itself. + ## Flow chart diff --git a/cosmo_inference/README.md b/cosmo_inference/README.md index 6b460375..5e5989cc 100644 --- a/cosmo_inference/README.md +++ b/cosmo_inference/README.md @@ -16,12 +16,21 @@ cosmosis 3.16.1 or newer, and prefer `--mpi`, which is what the upstream maintainer recommends. ### To Run -The inference pipeline is now orchestrated through Python. Run the main Snakemake workflow from the parent directory: +The inference pipeline is orchestrated through Snakemake. On the candide +cluster, drive it with the committed profile — see +[`workflow/README.md`](../workflow/README.md) for the one-time +`uv tool install` setup and the full explanation. From the repository root: ```bash -snakemake -j inference_fiducial +snakemake --profile workflow/profiles/candide \ + -s workflow/Snakefile \ + inference_fiducial --configfile ``` +Off-cluster, drop `--profile` and add `-j ` instead. Each job runs +inside the sp_validation container automatically — no `apptainer shell` or +`apptainer exec` needed by hand. + This will automatically execute all steps: 1. Calculate 2PCF ($\xi_{pm}$) via `cosmo_val.py` 2. Compute covariance matrices using CosmoCov diff --git a/workflow/README.md b/workflow/README.md index 96262a6a..1e6084a5 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -51,12 +51,20 @@ Give the target *before* `--configfile`: `--configfile` takes one-or-more paths, so a target after it is read as a config file ("No such file: im_mbias"). Always dry-run first with `-n`. -The profile carries only cluster policy — no container settings (the image-sims -rules own their `apptainer exec` call) and no `OMP_NUM_THREADS` (pinned to 1 at -that same `apptainer exec` line, since the slurm executor's `--export=ALL` -propagates the driver's env, not a profile flag). Per-rule `mem_mb` / `runtime` -stay on the rules. Off-cluster, drop `--profile` and add `-j N`. See the -profile's own comments for the full rationale. +Every rule runs inside the sp_validation container: the profile sets +`software-deployment-method: apptainer` and `apptainer-args` (the bind mounts), +and Snakemake wraps each job's `shell:`/`script:` command in `apptainer exec` +itself — no rule writes its own `apptainer exec` call. The image name comes +from the `container:` directive in `workflow/Snakefile` (or a rule's own +override, e.g. the image-sims `SIF`/`SIF_PIPELINE` pair). Two rules are +explicit, documented exceptions and keep `container: None` with an inline +`apptainer exec`/host-toolchain call — `xi_highres` (multi-node MPI) and +`covariance_cosmocov` (a host-compiled binary) — see their docstrings in +`workflow/rules/`. `OMP_NUM_THREADS` is not set by the profile either: the +slurm executor's `--export=ALL` propagates the driver's env, not a profile +flag, so a rule that needs it pinned sets it itself. Per-rule `mem_mb` / +`runtime` stay on the rules. Off-cluster, drop `--profile` and add `-j N`. See +the profile's own comments for the full rationale. ### Never write `/automnt/nXXdataN` in a path @@ -68,11 +76,26 @@ second after the allocation starts, before any log file is written. This is why `n17` is in the profile's exclude list. Every canonical path in `common.py` already uses the plain form; keep new paths the same. -### Snakemake version: host or container, not both +### Run Snakemake from the host, never from inside the container -Apptainer passes your `PATH` and mounts your `$HOME` into the container, so a -host-side `pip install --user snakemake` in `~/.local` can shadow the container's -own pinned Snakemake. Mixing the two versions in one session produces confusing -errors. Inside the container, check that `which snakemake` gives -`/app/.venv/bin/snakemake`. If you drive the workflow from the host instead, use -the same Snakemake version that `uv.lock` pins. +`snakemake` is a thin host-side tool, pinned once per machine: + +```bash +uv tool install snakemake==9.23.1 --with snakemake-executor-plugin-slurm +``` + +(match the version to `snakemake` in this repo's `uv.lock`). Run every +`snakemake` command directly on the host — do not `apptainer shell` first. +Snakemake itself never touches the science stack; it only reads rule +definitions and submits jobs. Each job carries its own `apptainer exec` +wrapping from the profile (see above), so the container is where the science +code runs, not where the orchestrator runs — one container per job, never a +nested one. + +Driving Snakemake from inside a container shell used to be the recommended +path, and is why an old `~/.local/bin/snakemake` (or any host-side `pip +install --user snakemake`) is worth checking for: Apptainer passes your `PATH` +and mounts your `$HOME` by default, so a leftover host install can silently +shadow the one `uv tool install` just set up. Run `which snakemake` and +confirm it resolves under `uv`'s tool directory (`uv tool dir`), not +`~/.local/bin`. diff --git a/workflow/common.py b/workflow/common.py index 3df3413b..e684502e 100644 --- a/workflow/common.py +++ b/workflow/common.py @@ -5,6 +5,19 @@ import re from pathlib import Path +# workflow/scripts/, from this file's own location. Plain Python `__file__`, +# so it is correct no matter how a rule file that references it was reached -- +# standalone (`-s workflow/Snakefile`) or composed via `module:` from a paper +# Snakefile (each paper Snakefile also derives its own path to workflow/ the +# same way, from `workflow.basedir`, under the *different* name WORKFLOW_DIR +# -- keep this one distinct so `from common import *` in a paper Snakefile +# cannot shadow it). Use this, not a hardcoded absolute path, whenever a +# `shell:` block needs to call a script under workflow/scripts/ directly +# (`script:` is preferred and already resolves relative to its own rule file; +# this constant is only for the few rules that cannot use `script:`, e.g. +# because they wrap the call in `mpiexec`). +WORKFLOW_SCRIPTS = Path(__file__).resolve().parent / "scripts" + # Output roots are env-overridable so a reproduction run can write into a # fresh tree without clobbering (or silently reusing) prior products. COSMO_VAL = Path( diff --git a/workflow/rules/covariance.smk b/workflow/rules/covariance.smk index 8a30709f..1849f81a 100644 --- a/workflow/rules/covariance.smk +++ b/workflow/rules/covariance.smk @@ -145,6 +145,14 @@ EOF rule covariance_cosmocov: + """Run the host-compiled CosmoCov binary. + + Exception to the profile-driven container model (see + workflow/profiles/candide/config.yaml): CosmoCov is a host-compiled + Fortran/C binary loaded through environment-modules (`module load gcc + intelpython openmpi`), not a Python entry point the container ships. + `container: None` is required here, not a leftover of the old convention. + """ input: rules.covariance_ini.output, output: @@ -203,13 +211,13 @@ rule covariance_glass_mock: seed=range(config["glass_mocks"]["seed_range"][0], config["glass_mocks"]["seed_range"][1] + 1), ), output: - xi_covariance="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/xi_covariance.npy", - cl_covariance="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/cl_covariance.npy", - combined_covariance="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/combined_covariance.npy", - correlation_plot="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/combined_correlation.png", - xi_mean="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/xi_mean.npy", - cl_mean="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/cl_mean.npy", - combined_mean="/n17data/cdaley/unions/pure_eb/results/covariance/glass_mock_v1.4.6/combined_mean.npy", + xi_covariance="results/covariance/glass_mock_v1.4.6/xi_covariance.npy", + cl_covariance="results/covariance/glass_mock_v1.4.6/cl_covariance.npy", + combined_covariance="results/covariance/glass_mock_v1.4.6/combined_covariance.npy", + correlation_plot="results/covariance/glass_mock_v1.4.6/combined_correlation.png", + xi_mean="results/covariance/glass_mock_v1.4.6/xi_mean.npy", + cl_mean="results/covariance/glass_mock_v1.4.6/cl_mean.npy", + combined_mean="results/covariance/glass_mock_v1.4.6/combined_mean.npy", script: "../scripts/compute_glass_mock_covariance.py" @@ -232,8 +240,13 @@ rule generate_glass_mock_rhotau_samples: output_dir="results/glass_mock_rhotau_samples", threads: 1 shell: + # A CLI script, not `script:`: it takes a single `--mock-ids` range and + # this rule wants one call per mock_id wildcard, so `script:` (which + # only sees this one job's input/output) would need the same argparse + # rewritten as snakemake.* access for no behavior change. Left as a + # plain shell call. """ - python /n17data/cdaley/unions/pure_eb/code/sp_validation/workflow/scripts/generate_glass_mock_rhotau_samples.py \ + python {WORKFLOW_SCRIPTS}/generate_glass_mock_rhotau_samples.py \ --cov-tau {input.cov_tau} \ --ref-tau {input.ref_tau} \ --output-dir {params.output_dir} \ @@ -242,6 +255,15 @@ rule generate_glass_mock_rhotau_samples: rule covariance_process: + """Post-process a raw CosmoCov matrix into the analysis-ready form. + + NOTE (pre-existing, unrelated to containerization): the script this rule + calls, cosmo_inference/scripts/cosmocov_process.py, was deleted in the + cosmo_inference cleanup (#236) and was never restored. This rule -- on + the default path via fiducial_covariance_outputs() -- currently fails + with FileNotFoundError. Flagging here rather than silently working + around it; needs either restoring the script or rewriting this rule. + """ input: str(COSMO_INFERENCE / "data/covariance/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}.txt") output: @@ -253,7 +275,7 @@ rule covariance_process: threads: 1 shell: """ - python /n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_inference/scripts/cosmocov_process.py {input} {params.output_stub} + python {COSMO_INFERENCE}/scripts/cosmocov_process.py {input} {params.output_stub} """ diff --git a/workflow/rules/glass_mock.smk b/workflow/rules/glass_mock.smk index da8cb491..243076d6 100644 --- a/workflow/rules/glass_mock.smk +++ b/workflow/rules/glass_mock.smk @@ -119,7 +119,7 @@ rule mock_cosebis_bias_test: ), xi_ref=f"{MOCK_RESULTS}/gg_glass_mock_00001_nbins=1000.fits", cov=str( - Path("/n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_inference/data/covariance") + COSMO_INFERENCE / "data/covariance" / "covariance_SP_v1.4.6_leak_corr_A_g_minsep=0.5_maxsep=500.0_nbins=1000_masked" / "covariance_SP_v1.4.6_leak_corr_A_g_minsep=0.5_maxsep=500.0_nbins=1000_masked_processed.txt" ), diff --git a/workflow/rules/twopoint.smk b/workflow/rules/twopoint.smk index 313b175f..3f5820f5 100644 --- a/workflow/rules/twopoint.smk +++ b/workflow/rules/twopoint.smk @@ -25,7 +25,17 @@ rule xi: rule xi_highres: - """High-resolution xi for COSEBIS integration.""" + """High-resolution xi for COSEBIS integration. + + Exception to the profile-driven container model (see + workflow/profiles/candide/config.yaml): this is multi-node MPI, one + `apptainer exec` per rank. Snakemake's own container wrapping puts the + *whole* shell command -- `mpiexec` included -- inside a single container + instance, so only rank 0's node would run inside it; the other ranks, + spawned by SLURM/PMI on their own nodes, would land bare on the host. + `container: None` plus an explicit `mpiexec -n N apptainer exec ...` + per-rank is therefore required, not a leftover of the old convention. + """ container: None output: txt=str(COSMO_VAL / f"{FIDUCIAL['version']}_xi_minsep={FIDUCIAL['min_sep_int']}_maxsep={FIDUCIAL['max_sep_int']}_nbins=10000_npatch=1.txt"), @@ -40,30 +50,14 @@ rule xi_highres: slurm_extra="'--exclude=n17,n09,n36 --partition=pscomp'", mpi="/softs/openmpi/5.0.5-slurm-CentOS8/bin/mpiexec", shell: + # Container path kept in sync by hand with the top-level `container:` + # in workflow/Snakefile -- this rule cannot inherit it, see docstring. "{resources.mpi} -n {resources.tasks} " "apptainer exec " "--bind /home,/n09data,/n17data,/n23data1,/softs " "--env LD_LIBRARY_PATH=/softs/openmpi/5.0.5-slurm-CentOS8/lib " "/n17data/cdaley/containers/containers " - "python /n17data/cdaley/unions/pure_eb/code/sp_validation/workflow/scripts/run_2pcf_highres.py" - - -rule run_cosmo_val: - """Full CosmoVal diagnostic suite.""" - output: - sentinel=str(COSMO_VAL / "run_cosmo_val.done"), - threads: 24 - resources: - mem_mb=60000, - disk_mb=20000, - runtime=360, - shell: - """ - export PYTHONPATH="/home/cdaley/.local/lib/python3.12/site-packages:${{PYTHONPATH:-}}" - cd /n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_val \ - && python run_cosmo_val.py \ - && touch {output.sentinel} - """ + "python {WORKFLOW_SCRIPTS}/run_2pcf_highres.py" rule rho_tau_stats: From a2fc7d8351565ad394b9bc0c80fe451a3d42fa2b Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Tue, 25 Aug 2026 02:23:02 +0200 Subject: [PATCH 05/37] workflow: fix script: directive under the profile-driven container container_smoke failed for real (SLURM jobs 839818/839819) with ModuleNotFoundError: snakemake.iocontainers -- the container image had its own snakemake==9.16.3 pip-installed directly (leftover from the old apptainer-shell-then-snakemake-inside pattern this pivot retires), shadowing the host-mounted 9.23.1 orchestrator that script: bind-mounts in and sys.path.extends (appended, not prepended). Removed it and its snakemake-executor-plugin-slurm/-slurm-jobstep/-interface-* family from the image (verified Required-by: none outside the family itself). Separately, apptainer-args never actually isolated host tooling: the image's own /.singularity.d/env/50-bashrc.sh unconditionally sourced the host ~/.bashrc for every apptainer action, not just an interactive `apptainer shell` -- so a host dotfile (asdf init) ran on every exec too, pushing host PATH entries (~/.local/bin) ahead of the image's own /usr/local/bin. A bare `python` in any shell:/script: rule was silently running the host's interpreter, invisibly, surviving --cleanenv. Gated the bashrc sourcing on APPTAINER_COMMAND=shell (set by apptainer itself before these scripts run). Fixing both surfaced a third, previously-masked bug: every script: rule (19 files) imports `from snakemake.script import snakemake`, which is IDE-hint-only in this snakemake version -- snakemake.script exposes no such runtime attribute (only the Snakemake class), and the preamble that actually gets pickled in already provides `snakemake` as a plain global before the rest of the file executes. Removed the broken import repo-wide; the object resolves via normal global lookup exactly as before, including inside the functions/branches a few scripts defer it into. Verified end to end: container_smoke now completes for real through SLURM (jobid 839822, python 3.12.12, sp_validation editable install resolved, correct HEAD commit read from inside the job) with no apptainer exec left in any rule. Dry-run coverage for every rule that owns an edited script (masks_only, cv_weights, and friends) shows clean DAGs. The container-image invariants (no in-image snakemake, exec/run must not source host dotfiles) aren't reproducible from this repo -- the sandbox at /n17data/cdaley/containers/containers has no tracked build recipe -- so they're now documented in workflow/README.md for the next rebuild. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU --- .../scripts/filter_catalog_ellipticity.py | 5 +++- .../bmodes/scripts/plot_pure_eb_covariance.py | 7 +++-- workflow/README.md | 30 +++++++++++++++++++ .../scripts/analyze_mask_power_spectrum.py | 6 ++-- workflow/scripts/container_smoke.py | 6 +++- workflow/scripts/cv_additive_bias.py | 5 +++- workflow/scripts/cv_cosebis.py | 5 +++- workflow/scripts/cv_footprints.py | 5 +++- workflow/scripts/cv_objectwise_leakage.py | 5 +++- workflow/scripts/cv_plot_2pcf.py | 5 +++- workflow/scripts/cv_plot_rho_stats.py | 5 +++- workflow/scripts/cv_plot_tau_stats.py | 5 +++- workflow/scripts/cv_pseudo_cl.py | 5 +++- workflow/scripts/cv_pure_eb.py | 5 +++- workflow/scripts/cv_ratio_xi_sys_xi.py | 5 +++- workflow/scripts/cv_rho_tau_fits.py | 5 +++- workflow/scripts/cv_summarize_bmodes.py | 5 +++- workflow/scripts/cv_weights.py | 5 +++- workflow/scripts/process_mask.py | 7 +++-- workflow/scripts/run_rho_tau.py | 6 ++-- 20 files changed, 109 insertions(+), 23 deletions(-) diff --git a/papers/bmodes/scripts/filter_catalog_ellipticity.py b/papers/bmodes/scripts/filter_catalog_ellipticity.py index ade5dd6d..31fa76c6 100644 --- a/papers/bmodes/scripts/filter_catalog_ellipticity.py +++ b/papers/bmodes/scripts/filter_catalog_ellipticity.py @@ -18,7 +18,10 @@ sys.stderr if hasattr(sys, "ps1") else open(sys.stderr.fileno(), "w", buffering=1) ) -from snakemake.script import snakemake # noqa: E402 +# `snakemake` is injected as a module global by Snakemake's `script:` +# preamble before this file runs (`from snakemake.script import snakemake` +# is IDE-hint-only and raises ImportError if actually executed -- +# snakemake.script has no such runtime attribute). input_path = snakemake.input["catalog"] output_fits = snakemake.output["catalog"] diff --git a/papers/bmodes/scripts/plot_pure_eb_covariance.py b/papers/bmodes/scripts/plot_pure_eb_covariance.py index 3fdba8db..11c93e4f 100644 --- a/papers/bmodes/scripts/plot_pure_eb_covariance.py +++ b/papers/bmodes/scripts/plot_pure_eb_covariance.py @@ -27,8 +27,11 @@ def _load_snakemake(): "results/paper_plots/pure_eb_covariance.png", str(Path.cwd()), ) - from snakemake.script import snakemake - + # `snakemake` is already the module global Snakemake's `script:` + # preamble injected before this file began executing (`from + # snakemake.script import snakemake` is IDE-hint-only and raises + # ImportError if actually executed -- snakemake.script has no such + # runtime attribute). return snakemake diff --git a/workflow/README.md b/workflow/README.md index 1e6084a5..3578f5f1 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -99,3 +99,33 @@ and mounts your `$HOME` by default, so a leftover host install can silently shadow the one `uv tool install` just set up. Run `which snakemake` and confirm it resolves under `uv`'s tool directory (`uv tool dir`), not `~/.local/bin`. + +### Container image invariants (not tracked in this repo) + +The `script:` directive works by bind-mounting the host orchestrator's own +`snakemake` install into the job's container and `sys.path.extend`-ing it in +(appended, not prepended) — so anything already importable inside the image +under that name wins the lookup instead. The image at +`/n17data/cdaley/containers/containers` is a writable sandbox (see the +top-level UNIONS `CLAUDE.md`), not built from a tracked recipe, so these two +invariants live only in the image itself and must be re-applied by hand after +any rebuild: + +- **No `snakemake` (or `snakemake-executor-plugin-slurm`) pip-installed + inside the image.** A leftover in-image install — from the old + apptainer-shell-then-snakemake-inside pattern this profile-driven setup + retired — shadows the host-mounted orchestrator ahead of it on `sys.path` + and breaks `script:`'s own unpickling preamble (`ModuleNotFoundError: No + module named 'snakemake.iocontainers'` if the in-image version predates + that submodule). Check with `apptainer exec ... python3 -m pip show + snakemake` — `Required-by:` should list nothing outside the snakemake + family itself before removing it. +- **`/.singularity.d/env/50-bashrc.sh` must not source the host `~/.bashrc` + for `apptainer exec`/`run`, only for an interactive `apptainer shell`.** + Apptainer sources every `/.singularity.d/env/*.sh` for all three actions; + gate any host-dotfile sourcing on `[ "$APPTAINER_COMMAND" = "shell" ]` (set + by Apptainer itself before these scripts run). Without the guard, a host + dotfile that mutates `PATH` (e.g. an `asdf` init) runs on every job too and + can push host tools — including a host-side `~/.local/bin/python` — ahead + of the image's own `/usr/local/bin`, so a bare `python` in a rule's + `shell:`/`script:` silently executes outside the container. diff --git a/workflow/scripts/analyze_mask_power_spectrum.py b/workflow/scripts/analyze_mask_power_spectrum.py index 2e0e5b28..25302a74 100644 --- a/workflow/scripts/analyze_mask_power_spectrum.py +++ b/workflow/scripts/analyze_mask_power_spectrum.py @@ -72,8 +72,10 @@ def export_power_spectrum( def main(): """Process single mask power spectrum (Snakemake script entry point).""" - from snakemake.script import snakemake - + # `snakemake` is injected as a module global by Snakemake's `script:` + # preamble before this file runs (`from snakemake.script import + # snakemake` is IDE-hint-only and raises ImportError if actually + # executed -- snakemake.script has no such runtime attribute). mask_path = snakemake.input.mask output_path = str(snakemake.output.power_spectrum) diff --git a/workflow/scripts/container_smoke.py b/workflow/scripts/container_smoke.py index a1276d33..cfafcf9d 100644 --- a/workflow/scripts/container_smoke.py +++ b/workflow/scripts/container_smoke.py @@ -23,7 +23,11 @@ import numpy as np import yaml -from snakemake.script import snakemake + +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). # --- editable install resolves inside the container ------------------------ import sp_validation diff --git a/workflow/scripts/cv_additive_bias.py b/workflow/scripts/cv_additive_bias.py index 3fdec487..f66d769b 100644 --- a/workflow/scripts/cv_additive_bias.py +++ b/workflow/scripts/cv_additive_bias.py @@ -10,8 +10,11 @@ import json from cv_runner import _unbuffer_streams, make_cv, verify_outputs -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.calculate_additive_bias() diff --git a/workflow/scripts/cv_cosebis.py b/workflow/scripts/cv_cosebis.py index 182cda99..78f22b3b 100644 --- a/workflow/scripts/cv_cosebis.py +++ b/workflow/scripts/cv_cosebis.py @@ -8,8 +8,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) p = snakemake.params diff --git a/workflow/scripts/cv_footprints.py b/workflow/scripts/cv_footprints.py index 5ed89f07..34ee1f4c 100644 --- a/workflow/scripts/cv_footprints.py +++ b/workflow/scripts/cv_footprints.py @@ -6,8 +6,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_footprints() diff --git a/workflow/scripts/cv_objectwise_leakage.py b/workflow/scripts/cv_objectwise_leakage.py index 8b6d5694..825a1175 100644 --- a/workflow/scripts/cv_objectwise_leakage.py +++ b/workflow/scripts/cv_objectwise_leakage.py @@ -8,8 +8,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_objectwise_leakage() diff --git a/workflow/scripts/cv_plot_2pcf.py b/workflow/scripts/cv_plot_2pcf.py index 9d5c8900..d33de611 100644 --- a/workflow/scripts/cv_plot_2pcf.py +++ b/workflow/scripts/cv_plot_2pcf.py @@ -7,8 +7,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_2pcf() diff --git a/workflow/scripts/cv_plot_rho_stats.py b/workflow/scripts/cv_plot_rho_stats.py index 95a37b09..fdca6400 100644 --- a/workflow/scripts/cv_plot_rho_stats.py +++ b/workflow/scripts/cv_plot_rho_stats.py @@ -6,8 +6,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_rho_stats() diff --git a/workflow/scripts/cv_plot_tau_stats.py b/workflow/scripts/cv_plot_tau_stats.py index ed90e334..7f496c26 100644 --- a/workflow/scripts/cv_plot_tau_stats.py +++ b/workflow/scripts/cv_plot_tau_stats.py @@ -5,8 +5,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_tau_stats() diff --git a/workflow/scripts/cv_pseudo_cl.py b/workflow/scripts/cv_pseudo_cl.py index cf04e8e8..24279f5a 100644 --- a/workflow/scripts/cv_pseudo_cl.py +++ b/workflow/scripts/cv_pseudo_cl.py @@ -6,8 +6,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_pseudo_cl() diff --git a/workflow/scripts/cv_pure_eb.py b/workflow/scripts/cv_pure_eb.py index d15a763f..6d4741ec 100644 --- a/workflow/scripts/cv_pure_eb.py +++ b/workflow/scripts/cv_pure_eb.py @@ -9,8 +9,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) p = snakemake.params diff --git a/workflow/scripts/cv_ratio_xi_sys_xi.py b/workflow/scripts/cv_ratio_xi_sys_xi.py index ba737d7e..6298f507 100644 --- a/workflow/scripts/cv_ratio_xi_sys_xi.py +++ b/workflow/scripts/cv_ratio_xi_sys_xi.py @@ -8,8 +8,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_ratio_xi_sys_xi(offset=snakemake.params.get("offset", 0.1)) diff --git a/workflow/scripts/cv_rho_tau_fits.py b/workflow/scripts/cv_rho_tau_fits.py index 39d2e371..a81532b5 100644 --- a/workflow/scripts/cv_rho_tau_fits.py +++ b/workflow/scripts/cv_rho_tau_fits.py @@ -9,8 +9,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) if cv.rho_tau_method != "none": diff --git a/workflow/scripts/cv_summarize_bmodes.py b/workflow/scripts/cv_summarize_bmodes.py index 90df5999..8bb3b833 100644 --- a/workflow/scripts/cv_summarize_bmodes.py +++ b/workflow/scripts/cv_summarize_bmodes.py @@ -17,8 +17,11 @@ import json from cv_runner import _unbuffer_streams, make_cv, verify_outputs -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) p = snakemake.params diff --git a/workflow/scripts/cv_weights.py b/workflow/scripts/cv_weights.py index 3cb3f3ca..f8a50c02 100644 --- a/workflow/scripts/cv_weights.py +++ b/workflow/scripts/cv_weights.py @@ -5,8 +5,11 @@ """ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -from snakemake.script import snakemake +# `snakemake` is injected as a module global by Snakemake's `script:` preamble +# before this file runs; no import is needed (and `from snakemake.script +# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime +# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_weights() diff --git a/workflow/scripts/process_mask.py b/workflow/scripts/process_mask.py index c0368147..2aea6c73 100644 --- a/workflow/scripts/process_mask.py +++ b/workflow/scripts/process_mask.py @@ -138,8 +138,11 @@ def save_area_summary( def main(): """Main processing function.""" - # Snakemake script execution only (no interactive mode) - from snakemake.script import snakemake + # Snakemake script execution only (no interactive mode). `snakemake` is + # injected as a module global by Snakemake's `script:` preamble before + # this file runs (`from snakemake.script import snakemake` is + # IDE-hint-only and raises ImportError if actually executed -- + # snakemake.script has no such runtime attribute). # Get parameters from Snakemake source_mask_path = snakemake.input.mask diff --git a/workflow/scripts/run_rho_tau.py b/workflow/scripts/run_rho_tau.py index ea2f35bc..832c7f53 100644 --- a/workflow/scripts/run_rho_tau.py +++ b/workflow/scripts/run_rho_tau.py @@ -28,8 +28,10 @@ "/n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_val/output/rho_tau_stats/rho_stats_SP_v1.4.5.fits", "/home/cdaley/n17data/unions/pure_eb", ) -else: - from snakemake.script import snakemake +# else: `snakemake` is already the module global Snakemake's `script:` +# preamble injected before this file began executing (`from snakemake.script +# import snakemake` is IDE-hint-only and raises ImportError if actually +# executed -- snakemake.script has no such runtime attribute). params = snakemake.params # type: ignore From 0c65327ad1a21586e813571226843f4ebe233e94 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Tue, 25 Aug 2026 02:39:41 +0200 Subject: [PATCH 06/37] workflow: remove last live apptainer-exec shell call + stale sweep-script docs presentation_pte_cosebis was the one rule left with an inline apptainer exec in its shell: block (container: None override); every sibling presentation_* rule already relies on the module-level container default. Drop the override and the raw call so it's wrapped like the rest. Also update the three sweep-script docstrings (run_xi_sweep, run_cosebis_ptes_sweep, run_cl_sweep) whose example invocations still showed the retired apptainer-exec-then-python pattern, to match the plain `python script.py ...` convention already used by their sibling CLI scripts. --- papers/bmodes/rules/presentation.smk | 8 +------- papers/bmodes/scripts/run_cl_sweep.py | 2 +- papers/bmodes/scripts/run_cosebis_ptes_sweep.py | 2 +- papers/bmodes/scripts/run_xi_sweep.py | 2 +- 4 files changed, 4 insertions(+), 10 deletions(-) diff --git a/papers/bmodes/rules/presentation.smk b/papers/bmodes/rules/presentation.smk index a9ef5dc1..984172bb 100644 --- a/papers/bmodes/rules/presentation.smk +++ b/papers/bmodes/rules/presentation.smk @@ -147,11 +147,5 @@ rule presentation_pte_cosebis: ], output: f"{TALK_DIR}/images/pte_cosebis_talk.png", - container: - None shell: - """ - apptainer exec --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data \ - /n17data/cdaley/containers/containers/ \ - python {TALK_DIR}/plot_pte_cosebis_talk.py - """ + "python {TALK_DIR}/plot_pte_cosebis_talk.py" diff --git a/papers/bmodes/scripts/run_cl_sweep.py b/papers/bmodes/scripts/run_cl_sweep.py index 8cb2106e..d5fd6eb0 100644 --- a/papers/bmodes/scripts/run_cl_sweep.py +++ b/papers/bmodes/scripts/run_cl_sweep.py @@ -13,7 +13,7 @@ ``mask:`` entry), so no per-version mask wiring is needed here. Serial over versions; the estimator uses the recipe's full core allocation per call. - apptainer exec ... /usr/local/bin/python run_cl_sweep.py \ + python run_cl_sweep.py \ --config .../config.yaml --cat-config .../cat_config.yaml \ --nside 1024 --npatch 1 --binning powspace --nbins 32 --power 0.5 \ --blind A --out diff --git a/papers/bmodes/scripts/run_cosebis_ptes_sweep.py b/papers/bmodes/scripts/run_cosebis_ptes_sweep.py index 26b3d996..df3b9b79 100644 --- a/papers/bmodes/scripts/run_cosebis_ptes_sweep.py +++ b/papers/bmodes/scripts/run_cosebis_ptes_sweep.py @@ -14,7 +14,7 @@ config_space_pte_matrices.py adapts via ``_cosebis_matrix_from_npz`` — into ``--out``. Serial over versions (~25 min/version, 206 pairs). - apptainer exec ... /usr/local/bin/python run_cosebis_ptes_sweep.py \ + python run_cosebis_ptes_sweep.py \ --config .../config.yaml \ --xi-sweep-dir \ --cov-sweep-dir \ diff --git a/papers/bmodes/scripts/run_xi_sweep.py b/papers/bmodes/scripts/run_xi_sweep.py index b02283a5..dd7219a3 100644 --- a/papers/bmodes/scripts/run_xi_sweep.py +++ b/papers/bmodes/scripts/run_xi_sweep.py @@ -13,7 +13,7 @@ binning. Serial over versions — lc's dask handles cross-output concurrency, and TreeCorr already uses the recipe's full OpenMP allocation per call. - apptainer exec ... /usr/local/bin/python run_xi_sweep.py \ + python run_xi_sweep.py \ --config .../config.yaml --cat-config .../cat_config.yaml --out """ From 1f6a70aff39a557b2679c728c93cda8bbbd2a334 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Tue, 25 Aug 2026 02:53:38 +0200 Subject: [PATCH 07/37] profile: add --cleanenv to apptainer-args MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per-node smoke tests showed default env passthrough makes container python resolution nondeterministic (host ~/.local shadowing — the #302 mechanism). --cleanenv makes every job's environment container-defined. Verified: container_smoke green through the profile with it. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU --- workflow/profiles/candide/config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/workflow/profiles/candide/config.yaml b/workflow/profiles/candide/config.yaml index c9dceca6..a59c2460 100644 --- a/workflow/profiles/candide/config.yaml +++ b/workflow/profiles/candide/config.yaml @@ -50,7 +50,7 @@ executor: slurm # in sync with the ``app`` bash function's raw ``apptainer exec`` in the # top-level UNIONS CLAUDE.md -- update both together. software-deployment-method: apptainer -apptainer-args: "--bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data" +apptainer-args: "--cleanenv --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data" # Cluster policy applied to every job unless a rule overrides it. The excludes # are the flaky/no-internet candide nodes (n17 mount issues, n09 no internet, From 54a9c09eb01785c6595ed52ef1f6cc17d27127d4 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Tue, 25 Aug 2026 18:33:11 +0200 Subject: [PATCH 08/37] tests: adapt pure-E/B and glass-mock tests to the tomographic API calculate_pure_eb now returns one results dict per tomographic bin pair (#297); the pure-E/B integration test still indexed the flat mode keys. Unwrap the non-tomographic "tomo_bin_all_tomo_bin_all" entry and correct the docstring that still advertised the flat return. glass_mock's map path imports cosmology.compat.camb, which is absent in the image, so the xfail's raises=AttributeError no longer matched. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01QndBZicN3QvyZG4XDDPmGs --- src/sp_validation/cosmo_val/pure_eb.py | 5 ++++- src/sp_validation/tests/test_cosmo_val.py | 4 +++- src/sp_validation/tests/test_glass_mock.py | 5 +++-- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/src/sp_validation/cosmo_val/pure_eb.py b/src/sp_validation/cosmo_val/pure_eb.py index 71f5d76a..9c075031 100644 --- a/src/sp_validation/cosmo_val/pure_eb.py +++ b/src/sp_validation/cosmo_val/pure_eb.py @@ -79,7 +79,10 @@ def calculate_pure_eb( Returns ------- dict - A dictionary containing the following keys: + One entry per tomographic bin pair, keyed + ``"tomo_bin_{b1}_tomo_bin_{b2}"`` (non-tomographic runs have the + single key ``"tomo_bin_all_tomo_bin_all"``). Each value is a + dictionary containing the following keys: - "xip_E": Pure E-mode correlation function for xi+. - "xim_E": Pure E-mode correlation function for xi-. diff --git a/src/sp_validation/tests/test_cosmo_val.py b/src/sp_validation/tests/test_cosmo_val.py index f50992d4..7d49e41a 100644 --- a/src/sp_validation/tests/test_cosmo_val.py +++ b/src/sp_validation/tests/test_cosmo_val.py @@ -622,13 +622,15 @@ def test_calculate_pure_eb_runs_on_synthetic_catalog(self, tmp_path): # mirrors the bmodes workflow's broad-and-fine integration grid; every # reporting bin is well-defined (no edge NaNs). nbins_int~80 here would # NaN the edge bins -- confirmed -- which is the finiteness teeth. + # calculate_pure_eb returns one results dict per tomographic bin pair; + # the non-tomographic run has the single "all x all" key. results = cv.calculate_pure_eb( version, npatch=npatch, min_sep_int=1.0, max_sep_int=300.0, nbins_int=600, - ) + )["tomo_bin_all_tomo_bin_all"] # Reference mode vectors from the seeded synthetic catalog + Schneider # transform. Deterministic (full-sample treecorr, no RNG); regenerate by diff --git a/src/sp_validation/tests/test_glass_mock.py b/src/sp_validation/tests/test_glass_mock.py index a056318b..bf9e0c2c 100644 --- a/src/sp_validation/tests/test_glass_mock.py +++ b/src/sp_validation/tests/test_glass_mock.py @@ -134,7 +134,8 @@ def test_config_change_breaks_reference(): @pytest.mark.xfail( reason=( "glass_mock map path is incompatible with the installed glass/cosmology " - "API: cosmology.Cosmology.from_camb returns a CambCosmology lacking " + "API: cosmology.compat.camb is missing entirely in the image, and where " + "it exists cosmology.Cosmology.from_camb returns a CambCosmology lacking " "comoving_distance, which glass.distance_grid / MultiPlaneConvergence " "require. The map path was never exercised before GLASS was added to the " "image. Fix = pin a compatible glass+cosmology pair (or adapt the API " @@ -142,7 +143,7 @@ def test_config_change_breaks_reference(): "See fiber shapepipe/sp_validation glass-cosmology-api-pin." ), strict=False, - raises=AttributeError, + raises=(AttributeError, ModuleNotFoundError), ) def test_matter_maps_are_seed_deterministic(): """Same config + seed → bit-identical matter/lensing maps. From f9b705b49235d61bd19ceb5c7cf56be8a66d8b44 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 04:17:03 +0200 Subject: [PATCH 09/37] docs: sweep comment bloat across the pivot diff Remove the 'snakemake is injected' comment repeated in 19 script: files (one note in workflow/README.md instead), trim the candide profile header to the operational lessons, and cut re-narrations of the container model in Snakefile/common.py/README. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- .../scripts/filter_catalog_ellipticity.py | 4 - .../bmodes/scripts/plot_pure_eb_covariance.py | 5 - workflow/README.md | 9 +- workflow/Snakefile | 6 -- workflow/common.py | 15 +-- workflow/profiles/candide/config.yaml | 92 ++++++------------- .../scripts/analyze_mask_power_spectrum.py | 4 - workflow/scripts/cv_additive_bias.py | 4 - workflow/scripts/cv_cosebis.py | 4 - workflow/scripts/cv_footprints.py | 4 - workflow/scripts/cv_objectwise_leakage.py | 4 - workflow/scripts/cv_plot_2pcf.py | 4 - workflow/scripts/cv_plot_rho_stats.py | 4 - workflow/scripts/cv_plot_tau_stats.py | 4 - workflow/scripts/cv_pseudo_cl.py | 4 - workflow/scripts/cv_pure_eb.py | 4 - workflow/scripts/cv_ratio_xi_sys_xi.py | 4 - workflow/scripts/cv_rho_tau_fits.py | 4 - workflow/scripts/cv_summarize_bmodes.py | 4 - workflow/scripts/cv_weights.py | 4 - workflow/scripts/process_mask.py | 5 - workflow/scripts/run_rho_tau.py | 4 - 22 files changed, 39 insertions(+), 157 deletions(-) diff --git a/papers/bmodes/scripts/filter_catalog_ellipticity.py b/papers/bmodes/scripts/filter_catalog_ellipticity.py index 31fa76c6..baab3ffb 100644 --- a/papers/bmodes/scripts/filter_catalog_ellipticity.py +++ b/papers/bmodes/scripts/filter_catalog_ellipticity.py @@ -18,10 +18,6 @@ sys.stderr if hasattr(sys, "ps1") else open(sys.stderr.fileno(), "w", buffering=1) ) -# `snakemake` is injected as a module global by Snakemake's `script:` -# preamble before this file runs (`from snakemake.script import snakemake` -# is IDE-hint-only and raises ImportError if actually executed -- -# snakemake.script has no such runtime attribute). input_path = snakemake.input["catalog"] output_fits = snakemake.output["catalog"] diff --git a/papers/bmodes/scripts/plot_pure_eb_covariance.py b/papers/bmodes/scripts/plot_pure_eb_covariance.py index 11c93e4f..5522ef59 100644 --- a/papers/bmodes/scripts/plot_pure_eb_covariance.py +++ b/papers/bmodes/scripts/plot_pure_eb_covariance.py @@ -27,11 +27,6 @@ def _load_snakemake(): "results/paper_plots/pure_eb_covariance.png", str(Path.cwd()), ) - # `snakemake` is already the module global Snakemake's `script:` - # preamble injected before this file began executing (`from - # snakemake.script import snakemake` is IDE-hint-only and raises - # ImportError if actually executed -- snakemake.script has no such - # runtime attribute). return snakemake diff --git a/workflow/README.md b/workflow/README.md index 3578f5f1..07cdb4bd 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -56,7 +56,7 @@ Every rule runs inside the sp_validation container: the profile sets and Snakemake wraps each job's `shell:`/`script:` command in `apptainer exec` itself — no rule writes its own `apptainer exec` call. The image name comes from the `container:` directive in `workflow/Snakefile` (or a rule's own -override, e.g. the image-sims `SIF`/`SIF_PIPELINE` pair). Two rules are +override, e.g. the image-sims `SIF`). Two rules are explicit, documented exceptions and keep `container: None` with an inline `apptainer exec`/host-toolchain call — `xi_highres` (multi-node MPI) and `covariance_cosmocov` (a host-compiled binary) — see their docstrings in @@ -129,3 +129,10 @@ any rebuild: can push host tools — including a host-side `~/.local/bin/python` — ahead of the image's own `/usr/local/bin`, so a bare `python` in a rule's `shell:`/`script:` silently executes outside the container. + +### `snakemake` in `script:` files + +Every script run via a rule's `script:` directive uses a bare `snakemake` +name (`snakemake.input[...]`, etc.) with no import — Snakemake injects it as +a module global before the script runs. `from snakemake.script import +snakemake` is IDE-hint-only and raises `ImportError` if actually executed. diff --git a/workflow/Snakefile b/workflow/Snakefile index 09cd38e3..7d91db81 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -35,12 +35,6 @@ from common import * wildcard_constraints: **WILDCARD_CONSTRAINTS -# Container smoke test — validates the profile-driven container contract -# (executor: slurm + software-deployment-method: apptainer + apptainer-args) -# that every rule below inherits. No config gate: it needs nothing from a -# run config, so it is always available as a target. -include: "rules/container_smoke.smk" - # Compute rules (infrastructure — raw outputs, no evidence.json) include: "rules/twopoint.smk" include: "rules/covariance.smk" diff --git a/workflow/common.py b/workflow/common.py index e684502e..6635e1f9 100644 --- a/workflow/common.py +++ b/workflow/common.py @@ -5,17 +5,10 @@ import re from pathlib import Path -# workflow/scripts/, from this file's own location. Plain Python `__file__`, -# so it is correct no matter how a rule file that references it was reached -- -# standalone (`-s workflow/Snakefile`) or composed via `module:` from a paper -# Snakefile (each paper Snakefile also derives its own path to workflow/ the -# same way, from `workflow.basedir`, under the *different* name WORKFLOW_DIR -# -- keep this one distinct so `from common import *` in a paper Snakefile -# cannot shadow it). Use this, not a hardcoded absolute path, whenever a -# `shell:` block needs to call a script under workflow/scripts/ directly -# (`script:` is preferred and already resolves relative to its own rule file; -# this constant is only for the few rules that cannot use `script:`, e.g. -# because they wrap the call in `mpiexec`). +# Absolute path to workflow/scripts/ for the few rules that must call a +# script from `shell:` (e.g. wrapped in `mpiexec`): a composing paper +# Snakefile runs with its own directory as the workdir, so a relative path +# would miss. WORKFLOW_SCRIPTS = Path(__file__).resolve().parent / "scripts" # Output roots are env-overridable so a reproduction run can write into a diff --git a/workflow/profiles/candide/config.yaml b/workflow/profiles/candide/config.yaml index a59c2460..1c96a2b1 100644 --- a/workflow/profiles/candide/config.yaml +++ b/workflow/profiles/candide/config.yaml @@ -1,88 +1,50 @@ # Committed SLURM profile for the candide cluster (IAP). # -# This is the "one run command" half of the workflow: drive any target with +# Drive any target with # # snakemake --profile workflow/profiles/candide \ # -s workflow/image_sims/Snakefile \ # --configfile # -# and Snakemake owns everything: scheduling (one SLURM job per branch x tile, -# fanned out against the cluster, MPI-free) *and* the container -- every rule -# runs through Snakemake's own container wrapping (``container:`` on the rule -# or an inherited module-level default; see workflow/Snakefile), never a -# rule's own ``apptainer exec`` shell call. This file is the single home for -# that: ``software-deployment-method: apptainer`` turns the wrapping on, -# ``apptainer-args`` carries the bind mounts every rule needs. Rules stay -# plain ``shell:``/``script:`` commands with no container knowledge at all. +# Snakemake owns scheduling (one SLURM job per branch x tile) and the +# container: every rule runs through Snakemake's own container wrapping +# (``container:`` on the rule or the module-level default in +# workflow/Snakefile), never a rule's own ``apptainer exec`` shell call. +# ``software-deployment-method: apptainer`` below turns that wrapping on; +# ``apptainer-args`` carries the bind mounts every rule needs. Exceptions +# (``xi_highres``, ``covariance_cosmocov``) are documented at their rule +# definitions in workflow/rules/. # -# The orchestrator itself (``snakemake``) is a thin host-side tool, pinned via -# ``uv tool install`` (see workflow/README.md) -- it is never run from inside -# an ``apptainer shell``. It spawns the apptainer wrapping per job; there is -# exactly one container per job, never a nested one. -# -# Two explicit exceptions still set ``container: None`` and own an inline -# ``apptainer exec``/host-toolchain call directly in their rule: ``xi_highres`` -# (multi-node MPI -- Snakemake's wrapping puts the whole shell command in one -# container, which cannot spawn per-rank containers across nodes) and -# ``covariance_cosmocov`` (calls a host-compiled Fortran/C binary via -# environment-modules, not a Python entry point). Both are documented at their -# rule definition in workflow/rules/. -# -# What is still deliberately NOT here: -# -# * OMP_NUM_THREADS. The slurm executor submits with ``--export=ALL``, -# which propagates the *driver's* ambient environment; a profile only sets -# CLI flags, never the driver's own env, so an ``OMP_NUM_THREADS`` set here -# would silently depend on the operator having exported it by hand. Rules -# that need it pinned set it themselves (``envvars:`` / ``params:``), which -# keeps it committed and independent of the launching shell. -# -# * Per-rule resources (mem_mb, runtime). Those live on each rule in the -# .smk; the ``default-resources`` below are only the floor for rules that -# set none. +# ``snakemake`` itself is a thin host-side tool, pinned via ``uv tool +# install`` (see workflow/README.md); run it on the host, never inside an +# ``apptainer shell``. executor: slurm -# Every rule runs inside the sp_validation container -- the top-level -# ``container:`` in workflow/Snakefile / workflow/image_sims/Snakefile (or a -# rule's own override, e.g. the image-sims SIF/SIF_PIPELINE pair) names the -# image; this just turns Snakemake's wrapping on and supplies the binds. Kept -# in sync with the ``app`` bash function's raw ``apptainer exec`` in the +# Kept in sync with the ``app`` bash function's raw ``apptainer exec`` in the # top-level UNIONS CLAUDE.md -- update both together. software-deployment-method: apptainer apptainer-args: "--cleanenv --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data" -# Cluster policy applied to every job unless a rule overrides it. The excludes -# are the flaky/no-internet candide nodes (n17 mount issues, n09 no internet, -# n36); ``slurm_extra`` is passed verbatim onto the sbatch line by the executor -# plugin, so the quoting is what sbatch must see. -# -# Three candide-specific SLURM lessons are baked into the values below (learned -# the hard way on the earlier hand-driven im-sims runs; see shapepipe's retired -# image_sims_pipeline/Snakefile docstring): +# Cluster policy applied to every job unless a rule overrides it. Excludes are +# the flaky/no-internet candide nodes (n17 mount issues, n09 no internet, n36). # -# * ``runtime`` MUST carry a unit (``60m``, ``6h``, ``2d``). Snakemake's -# resource parser reads a *bare* number as SECONDS, so ``runtime: 60`` would -# silently give every job a 60-second wall clock and kill it on start. The -# quoted-with-unit form here is deliberate; keep it that way, and prefer the -# same in any ``--default-resources`` passed on the command line. (A bare -# integer in a *rule's* ``resources: runtime=720`` is fine -- snakemake -# reads rule-level numeric runtime as minutes -- the seconds trap is only +# * ``runtime`` MUST carry a unit (``60m``, ``6h``, ``2d``). Snakemake's +# resource parser reads a bare number as SECONDS, so ``runtime: 60`` would +# silently give every job a 60-second wall clock and kill it on start. +# (A bare integer in a rule's own ``resources: runtime=720`` is fine -- +# rule-level numeric runtime is read as minutes; the seconds trap is only # the CLI/default-resources parser.) # # * ``cpus_per_task`` is pinned to 12 to CAP JOBS PER NODE, not because a job -# needs 12 cores (the chain is MPI-free and pins ``OMP_NUM_THREADS=1`` at -# the container). candide's per-user process limit is ``ulimit -u 1200`` -# *per node*, and apptainer crashes ("can't start new thread") beyond ~4 -# concurrent jobs on a 48-core node. Requesting 12 CPUs/job holds SLURM to -# ~4 jobs per 48-core node, under the ceiling. Dropping this to 1 would let -# SLURM pack ~48 jobs onto a node and crash the compute-heavy im_pipeline -# stage (which inherits this default -- it sets mem/runtime but not cpus). +# needs 12 cores. candide's per-user process limit is ``ulimit -u 1200`` +# per node, and apptainer crashes ("can't start new thread") beyond ~4 +# concurrent jobs on a 48-core node; 12 CPUs/job holds SLURM to ~4 jobs per +# node. # -# * After launching a real fan-out, VERIFY the request actually landed: -# ``squeue -u $USER -o "%C %l"`` must show 12 (CPUs) and the wall clock you -# intended (e.g. 12:00:00 for im_pipeline). A silently-misparsed runtime or -# cpus shows up here before it wastes a queue slot. +# * After launching a real fan-out, verify the request landed: +# ``squeue -u $USER -o "%C %l"`` should show 12 (CPUs) and the intended +# wall clock. default-resources: slurm_account: "cusers" slurm_partition: "comp,pscomp" diff --git a/workflow/scripts/analyze_mask_power_spectrum.py b/workflow/scripts/analyze_mask_power_spectrum.py index 25302a74..65c02132 100644 --- a/workflow/scripts/analyze_mask_power_spectrum.py +++ b/workflow/scripts/analyze_mask_power_spectrum.py @@ -72,10 +72,6 @@ def export_power_spectrum( def main(): """Process single mask power spectrum (Snakemake script entry point).""" - # `snakemake` is injected as a module global by Snakemake's `script:` - # preamble before this file runs (`from snakemake.script import - # snakemake` is IDE-hint-only and raises ImportError if actually - # executed -- snakemake.script has no such runtime attribute). mask_path = snakemake.input.mask output_path = str(snakemake.output.power_spectrum) diff --git a/workflow/scripts/cv_additive_bias.py b/workflow/scripts/cv_additive_bias.py index f66d769b..60df4c3d 100644 --- a/workflow/scripts/cv_additive_bias.py +++ b/workflow/scripts/cv_additive_bias.py @@ -11,10 +11,6 @@ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.calculate_additive_bias() diff --git a/workflow/scripts/cv_cosebis.py b/workflow/scripts/cv_cosebis.py index 78f22b3b..c1ca8275 100644 --- a/workflow/scripts/cv_cosebis.py +++ b/workflow/scripts/cv_cosebis.py @@ -9,10 +9,6 @@ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) p = snakemake.params diff --git a/workflow/scripts/cv_footprints.py b/workflow/scripts/cv_footprints.py index 34ee1f4c..e4a1af6a 100644 --- a/workflow/scripts/cv_footprints.py +++ b/workflow/scripts/cv_footprints.py @@ -7,10 +7,6 @@ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_footprints() diff --git a/workflow/scripts/cv_objectwise_leakage.py b/workflow/scripts/cv_objectwise_leakage.py index 825a1175..ae0f012a 100644 --- a/workflow/scripts/cv_objectwise_leakage.py +++ b/workflow/scripts/cv_objectwise_leakage.py @@ -9,10 +9,6 @@ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_objectwise_leakage() diff --git a/workflow/scripts/cv_plot_2pcf.py b/workflow/scripts/cv_plot_2pcf.py index d33de611..5251e7a5 100644 --- a/workflow/scripts/cv_plot_2pcf.py +++ b/workflow/scripts/cv_plot_2pcf.py @@ -8,10 +8,6 @@ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_2pcf() diff --git a/workflow/scripts/cv_plot_rho_stats.py b/workflow/scripts/cv_plot_rho_stats.py index fdca6400..c99bd646 100644 --- a/workflow/scripts/cv_plot_rho_stats.py +++ b/workflow/scripts/cv_plot_rho_stats.py @@ -7,10 +7,6 @@ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_rho_stats() diff --git a/workflow/scripts/cv_plot_tau_stats.py b/workflow/scripts/cv_plot_tau_stats.py index 7f496c26..2a00b3a5 100644 --- a/workflow/scripts/cv_plot_tau_stats.py +++ b/workflow/scripts/cv_plot_tau_stats.py @@ -6,10 +6,6 @@ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_tau_stats() diff --git a/workflow/scripts/cv_pseudo_cl.py b/workflow/scripts/cv_pseudo_cl.py index 24279f5a..7dfebc54 100644 --- a/workflow/scripts/cv_pseudo_cl.py +++ b/workflow/scripts/cv_pseudo_cl.py @@ -7,10 +7,6 @@ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_pseudo_cl() diff --git a/workflow/scripts/cv_pure_eb.py b/workflow/scripts/cv_pure_eb.py index 6d4741ec..bcc46d84 100644 --- a/workflow/scripts/cv_pure_eb.py +++ b/workflow/scripts/cv_pure_eb.py @@ -10,10 +10,6 @@ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) p = snakemake.params diff --git a/workflow/scripts/cv_ratio_xi_sys_xi.py b/workflow/scripts/cv_ratio_xi_sys_xi.py index 6298f507..79afd89a 100644 --- a/workflow/scripts/cv_ratio_xi_sys_xi.py +++ b/workflow/scripts/cv_ratio_xi_sys_xi.py @@ -9,10 +9,6 @@ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_ratio_xi_sys_xi(offset=snakemake.params.get("offset", 0.1)) diff --git a/workflow/scripts/cv_rho_tau_fits.py b/workflow/scripts/cv_rho_tau_fits.py index a81532b5..64b24b67 100644 --- a/workflow/scripts/cv_rho_tau_fits.py +++ b/workflow/scripts/cv_rho_tau_fits.py @@ -10,10 +10,6 @@ from cv_runner import _unbuffer_streams, make_cv, touch_sentinels -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) if cv.rho_tau_method != "none": diff --git a/workflow/scripts/cv_summarize_bmodes.py b/workflow/scripts/cv_summarize_bmodes.py index 8bb3b833..85f0fa1d 100644 --- a/workflow/scripts/cv_summarize_bmodes.py +++ b/workflow/scripts/cv_summarize_bmodes.py @@ -18,10 +18,6 @@ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) p = snakemake.params diff --git a/workflow/scripts/cv_weights.py b/workflow/scripts/cv_weights.py index f8a50c02..0316fe1e 100644 --- a/workflow/scripts/cv_weights.py +++ b/workflow/scripts/cv_weights.py @@ -6,10 +6,6 @@ from cv_runner import _unbuffer_streams, make_cv, verify_outputs -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). _unbuffer_streams() cv = make_cv(snakemake) cv.plot_weights() diff --git a/workflow/scripts/process_mask.py b/workflow/scripts/process_mask.py index 2aea6c73..c9590368 100644 --- a/workflow/scripts/process_mask.py +++ b/workflow/scripts/process_mask.py @@ -138,11 +138,6 @@ def save_area_summary( def main(): """Main processing function.""" - # Snakemake script execution only (no interactive mode). `snakemake` is - # injected as a module global by Snakemake's `script:` preamble before - # this file runs (`from snakemake.script import snakemake` is - # IDE-hint-only and raises ImportError if actually executed -- - # snakemake.script has no such runtime attribute). # Get parameters from Snakemake source_mask_path = snakemake.input.mask diff --git a/workflow/scripts/run_rho_tau.py b/workflow/scripts/run_rho_tau.py index 832c7f53..fbf62a3c 100644 --- a/workflow/scripts/run_rho_tau.py +++ b/workflow/scripts/run_rho_tau.py @@ -28,10 +28,6 @@ "/n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_val/output/rho_tau_stats/rho_stats_SP_v1.4.5.fits", "/home/cdaley/n17data/unions/pure_eb", ) -# else: `snakemake` is already the module global Snakemake's `script:` -# preamble injected before this file began executing (`from snakemake.script -# import snakemake` is IDE-hint-only and raises ImportError if actually -# executed -- snakemake.script has no such runtime attribute). params = snakemake.params # type: ignore From aaadd293fabef8395f7b77bfced15282ed3a5567 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 04:17:22 +0200 Subject: [PATCH 10/37] image_sims: one image, drop dead SLURM env strip, im_mbias_config as script MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Collapse sif/sif_pipeline to the single sp_validation image (it ships shapepipe; must be rebuilt from uv.lock — the current 2026-07-04 image predates the lock and its numpy 2.5 breaks numba/ngmix). Remove the env -u SLURM_* prefix: shapepipe#744 gates mpi4py on OMPI/PMI vars, and --cleanenv strips the host env anyway (verified in-container). Convert im_mbias_config from a run: block to script:, drop the redundant os.makedirs, and trim pivot re-narration from comments. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- workflow/image_sims/Snakefile | 40 +--- workflow/image_sims/config.yaml | 49 ++--- workflow/rules/image_sims.smk | 278 ++++++---------------------- workflow/scripts/im_mbias_config.py | 95 ++++++++++ 4 files changed, 174 insertions(+), 288 deletions(-) create mode 100644 workflow/scripts/im_mbias_config.py diff --git a/workflow/image_sims/Snakefile b/workflow/image_sims/Snakefile index 113c0270..457ec750 100644 --- a/workflow/image_sims/Snakefile +++ b/workflow/image_sims/Snakefile @@ -4,48 +4,28 @@ Run the sp_validation-side chain (merge -> extract -> calibrate -> m-bias), optionally including the ShapePipe pipeline stage, without pulling in the cosmology-validation config the top-level ``workflow/Snakefile`` requires. -Layer a run config over the operational defaults -- the workflow config.yaml -carries operational defaults but *no* science keys, so it is incomplete on its -own (by design); the run config supplies the science knobs. The one drive -command on candide, with the committed SLURM profile owning all scheduling: +The one drive command on candide, with the committed SLURM profile owning +scheduling and container wrapping: snakemake --profile workflow/profiles/candide \\ -s workflow/image_sims/Snakefile \\ im_mbias --configfile my_run.yaml -``configfile: "workflow/image_sims/config.yaml"`` below loads the operational -defaults automatically, so only ``my_run.yaml`` (the science knobs, and any -operational override that run wants) is passed on the command line; Snakemake -deep-merges the two. The profile supplies the executor, account, partition, -node excludes and job floor -- no ``-j`` needed (the slurm executor sets the -job cap). Off-cluster, drop ``--profile`` and add ``-j N`` to run locally. - -The target (``im_mbias``) is given *before* ``--configfile``: Snakemake's -``--configfile`` takes one-or-more paths, so a target placed after it is -swallowed as a config path ("No such file: im_mbias"). Put targets ahead of -``--configfile`` (or make ``--configfile`` the last flag on the line). Always +``configfile:`` below loads the operational defaults, so ``my_run.yaml`` need +only carry the science knobs (and any operational override); Snakemake +deep-merges the two. Off-cluster, drop ``--profile`` and add ``-j N``. + +Put targets *before* ``--configfile``: it takes one-or-more paths, so a target +after it is swallowed as a config path ("No such file: im_mbias"). Always dry-run first with ``-n``. -The same rules are also available inside the main workflow: they are included -there under ``if "image_sims" in config``. +The same rules are also included in the main workflow under +``if "image_sims" in config``. """ configfile: "workflow/image_sims/config.yaml" -# No single image covers this chain -- the ShapePipe stages (pipeline, merge) -# and the sp_validation stages (manifest, extract, calibrate, m_bias) run in -# two different images (config["image_sims"]["sif"] / ["sif_pipeline"]), so -# there is no correct *default* to set here. Each rule in -# workflow/rules/image_sims.smk instead declares its own ``container: SIF`` / -# ``container: SIF_PIPELINE``; Snakemake wraps every job through the -# profile's ``software-deployment-method: apptainer`` exactly as it does for -# the single-image rules elsewhere -- no rule shells out to ``apptainer exec`` -# itself. ``container: None`` here just means "no module-level fallback", -# not "unwrapped": every included rule sets its own. -container: None - - include: "../rules/image_sims.smk" diff --git a/workflow/image_sims/config.yaml b/workflow/image_sims/config.yaml index 28fb6277..7226e654 100644 --- a/workflow/image_sims/config.yaml +++ b/workflow/image_sims/config.yaml @@ -1,47 +1,24 @@ # Image-simulation m-bias workflow configuration. # -# Two kinds of keys live under `image_sims:`, and the split is the point: -# -# * OPERATIONAL keys default here (active lines below) and *nowhere else* -- -# the .smk reads them bare, so this file is their single home. Override in -# a run config only when a run genuinely differs from the shared setup. -# -# * SCIENCE keys have NO default -- not here, not in code. They fix the -# estimator's scientific behaviour and must be stated per run, so they -# appear below only as commented template lines. Supply them in a run -# config layered on top: -# -# snakemake -s workflow/image_sims/Snakefile \ -# --configfile workflow/image_sims/config.yaml \ -# --configfile my_run.yaml \ -# -j 4 im_mbias -# -# A run config that omits a science key fails at DAG parse, naming the key; an -# unknown key under `image_sims:` fails as a typo. The structural keys below -# (sif, repos, data roots, num, tile_ids) also have no default and must be set. +# OPERATIONAL keys default here and nowhere else -- the .smk reads them bare, so +# this file is their single home. SCIENCE keys have no default anywhere and +# appear below only as commented template lines: they fix the estimator's +# behaviour, so each run must state them in a run config layered on top. A run +# config that omits one fails at DAG parse, naming the key; an unknown key under +# `image_sims:` fails as a typo. image_sims: - # --- containers ------------------------------------------------------- - # Two images, one per half of the chain (the split gate766 ran). One image - # is the eventual target -- the sp_validation image is FROM the ShapePipe - # image -- but until sp_validation is uv-locked with cosmo_numba declared, - # its published image can drift NumPy past numba's window (seen 2026-07-11: - # "Numba needs NumPy 2.4 or less. Got NumPy 2.5" at ngmix). PYTHONPATH - # shadows pure-Python code only, never binary deps. - sif: /n17data/cdaley/containers/sp_validation_im_sims.sif # extract/calibrate/m-bias - sif_pipeline: /n17data/cdaley/containers/shapepipe_im_sims-runtime.sif # pipeline/merge - # Bind mounts are no longer set per-run: each rule below carries a plain - # ``container: sif``/``container: sif_pipeline`` directive, and Snakemake - # wraps its ``shell:`` command in that image via the driving profile's - # ``apptainer-args`` (workflow/profiles/candide/config.yaml) -- one bind - # list for every containerized rule in the whole workflow, not a value - # threaded through image_sims config. + # --- container -------------------------------------------------------- + # One image for the whole chain: the sp_validation image is built FROM the + # ShapePipe image, so it carries both stacks. It must be built from the uv + # lock -- an unlocked build drifts NumPy past numba's ceiling and the ngmix + # stage dies ("Numba needs NumPy 2.4 or less"). + sif: /n17data/cdaley/containers/sp_validation_im_sims.sif # --- repositories ----------------------------------------------------- # Bound into the image; both repos' src go on PYTHONPATH so this branch's - # code wins over the baked copies: ShapePipe's #766 build, and sp_validation's - # image_sims.py / catalog.match_catalogs_radec. + # code wins over the baked copies. shapepipe_repo: /n17data/cdaley/unions/code/shapepipe sp_validation_repo: /n17data/cdaley/unions/code/sp_validation diff --git a/workflow/rules/image_sims.smk b/workflow/rules/image_sims.smk index d1fcf120..855987c3 100644 --- a/workflow/rules/image_sims.smk +++ b/workflow/rules/image_sims.smk @@ -1,46 +1,18 @@ """Image-simulation orchestration: raw SKiLLS sim images -> shear m/c bias. -This rule set drives the image-simulation validation chain end to end and is -the sp_validation-side half of the split described in +The sp_validation-side half of the split described in ``UNIONS-WL/MultiBand_ImSim#1``: ShapePipe turns the simulated tiles into -per-tile shape catalogues, then sp_validation merges, extracts, calibrates and -finally measures the multiplicative/additive shear bias. - -Two images, one prefix shape. Architecturally one image could run every -stage -- the sp_validation image is built ``FROM`` the ShapePipe image, so it -carries both stacks -- but the *published* sp_validation image's environment -is not yet trustworthy for the ShapePipe half: sp_validation has no lockfile -and does not declare its numba-bearing dependency (``cosmo_numba``), so -unpinned install layers can drift NumPy past numba's window (a 2026-07-11 -gate run hit exactly this: ``Numba needs NumPy 2.4 or less. Got NumPy 2.5`` -at the ngmix stage). PYTHONPATH shadowing covers pure-Python *code*, never -binary deps, so until sp_validation is uv-locked with its deps declared -(spun off as its own task), each half runs in its own repo's image -- the -same split the gate766 baseline ran: - -* ShapePipe stages -> ``pipeline`` (raw images -> per-tile cats) and ``merge`` - (``create_final_cat`` -> ``final_cat_{sim}.hdf5``) run in ``sif_pipeline`` - (the ShapePipe image). -* sp_validation stages -> ``manifest``, ``extract`` (-> comprehensive cat), - ``calibrate`` (-> cut cat) and ``m_bias`` (-> ``m_bias_results.yaml``) run - in ``sif`` (the sp_validation image). - -Every compute rule declares its own ``container: SIF`` or -``container: SIF_PIPELINE`` (a plain per-rule Snakemake directive -- no rule -shells out to ``apptainer`` itself); Snakemake wraps the rule's ``shell:`` -command in the right image via the driving profile's -``software-deployment-method: apptainer`` + ``apptainer-args`` (binds live -there now, not in this file). Two images, because the images are not the -workflow's top-level container. Everything is parameterised under -``config["image_sims"]`` -- the two ``sif`` keys, repository roots, data roots, -the PSF dictionary, the explicit ``tile_ids`` list and the sim/calibration -knobs -- so a fresh user drives it from config alone, with no hard-coded clone -layout. Configuration is fail-fast: a schema check at load rejects an unknown -key (typo) and a missing science key (see ``workflow/image_sims/config.yaml`` -for the operational/science split). The ``PYTHONPATH`` override injects both -repos' ``src`` so the *branch* source (ShapePipe's ``#766`` build; -sp_validation's ``image_sims.py``, ``catalog.match_catalogs_radec``) wins over -whatever is baked into the image. +per-tile shape catalogues (``pipeline``, ``merge``), then sp_validation +extracts, calibrates and measures the multiplicative/additive shear bias +(``manifest``, ``extract``, ``calibrate``, ``m_bias``). + +One image runs the whole chain: the sp_validation image is built ``FROM`` the +ShapePipe image, so it carries both stacks. Everything else is parameterised +under ``config["image_sims"]`` -- repository roots, data roots, the PSF +dictionary, the explicit ``tile_ids`` list and the sim/calibration knobs -- so +a fresh user drives it from config alone. Configuration is fail-fast: a schema +check at load rejects an unknown key (typo) and a missing science key (see +``workflow/image_sims/config.yaml`` for the operational/science split). The five simulations per grid are the reference ``1z2z`` (no input shear) plus the ``+/-`` shear pairs ``1p2z``/``1m2z`` (g1) and ``1z2p``/``1z2m`` (g2); the @@ -48,24 +20,15 @@ m-bias estimator matches each to the reference by RA/Dec. """ import os -from pathlib import Path IMSIM = config["image_sims"] # --- fail-fast schema check ---------------------------------------------- -# One home for every fact: the run config carries the science knobs, the -# workflow config.yaml carries the operational defaults, and *this* block is -# where a typo or a missing knob dies -- at DAG parse, before any compute. +# An unknown key under ``image_sims:`` is a hard error (typo protection); a +# missing key is a hard error naming it. Both fire at DAG parse, before compute. # -# Every key must be declared below. An unknown key under ``image_sims:`` is a -# hard error (typo protection); a missing *science* key is a hard error naming -# the key (no silent code default anywhere). Operational keys default in the -# workflow config.yaml and nowhere else: the .smk reads them as bare -# ``IMSIM[key]`` (never ``.get`` with a second literal), so their value comes -# from config.yaml alone -- the single home for an operational default. -# -# Science keys: required from the *run* config; no default in config.yaml (only -# a commented template line) and no default in code. These fix the estimator's +# Science keys: required from the *run* config; no default here or in +# config.yaml (only a commented template line). These fix the estimator's # scientific behaviour, so they must be stated per run, never inherited. _SCIENCE_KEYS = { "w_cols", @@ -75,14 +38,13 @@ _SCIENCE_KEYS = { "bootstrap_seed", "mask_config", } -# Deprecated science keys: accepted (so a pre-``w_cols`` run config still parses -# and the estimator's back-compat path runs) but not *required* -- our configs -# state ``w_cols``. Listed here only to keep them out of the unknown-key error. +# Deprecated but still accepted, so a pre-``w_cols`` run config keeps parsing +# into the estimator's back-compat path. _DEPRECATED_KEYS = { "w_col", } -# Operational keys: default (visibly) in the workflow config.yaml; the .smk -# reads them bare, so config.yaml is their one home. +# Operational keys: default in the workflow config.yaml; the .smk reads them +# bare (never ``.get`` with a literal), so config.yaml is their one home. _OPERATIONAL_KEYS = { "sims_type", "branches", @@ -96,7 +58,6 @@ _OPERATIONAL_KEYS = { # Structural keys: paths/identifiers the run must supply (no sensible default). _STRUCTURAL_KEYS = { "sif", - "sif_pipeline", "shapepipe_repo", "sp_validation_repo", "grids_base", @@ -130,16 +91,12 @@ if _missing_structural: f"{_missing_structural} -- set them in the run config" ) -# --- containers ----------------------------------------------------------- -# Two images (see module docstring): the ShapePipe image for the pipeline and -# merge stages, the sp_validation image for everything downstream. Collapse -# back to one image once sp_validation's env is lock-managed. Each compute -# rule below carries its own ``container: SIF`` / ``container: SIF_PIPELINE`` -# directive; the bind mounts these images need are the driving profile's -# ``apptainer-args`` (workflow/profiles/candide/config.yaml), not a value read -# from this config -- there is no per-rule bind string left to own. -SIF = IMSIM["sif"] # sp_validation stages -SIF_PIPELINE = IMSIM["sif_pipeline"] # ShapePipe stages +# --- container ------------------------------------------------------------ +# Every compute rule carries ``container: SIF`` rather than inheriting a +# module-level default: these rules are also included from the top-level +# workflow/Snakefile, whose module default is the cosmology image (no ShapePipe +# stack). Binds come from the driving profile's ``apptainer-args``. +SIF = IMSIM["sif"] # --- repositories (bound into the image; branch code overrides) ----------- SHAPEPIPE_REPO = IMSIM["shapepipe_repo"] @@ -187,42 +144,25 @@ CALIBRATE = IMSIM["calibrate_script"] COMPUTE_M_BIAS = f"{SPV_REPO}/scripts/compute_m_bias_image_sims.py" # --- in-command env prefix ------------------------------------------------- -# One prefix *shape* for every stage, now expressed as plain shell -# ``VAR=value`` / ``env -u`` syntax at the front of each rule's ``shell:`` -# string rather than as ``apptainer exec --env``/``-u`` flags -- Snakemake -# wraps the whole shell string inside the container (see each rule's -# ``container:``), so setting/unsetting the vars as the first shell tokens -# lands them exactly where the ``--env``/``-u`` flags used to, with no -# apptainer-specific mechanism required. Three things it does: +# Snakemake wraps each rule's whole ``shell:`` string inside the container, so +# these ``VAR=value`` tokens land inside it. Three settings: # -# * PYTHONPATH prepends BOTH repos' ``src`` (ShapePipe first, then -# sp_validation), so Python resolves the worktree build before -# ``/app``/``/sp_validation`` -- the local-testing counterpart of the -# git-ref deps, letting the branch code run without an image rebuild. This -# covers the Python *packages* only: the bash entry points (run_job) and -# the ShapePipe/sp_validation *scripts* are still invoked at the repo paths -# resolved from config (RUN_JOB, CREATE_FINAL_CAT, EXTRACT_INFO, ...), not -# shadowed by PYTHONPATH. +# * PYTHONPATH prepends both repos' ``src`` so Python resolves the worktree +# build ahead of the copies baked into the image -- the branch's code runs +# without an image rebuild. Packages only: the bash and python entry points +# are invoked at the repo paths from config (RUN_JOB, CREATE_FINAL_CAT, +# EXTRACT_INFO, ...), not shadowed by PYTHONPATH. # * PSF_DICT points the fake_psf module (PSF_DICT_PATH = $PSF_DICT, expanded # via getexpanded) at this run's PSF dictionary. -# * The SLURM env vars are stripped (``env -u ...``) so that when the -# ShapePipe pipeline stage's OpenMPI initialises inside the image it does -# not try to attach to the host SLURM launcher (cf. apptainer_noslurm.sh). -# The strip is harmless for the pure-Python sp_validation stages, so one -# prefix serves all. -# -# ``OMP_NUM_THREADS=1`` rides the same prefix, and not the SLURM profile, for -# the same reason as before: the chain is MPI-free (Snakemake fans out one job -# per branch x tile; parallelism inside a job is ShapePipe's own -# ``-N n_smp``), so the OpenMP/BLAS thread pool must be pinned to 1 to avoid -# oversubscription, and the slurm executor's ``--export=ALL`` only propagates -# the *driver's* ambient environment, not a value a profile could pin. Setting -# it here, inside the command every rule actually runs, keeps it committed and -# independent of both the driver's env and which container wraps the job. +# * OMP_NUM_THREADS=1 rides here rather than in the SLURM profile: the chain +# is MPI-free (Snakemake fans out one job per branch x tile; in-job +# parallelism is ShapePipe's own ``-N n_smp``), so the OpenMP/BLAS pool must +# be pinned to 1 to avoid oversubscription, and a profile can only set CLI +# flags, never the driver env the slurm executor's ``--export=ALL`` +# propagates. _ENV_PREFIX = ( f"PYTHONPATH={SHAPEPIPE_REPO}/src:{SPV_REPO}/src " f"PSF_DICT={PSF_DICT} OMP_NUM_THREADS=1 " - "env -u SLURM_JOBID -u SLURM_JOB_ID -u SLURM_PROCID " ) JOB_MASK = sum([1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]) @@ -352,22 +292,17 @@ rule im_init: rule im_pipeline: - """Run ShapePipe on one simulated tile (ShapePipe stage). + """Run ShapePipe on one simulated tile. Delegates the module DAG to ShapePipe's own job runner; the sentinel log - marks tile completion for the merge step. This is the compute-heavy, - MPI-bearing stage. + marks tile completion for the merge step. The compute-heavy stage. """ input: - # ``params.py`` is a *tracked* output of ``im_init``, so this one input - # supplies the im_init -> im_pipeline edge. The ``cfis`` symlink the - # shell reads (via {RUN_JOB}) is created by that same im_init shell block - # as an *untracked* side effect -- no rule declares it as an output - # (snakemake will not track a symlink/directory output). Declaring it an - # input here therefore asked the DAG for a file no rule produces: on a - # fresh grids_base it aborted the build with MissingInputException before - # any job ran. It is safe to drop -- cfis exists whenever params does, - # since im_init stages both together. + # params.py alone supplies the im_init -> im_pipeline edge. The `cfis` + # symlink {RUN_JOB} also reads is an untracked side effect of the same + # im_init shell (snakemake will not track a symlink output), so it must + # not be declared here -- doing so asks the DAG for a file no rule + # produces and aborts on a fresh grids_base. params=f"{GRIDS_BASE}/{{sim}}/params.py", output: done=touch(f"{GRIDS_BASE}/{{sim}}/logs/pipeline_{{tile}}.done"), @@ -379,7 +314,7 @@ rule im_pipeline: mem_mb=16000, runtime=720, container: - SIF_PIPELINE + SIF shell: "cd {params.run_dir} && " "{_ENV_PREFIX} bash {RUN_JOB} " @@ -403,7 +338,7 @@ rule im_merge: params: run_dir=lambda wc: f"{GRIDS_BASE}/{wc.sim}", container: - SIF_PIPELINE + SIF shell: "cd {params.run_dir} && " "{_ENV_PREFIX} python {CREATE_FINAL_CAT} " @@ -453,18 +388,8 @@ rule im_calibrate: rule im_mbias_config: - """Assemble ``m_bias_config.yaml`` -- the manifest's shear/branch facts, - this run's science knobs, and git/container provenance -- ahead of the - m-bias compute step. - - Pure host-side introspection (``git -C``, a plain-text scan of the SIFs' - OCI labels, PyYAML) -- no sp_validation/ShapePipe import, so it stays a - ``run:`` block with no container. Snakemake never containerizes ``run:`` - regardless of a rule's ``container:`` directive, which is exactly why this - step is split out of the compute rule below rather than left as a - ``run:`` block that shells out to ``{EXEC}`` at the end: a ``run:`` rule - can't carry the container the *compute* actually needs, but a ``shell:`` - rule can. + """Assemble ``m_bias_config.yaml`` for the m-bias step: the manifest's + shear/branch facts, this run's science knobs, and git/container provenance. """ input: manifest=MANIFEST, @@ -478,7 +403,6 @@ rule im_mbias_config: num=NUM, cat_name=f"shape_catalog_cut_{SHAPE}.fits", sif=SIF, - sif_pipeline=SIF_PIPELINE, shapepipe_repo=SHAPEPIPE_REPO, sp_validation_repo=SPV_REPO, results_dir=f"{GRIDS_BASE}/results", @@ -489,107 +413,17 @@ rule im_mbias_config: n_bootstrap=IMSIM["n_bootstrap"], pair_match=IMSIM["pair_match"], bootstrap_seed=IMSIM["bootstrap_seed"], - run: - import hashlib - import re - import subprocess - - import yaml - - with open(input.manifest) as fh: - manifest = yaml.safe_load(fh) - - def _git(repo, *args): - """Read a git fact from ``repo``; ``None`` if it is not a checkout.""" - try: - return subprocess.run( - ["git", "-C", repo, *args], - capture_output=True, - text=True, - check=True, - ).stdout.strip() - except (subprocess.CalledProcessError, FileNotFoundError): - return None - - def _sif_revision(sif_path): - """GHCR revision baked into the SIF's OCI labels. - - A plain-text scan of the image file (login-safe: no exec, no - container start), reading org.opencontainers.image.revision -- the - source commit GHCR built the image from. ``None`` if absent. - """ - try: - with open(sif_path, "rb") as fh: - blob = fh.read() - except OSError: - return None - m = re.search( - rb'org\.opencontainers\.image\.revision"?[:=]"?([0-9a-f]{7,40})', - blob, - ) - return m.group(1).decode() if m else None - - # Manifest hash: sha256 of the exact bytes im_manifest wrote, so the - # result records which injected-shear facts it was computed against. - with open(input.manifest, "rb") as fh: - manifest_sha256 = hashlib.sha256(fh.read()).hexdigest() - - provenance = { - "manifest_sha256": manifest_sha256, - "sp_validation": { - "branch": _git(params.sp_validation_repo, "rev-parse", "--abbrev-ref", "HEAD"), - "commit": _git(params.sp_validation_repo, "rev-parse", "HEAD"), - }, - "shapepipe": { - "branch": _git(params.shapepipe_repo, "rev-parse", "--abbrev-ref", "HEAD"), - "commit": _git(params.shapepipe_repo, "rev-parse", "HEAD"), - }, - "containers": { - "sif": params.sif, - "ghcr_revision": _sif_revision(params.sif), - "sif_pipeline": params.sif_pipeline, - "ghcr_revision_pipeline": _sif_revision(params.sif_pipeline), - }, - } - - os.makedirs(params.results_dir, exist_ok=True) - # Emit *every* key the estimator requires -- pair_match and - # bootstrap_seed included. Requiring a key without emitting it would - # be a KeyError at run time, so the generated config is the complete - # contract between rule and estimator. ``provenance`` rides along as a - # top-level block: the compute script copies it verbatim into the output - # results yaml, so a result file is self-describing (which manifest, - # which repo commits, which container built the number). - mbias_cfg = { - "grids_dir": params.grids_base, - "num": params.num, - "catalog_name": params.cat_name, - # Injected shear: from the manifest, the single source of truth. - "shear_amplitude": manifest["shear_amplitude"], - "branches": list(manifest["branches"]), - "pairs": manifest["pairs"], - "match_radius_deg": params.match_radius_deg, - "w_cols": list(params.w_cols), - "pair_match": params.pair_match, - "n_bootstrap": params.n_bootstrap, - "bootstrap_seed": params.bootstrap_seed, - "results_dir": params.results_dir, - "output_path": params.results, - "provenance": provenance, - } - with open(output.cfg, "w") as fh: - yaml.safe_dump(mbias_cfg, fh) + container: + SIF + script: + "../scripts/im_mbias_config.py" rule im_mbias: """Multiplicative/additive shear bias from the calibrated grids. - Produces the workflow's headline artifact, ``m_bias_results.yaml``, by - running the estimator against the config ``im_mbias_config`` assembled - (manifest's shear/branch facts, science knobs, provenance) -- the one - sp_validation-stage compute call in the chain, so it is the one place a - real ``container:``/``shell:`` split (rather than a ``run:`` block's - trailing ``shell()``) is required to containerize it at all. + Produces the workflow's headline artifact, ``m_bias_results.yaml``, running + the estimator against the config ``im_mbias_config`` assembled. """ input: cfg=f"{GRIDS_BASE}/results/m_bias_config.yaml", diff --git a/workflow/scripts/im_mbias_config.py b/workflow/scripts/im_mbias_config.py new file mode 100644 index 00000000..147ba8bb --- /dev/null +++ b/workflow/scripts/im_mbias_config.py @@ -0,0 +1,95 @@ +"""Assemble ``m_bias_config.yaml`` for the image-sims m-bias estimator. + +Combines the manifest's injected-shear facts, this run's science knobs and +git/container provenance into the single config ``compute_m_bias_image_sims.py`` +reads. ``provenance`` rides along as a top-level block: the estimator copies it +verbatim into its results yaml, so a result file records which manifest, repo +commits and container produced the number. + +`snakemake` is injected as a module global by Snakemake's `script:` preamble +before this file runs (`from snakemake.script import snakemake` is +IDE-hint-only and raises ImportError if actually executed). +""" + +import hashlib +import re +import subprocess + +import yaml + +params = snakemake.params # noqa: F821 +manifest_path = snakemake.input["manifest"] # noqa: F821 + + +def _git(repo, *args): + """Read a git fact from ``repo``; ``None`` if it is not a checkout.""" + try: + return subprocess.run( + ["git", "-C", repo, *args], + capture_output=True, + text=True, + check=True, + ).stdout.strip() + except (subprocess.CalledProcessError, FileNotFoundError): + return None + + +def _sif_revision(sif_path): + """``org.opencontainers.image.revision`` from the SIF's OCI labels. + + Plain-text scan of the image file -- no exec, no container start. + """ + try: + with open(sif_path, "rb") as fh: + blob = fh.read() + except OSError: + return None + m = re.search( + rb'org\.opencontainers\.image\.revision"?[:=]"?([0-9a-f]{7,40})', blob + ) + return m.group(1).decode() if m else None + + +with open(manifest_path) as fh: + manifest = yaml.safe_load(fh) +with open(manifest_path, "rb") as fh: + manifest_sha256 = hashlib.sha256(fh.read()).hexdigest() + +mbias_cfg = { + "grids_dir": params.grids_base, + "num": params.num, + "catalog_name": params.cat_name, + # Injected shear: from the manifest, the single source of truth. + "shear_amplitude": manifest["shear_amplitude"], + "branches": list(manifest["branches"]), + "pairs": manifest["pairs"], + "match_radius_deg": params.match_radius_deg, + "w_cols": list(params.w_cols), + "pair_match": params.pair_match, + "n_bootstrap": params.n_bootstrap, + "bootstrap_seed": params.bootstrap_seed, + "results_dir": params.results_dir, + "output_path": params.results, + "provenance": { + "manifest_sha256": manifest_sha256, + "sp_validation": { + "branch": _git( + params.sp_validation_repo, "rev-parse", "--abbrev-ref", "HEAD" + ), + "commit": _git(params.sp_validation_repo, "rev-parse", "HEAD"), + }, + "shapepipe": { + "branch": _git( + params.shapepipe_repo, "rev-parse", "--abbrev-ref", "HEAD" + ), + "commit": _git(params.shapepipe_repo, "rev-parse", "HEAD"), + }, + "container": { + "sif": params.sif, + "ghcr_revision": _sif_revision(params.sif), + }, + }, +} + +with open(snakemake.output["cfg"], "w") as fh: # noqa: F821 + yaml.safe_dump(mbias_cfg, fh) From 2176894aca467fb2f45fd439628141f9a4927b05 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 04:17:22 +0200 Subject: [PATCH 11/37] covariance: restore cosmocov_process.py as a containerized script: rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deleted in the #236 cleanup with no replacement; restored from history to workflow/scripts/ (the rule is its only caller) and converted the rule from shell: to script:. Fixes on the way: bare exit() on a non-PD matrix returned 0 (Snakemake saw success) — now sys.exit(1); eigvalsh for the symmetric matrix; Agg backend; plot dpi 2000 -> 300. Verified round-trip on synthetic input in the container. Drop the NOTE and stale comments. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- workflow/rules/covariance.smk | 24 +------- workflow/scripts/cosmocov_process.py | 89 ++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+), 21 deletions(-) create mode 100644 workflow/scripts/cosmocov_process.py diff --git a/workflow/rules/covariance.smk b/workflow/rules/covariance.smk index 1849f81a..46e21df5 100644 --- a/workflow/rules/covariance.smk +++ b/workflow/rules/covariance.smk @@ -11,7 +11,6 @@ def get_cat_params(version): # covariance_dir(), covariance_base(), covariance_path() defined in Snakefile -# Additional wildcard constraints defined locally for pseudo-Cl rules (line 327) # DEFAULT_MASK_SUFFIX defined in Snakefile # Footprint mask power spectra (nside=4096, from comprehensive catalog with spatial cuts only) @@ -240,11 +239,6 @@ rule generate_glass_mock_rhotau_samples: output_dir="results/glass_mock_rhotau_samples", threads: 1 shell: - # A CLI script, not `script:`: it takes a single `--mock-ids` range and - # this rule wants one call per mock_id wildcard, so `script:` (which - # only sees this one job's input/output) would need the same argparse - # rewritten as snakemake.* access for no behavior change. Left as a - # plain shell call. """ python {WORKFLOW_SCRIPTS}/generate_glass_mock_rhotau_samples.py \ --cov-tau {input.cov_tau} \ @@ -255,28 +249,16 @@ rule generate_glass_mock_rhotau_samples: rule covariance_process: - """Post-process a raw CosmoCov matrix into the analysis-ready form. - - NOTE (pre-existing, unrelated to containerization): the script this rule - calls, cosmo_inference/scripts/cosmocov_process.py, was deleted in the - cosmo_inference cleanup (#236) and was never restored. This rule -- on - the default path via fiducial_covariance_outputs() -- currently fails - with FileNotFoundError. Flagging here rather than silently working - around it; needs either restoring the script or rewriting this rule. - """ + """Post-process a raw CosmoCov matrix into the analysis-ready form.""" input: str(COSMO_INFERENCE / "data/covariance/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}.txt") output: matrix=str(COSMO_INFERENCE / "data/covariance/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}_processed.txt"), gaussian=str(COSMO_INFERENCE / "data/covariance/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}_processed_g.txt"), plot=str(COSMO_INFERENCE / "data/covariance/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}_processed_plot.pdf") - params: - output_stub=str(COSMO_INFERENCE / "data/covariance/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}/covariance_{version}_{blind}_{gaussian}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}_processed") threads: 1 - shell: - """ - python {COSMO_INFERENCE}/scripts/cosmocov_process.py {input} {params.output_stub} - """ + script: + "../scripts/cosmocov_process.py" def fiducial_covariance_outputs(mask_suffix=""): diff --git a/workflow/scripts/cosmocov_process.py b/workflow/scripts/cosmocov_process.py new file mode 100644 index 00000000..76ea7e0a --- /dev/null +++ b/workflow/scripts/cosmocov_process.py @@ -0,0 +1,89 @@ +"""Assemble a raw CosmoCov block dump into an analysis-ready covariance matrix. + +CosmoCov writes one row per (i, j) element with the Gaussian term in column 8 +and the non-Gaussian term in column 9; this rebuilds the symmetric matrices, +checks positive-definiteness, and plots the correlation matrix. +""" + +import sys + +import matplotlib + +matplotlib.use("Agg") + +import matplotlib.pyplot as plt # noqa: E402 +import numpy as np # noqa: E402 + + +def get_cov(filename): + """Return (gaussian, non-gaussian, ndata) from a CosmoCov element list.""" + + data = np.loadtxt(filename) + ndata = int(np.max(data[:, 0])) + 1 + + cov_g = np.zeros((ndata, ndata)) + cov_ng = np.zeros((ndata, ndata)) + for i in range(data.shape[0]): + row, col = int(data[i, 0]), int(data[i, 1]) + cov_g[row, col] = cov_g[col, row] = data[i, 8] + cov_ng[row, col] = cov_ng[col, row] = data[i, 9] + + return cov_g, cov_ng, ndata + + +def plot_correlation(cov, ndata, plot_path): + """Save the correlation matrix with xi+/xi- block annotations.""" + + diag = np.sqrt(np.diag(cov)) + correlation = cov / np.outer(diag, diag) + + fig, ax = plt.subplots() + extent = (0, ndata, ndata, 0) + image = ax.imshow(correlation, cmap="seismic", vmin=-1, vmax=1, extent=extent) + + ax.axvline(x=ndata // 2, color="black", linewidth=1.0) + ax.axhline(y=ndata // 2, color="black", linewidth=1.0) + + fig.colorbar(image, orientation="vertical") + + ax.text(ndata // 4, ndata + 5, r"$\xi_+^{ij}(\theta)$", fontsize=12) + ax.text(3 * (ndata // 4), ndata + 5, r"$\xi_-^{ij}(\theta)$", fontsize=12) + ax.text(-9, ndata // 4, r"$\xi_+^{ij}(\theta)$", fontsize=12) + ax.text(-9, 3 * (ndata // 4), r"$\xi_-^{ij}(\theta)$", fontsize=12) + + fig.savefig(plot_path, dpi=300) + plt.close(fig) + + +def main(covfile, matrix_path, gaussian_path, plot_path): + cov_g, cov_ng, ndata = get_cov(covfile) + print(f"Dimension of cov: {ndata}x{ndata}") + + cov = cov_g + cov_ng + + eigenvalues = np.linalg.eigvalsh(cov) + print(f"min+max eigenvalues cov: {eigenvalues.min():e}, {eigenvalues.max():e}") + if eigenvalues.min() <= 0.0: + sys.exit("non-positive eigenvalue encountered! Covariance invalid!") + + np.savetxt(matrix_path, cov) + np.savetxt(gaussian_path, cov_g) + plot_correlation(cov, ndata, plot_path) + + +if __name__ == "__main__": + try: + snakemake # noqa: F821 - injected by snakemake at runtime + except NameError: + if len(sys.argv) != 3: + print("Usage: python cosmocov_process.py ") + sys.exit(1) + stub = sys.argv[2] + main(sys.argv[1], f"{stub}.txt", f"{stub}_g.txt", f"{stub}_plot.pdf") + else: + main( + snakemake.input[0], # noqa: F821 + snakemake.output.matrix, # noqa: F821 + snakemake.output.gaussian, # noqa: F821 + snakemake.output.plot, # noqa: F821 + ) From 48c7720fe80905f1ced9b3f7e7c520d7a43bc6b4 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 04:17:22 +0200 Subject: [PATCH 12/37] tests: container_smoke becomes a real pytest, out of the main workflow MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rule asserted nothing and put a non-scientific artifact in every paper's results/. Now: tests/data/container_smoke/{Snakefile,script} driven by test_container_smoke.py (@slow, skipped off-cluster), which submits one tiny SLURM job through the committed candide profile and asserts APPTAINER_CONTAINER is set (the job really ran in the image), the editable install resolved, seed-42 eigh values match, and git works inside the container. OMP_NUM_THREADS is recorded, not asserted — unset is the profile's designed state. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- .../tests/data/container_smoke/Snakefile | 24 ++++ .../data/container_smoke}/container_smoke.py | 35 ++++-- .../tests/test_container_smoke.py | 106 ++++++++++++++++++ workflow/rules/container_smoke.smk | 19 ---- 4 files changed, 154 insertions(+), 30 deletions(-) create mode 100644 src/sp_validation/tests/data/container_smoke/Snakefile rename {workflow/scripts => src/sp_validation/tests/data/container_smoke}/container_smoke.py (58%) create mode 100644 src/sp_validation/tests/test_container_smoke.py delete mode 100644 workflow/rules/container_smoke.smk diff --git a/src/sp_validation/tests/data/container_smoke/Snakefile b/src/sp_validation/tests/data/container_smoke/Snakefile new file mode 100644 index 00000000..619a1aee --- /dev/null +++ b/src/sp_validation/tests/data/container_smoke/Snakefile @@ -0,0 +1,24 @@ +# Standalone workflow exercised by src/sp_validation/tests/test_container_smoke.py. +# +# The module-level `container:` below mirrors what every real workflow does +# (workflow/Snakefile line 11) -- Snakemake has no way to take a default image +# from a profile. Everything else under test arrives from the driving profile +# (workflow/profiles/candide): the slurm executor, `software-deployment-method: +# apptainer` that turns container wrapping on, and the `apptainer-args` binds. +# No rule-level `container:` and no `apptainer exec` shell call -- Snakemake +# wraps the job itself, and the test asserts APPTAINER_CONTAINER was visible +# inside the job to prove the wrapping actually happened. +# +# See container_smoke.py for what the job checks and why. + + +container: "/n17data/cdaley/containers/containers" + + +rule container_smoke: + output: + "results/container_smoke.yaml", + resources: + runtime=5, + script: + "container_smoke.py" diff --git a/workflow/scripts/container_smoke.py b/src/sp_validation/tests/data/container_smoke/container_smoke.py similarity index 58% rename from workflow/scripts/container_smoke.py rename to src/sp_validation/tests/data/container_smoke/container_smoke.py index cfafcf9d..98e5896f 100644 --- a/workflow/scripts/container_smoke.py +++ b/src/sp_validation/tests/data/container_smoke/container_smoke.py @@ -3,18 +3,22 @@ Cheap sanity check for the profile-driven-container pivot -- same executor (slurm), same software-deployment-method (apptainer), same apptainer-args binds, same container image every real rule uses. No rule-level `container:` -or `apptainer exec` anywhere here; Snakemake wraps the job entirely from the -profile. Three things it proves, each written to the output YAML: +or `apptainer exec` anywhere here; Snakemake wraps the job itself. Four things +it proves, each written to the output YAML: + * the job really ran inside the image (``APPTAINER_CONTAINER``, set by + apptainer itself -- without it the rest could all pass on the bare host); * the editable ``sp_validation`` install resolves on the container's PYTHONPATH (import provenance: file + version, not just import success); - * the numeric stack works and honours threading env (numpy eigh on a small - fixed matrix, plus OMP_NUM_THREADS as seen inside the job); + * the numeric stack works (numpy eigh on a small fixed matrix). The + ``OMP_NUM_THREADS`` the job sees is recorded but NOT asserted: the profile + deliberately leaves it unset, and rules needing it pinned set it themselves + (see the image_sims rules' env prefix), so "unset" here is correct; * which commit of this checkout is running (git rev-parse from inside the container -- proves /home is bound and usable, not just readable). -Run it directly with `snakemake ... container_smoke` before trusting the -pivot on real compute. +Driven by the co-located Snakefile; the assertions on the output YAML live in +src/sp_validation/tests/test_container_smoke.py (marked ``slow``, cluster only). """ import os @@ -24,10 +28,13 @@ import numpy as np import yaml -# `snakemake` is injected as a module global by Snakemake's `script:` preamble -# before this file runs; no import is needed (and `from snakemake.script -# import snakemake` is IDE-hint-only -- snakemake.script has no such runtime -# attribute and raises ImportError if actually executed). + +# --- the job is actually inside the image --------------------------------- +# apptainer sets APPTAINER_CONTAINER (path of the running image) in every +# process it starts, and it survives --cleanenv. Absent => ran on the bare host. +container_info = { + "apptainer_container": os.environ.get("APPTAINER_CONTAINER", "unset"), +} # --- editable install resolves inside the container ------------------------ import sp_validation @@ -50,7 +57,12 @@ } # --- provenance: what commit is actually running in the container --------- -repo_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +# src/sp_validation/tests/data/container_smoke/ -> repo root, five levels up. +# (This is the checkout the Snakefile came from, which is what we want to +# report; the editable install may well resolve to a *different* checkout.) +repo_dir = os.path.abspath( + os.path.join(os.path.dirname(os.path.abspath(__file__)), *([os.pardir] * 5)) +) try: commit = subprocess.run( ["git", "-C", repo_dir, "rev-parse", "HEAD"], @@ -71,6 +83,7 @@ with open(snakemake.output[0], "w") as f: yaml.safe_dump( { + "container": container_info, "sp_validation": sp_validation_info, "numeric": numeric_info, "provenance": provenance, diff --git a/src/sp_validation/tests/test_container_smoke.py b/src/sp_validation/tests/test_container_smoke.py new file mode 100644 index 00000000..236b5479 --- /dev/null +++ b/src/sp_validation/tests/test_container_smoke.py @@ -0,0 +1,106 @@ +"""Smoke test of the profile-driven containerized-SLURM path. + +Submits one real (tiny, 5-minute) SLURM job through the committed candide +profile. The executor, the apptainer deployment method and the bind mounts come +from that profile; the image is the module-level ``container:`` in the test +Snakefile, exactly as real workflows declare it. That contract is what's under +test, so this can only run on candide -- marked ``slow``, skipped elsewhere. + +The job writes a YAML report (see data/container_smoke/container_smoke.py); the +assertions below check what it reports. +""" + +import os +import re +import shutil +import subprocess +import tempfile +from pathlib import Path + +import numpy as np +import pytest +import yaml + +requires_cluster = pytest.mark.skipif( + not Path("/n17data/cdaley/unions").exists() or shutil.which("sbatch") is None, + reason="needs candide: /n17data and a SLURM submit host", +) + + +def _repo_root() -> Path: + for parent in Path(__file__).resolve().parents: + if (parent / "pyproject.toml").exists(): + return parent + raise RuntimeError("could not locate repo root (no pyproject.toml above test)") + + +def _reference_eigenvalues() -> np.ndarray: + """The same deterministic computation the job runs inside the container.""" + rng = np.random.default_rng(seed=42) + a = rng.standard_normal((8, 8)) + return np.linalg.eigh(a + a.T)[0] + + +@pytest.mark.slow +@requires_cluster +def test_container_smoke(): + repo_root = _repo_root() + workflow_dir = repo_root / "src/sp_validation/tests/data/container_smoke" + + # Not pytest's tmp_path: that lives in the login node's /tmp, which the + # compute node cannot see, so the job's output would "go missing". The + # workdir must be on a shared filesystem. + tmp_path = Path(tempfile.mkdtemp(prefix="container_smoke_", dir=Path.home())) + + env = os.environ | {"PYTHONNOUSERSITE": "1", "PYTHONUNBUFFERED": "1"} + result = subprocess.run( + [ + "snakemake", + "--profile", + str(repo_root / "workflow/profiles/candide"), + "-s", + str(workflow_dir / "Snakefile"), + "--directory", + str(tmp_path), + "--jobs", + "1", + "container_smoke", + ], + env=env, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + timeout=600, + check=False, + ) + assert result.returncode == 0, result.stdout + + report = yaml.safe_load((tmp_path / "results/container_smoke.yaml").read_text()) + + # The job ran inside the image, not on the bare host. Everything below would + # pass on the host too, so this is the assertion that makes them mean + # something: apptainer sets APPTAINER_CONTAINER in every process it starts. + assert report["container"]["apptainer_container"] != "unset", report["container"] + + # The install must resolve to an editable src/ checkout, not a site-packages + # copy. Note it need not be *this* checkout: the container's editable install + # points at the shared /n17data working tree, while the Snakefile under test + # is read from wherever the test runs. + module_file = Path(report["sp_validation"]["file"]) + assert module_file.parts[-3:] == ("src", "sp_validation", "__init__.py"), module_file + assert "site-packages" not in module_file.parts, module_file + + # The numeric stack agrees with the same computation run here. + np.testing.assert_allclose( + report["numeric"]["eigenvalues"], _reference_eigenvalues(), rtol=1e-10, atol=1e-12 + ) + + # numeric.omp_num_threads is recorded for observability but deliberately NOT + # asserted: the profile leaves OMP_NUM_THREADS unset by design, and rules + # that need it pinned set it themselves (image_sims' env prefix), so "unset" + # here is the correct state rather than a gap. + + # git worked inside the container, so /home is bound and usable. + assert re.fullmatch(r"[0-9a-f]{40}", report["provenance"]["commit"]), report["provenance"] + + shutil.rmtree(tmp_path) # keep only on failure, for post-mortem diff --git a/workflow/rules/container_smoke.smk b/workflow/rules/container_smoke.smk deleted file mode 100644 index eef00730..00000000 --- a/workflow/rules/container_smoke.smk +++ /dev/null @@ -1,19 +0,0 @@ -# Container smoke test -- validates the base container contract every -# composed workflow inherits: profile executor (slurm) + software-deployment -# -method (apptainer) + apptainer-args (binds), no rule-level `container:` -# override and no `apptainer exec` in the shell. See -# workflow/scripts/container_smoke.py for what it actually checks. -# -# Run standalone before trusting the pivot on real compute: -# -# snakemake --profile workflow/profiles/candide -s workflow/Snakefile \ -# --configfile container_smoke - - -rule container_smoke: - output: - "results/container_smoke.yaml", - resources: - runtime=5, - script: - "../scripts/container_smoke.py" From 0ab121d19e5068dc85ea5c9f7721b5bb03ac1c2b Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 14:59:12 +0200 Subject: [PATCH 13/37] workflow: locate shell-invoked scripts via workflow.source_path Replace the hand-rolled WORKFLOW_SCRIPTS constant with Snakemake's first-class mechanism, which the docs specifically prescribe because manual path construction breaks under module composition. The script goes in input: (not params:, which would cause spurious reruns), so it also becomes an honest dependency. Fixes a live bug on the way: papers/bmodes unblinding_ceremony called 'python workflow/scripts/unblinding_ceremony.py', which from the paper workdir resolves into the paper's own scripts/ dir -- stale since 091bba8 and only detectable at run time. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- papers/bmodes/rules/synthesis.smk | 5 +++-- workflow/common.py | 6 ------ workflow/rules/covariance.smk | 3 ++- workflow/rules/twopoint.smk | 4 +++- 4 files changed, 8 insertions(+), 10 deletions(-) diff --git a/papers/bmodes/rules/synthesis.smk b/papers/bmodes/rules/synthesis.smk index fb9208e1..7e4b17db 100644 --- a/papers/bmodes/rules/synthesis.smk +++ b/papers/bmodes/rules/synthesis.smk @@ -138,9 +138,10 @@ rule unblinding_ceremony: Via snakemake: snakemake unblinding_ceremony --config ceremony_blind=B --nolock Standalone: - app python workflow/scripts/unblinding_ceremony.py B + app python scripts/unblinding_ceremony.py B """ input: + script=workflow.source_path("../scripts/unblinding_ceremony.py"), xi_data=f"{_CEREMONY_COSMOSIS_DIR}/{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}/cosmosis_{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}.fits", pure_eb=f"results/paper_plots/intermediate/{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_pure_eb_semianalytic.npz", pseudo_cl=_pseudo_cl_path(FIDUCIAL_VERSION, blind=_CEREMONY_BLIND), @@ -159,7 +160,7 @@ rule unblinding_ceremony: bestfit_root_halofit_cell=f"{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_halofit_cell", bestfit_root_config=f"{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_10_80", shell: - "python workflow/scripts/unblinding_ceremony.py {params.blind} --chain-version {params.chain_version}" + "python {input.script} {params.blind} --chain-version {params.chain_version}" rule all_tapestry: diff --git a/workflow/common.py b/workflow/common.py index 6635e1f9..3df3413b 100644 --- a/workflow/common.py +++ b/workflow/common.py @@ -5,12 +5,6 @@ import re from pathlib import Path -# Absolute path to workflow/scripts/ for the few rules that must call a -# script from `shell:` (e.g. wrapped in `mpiexec`): a composing paper -# Snakefile runs with its own directory as the workdir, so a relative path -# would miss. -WORKFLOW_SCRIPTS = Path(__file__).resolve().parent / "scripts" - # Output roots are env-overridable so a reproduction run can write into a # fresh tree without clobbering (or silently reusing) prior products. COSMO_VAL = Path( diff --git a/workflow/rules/covariance.smk b/workflow/rules/covariance.smk index 46e21df5..7b58ec36 100644 --- a/workflow/rules/covariance.smk +++ b/workflow/rules/covariance.smk @@ -230,6 +230,7 @@ rule generate_glass_mock_rhotau_samples: Only tau is sampled; inference_prep_glass_mock uses real rho data. """ input: + script=workflow.source_path("../scripts/generate_glass_mock_rhotau_samples.py"), cov_tau=str(COSMO_VAL / f"rho_tau_stats/cov_tau_{FIDUCIAL['mock_version']}{fiducial_binning_suffix()}_th.npy"), ref_tau=str(COSMO_VAL / f"rho_tau_stats/tau_stats_{FIDUCIAL['mock_version']}{fiducial_binning_suffix()}.fits"), output: @@ -240,7 +241,7 @@ rule generate_glass_mock_rhotau_samples: threads: 1 shell: """ - python {WORKFLOW_SCRIPTS}/generate_glass_mock_rhotau_samples.py \ + python {input.script} \ --cov-tau {input.cov_tau} \ --ref-tau {input.ref_tau} \ --output-dir {params.output_dir} \ diff --git a/workflow/rules/twopoint.smk b/workflow/rules/twopoint.smk index 3f5820f5..5d061057 100644 --- a/workflow/rules/twopoint.smk +++ b/workflow/rules/twopoint.smk @@ -37,6 +37,8 @@ rule xi_highres: per-rank is therefore required, not a leftover of the old convention. """ container: None + input: + script=workflow.source_path("../scripts/run_2pcf_highres.py"), output: txt=str(COSMO_VAL / f"{FIDUCIAL['version']}_xi_minsep={FIDUCIAL['min_sep_int']}_maxsep={FIDUCIAL['max_sep_int']}_nbins=10000_npatch=1.txt"), xi_plus=str(COSMO_VAL / f"xi_plus_{FIDUCIAL['version']}_minsep={FIDUCIAL['min_sep_int']}_maxsep={FIDUCIAL['max_sep_int']}_nbins=10000_npatch=1.fits"), @@ -57,7 +59,7 @@ rule xi_highres: "--bind /home,/n09data,/n17data,/n23data1,/softs " "--env LD_LIBRARY_PATH=/softs/openmpi/5.0.5-slurm-CentOS8/lib " "/n17data/cdaley/containers/containers " - "python {WORKFLOW_SCRIPTS}/run_2pcf_highres.py" + "python {input.script}" rule rho_tau_stats: From 5b7287622afc87f30ea4e7a04187ef415b36b4cf Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 16:32:51 +0200 Subject: [PATCH 14/37] workflow: use script: for single-process rules, source_path only for MPI script: resolves relative to the .smk defining it, module composition included, so it defeats the paper-workdir trap without making scripts into input files -- and matches what the other 20+ rules already do. Only xi_highres keeps source_path, where script: is structurally impossible (snakemake would wrap the whole mpiexec line in one container). unblinding_ceremony was already written for script: -- its _config_from_snakemake was dead code because the rule invoked it via shell:, so it silently ran _config_from_cli, which re-derives paths from constants that no longer exist (a cosmo_val dir deleted from the referenced checkout, and a _PROJECT_ROOT off by one since the script moved to papers/bmodes/scripts/). The rule declared 6 inputs and 9 params while passing 2 on the command line. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- papers/bmodes/rules/synthesis.smk | 5 +- workflow/rules/covariance.smk | 11 +-- workflow/rules/twopoint.smk | 3 + .../generate_glass_mock_rhotau_samples.py | 73 +++++++++++-------- 4 files changed, 51 insertions(+), 41 deletions(-) diff --git a/papers/bmodes/rules/synthesis.smk b/papers/bmodes/rules/synthesis.smk index 7e4b17db..8c6f7e38 100644 --- a/papers/bmodes/rules/synthesis.smk +++ b/papers/bmodes/rules/synthesis.smk @@ -141,7 +141,6 @@ rule unblinding_ceremony: app python scripts/unblinding_ceremony.py B """ input: - script=workflow.source_path("../scripts/unblinding_ceremony.py"), xi_data=f"{_CEREMONY_COSMOSIS_DIR}/{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}/cosmosis_{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}.fits", pure_eb=f"results/paper_plots/intermediate/{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_pure_eb_semianalytic.npz", pseudo_cl=_pseudo_cl_path(FIDUCIAL_VERSION, blind=_CEREMONY_BLIND), @@ -159,8 +158,8 @@ rule unblinding_ceremony: bestfit_root_fid_cell=f"{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_fid_cell", bestfit_root_halofit_cell=f"{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_halofit_cell", bestfit_root_config=f"{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_10_80", - shell: - "python {input.script} {params.blind} --chain-version {params.chain_version}" + script: + "../scripts/unblinding_ceremony.py" rule all_tapestry: diff --git a/workflow/rules/covariance.smk b/workflow/rules/covariance.smk index 7b58ec36..9a78dd78 100644 --- a/workflow/rules/covariance.smk +++ b/workflow/rules/covariance.smk @@ -230,7 +230,6 @@ rule generate_glass_mock_rhotau_samples: Only tau is sampled; inference_prep_glass_mock uses real rho data. """ input: - script=workflow.source_path("../scripts/generate_glass_mock_rhotau_samples.py"), cov_tau=str(COSMO_VAL / f"rho_tau_stats/cov_tau_{FIDUCIAL['mock_version']}{fiducial_binning_suffix()}_th.npy"), ref_tau=str(COSMO_VAL / f"rho_tau_stats/tau_stats_{FIDUCIAL['mock_version']}{fiducial_binning_suffix()}.fits"), output: @@ -239,14 +238,8 @@ rule generate_glass_mock_rhotau_samples: mock_id="{mock_id}", output_dir="results/glass_mock_rhotau_samples", threads: 1 - shell: - """ - python {input.script} \ - --cov-tau {input.cov_tau} \ - --ref-tau {input.ref_tau} \ - --output-dir {params.output_dir} \ - --mock-ids {params.mock_id} - """ + script: + "../scripts/generate_glass_mock_rhotau_samples.py" rule covariance_process: diff --git a/workflow/rules/twopoint.smk b/workflow/rules/twopoint.smk index 5d061057..71e62925 100644 --- a/workflow/rules/twopoint.smk +++ b/workflow/rules/twopoint.smk @@ -35,6 +35,9 @@ rule xi_highres: spawned by SLURM/PMI on their own nodes, would land bare on the host. `container: None` plus an explicit `mpiexec -n N apptainer exec ...` per-rank is therefore required, not a leftover of the old convention. + Because this rule builds its own apptainer call, reaching the source-cache + copy of the script relies on our `--bind /home` rather than on Snakemake's + automatic mount. """ container: None input: diff --git a/workflow/scripts/generate_glass_mock_rhotau_samples.py b/workflow/scripts/generate_glass_mock_rhotau_samples.py index 63d99095..87b7d06c 100644 --- a/workflow/scripts/generate_glass_mock_rhotau_samples.py +++ b/workflow/scripts/generate_glass_mock_rhotau_samples.py @@ -116,6 +116,36 @@ def generate_samples_for_mock(mock_id, cov_tau, theta, ref_tau_header, output_di return f"Generated tau samples for mock {mock_id:05d}" +def run(cov_tau_path, ref_tau_path, output_dir, mock_ids): + """Generate sampled tau statistics for every mock in ``mock_ids``.""" + print("Loading tau covariance...") + cov_tau = np.load(cov_tau_path) + print(f" cov_tau shape: {cov_tau.shape}") + + print("Loading reference FITS...") + ref_tau_data, ref_tau_header = load_reference_fits(ref_tau_path) + theta = ref_tau_data["theta"] + print( + f" theta range: {theta.min():.3f} - {theta.max():.3f} arcmin, nbins: {len(theta)}" + ) + + print(f"Generating tau samples for {len(mock_ids)} mocks...") + for mock_id in mock_ids: + msg = generate_samples_for_mock( + mock_id, cov_tau, theta, ref_tau_header, output_dir + ) + print(msg) + print("Done!") + + +def parse_mock_ids(spec): + """Parse a mock-ID spec, either a single ID or an inclusive ``lo-hi`` range.""" + if "-" in spec: + lo, hi = spec.split("-") + return list(range(int(lo), int(hi) + 1)) + return [int(spec)] + + def main(): parser = argparse.ArgumentParser( description="Generate zero-mean tau samples for GLASS mocks" @@ -146,37 +176,22 @@ def main(): ) args = parser.parse_args() - # Load tau covariance and reference - print("Loading tau covariance...") - cov_tau = np.load(args.cov_tau) - print(f" cov_tau shape: {cov_tau.shape}") - - print("Loading reference FITS...") - ref_tau_data, ref_tau_header = load_reference_fits(args.ref_tau) - theta = ref_tau_data["theta"] - print( - f" theta range: {theta.min():.3f} - {theta.max():.3f} arcmin, nbins: {len(theta)}" - ) + run(args.cov_tau, args.ref_tau, args.output_dir, parse_mock_ids(args.mock_ids)) - # Parse mock ID range - mock_ids = ( - list( - range( - int(args.mock_ids.split("-")[0]), int(args.mock_ids.split("-")[1]) + 1 - ) - ) - if "-" in args.mock_ids - else [int(args.mock_ids)] - ) - print(f"Generating tau samples for {len(mock_ids)} mocks...") - for mock_id in mock_ids: - msg = generate_samples_for_mock( - mock_id, cov_tau, theta, ref_tau_header, args.output_dir - ) - print(msg) - print("Done!") +# ── Entry point dispatch ───────────────────────────────────────────────────── +try: + snakemake # injected by snakemake's script: directive +except NameError: + snakemake = None -if __name__ == "__main__": +if snakemake is not None: + run( + snakemake.input.cov_tau, + snakemake.input.ref_tau, + snakemake.params.output_dir, + parse_mock_ids(snakemake.params.mock_id), + ) +elif __name__ == "__main__": main() From f08aca0f5209b7e11b0de5f7e7120dab86063435 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 16:38:18 +0200 Subject: [PATCH 15/37] papers/bmodes: delete the unblinding ceremony Never used, and silently broken: the rule invoked the script via shell:, so the script's snakemake branch was dead code and its CLI branch re-derived paths from constants that no longer exist. Nothing imported it and no rule consumed its output. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- papers/bmodes/Snakefile | 2 +- papers/bmodes/rules/synthesis.smk | 36 +- papers/bmodes/scripts/unblinding_ceremony.py | 1389 ------------------ 3 files changed, 2 insertions(+), 1425 deletions(-) delete mode 100644 papers/bmodes/scripts/unblinding_ceremony.py diff --git a/papers/bmodes/Snakefile b/papers/bmodes/Snakefile index 0adf66a1..9ead91bd 100644 --- a/papers/bmodes/Snakefile +++ b/papers/bmodes/Snakefile @@ -4,7 +4,7 @@ configfile: "config/config.yaml" configfile: "/n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_val/cat_config.yaml" -container: "/n17data/cdaley/containers/containers" +container: "docker://ghcr.io/cosmostat/sp_validation:develop" envvars: "PYTHONUNBUFFERED", diff --git a/papers/bmodes/rules/synthesis.smk b/papers/bmodes/rules/synthesis.smk index 8c6f7e38..64b60770 100644 --- a/papers/bmodes/rules/synthesis.smk +++ b/papers/bmodes/rules/synthesis.smk @@ -39,7 +39,7 @@ def _claim_outputs(): # Paper Macros # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -localrules: xi_cosmology_paper, paper_macros, bmodes_paper_spec, all_tapestry, unblinding_ceremony +localrules: xi_cosmology_paper, paper_macros, bmodes_paper_spec, all_tapestry rule xi_cosmology_paper: """Spec for B-mode reporting in configuration-space paper (Goh et al.). @@ -128,40 +128,6 @@ rule bmodes_paper_spec: # Aggregate Targets # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -_CEREMONY_BLIND = config.get("ceremony_blind", "A") -_CEREMONY_CHAIN_ROOT = "/n09data/guerrini/output_chains" -_CEREMONY_COSMOSIS_DIR = "/home/guerrini/sp_validation/cosmo_inference/data" - -rule unblinding_ceremony: - """Generate unblinding ceremony figure sequence. - - Via snakemake: - snakemake unblinding_ceremony --config ceremony_blind=B --nolock - Standalone: - app python scripts/unblinding_ceremony.py B - """ - input: - xi_data=f"{_CEREMONY_COSMOSIS_DIR}/{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}/cosmosis_{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}.fits", - pure_eb=f"results/paper_plots/intermediate/{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_pure_eb_semianalytic.npz", - pseudo_cl=_pseudo_cl_path(FIDUCIAL_VERSION, blind=_CEREMONY_BLIND), - pseudo_cl_cov=_pseudo_cl_cov_path(FIDUCIAL_VERSION, blind=_CEREMONY_BLIND), - cosmosis_cell_fits=f"{_CEREMONY_COSMOSIS_DIR}/{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_fid/cosmosis_{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_fid_cell.fits", - bestfit_dir=f"{_CEREMONY_CHAIN_ROOT}/best_fit/{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_10_80/shear_xi_plus/theta.txt", - output: - evidence=f"{TAPESTRY_DIR}/unblinding_ceremony/evidence.json", - params: - blind=_CEREMONY_BLIND, - chain_version=FIDUCIAL_VERSION.replace("SP_", "").replace("_leak_corr", ""), - chain_prefix=FIDUCIAL_VERSION, - chain_root_dir=_CEREMONY_CHAIN_ROOT, - results_dir="results/unblinding", - bestfit_root_fid_cell=f"{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_fid_cell", - bestfit_root_halofit_cell=f"{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_halofit_cell", - bestfit_root_config=f"{FIDUCIAL_VERSION}_{_CEREMONY_BLIND}_10_80", - script: - "../scripts/unblinding_ceremony.py" - - rule all_tapestry: """Aggregate target for all claim evidence and paper outputs.""" input: diff --git a/papers/bmodes/scripts/unblinding_ceremony.py b/papers/bmodes/scripts/unblinding_ceremony.py deleted file mode 100644 index 61ab3a95..00000000 --- a/papers/bmodes/scripts/unblinding_ceremony.py +++ /dev/null @@ -1,1389 +0,0 @@ -#!/usr/bin/env python3 -"""Run the UNIONS unblinding ceremony figure sequence. - -Usage (standalone) ------------------- -app python workflow/scripts/unblinding_ceremony.py A -app python workflow/scripts/unblinding_ceremony.py A --output-dir /path/to/output - -Usage (snakemake) ------------------ -snakemake unblinding_ceremony --config ceremony_blind=A -""" - -import argparse -import json -import shutil -import sys -from dataclasses import dataclass -from datetime import datetime, timezone -from pathlib import Path - -_SCRIPT_DIR = Path(__file__).resolve().parent -if str(_SCRIPT_DIR) not in sys.path: - sys.path.insert(0, str(_SCRIPT_DIR)) - -import matplotlib.pyplot as plt -import numpy as np -from astropy.io import fits -from getdist import plots -from matplotlib import scale as mscale -from matplotlib.gridspec import GridSpec -from plotting_utils import PAPER_MPLSTYLE, SquareRootScale -from scipy.interpolate import interp1d - -# ── Plotting environment ──────────────────────────────────────────────────── -mscale.register_scale(SquareRootScale) -plt.style.use(PAPER_MPLSTYLE) -plt.rc("text", usetex=True) - - -# ── Data types ────────────────────────────────────────────────────────────── - - -@dataclass(frozen=True) -class ChainSpec: - root: str - label: str - color: str - base_dir: Path - alpha: float = 1.0 - - -FULL_PARAMS = [ - "OMEGA_M", - "ombh2", - "h0", - "n_s", - "SIGMA_8", - "s_8_input", - "logt_agn", - "a", - "m1", - "bias_1", -] -COSMO_PARAMS = ["OMEGA_M", "s_8_input", "SIGMA_8", "a"] - -MUTED_ALPHA = 0.25 - - -@dataclass -class CeremonyConfig: - """All paths and parameters needed by the ceremony, sourced from snakemake or CLI.""" - - blind: str - chain_version: str - chain_prefix: str - chain_root_dir: Path - external_root_dir: Path - results_dir: Path - evidence_dir: Path - - xi_data_path: Path - pure_eb_path: Path - pseudo_cl_path: Path - pseudo_cl_cov_path: Path - cosmosis_cell_fits: Path - bestfit_dir: Path - bestfit_root_fid_cell: str - bestfit_root_halofit_cell: str - bestfit_root_config: str - - -def _save_path(cfg: CeremonyConfig, index: int, slug: str) -> Path: - return cfg.results_dir / f"{index:02d}_{slug}.pdf" - - -# ── Chain loading (extracted from Sasha's notebooks) ──────────────────────── - - -def _load_xi_table(path: Path | str, nrows: int = 20) -> np.ndarray: - rows: list[np.ndarray] = [] - with Path(path).open("r", encoding="utf-8") as handle: - for line in handle: - stripped = line.strip() - if not stripped or stripped.startswith("#"): - continue - values = np.fromstring(stripped, sep=" ") - if values.size < 9: - continue - rows.append(values) - if len(rows) >= nrows: - break - if len(rows) < nrows: - raise ValueError( - f"Expected at least {nrows} xi rows in {path}, found {len(rows)}" - ) - return np.vstack(rows) - - -def ensure_getdist_chain(base_dir: Path, root: str) -> Path: - """MAKE PARAMNAMES FILE + READ CHAIN conversion (from notebooks).""" - chain_dir = base_dir / root - samples_path = chain_dir / f"samples_{root}.txt" - gd_samples_path = chain_dir / f"getdist_{root}.txt" - paramnames_path = chain_dir / f"getdist_{root}.paramnames" - - if not samples_path.exists(): - return gd_samples_path - - with samples_path.open("r", encoding="utf-8") as file: - params = file.readline()[1:].split("\t")[:-4] - - with paramnames_path.open("w", encoding="utf-8") as file: - for param in params: - if len(param.split("--")) > 1: - file.write(param.split("--")[1] + "\n") - else: - file.write(param.split("--")[0] + "\n") - - samples = np.loadtxt(samples_path) - if "nautilus" in root: - samples = np.column_stack( - (np.exp(samples[:, -3]), samples[:, -1] - samples[:, -2], samples[:, 0:-3]) - ) - else: - samples = np.column_stack((samples[:, -1], samples[:, -3], samples[:, 0:-4])) - np.savetxt(gd_samples_path, samples) - return gd_samples_path - - -def _build_plotter( - width_inch: float, - axes_fontsize: float, - axes_labelsize: float, - legend_fontsize: float, -): - g = plots.get_subplot_plotter(width_inch=width_inch) - g.settings.axes_fontsize = axes_fontsize - g.settings.axes_labelsize = axes_labelsize - g.settings.alpha_filled_add = 0.7 - g.settings.legend_fontsize = legend_fontsize - return g - - -def _set_param_labels(chain) -> None: - name_list = [ - "OMEGA_M", - "ombh2", - "h0", - "n_s", - "SIGMA_8", - "S_8", - "s_8_input", - "logt_agn", - "a", - "m1", - "bias_1", - ] - label_list = [ - r"\Omega_{\rm m}", - r"\omega_{\rm b} h^2", - r"h_0", - r"n_{\rm s}", - r"\sigma_8", - r"S_8", - r"S_8", - r"\log T_{\rm AGN}", - r"A_{\rm IA}", - r"m_1", - r"\Delta z_1", - ] - - param_names = chain.getParamNames() - for name, label in zip(name_list, label_list): - try: - param_names.parWithName(name).label = label - except Exception: - pass - - try: - param_names.parWithName("S_8") - except Exception: - try: - s8_input = chain.getParams().s_8_input - chain.addDerived(s8_input, name="S_8", label=r"S_8") - except Exception: - pass - - -def _adjust_paramname_chain( - chain, current_name: str, target_name: str, label: str -) -> None: - try: - param_names = chain.getParamNames() - par = param_names.parWithName(current_name) - par.label = label - par.name = target_name - chain.setParamNames(param_names) - except Exception: - pass - - -def _derive_parameter_s8(chain): - if "S_8" in chain.getParamNames().list(): - return chain - omega_m = chain.getParams().OMEGA_M - sigma_8 = chain.getParams().SIGMA_8 - s_8 = sigma_8 * (omega_m / 0.3) ** 0.5 - chain.addDerived(s_8, name="S_8", label=r"S_8") - return chain - - -def _harmonize_external_chain(chain, root: str) -> None: - if root in {"Planck18", "KiDS-1000", "HSC_Y3", "HSC_Y3_cell", "DES+KiDS", "DES_Y3"}: - _adjust_paramname_chain(chain, "omega_m", "OMEGA_M", r"\Omega_{\rm m}") - if root == "DES_Y3": - _derive_parameter_s8(chain) - - -def load_getdist_chains( - chain_specs: list[ChainSpec], - width_inch: float, - axes_fontsize: float, - axes_labelsize: float, - legend_fontsize: float, -): - g = _build_plotter( - width_inch=width_inch, - axes_fontsize=axes_fontsize, - axes_labelsize=axes_labelsize, - legend_fontsize=legend_fontsize, - ) - - chains = [] - for spec in chain_specs: - ensure_getdist_chain(spec.base_dir, spec.root) - chain = g.samples_for_root( - str(spec.base_dir / spec.root / f"getdist_{spec.root}"), - cache=False, - settings={"ignore_rows": 0, "smooth_scale_2D": 0.5, "smooth_scale_1D": 0.5}, - ) - _set_param_labels(chain) - if spec.base_dir.name == "ext_data" or spec.root in { - "Planck18", - "DES_Y3", - "KiDS-1000", - "DES+KiDS", - "HSC_Y3", - "HSC_Y3_cell", - }: - _harmonize_external_chain(chain, spec.root) - chains.append(chain) - - return g, chains - - -# ── Plot functions (extracted from Sasha's notebooks) ─────────────────────── - - -def plot_triangle( - chain_specs: list[ChainSpec], - param_names: list[str], - output_path: Path, - width_inch: float = 20.0, - axes_fontsize: float = 35.0, - axes_labelsize: float = 50.0, - legend_fontsize: float = 40.0, - legend_loc: str = "upper right", -) -> None: - """Extracted triangle_plot pattern from contour notebooks.""" - g, chains = load_getdist_chains( - chain_specs, width_inch, axes_fontsize, axes_labelsize, legend_fontsize - ) - - colours = [spec.color for spec in chain_specs] - linestyle = ["solid" for _ in chain_specs] - line_args = [dict(color=col, ls=ls) for col, ls in zip(colours, linestyle)] - - g.triangle_plot( - chains, - param_names, - legend_labels=[spec.label for spec in chain_specs], - line_args=line_args, - contour_colors=colours, - legend_loc=legend_loc, - filled=True, - ) - - output_path.parent.mkdir(parents=True, exist_ok=True) - g.export(str(output_path)) - plt.close(g.fig) - - -def plot_xipm_data_vector(xipm_path: str, output_path: Path) -> None: - """Extracted from 2D_cosmic_shear_paper_plots/corr_func.ipynb (xi+ / xi- blocks).""" - xipm = _load_xi_table(xipm_path, nrows=20) - theta = xipm[:, 1] - xip = xipm[:, 3] - xim = xipm[:, 4] - varxip = xipm[:, 7] - varxim = xipm[:, 8] - - fig, (ax1, ax2) = plt.subplots(ncols=2, nrows=1, figsize=(10, 4.5)) - - ax1.tick_params( - axis="both", - which="both", - direction="in", - length=6, - width=1, - top=True, - bottom=True, - left=True, - right=True, - ) - ax1.yaxis.minorticks_on() - ax1.plot( - theta, - xip * 1e4, - marker="o", - markersize=4, - ls="solid", - lw=1.8, - color="royalblue", - ) - ax1.fill_between( - theta, (xip - varxip) * 1e4, (xip + varxip) * 1e4, color="powderblue", alpha=0.7 - ) - ax1.text( - 0.85, - 0.88, - "1-1", - transform=ax1.transAxes, - bbox=dict(facecolor="white", edgecolor="black", boxstyle="round", pad=0.5), - ) - ax1.axvspan(0, 10, color="gray", alpha=0.3) - ax1.axvspan(150, 200, color="gray", alpha=0.3) - ax1.set_xscale("log") - ax1.set_xlabel(r"$\theta$ [arcmin]") - ax1.set_ylabel(r"$\xi_+\times 10^4$") - - ax2.tick_params( - axis="both", - which="both", - direction="in", - length=6, - width=1, - top=True, - bottom=True, - left=True, - right=True, - ) - ax2.yaxis.minorticks_on() - ax2.plot( - theta, - xim * 1e4, - marker="o", - markersize=4, - ls="solid", - lw=1.8, - color="orangered", - ) - ax2.fill_between( - theta, (xim - varxim) * 1e4, (xim + varxim) * 1e4, color="pink", alpha=0.7 - ) - ax2.text( - 0.85, - 0.88, - "1-1", - transform=ax2.transAxes, - bbox=dict(facecolor="white", edgecolor="black", boxstyle="round", pad=0.5), - ) - ax2.axvspan(0, 10, color="gray", alpha=0.3) - ax2.axvspan(150, 200, color="gray", alpha=0.3) - ax2.set_xscale("log") - ax2.set_xlabel(r"$\theta$ [arcmin]") - ax2.set_ylabel(r"$\xi_-\times 10^4$") - - fig.tight_layout() - output_path.parent.mkdir(parents=True, exist_ok=True) - fig.savefig(output_path, dpi=300, bbox_inches="tight") - plt.close(fig) - - -def plot_cell_ee_data_vector( - pseudo_cl_path: str, pseudo_cl_cov_path: str, output_path: Path -) -> None: - """Extracted from 2025_10_08_plot_data_vectors.py (EE panel logic).""" - cell = fits.getdata(pseudo_cl_path) - cov_cell = fits.open(pseudo_cl_cov_path) - - ell = cell["ell"] - cl_ee = cell["EE"] - cov_cl_ee = cov_cell["COVAR_EE_EE"].data - cov_cell.close() - - fig, ax0 = plt.subplots(ncols=1, nrows=1, figsize=(7, 5)) - ax0.errorbar( - ell, - cl_ee * ell, - yerr=np.sqrt(np.diag(cov_cl_ee)) * ell, - label=r"$C_\ell^{EE}$", - color="royalblue", - fmt="o", - capsize=2, - ) - - ax0.set_xscale("squareroot") - ax0.set_xticks(np.array([100, 400, 900, 1600])) - ax0.minorticks_on() - ax0.tick_params(axis="x", which="minor", length=2, width=0.8) - minor_ticks = [i * 10 for i in range(1, 10)] + [i * 100 for i in range(1, 21)] - ax0.set_xticks(minor_ticks, minor=True) - ax0.legend() - - ax0.set_xlabel(r"$\ell$") - ax0.set_ylabel(r"$\ell \, C_\ell^{EE}$") - - plt.tight_layout() - output_path.parent.mkdir(parents=True, exist_ok=True) - plt.savefig(output_path, dpi=300, bbox_inches="tight") - plt.close(fig) - - -def plot_xipm_bestfit_with_bmodes( - xi_data_path: Path | str, - pure_eb_data_path: Path, - bestfit_dir: Path | None, - output_path: Path, - scale_cut_xip: tuple[float, float] = (12.0, 83.0), - scale_cut_xim: tuple[float, float] = (12.0, 83.0), -) -> None: - """Extracted from 2D_cosmic_shear_paper_plots/workflow/scripts/plot_xi_bestfit.py. - - If bestfit_dir is None, plots data + B-modes only (no theory curves). - - xi_data_path can be: - - A CosmoSIS FITS file with XI_PLUS/XI_MINUS HDUs and COVMAT (preferred) - - A plain-text TreeCorr output table (legacy) - """ - xi_path = Path(xi_data_path) - if xi_path.suffix == ".fits": - xip_hdu = fits.getdata(str(xi_path), "XI_PLUS") - xim_hdu = fits.getdata(str(xi_path), "XI_MINUS") - cov = fits.getdata(str(xi_path), "COVMAT") - theta_data = xip_hdu["ANG"] - xip_data = xip_hdu["VALUE"] - xim_data = xim_hdu["VALUE"] - n = len(xip_data) - sigma_xip = np.sqrt(np.diag(cov[:n, :n])) - sigma_xim = np.sqrt(np.diag(cov[n : 2 * n, n : 2 * n])) - else: - data = _load_xi_table(xi_path, nrows=20) - theta_data = data[:, 1] - xip_data = data[:, 3] - xim_data = data[:, 4] - sigma_xip = data[:, 7] - sigma_xim = data[:, 8] - - eb_data = np.load(pure_eb_data_path) - theta_eb = eb_data["theta"] - xip_B = eb_data["xip_B"] - xim_B = eb_data["xim_B"] - cov_pure_eb = eb_data["cov_pure_eb"] - - nbins = len(theta_eb) - sigma_xip_B = np.sqrt( - np.diag(cov_pure_eb[2 * nbins : 3 * nbins, 2 * nbins : 3 * nbins]) - ) - sigma_xim_B = np.sqrt( - np.diag(cov_pure_eb[3 * nbins : 4 * nbins, 3 * nbins : 4 * nbins]) - ) - - min_sep, max_sep = 1.0, 250.0 - bin_edges = np.geomspace(min_sep, max_sep, nbins + 1) - bin_centers_nominal = np.sqrt(bin_edges[:-1] * bin_edges[1:]) - - def get_bin_edge_cuts(centers, edges, scale_cut): - mask = (centers >= scale_cut[0]) & (centers <= scale_cut[1]) - idx_first = np.where(mask)[0][0] - idx_last = np.where(mask)[0][-1] - return edges[idx_first], edges[idx_last + 1] - - edge_cut_xip = get_bin_edge_cuts(bin_centers_nominal, bin_edges, scale_cut_xip) - edge_cut_xim = get_bin_edge_cuts(bin_centers_nominal, bin_edges, scale_cut_xim) - - has_theory = bestfit_dir is not None - if has_theory: - theta_theory_rad = np.loadtxt( - bestfit_dir / "shear_xi_plus" / "theta.txt", comments="#" - ) - theta_theory = np.rad2deg(theta_theory_rad) * 60 - xip_theory = np.loadtxt( - bestfit_dir / "shear_xi_plus" / "bin_1_1.txt", comments="#" - ) - xim_theory = np.loadtxt( - bestfit_dir / "shear_xi_minus" / "bin_1_1.txt", comments="#" - ) - - theta_sys_rad = np.loadtxt(bestfit_dir / "xi_sys" / "theta.txt", comments="#") - theta_sys = np.rad2deg(theta_sys_rad) * 60 - xip_sys = np.loadtxt(bestfit_dir / "xi_sys" / "shear_xi_plus.txt", comments="#") - xim_sys = np.loadtxt( - bestfit_dir / "xi_sys" / "shear_xi_minus.txt", comments="#" - ) - - theta_fine = np.geomspace(0.5, 300, 500) - if has_theory: - xip_th_interp = interp1d( - theta_theory, xip_theory, kind="cubic", fill_value="extrapolate" - )(theta_fine) - xim_th_interp = interp1d( - theta_theory, xim_theory, kind="cubic", fill_value="extrapolate" - )(theta_fine) - xip_sys_interp = interp1d( - theta_sys, xip_sys, kind="cubic", fill_value="extrapolate" - )(theta_fine) - xim_sys_interp = interp1d( - theta_sys, xim_sys, kind="cubic", fill_value="extrapolate" - )(theta_fine) - - scale_factor = 1e-4 - xlim = [1, 250] - ylim = [-0.15, 1.25] - - ms_data = 3 - ms_bmode = 3 - capsize = 1.5 - elinewidth = 0.8 - - fig, axes = plt.subplots(1, 2, figsize=(10, 4.5), sharey=True) - - plot_configs = [ - ( - axes[0], - xip_data, - sigma_xip, - xip_B, - sigma_xip_B, - xip_th_interp if has_theory else None, - xip_sys_interp if has_theory else None, - edge_cut_xip, - r"$\xi_+$", - "+", - ), - ( - axes[1], - xim_data, - sigma_xim, - xim_B, - sigma_xim_B, - xim_th_interp if has_theory else None, - xim_sys_interp if has_theory else None, - edge_cut_xim, - r"$\xi_-$", - "-", - ), - ] - - for idx, ( - ax, - xi_data_arr, - sigma_xi, - xi_B, - sigma_B, - xi_th, - xi_sys_arr, - edge_cut, - label, - _pm, - ) in enumerate(plot_configs): - show_legend = idx == 1 - - ax.axvspan(xlim[0], edge_cut[0], color="0.90", zorder=0, alpha=0.7) - ax.axvspan(edge_cut[1], xlim[1], color="0.90", zorder=0, alpha=0.7) - - if has_theory: - ax.plot( - theta_fine, - theta_fine * (xi_th + xi_sys_arr) / scale_factor, - "-", - color="k", - lw=1.5, - label=r"Best-fit $\xi^{\mathrm{th}}_\pm + \xi^{\mathrm{sys}}_\pm$" - if show_legend - else None, - zorder=2, - ) - ax.plot( - theta_fine, - theta_fine * xi_sys_arr / scale_factor, - "-", - color="C0", - lw=1.2, - label=r"Best-fit $\xi^{\mathrm{sys}}_\pm$" if show_legend else None, - zorder=2, - ) - - ax.errorbar( - theta_data, - theta_data * xi_data_arr / scale_factor, - yerr=theta_data * sigma_xi / scale_factor, - fmt="o", - color="k", - markersize=ms_data, - capsize=capsize, - elinewidth=elinewidth, - label=r"$\xi_\pm$" if show_legend else None, - zorder=3, - ) - - theta_eb_offset = theta_eb * 1.03 - ax.errorbar( - theta_eb_offset, - theta_eb_offset * xi_B / scale_factor, - yerr=theta_eb_offset * sigma_B / scale_factor, - fmt="o", - color="C3", - markersize=ms_bmode, - capsize=capsize, - elinewidth=elinewidth, - alpha=0.85, - label=r"$\xi^B_\pm$" if show_legend else None, - zorder=3, - ) - - ax.axhline(0, color="gray", linestyle="--", alpha=0.8, linewidth=0.8, zorder=1) - ax.set_xscale("log") - ax.set_xlim(xlim) - ax.set_ylim(ylim) - ax.set_xlabel(r"$\theta$ (arcmin)") - ax.set_title(label) - if show_legend: - ax.legend(loc="upper left") - - axes[0].set_ylabel(r"$\theta\xi \times 10^4$") - - fig.tight_layout() - output_path.parent.mkdir(parents=True, exist_ok=True) - fig.savefig(output_path, dpi=150, bbox_inches="tight") - plt.close(fig) - - -def _get_stats_row(chain, label: str, color: str) -> list[str | float]: - margestats = chain.getMargeStats() - try: - s8_stats = margestats.parWithName("S_8") - except Exception: - s8_stats = margestats.parWithName("s_8_input") - sigma8_stats = margestats.parWithName("SIGMA_8") - omegam_stats = margestats.parWithName("OMEGA_M") - return [ - label, - color, - s8_stats.mean, - s8_stats.mean - s8_stats.limits[0].lower, - s8_stats.limits[0].upper - s8_stats.mean, - sigma8_stats.mean, - sigma8_stats.mean - sigma8_stats.limits[0].lower, - sigma8_stats.limits[0].upper - sigma8_stats.mean, - omegam_stats.mean, - omegam_stats.mean - omegam_stats.limits[0].lower, - omegam_stats.limits[0].upper - omegam_stats.mean, - ] - - -def get_sigma_tension(mean1, low1, high1, mean2, low2, high2): - sigma1 = 0.5 * (high1 + low1) - sigma2 = 0.5 * (high2 + low2) - delta_mean = np.abs(mean1 - mean2) - sigma_tension = delta_mean / np.sqrt(sigma1**2 + sigma2**2) - sign = 1 if mean1 > mean2 else -1 - return sigma_tension * sign - - -def plot_cell_ee_with_bestfit( - cosmosis_data_path: str, - bestfit_specs: list[tuple[str, str, dict]], - output_folder: str, - output_path: Path, - ell_min: float = 10.0, - ell_max: float = 2048.0, - label_data: str = "Fiducial data", -) -> None: - """Extracted from get_chi2_cell.ipynb plot_best_fit() (cell 21-22). - - Parameters - ---------- - cosmosis_data_path : str - Path to CosmoSIS FITS file with CELL_EE and COVMAT HDUs. - bestfit_specs : list of (label, root, line_args_dict) - Each entry is (legend label, chain root name, dict of plot kwargs). - output_folder : str - Base path to best_fit directories (e.g. /n09data/guerrini/output_chains/). - output_path : Path - Where to save the figure. - """ - data = fits.getdata(cosmosis_data_path, "CELL_EE") - cov_mat = fits.getdata(cosmosis_data_path, "COVMAT") - - fig, ax = plt.subplots(1, 1, figsize=(8, 5)) - - ell = data["ANG"] - cell = data["VALUE"] - ax.errorbar( - ell, - ell * cell, - yerr=ell * np.sqrt(np.diag(cov_mat)), - fmt="o", - label=label_data, - color="black", - capsize=2, - ) - - for label, root, line_kw in bestfit_specs: - ell_th = np.loadtxt(f"{output_folder}/best_fit/{root}/shear_cl/ell.txt") - shear_cl = np.loadtxt(f"{output_folder}/best_fit/{root}/shear_cl/bin_1_1.txt") - mask = (ell_th > ell_min) & (ell_th < ell_max) - ax.plot(ell_th[mask], ell_th[mask] * shear_cl[mask], label=label, **line_kw) - - ax.axvline(x=1800, color="black", linestyle="--", alpha=0.5) - ax.axvline(x=2048, color="black", linestyle="--", alpha=1.0) - ax.axvline(x=500, color="black", linestyle="--", alpha=0.3) - - ax.text( - 1740, - 0.90, - r"$k_\mathrm{max} = 3 h$ Mpc$^{-1}$", - transform=ax.get_xaxis_transform(), - ha="center", - va="top", - fontsize=10, - rotation=90, - ) - ax.text( - 1978, - 0.90, - r"$k_\mathrm{max} = 5 h$ Mpc$^{-1}$", - transform=ax.get_xaxis_transform(), - ha="center", - va="top", - fontsize=10, - rotation=90, - ) - ax.text( - 470, - 0.90, - r"$k_\mathrm{max} = 1 h$ Mpc$^{-1}$", - transform=ax.get_xaxis_transform(), - ha="center", - va="top", - fontsize=10, - rotation=90, - ) - - ax.set_ylabel(r"$\ell C_\ell$", fontsize=16) - ax.set_xlabel(r"$\ell$", fontsize=16) - ax.set_xlim(ell.min() - 10, ell.max() + 100) - ax.set_xscale("squareroot") - ax.set_xticks(np.array([100, 400, 900, 1600])) - ax.minorticks_on() - ax.tick_params(axis="x", which="minor", length=2, width=0.8) - minor_ticks = [i * 10 for i in range(1, 10)] + [i * 100 for i in range(1, 21)] - ax.xaxis.set_ticks(minor_ticks, minor=True) - ax.tick_params(axis="both", which="major", labelsize=14) - ax.tick_params(axis="both", which="minor", labelsize=10) - ax.yaxis.get_offset_text().set_fontsize(14) - - plt.legend(loc="lower center", bbox_to_anchor=(0.685, 0.70), fontsize=12) - - output_path.parent.mkdir(parents=True, exist_ok=True) - plt.savefig(output_path, bbox_inches="tight") - plt.close(fig) - - -def plot_s8_whisker( - chain_specs: list[ChainSpec], - output_path: Path, - reference_labels: list[str] | None = None, - reference_colors: list[str] | None = None, - reference_label: str | None = None, -) -> None: - """Extracted from 2025_10_28_plot_whisker.ipynb (cells 5-11). - - Supports multiple reference bands: pass reference_labels and - reference_colors as parallel lists. Each reference gets its own - shaded band in the corresponding color. For backwards compatibility, - a single reference_label still works. - """ - if reference_labels is None and reference_label is not None: - reference_labels = [reference_label] - reference_colors = reference_colors or [None] - - g, chains = load_getdist_chains( - chain_specs, - width_inch=30, - axes_fontsize=60, - axes_labelsize=60, - legend_fontsize=60, - ) - plt.close(g.fig) - - labels = [spec.label for spec in chain_specs] - colours = [spec.color for spec in chain_specs] - alphas = [spec.alpha for spec in chain_specs] - - param_values = np.array( - [ - [ - "# Expt", - "Colour", - "S8_Mean", - "S8_low", - "S8_high", - "sigma_8_Mean", - "sigma_8_low", - "sigma_8_high", - "Omega_m_Mean", - "Omega_m_low", - "Omega_m_high", - ] - ], - dtype=object, - ) - - escaped_labels = np.char.replace(np.array(labels), "\\", "\\\\") - for i, chain in enumerate(chains): - row = _get_stats_row(chain, escaped_labels[i], colours[i]) - param_values = np.vstack((param_values, row)) - - expt = np.char.replace(param_values[1:, 0].astype(str), "\\\\", "\\") - colours_arr = param_values[1:, 1].astype(str) - s8_mean = param_values[1:, 2].astype(np.float64) - s8_low = param_values[1:, 3].astype(np.float64) - s8_high = param_values[1:, 4].astype(np.float64) - sigma8_mean = param_values[1:, 5].astype(np.float64) - sigma8_low = param_values[1:, 6].astype(np.float64) - sigma8_high = param_values[1:, 7].astype(np.float64) - omegam_mean = param_values[1:, 8].astype(np.float64) - omegam_low = param_values[1:, 9].astype(np.float64) - omegam_high = param_values[1:, 10].astype(np.float64) - - ref_indices = [] - for rl in reference_labels or []: - matches = np.where(expt == rl)[0] - if len(matches): - ref_indices.append(matches[0]) - ref_label_set = set(reference_labels or []) - - n_rows = len(expt) - fig_height = max(6, 0.5 * n_rows + 1) - fig = plt.figure(figsize=(10, fig_height)) - gs = GridSpec(1, 3, width_ratios=[1, 0.5, 0.5]) - ax1 = fig.add_subplot(gs[0]) - ax2 = fig.add_subplot(gs[1], sharey=ax1) - ax3 = fig.add_subplot(gs[2], sharey=ax1) - - axs = [ax1, ax2, ax3] - - params = [ - (s8_mean, s8_low, s8_high, r"$S_8$"), - (sigma8_mean, sigma8_low, sigma8_high, r"$\sigma_8$"), - (omegam_mean, omegam_low, omegam_high, r"$\Omega_{\rm m}$"), - ] - - row_spacing = 0.1 - y = np.arange(len(expt)) - - for ax, param in zip(axs, params): - means, lows, highs, label = param - for i, mean, low, high, color, alpha in zip( - y, means, lows, highs, colours_arr, alphas - ): - ax.errorbar( - mean, - 0.05 + i * row_spacing, - xerr=np.array([low, high])[:, None], - fmt="o", - color=color, - ecolor=color, - elinewidth=2, - capsize=3, - alpha=alpha, - ) - ax.set_xlabel(label, fontsize=14) - - for ri, ref_idx in enumerate(ref_indices): - band_color = ( - reference_colors[ri] - if reference_colors and ri < len(reference_colors) - else colours_arr[ref_idx] - ) or colours_arr[ref_idx] - ax.axvspan( - means[ref_idx] - lows[ref_idx], - means[ref_idx] + highs[ref_idx], - color=band_color, - alpha=0.15, - zorder=0, - ) - - ax.grid(False) - ax.tick_params(axis="y", left=False, labelleft=False) - if label == r"$S_8$": - ax.set_xlim(0.25, 1.05) - elif label == r"$\sigma_8$": - ax.set_xlim(0.5, 1.2) - elif label == r"$\Omega_{\rm m}$": - ax.set_xlim(0.1, 0.5) - - axs[0].set_yticks(0.05 + y * row_spacing) - axs[0].set_yticklabels([]) - for label, color, alpha in zip(expt, colours_arr, alphas): - idx = np.where(expt == label)[0][0] - yloc = 0.05 + row_spacing * idx - axs[0].text( - 0.26, - yloc, - label, - fontsize=12, - ha="left", - va="center", - color=color, - alpha=alpha, - ) - if label not in ref_label_set and ref_indices: - ri0 = ref_indices[0] - s8_tension = get_sigma_tension( - s8_mean[idx], - s8_low[idx], - s8_high[idx], - s8_mean[ri0], - s8_low[ri0], - s8_high[ri0], - ) - sign_str = "+" if s8_tension > 0 else "-" - axs[0].text( - 1.045, - yloc, - rf"${sign_str}{np.abs(s8_tension):.2f}" + r"\, \sigma$", - fontsize=10, - ha="right", - va="center", - color=color, - alpha=alpha, - ) - - plt.gca().invert_yaxis() - plt.tight_layout() - - output_path.parent.mkdir(parents=True, exist_ok=True) - plt.savefig(output_path, dpi=300, bbox_inches="tight") - plt.close(fig) - - -# ── Snakemake entry ────────────────────────────────────────────────────────── - - -def _config_from_snakemake(smk) -> CeremonyConfig: - """Build config from snakemake.input / snakemake.output / snakemake.params.""" - chain_root_dir = Path(smk.params.chain_root_dir) - - return CeremonyConfig( - blind=smk.params.blind, - chain_version=smk.params.chain_version, - chain_prefix=smk.params.chain_prefix, - chain_root_dir=chain_root_dir, - external_root_dir=chain_root_dir / "ext_data", - results_dir=Path(smk.params.results_dir), - evidence_dir=Path(smk.output.evidence).parent, - xi_data_path=Path(smk.input.xi_data), - pure_eb_path=Path(smk.input.pure_eb), - pseudo_cl_path=Path(smk.input.pseudo_cl), - pseudo_cl_cov_path=Path(smk.input.pseudo_cl_cov), - cosmosis_cell_fits=Path(smk.input.cosmosis_cell_fits), - bestfit_dir=Path(smk.input.bestfit_dir).parent.parent, - bestfit_root_fid_cell=smk.params.bestfit_root_fid_cell, - bestfit_root_halofit_cell=smk.params.bestfit_root_halofit_cell, - bestfit_root_config=smk.params.bestfit_root_config, - ) - - -# ── CLI entry ──────────────────────────────────────────────────────────────── - -_CHAIN_ROOT_DIR = Path("/n09data/guerrini/output_chains") -_COSMOSIS_DATA_DIR = Path("/home/guerrini/sp_validation/cosmo_inference/data") -_DEFAULT_CHAIN_VERSION = "v1.4.6" - - -def _require_path(path: Path, label: str) -> Path: - if path.exists(): - return path - raise FileNotFoundError(f"{label}: {path}") - - -def _require_bestfit_root(chain_root_dir: Path, root: str) -> str: - if (chain_root_dir / "best_fit" / root / "shear_cl" / "ell.txt").exists(): - return root - raise FileNotFoundError(f"No best-fit shear_cl for {root}") - - -def _config_from_cli() -> CeremonyConfig: - """Build config from command-line arguments + path resolution. - - Uses the exact data vectors from inference (Lisa's xi_pm, Sasha's pseudo-Cl - and CosmoSIS FITS) — the same files the chains were fit to. - """ - _PROJECT_ROOT = _SCRIPT_DIR.parent.parent - - parser = argparse.ArgumentParser( - description="Run the UNIONS unblinding ceremony plot sequence." - ) - parser.add_argument("blind", choices=["A", "B", "C"], help="Revealed blind letter") - parser.add_argument( - "--chain-version", - default=_DEFAULT_CHAIN_VERSION, - help="Chain version (default: %(default)s)", - ) - parser.add_argument( - "--output-dir", - type=Path, - default=None, - help="Output directory for results (default: /results/unblinding)", - ) - args = parser.parse_args() - - blind = args.blind - chain_version = args.chain_version - chain_prefix = f"SP_{chain_version}_leak_corr" - output_dir = args.output_dir or (_PROJECT_ROOT / "results" / "unblinding") - - xi_data_path = _require_path( - _COSMOSIS_DATA_DIR - / f"{chain_prefix}_{blind}" - / f"cosmosis_{chain_prefix}_{blind}.fits", - f"CosmoSIS xi FITS for blind {blind}", - ) - - pseudo_cl_path = _require_path( - Path( - f"/home/guerrini/sp_validation/cosmo_val/output/pseudo_cl_{chain_prefix}.fits" - ), - f"pseudo-Cl for {chain_prefix} (Guerrini)", - ) - pseudo_cl_cov_path = _require_path( - Path( - f"/home/guerrini/sp_validation/cosmo_val/output/pseudo_cl_cov_{chain_prefix}.fits" - ), - f"pseudo-Cl covariance for {chain_prefix} (Guerrini)", - ) - - cosmosis_cell_fits = _require_path( - _COSMOSIS_DATA_DIR - / f"{chain_prefix}_{blind}_fid" - / f"cosmosis_{chain_prefix}_{blind}_fid_cell.fits", - f"CosmoSIS C_ell FITS for blind {blind}", - ) - - bestfit_root_config = f"{chain_prefix}_{blind}_10_80" - bestfit_dir = _require_path( - _CHAIN_ROOT_DIR / "best_fit" / bestfit_root_config, - f"Best-fit directory for blind {blind}", - ) - - return CeremonyConfig( - blind=blind, - chain_version=chain_version, - chain_prefix=chain_prefix, - chain_root_dir=_CHAIN_ROOT_DIR, - external_root_dir=_CHAIN_ROOT_DIR / "ext_data", - results_dir=output_dir, - evidence_dir=output_dir / "claims" / "unblinding_ceremony", - xi_data_path=xi_data_path, - pure_eb_path=_require_path( - _PROJECT_ROOT - / "results" - / "paper_plots" - / "intermediate" - / f"{chain_prefix}_{blind}_pure_eb_semianalytic.npz", - f"Pure E/B file for blind {blind}", - ), - pseudo_cl_path=pseudo_cl_path, - pseudo_cl_cov_path=pseudo_cl_cov_path, - cosmosis_cell_fits=cosmosis_cell_fits, - bestfit_dir=bestfit_dir, - bestfit_root_fid_cell=_require_bestfit_root( - _CHAIN_ROOT_DIR, f"{chain_prefix}_{blind}_fid_cell" - ), - bestfit_root_halofit_cell=_require_bestfit_root( - _CHAIN_ROOT_DIR, f"{chain_prefix}_{blind}_halofit_cell" - ), - bestfit_root_config=bestfit_root_config, - ) - - -# ── Main ceremony ──────────────────────────────────────────────────────────── - - -def run_ceremony(cfg: CeremonyConfig) -> None: - blind = cfg.blind - cfg.results_dir.mkdir(parents=True, exist_ok=True) - - reveal_harmonic_root = f"{cfg.chain_prefix}_{blind}_lmin=300_lmax=1600_cell" - reveal_config_root = f"{cfg.chain_prefix}_{blind}_10_80" - - # ── Act 1 — The Data (naked, then with fits) ────────────────────── - - # 01: xi+/- data + B-modes, no theory - plot_xipm_bestfit_with_bmodes( - xi_data_path=cfg.xi_data_path, - pure_eb_data_path=cfg.pure_eb_path, - bestfit_dir=None, - output_path=_save_path(cfg, 1, "xi_pm_data"), - ) - - # 02: C_ell^EE data, no theory - plot_cell_ee_data_vector( - str(cfg.pseudo_cl_path), - str(cfg.pseudo_cl_cov_path), - _save_path(cfg, 2, "cell_ee_data"), - ) - - # 03: xi+/- with config-space best-fit (Paper IV Fig 1) - plot_xipm_bestfit_with_bmodes( - xi_data_path=cfg.xi_data_path, - pure_eb_data_path=cfg.pure_eb_path, - bestfit_dir=cfg.bestfit_dir, - output_path=_save_path(cfg, 3, f"xi_bestfit_blind_{blind}"), - ) - - # 04: C_ell^EE with harmonic + config best-fit (Paper V Fig 2) - plot_cell_ee_with_bestfit( - cosmosis_data_path=str(cfg.cosmosis_cell_fits), - bestfit_specs=[ - ( - rf"UNIONS $C_\ell$, Blind {blind}", - cfg.bestfit_root_fid_cell, - {"color": "royalblue", "linestyle": "-"}, - ), - ( - r"UNIONS $C_\ell$, Halofit", - cfg.bestfit_root_halofit_cell, - {"color": "royalblue", "linestyle": "--"}, - ), - ( - r"UNIONS $\xi_\pm(\vartheta)$ (Goh et al., 2026)", - cfg.bestfit_root_config, - {"color": "orange", "linestyle": "-"}, - ), - ], - output_folder=str(cfg.chain_root_dir), - output_path=_save_path(cfg, 4, f"cell_ee_bestfit_blind_{blind}"), - ) - - # ── Act 2 — The Reveal ──────────────────────────────────────────── - - # 05: Consistency — Omega_m-S8, harmonic vs config vs Planck - plot_triangle( - [ - ChainSpec( - root=reveal_harmonic_root, - label=rf"UNIONS $C_\ell$, Blind {blind}", - color="royalblue", - base_dir=cfg.chain_root_dir, - ), - ChainSpec( - root=reveal_config_root, - label=rf"UNIONS $\xi_\pm(\vartheta)$, Blind {blind}", - color="orange", - base_dir=cfg.chain_root_dir, - ), - ChainSpec( - root="Planck18", - label=r"\textit{Planck} 2018", - color="violet", - base_dir=cfg.external_root_dir, - ), - ], - ["OMEGA_M", "S_8"], - _save_path(cfg, 5, f"consistency_blind_{blind}"), - width_inch=12, - axes_fontsize=24, - axes_labelsize=28, - ) - - # 06: 4-param triangle — revealed blind, harmonic + config overlaid - plot_triangle( - [ - ChainSpec( - root=reveal_harmonic_root, - label=rf"UNIONS $C_\ell$, Blind {blind}", - color="royalblue", - base_dir=cfg.chain_root_dir, - ), - ChainSpec( - root=reveal_config_root, - label=rf"UNIONS $\xi_\pm(\vartheta)$, Blind {blind}", - color="orange", - base_dir=cfg.chain_root_dir, - ), - ], - COSMO_PARAMS, - _save_path(cfg, 6, f"triangle_cosmo_blind_{blind}"), - width_inch=20, - ) - - # 07: Full-param triangle — revealed blind, harmonic + config overlaid - plot_triangle( - [ - ChainSpec( - root=reveal_harmonic_root, - label=rf"UNIONS $C_\ell$, Blind {blind}", - color="royalblue", - base_dir=cfg.chain_root_dir, - ), - ChainSpec( - root=reveal_config_root, - label=rf"UNIONS $\xi_\pm(\vartheta)$, Blind {blind}", - color="orange", - base_dir=cfg.chain_root_dir, - ), - ], - FULL_PARAMS, - _save_path(cfg, 7, f"triangle_full_blind_{blind}"), - width_inch=30, - axes_fontsize=26, - axes_labelsize=28, - ) - - # ── Act 3 — In Context ──────────────────────────────────────────── - - # 08: S8 whisker — all 6 UNIONS blinds (3 harmonic + 3 config), - # revealed blind highlighted, rest muted. Plus external surveys. - whisker_specs = [] - - for b in ("A", "B", "C"): - is_revealed = b == blind - alpha = 1.0 if is_revealed else MUTED_ALPHA - whisker_specs.append( - ChainSpec( - root=f"{cfg.chain_prefix}_{b}_lmin=300_lmax=1600_cell", - label=rf"UNIONS $C_\ell$, Blind {b}", - color="royalblue", - base_dir=cfg.chain_root_dir, - alpha=alpha, - ) - ) - whisker_specs.append( - ChainSpec( - root=f"{cfg.chain_prefix}_{b}_10_80", - label=rf"UNIONS $\xi_\pm$, Blind {b}", - color="orange", - base_dir=cfg.chain_root_dir, - alpha=alpha, - ) - ) - - whisker_specs.extend( - [ - ChainSpec( - root="Planck18", - label=r"\textit{Planck} 2018", - color="black", - base_dir=cfg.external_root_dir, - ), - ChainSpec( - root="DES_Y3", - label=r"DES Y3 $\xi_\pm$", - color="black", - base_dir=cfg.external_root_dir, - ), - ChainSpec( - root="KiDS-1000", - label=r"KiDS-1000 $\xi_\pm$", - color="black", - base_dir=cfg.external_root_dir, - ), - ChainSpec( - root="HSC_Y3", - label=r"HSC Y3 $\xi_\pm$", - color="black", - base_dir=cfg.external_root_dir, - ), - ] - ) - - plot_s8_whisker( - whisker_specs, - reference_labels=[ - rf"UNIONS $C_\ell$, Blind {blind}", - rf"UNIONS $\xi_\pm$, Blind {blind}", - ], - reference_colors=["royalblue", "orange"], - output_path=_save_path(cfg, 8, f"s8_whisker_blind_{blind}"), - ) - - # ── Write evidence ─────────────────────────────────────────────── - - produced_figures = [ - _save_path(cfg, 1, "xi_pm_data"), - _save_path(cfg, 2, "cell_ee_data"), - _save_path(cfg, 3, f"xi_bestfit_blind_{blind}"), - _save_path(cfg, 4, f"cell_ee_bestfit_blind_{blind}"), - _save_path(cfg, 5, f"consistency_blind_{blind}"), - _save_path(cfg, 6, f"triangle_cosmo_blind_{blind}"), - _save_path(cfg, 7, f"triangle_full_blind_{blind}"), - _save_path(cfg, 8, f"s8_whisker_blind_{blind}"), - ] - output_dict = {} - for fig_path in produced_figures: - output_dict[fig_path.stem] = fig_path.name - - cfg.evidence_dir.mkdir(parents=True, exist_ok=True) - - evidence = { - "id": "unblinding_ceremony", - "spec_id": "unblinding_ceremony", - "generated": datetime.now(timezone.utc).isoformat(timespec="seconds"), - "input": { - "xi_data": str(cfg.xi_data_path), - "pure_eb_data": str(cfg.pure_eb_path), - "pseudo_cl": str(cfg.pseudo_cl_path), - "pseudo_cl_cov": str(cfg.pseudo_cl_cov_path), - "cosmosis_cell_fits": str(cfg.cosmosis_cell_fits), - "harmonic_chains": str( - cfg.chain_root_dir - / f"{cfg.chain_prefix}_{{A,B,C}}_lmin=300_lmax=1600_cell" - ), - "config_chains": str( - cfg.chain_root_dir / f"{cfg.chain_prefix}_{{A,B,C}}_10_80" - ), - "external_chains": str( - cfg.external_root_dir / "{Planck18,DES_Y3,KiDS-1000,HSC_Y3}" - ), - }, - "output": output_dict, - "params": { - "chain_version": cfg.chain_version, - "blind": blind, - "scale_cut_arcmin": "12-83", - "ell_range": "300-1600", - "n_figures": len(output_dict), - }, - "evidence": { - "ceremony_date": "2026-02-27", - "script": "workflow/scripts/unblinding_ceremony.py", - }, - } - - evidence_path = cfg.evidence_dir / "evidence.json" - evidence_path.write_text(json.dumps(evidence, indent=2) + "\n") - - for fig_path in produced_figures: - shutil.copy2(fig_path, cfg.evidence_dir / fig_path.name) - - print(f"Saved ceremony figures to {cfg.results_dir.resolve()}") - print(f"Wrote evidence to {evidence_path}") - - -# ── Entry point dispatch ───────────────────────────────────────────────────── - -try: - snakemake # injected by snakemake's script: directive -except NameError: - snakemake = None - -if snakemake is not None: - run_ceremony(_config_from_snakemake(snakemake)) -elif __name__ == "__main__": - run_ceremony(_config_from_cli()) From c6162c8180a2a99ff89f6ce60713ffa08456c2c8 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 16:59:06 +0200 Subject: [PATCH 16/37] containers: run the CI-published image from one shared path CI builds ghcr.io/cosmostat/sp_validation on every push from uv.lock; the hand-built SIFs it replaces were stale in ways that only failed at run time (no shapepipe.modules in one, numpy 2.5 breaking numba in the other). Every call site -- the Snakefiles, image_sims, the MPI rule's own apptainer exec, the paper shell drivers, interactive use -- now names one file, refreshed deliberately (see workflow/README.md). Overridable with --config container=. im_mbias_config read the OCI revision by opening the configured sif path; it now reads APPTAINER_CONTAINER, since the image a job actually ran in may be overridden and current.sif is a moving target. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- papers/bmodes/Snakefile | 2 +- papers/bmodes/scripts/run_cov_sweep.sh | 2 +- .../bmodes/scripts/run_pure_eb_ptes_sweep.sh | 2 +- .../scripts/run_pure_eb_semianalytic.sh | 2 +- papers/bmodes/scripts/run_pure_eb_sweep.sh | 2 +- papers/cosmo_val/Snakefile | 2 +- .../tests/data/container_smoke/Snakefile | 4 +- workflow/README.md | 129 ++++++++++++++---- workflow/Snakefile | 5 +- workflow/image_sims/config.yaml | 9 +- workflow/rules/twopoint.smk | 6 +- workflow/scripts/im_mbias_config.py | 30 ++-- 12 files changed, 141 insertions(+), 54 deletions(-) diff --git a/papers/bmodes/Snakefile b/papers/bmodes/Snakefile index 9ead91bd..165c96f0 100644 --- a/papers/bmodes/Snakefile +++ b/papers/bmodes/Snakefile @@ -4,7 +4,7 @@ configfile: "config/config.yaml" configfile: "/n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_val/cat_config.yaml" -container: "docker://ghcr.io/cosmostat/sp_validation:develop" +container: config.get("container", "/n17data/cdaley/containers/snakemake-sif/current.sif") envvars: "PYTHONUNBUFFERED", diff --git a/papers/bmodes/scripts/run_cov_sweep.sh b/papers/bmodes/scripts/run_cov_sweep.sh index 88d7f593..174504ee 100755 --- a/papers/bmodes/scripts/run_cov_sweep.sh +++ b/papers/bmodes/scripts/run_cov_sweep.sh @@ -26,7 +26,7 @@ # [--blind A] [--versions "v1 v2 ..."] set -euo pipefail -CONTAINER=/n17data/cdaley/containers/containers/ +CONTAINER=/n17data/cdaley/containers/snakemake-sif/current.sif WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra SRC=$WT/src WSCRIPTS=$WT/workflow/scripts diff --git a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh index 34ba5f7b..ff46ed8b 100644 --- a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh @@ -19,7 +19,7 @@ # --out [--blind A] [--versions "v1 v2 ..."] set -euo pipefail -CONTAINER=/n17data/cdaley/containers/containers/ +CONTAINER=/n17data/cdaley/containers/snakemake-sif/current.sif WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra SRC=$WT/src WSCRIPTS=$WT/workflow/scripts diff --git a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh index 609f2a18..dbb5b212 100644 --- a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh +++ b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh @@ -14,7 +14,7 @@ # --out [--n-chunks 20] [--n-samples 2000] [--nproc 16] set -euo pipefail -CONTAINER=/n17data/cdaley/containers/containers/ +CONTAINER=/n17data/cdaley/containers/snakemake-sif/current.sif SRC=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra/src SCRIPTS=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra/papers/bmodes/scripts BIND=/home,/scratch,/automnt,/n17data,/n23data1,/n09data diff --git a/papers/bmodes/scripts/run_pure_eb_sweep.sh b/papers/bmodes/scripts/run_pure_eb_sweep.sh index 74b6dab3..9f8dc9a1 100755 --- a/papers/bmodes/scripts/run_pure_eb_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_sweep.sh @@ -19,7 +19,7 @@ # --out [--blind A] [--versions "v1 v2 ..."] set -euo pipefail -CONTAINER=/n17data/cdaley/containers/containers/ +CONTAINER=/n17data/cdaley/containers/snakemake-sif/current.sif WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra SRC=$WT/src WSCRIPTS=$WT/workflow/scripts diff --git a/papers/cosmo_val/Snakefile b/papers/cosmo_val/Snakefile index 2585bb7a..418a007c 100644 --- a/papers/cosmo_val/Snakefile +++ b/papers/cosmo_val/Snakefile @@ -9,7 +9,7 @@ configfile: "config/config.yaml" configfile: "/n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_val/cat_config.yaml" -container: "/n17data/cdaley/containers/containers" +container: config.get("container", "/n17data/cdaley/containers/snakemake-sif/current.sif") envvars: "PYTHONUNBUFFERED", diff --git a/src/sp_validation/tests/data/container_smoke/Snakefile b/src/sp_validation/tests/data/container_smoke/Snakefile index 619a1aee..1cdb82b8 100644 --- a/src/sp_validation/tests/data/container_smoke/Snakefile +++ b/src/sp_validation/tests/data/container_smoke/Snakefile @@ -1,7 +1,7 @@ # Standalone workflow exercised by src/sp_validation/tests/test_container_smoke.py. # # The module-level `container:` below mirrors what every real workflow does -# (workflow/Snakefile line 11) -- Snakemake has no way to take a default image +# (workflow/Snakefile) -- Snakemake has no way to take a default image # from a profile. Everything else under test arrives from the driving profile # (workflow/profiles/candide): the slurm executor, `software-deployment-method: # apptainer` that turns container wrapping on, and the `apptainer-args` binds. @@ -12,7 +12,7 @@ # See container_smoke.py for what the job checks and why. -container: "/n17data/cdaley/containers/containers" +container: config.get("container", "/n17data/cdaley/containers/snakemake-sif/current.sif") rule container_smoke: diff --git a/workflow/README.md b/workflow/README.md index 07cdb4bd..faaf6128 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -66,6 +66,25 @@ flag, so a rule that needs it pinned sets it itself. Per-rule `mem_mb` / `runtime` stay on the rules. Off-cluster, drop `--profile` and add `-j N`. See the profile's own comments for the full rationale. +### Running your own checkout instead of the image's code + +Rules import the `sp_validation` baked into the image. To run a working copy +instead — testing a branch without rebuilding — prepend it to `PYTHONPATH` at +the container boundary. Apptainer forwards `APPTAINERENV_`-prefixed host +variables into the job (this survives the profile's `--cleanenv`, which strips +everything else), so setting it on the `snakemake` invocation reaches every +rule: + +```bash +APPTAINERENV_PYTHONPATH=/path/to/your/sp_validation/src \ + snakemake --profile workflow/profiles/candide -s workflow/Snakefile +``` + +The checkout has to sit under one of the profile's bind mounts to be visible +inside the job. This is a user-side override on purpose: nothing in the +workflow sets it, so a run reproduces from the image alone unless you ask +otherwise. + ### Never write `/automnt/nXXdataN` in a path Use the plain form `/nXXdataN/...` in every rule, config, and invocation @@ -100,35 +119,87 @@ shadow the one `uv tool install` just set up. Run `which snakemake` and confirm it resolves under `uv`'s tool directory (`uv tool dir`), not `~/.local/bin`. -### Container image invariants (not tracked in this repo) - -The `script:` directive works by bind-mounting the host orchestrator's own -`snakemake` install into the job's container and `sys.path.extend`-ing it in -(appended, not prepended) — so anything already importable inside the image -under that name wins the lookup instead. The image at -`/n17data/cdaley/containers/containers` is a writable sandbox (see the -top-level UNIONS `CLAUDE.md`), not built from a tracked recipe, so these two -invariants live only in the image itself and must be re-applied by hand after -any rebuild: - -- **No `snakemake` (or `snakemake-executor-plugin-slurm`) pip-installed - inside the image.** A leftover in-image install — from the old - apptainer-shell-then-snakemake-inside pattern this profile-driven setup - retired — shadows the host-mounted orchestrator ahead of it on `sys.path` - and breaks `script:`'s own unpickling preamble (`ModuleNotFoundError: No - module named 'snakemake.iocontainers'` if the in-image version predates - that submodule). Check with `apptainer exec ... python3 -m pip show - snakemake` — `Required-by:` should list nothing outside the snakemake - family itself before removing it. -- **`/.singularity.d/env/50-bashrc.sh` must not source the host `~/.bashrc` - for `apptainer exec`/`run`, only for an interactive `apptainer shell`.** - Apptainer sources every `/.singularity.d/env/*.sh` for all three actions; - gate any host-dotfile sourcing on `[ "$APPTAINER_COMMAND" = "shell" ]` (set - by Apptainer itself before these scripts run). Without the guard, a host - dotfile that mutates `PATH` (e.g. an `asdf` init) runs on every job too and - can push host tools — including a host-side `~/.local/bin/python` — ahead - of the image's own `/usr/local/bin`, so a bare `python` in a rule's - `shell:`/`script:` silently executes outside the container. +### The container image + +Everything runs one image, reached by one path: + +``` +/n17data/cdaley/containers/snakemake-sif/current.sif +``` + +That single string is what `workflow/Snakefile`, the paper Snakefiles, the +image-sims `sif:` config key, the `xi_highres` MPI rule's own `apptainer exec`, +and the `papers/bmodes/scripts/run_*.sh` drivers all use. Snakemake treats a +local path as local: it never pulls, never consults a cache, and never touches +the network during a run. + +**Where the image comes from.** CI (`.github/workflows/deploy-image.yml`) builds +it on every push, `FROM ghcr.io/cosmostat/shapepipe:im_sims` with `uv sync +--frozen` against `uv.lock`, and publishes to +`ghcr.io/cosmostat/sp_validation` tagged by branch — so `:develop` tracks the +tip of `develop`. The package is public; no credentials are needed. Because +`current.sif` is a file rather than a tag, **CI publishing a new image does not +change what your jobs run.** Someone has to refresh it deliberately, which is +the point. + +**Refreshing** — one person does it for everybody: + +```bash +# From a compute node (~1.5 GB / ~15 min; never on the login node). +salloc -p comp -c 4 --time=01:00:00 --no-shell # note the job id +export APPTAINER_CACHEDIR=/n17data/cdaley/containers/.apptainer-cache/cache +export APPTAINER_TMPDIR=/n17data/cdaley/containers/.apptainer-cache/tmp +srun --jobid= bash -c 'cd /n17data/cdaley/containers/snakemake-sif && \ + apptainer pull --force --name next.sif \ + docker://ghcr.io/cosmostat/sp_validation:develop && \ + mv -f next.sif current.sif' +scancel +``` + +Pull to `next.sif` and `mv` — never pull straight onto `current.sif`. `mv` +within one directory is an atomic rename, so a job either gets the whole old +image or the whole new one. Pulling in place would leave `current.sif` a +half-written file for the ~15 minutes the pull takes, and any job starting in +that window would fail. Jobs already running hold the old inode open and finish +against it unharmed. + +To check what you have: + +```bash +apptainer inspect --labels /n17data/cdaley/containers/snakemake-sif/current.sif +``` + +`org.opencontainers.image.revision` is the sp_validation commit the image was +built from. The image-sims workflow records it in `m_bias_config.yaml` as +`ghcr_revision`, so a result file says which image produced the number. + +**Interactive use** — the same image, the same path: + +```bash +apptainer exec --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data \ + /n17data/cdaley/containers/snakemake-sif/current.sif +``` + +This is what Cail's `app` shell function points at (the function lives in his +shell config, not this repo). There is one image, not two — a refresh moves +interactive use and the workflow together, with nothing to keep in sync. + +**Running your own image** instead of the shared one: + +```bash +snakemake --profile workflow/profiles/candide --config container=/path/to/my.sif +``` + +For the image-sims workflow, set `image_sims: {sif: /path/to/my.sif}` in your run +config — it is already a config key. Your image has to sit under one of the +profile's bind mounts to be visible. + +One invariant survives from the old hand-built sandbox and still applies: the +`script:` directive bind-mounts the host orchestrator's `snakemake` into the job +and *appends* it to `sys.path`, so a `snakemake` importable inside the image +wins the lookup. If `script:` rules start failing with `ModuleNotFoundError: No +module named 'snakemake.iocontainers'` or similar, an in-image snakemake older +than the host's is the first thing to check. ### `snakemake` in `script:` files diff --git a/workflow/Snakefile b/workflow/Snakefile index 7d91db81..c517dd66 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -8,7 +8,10 @@ # or run standalone with --configfile pointing at a paper config # (e.g. papers/bmodes/config/config.yaml). -container: "/n17data/cdaley/containers/containers" +# The one image every call site uses, workflow and interactive alike. Built by +# CI from the GHCR image and refreshed deliberately -- see workflow/README.md. +# Override with `--config container=/path/to/my.sif`. +container: config.get("container", "/n17data/cdaley/containers/snakemake-sif/current.sif") envvars: "PYTHONUNBUFFERED", diff --git a/workflow/image_sims/config.yaml b/workflow/image_sims/config.yaml index 7226e654..fe8faf99 100644 --- a/workflow/image_sims/config.yaml +++ b/workflow/image_sims/config.yaml @@ -11,10 +11,11 @@ image_sims: # --- container -------------------------------------------------------- # One image for the whole chain: the sp_validation image is built FROM the - # ShapePipe image, so it carries both stacks. It must be built from the uv - # lock -- an unlocked build drifts NumPy past numba's ceiling and the ngmix - # stage dies ("Numba needs NumPy 2.4 or less"). - sif: /n17data/cdaley/containers/sp_validation_im_sims.sif + # ShapePipe image, so it carries both stacks. CI builds it from the uv lock -- + # an unlocked build drifts NumPy past numba's ceiling and the ngmix stage dies + # ("Numba needs NumPy 2.4 or less"). Same path every call site uses; refresh is + # deliberate (workflow/README.md). Override here to run your own image. + sif: /n17data/cdaley/containers/snakemake-sif/current.sif # --- repositories ----------------------------------------------------- # Bound into the image; both repos' src go on PYTHONPATH so this branch's diff --git a/workflow/rules/twopoint.smk b/workflow/rules/twopoint.smk index 71e62925..077f3faa 100644 --- a/workflow/rules/twopoint.smk +++ b/workflow/rules/twopoint.smk @@ -55,13 +55,13 @@ rule xi_highres: slurm_extra="'--exclude=n17,n09,n36 --partition=pscomp'", mpi="/softs/openmpi/5.0.5-slurm-CentOS8/bin/mpiexec", shell: - # Container path kept in sync by hand with the top-level `container:` - # in workflow/Snakefile -- this rule cannot inherit it, see docstring. + # Same image path as the top-level `container:` in workflow/Snakefile; + # this rule cannot inherit it, see docstring. "{resources.mpi} -n {resources.tasks} " "apptainer exec " "--bind /home,/n09data,/n17data,/n23data1,/softs " "--env LD_LIBRARY_PATH=/softs/openmpi/5.0.5-slurm-CentOS8/lib " - "/n17data/cdaley/containers/containers " + "/n17data/cdaley/containers/snakemake-sif/current.sif " "python {input.script}" diff --git a/workflow/scripts/im_mbias_config.py b/workflow/scripts/im_mbias_config.py index 147ba8bb..20fe6e8c 100644 --- a/workflow/scripts/im_mbias_config.py +++ b/workflow/scripts/im_mbias_config.py @@ -12,6 +12,7 @@ """ import hashlib +import os import re import subprocess @@ -34,20 +35,31 @@ def _git(repo, *args): return None -def _sif_revision(sif_path): - """``org.opencontainers.image.revision`` from the SIF's OCI labels. +def _sif_revision(): + """``org.opencontainers.image.revision`` from the running image's OCI labels. - Plain-text scan of the image file -- no exec, no container start. + Read the image actually mounted, which Apptainer names in + ``APPTAINER_CONTAINER``, rather than the configured ``sif`` -- a run may + override it, and ``current.sif`` is a moving target either way. + Chunked plain-text scan: no exec, no container start, no 1.5 GB in memory. """ + sif_path = os.environ.get("APPTAINER_CONTAINER") + if not sif_path: + return None + pattern = re.compile( + rb'org\.opencontainers\.image\.revision"?[:=]"?([0-9a-f]{7,40})' + ) + tail = b"" try: with open(sif_path, "rb") as fh: - blob = fh.read() + while chunk := fh.read(8 << 20): + m = pattern.search(tail + chunk) + if m: + return m.group(1).decode() + tail = chunk[-128:] except OSError: return None - m = re.search( - rb'org\.opencontainers\.image\.revision"?[:=]"?([0-9a-f]{7,40})', blob - ) - return m.group(1).decode() if m else None + return None with open(manifest_path) as fh: @@ -86,7 +98,7 @@ def _sif_revision(sif_path): }, "container": { "sif": params.sif, - "ghcr_revision": _sif_revision(params.sif), + "ghcr_revision": _sif_revision(), }, }, } From 776c607a83112a7d6a566e7d51893f74933994d6 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 16:59:06 +0200 Subject: [PATCH 17/37] deps: ship CosmoSIS in the image, drop the vestigial cosmology pin CosmoSIS was an undeclared, user-supplied dependency of the inference step -- which is why #303 was hand-patched in someone's ~/.local. It pip-installs into the image against the base gfortran/GSL/cfitsio in ~2 min, so declare it. MPIFC must be set at build time or the sampler Makefiles silently skip the MPI targets and --mpi fails at load; chains must run under MPI because the upstream --smp pool is still broken at 3.25.2. cosmology 2022.10.9 was vestigial: the cosmology.compat.camb adapter comes from cosmology-compat-camb via glass[examples]. Relocking drops it and nothing else. UV_PYTHON pins uv to the image's own interpreter, since $HOME is bind-mounted and uv would otherwise pick a host CPython carrying none of the stack. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- .gitignore | 5 +- Dockerfile | 13 +++ cosmo_inference/README.md | 19 ++-- pyproject.toml | 10 +- uv.lock | 198 +++++++++++++++++++++++++++++++++++--- 5 files changed, 221 insertions(+), 24 deletions(-) diff --git a/.gitignore b/.gitignore index 82fb2697..c5fbafd5 100644 --- a/.gitignore +++ b/.gitignore @@ -194,4 +194,7 @@ papers/catalog/plots/*.pdf papers/cosmo_val/logs/ # Ignore scratch notebooks -scratch/*/*.ipynb \ No newline at end of file +scratch/*/*.ipynb + +# Snakemake run state +.snakemake/ diff --git a/Dockerfile b/Dockerfile index ccc930bf..35ecf855 100644 --- a/Dockerfile +++ b/Dockerfile @@ -16,6 +16,11 @@ RUN apt-get update -y --quiet --fix-missing && \ # VIRTUAL_ENV); install sp_validation's deps into that same venv rather than # spawning a second one under /sp_validation. ENV UV_PROJECT_ENVIRONMENT=/app/.venv +# $HOME is bind-mounted under apptainer, so uv would otherwise discover the +# host's managed CPythons -- including newer ones that satisfy requires-python +# -- and build a venv against an interpreter carrying none of this stack. +ENV UV_PYTHON=/usr/local/bin/python3.12 \ + UV_PYTHON_DOWNLOADS=never WORKDIR /sp_validation @@ -30,6 +35,14 @@ WORKDIR /sp_validation # numba-safe numpy 2.4.6 come straight from the lock, so the old ad-hoc snakemake # and cs_util `--upgrade` layers are gone. COPY pyproject.toml uv.lock /sp_validation/ + +# cosmosis builds MPI-enabled polychord/multinest only when MPIFC is set: its +# setup.py exports MPIFC for conda builds only, and the sampler Makefiles gate on +# `which $(MPIFC)`. Without this the install succeeds but omits libchord_mpi.so, +# and `cosmosis --mpi` fails at load time -- which is how the pipeline runs, since +# the --smp pool is broken upstream. Must precede the sync that builds cosmosis. +ENV MPIFC=mpif90 + RUN uv sync --frozen --inexact --no-install-project \ --extra test --extra glass --extra workflow diff --git a/cosmo_inference/README.md b/cosmo_inference/README.md index 5e5989cc..94998849 100644 --- a/cosmo_inference/README.md +++ b/cosmo_inference/README.md @@ -4,16 +4,17 @@ by Lisa Goh and Sacha Guerrini, CEA Paris-Saclay This folder contains the files neccessary to run the cosmological inference pipeline on the UNIONS galaxy catalogues. ### Requirements -To run the pipeline, one would need to have installed [CosmoSIS](https://cosmosis.readthedocs.io/en/latest/). To sample the PSF leakage parameters, the fork of [cosmosis-standard-library](https://github.com/sachaguer/cosmosis-standard-library/) of Sacha Guerrini has to be used. +[CosmoSIS](https://cosmosis.readthedocs.io/en/latest/) ships in the container via +the `workflow` extra, built with MPI support. To sample the PSF leakage +parameters, the fork of +[cosmosis-standard-library](https://github.com/sachaguer/cosmosis-standard-library/) +of Sacha Guerrini has to be used; it is not packaged, so clone and build it +yourself and point `COSMOSIS_DIR` in the pipeline templates at your checkout. -Run CosmoSIS with `--mpi`, not `--smp`. The `--smp` process pool is fragile and -barely maintained upstream: its `bcast`, `gather` and `allreduce` methods all -return a `self.data` attribute that is never set, so a run can crash right after -sampling finishes (upstream issue -[cosmosis#170](https://github.com/cosmosis-developers/cosmosis/issues/170) — the -`allreduce` crash was fixed in cosmosis 3.16.1, the rest is still open). Use -cosmosis 3.16.1 or newer, and prefer `--mpi`, which is what the upstream -maintainer recommends. +Launch sampling under MPI (`mpiexec -n N cosmosis --mpi ...`), not `--smp`: +CosmoSIS's shared-memory pool is unmaintained and still crashes after sampling +completes (`Pool` has no attribute `data`, `runtime/process_pool.py`) as of +3.25.2. ### To Run The inference pipeline is orchestrated through Snakemake. On the candide diff --git a/pyproject.toml b/pyproject.toml index c6a68dd0..ecc4c92e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -155,8 +155,16 @@ glass = [ workflow = [ "snakemake", # run_2pcf_highres.py drives the MPI convergence run; the container ships - # OpenMPI (/opt/ompi) so mpi4py builds against it. + # OpenMPI 4.1.4 (mpicc/mpif90 on PATH in /usr/bin) so mpi4py builds against it. "mpi4py", + # CosmoSIS drives the cosmo_inference sampling step. Sdist-only, so this is a + # source build (~2 min) against the base image's gfortran/GSL/cfitsio/OpenMPI — + # no extra apt packages needed. Floor at 3.25 simply to stay current; there is + # no known-good older version to pin back to. Note the shared-memory pool is + # still broken upstream at 3.25.2 (bcast/gather/allreduce in + # runtime/process_pool.py return an unassigned `self.data`), so chains must run + # under MPI -- which is why the Dockerfile sets MPIFC before syncing this extra. + "cosmosis>=3.25", # NOTE: workflow/scripts/cv_*.py also import `cv_runner`, which is not # published or resolvable (no public repo found) — left undeclared pending # its source. Same for `unions_wl` (scripts/check_footprint.py). diff --git a/uv.lock b/uv.lock index bef8072e..1b5aca1b 100644 --- a/uv.lock +++ b/uv.lock @@ -112,6 +112,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/86/16/1a8fd2b19544b84575cf84ef7aa3ad4c173b756d5f087c91f85d1b295777/array_api_compat-1.15.0-py3-none-any.whl", hash = "sha256:7b1b9c53269061403fd5f45a8de349f16e7887653328bfa0c5f2d45299ff0a8e", size = 79113, upload-time = "2026-06-07T20:53:23.621Z" }, ] +[[package]] +name = "array-api-extra" +version = "0.11.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "array-api-compat", marker = "sys_platform == 'linux'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/94/40/e2e74294b894b115b05c052364fde283e3684e309c58c8f3e0463270051b/array_api_extra-0.11.1.tar.gz", hash = "sha256:360bc6faf858b1ef2ca0fb3cc86dbac0ed566fa1f78ae515cd234830c8b119f8", size = 102150, upload-time = "2026-08-12T11:29:24.981Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4f/ee/c0a6a4bb3c5d874c68f57c2471af931f5f4cf2f84fdcc86b8a7467eb3e66/array_api_extra-0.11.1-py3-none-any.whl", hash = "sha256:2da3eed8842ed14cdded9a2a82f11dcafae2aa2c0c1622a3b77e769f72331c64", size = 98046, upload-time = "2026-08-12T11:29:23.571Z" }, +] + [[package]] name = "arrow" version = "1.4.0" @@ -550,17 +562,48 @@ dependencies = [ ] [[package]] -name = "cosmology" -version = "2022.10.9" +name = "cosmology-api" +version = "0.3.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f0/f4/801cab91dab5b2e8102ed54cc4e2bd257f319c9047db182fcaf53706df6a/cosmology_api-0.3.2.tar.gz", hash = "sha256:7ccdfdf20f91dfc2282aee059adf2530bea6e194a0f8d01487e15a7497cf4694", size = 17338, upload-time = "2025-05-15T13:29:52.355Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bf/8c/6cc73fafff9f4d75e77a9c3cf5b9c34aa42841cfc7da5c89737daa2c3bd7/cosmology_api-0.3.2-py3-none-any.whl", hash = "sha256:9391ef0b2616bbf4217fefdb94a72370766d59a2f0558b5ade224a92ea093a61", size = 18685, upload-time = "2025-05-15T13:29:50.833Z" }, +] + +[[package]] +name = "cosmology-compat-camb" +version = "0.2.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "numpy", marker = "sys_platform == 'linux'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/c9/38/04099bb2b6a626bd69a3f117afc60f2c47c08899e0730e0e48c898bf4745/cosmology-2022.10.9.tar.gz", hash = "sha256:0c2857c9bf1fdd09f1f11ab5765df0389a4101f3a900fa11251c3f37696f02d4", size = 8488, upload-time = "2022-10-10T10:18:16.229Z" } +sdist = { url = "https://files.pythonhosted.org/packages/08/2e/a637aae1677d02337be371c9a7b7e9d5051199bb58a2386781230adcee45/cosmology_compat_camb-0.2.0.tar.gz", hash = "sha256:e36fda04a78e16fc1a5c4aa22c30ec1bf64bbe0b092aa1b894ca746ce5e5551b", size = 4562, upload-time = "2025-05-08T10:31:43.226Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/2e/2c/c80db37593ad593e3f6c60705fbdf690b33f22c89edd5507d2731346cd87/cosmology-2022.10.9-py3-none-any.whl", hash = "sha256:3903658c2474177a1a1c75771ae14458d93200c516f8fc6c3f4d776f3287b288", size = 9341, upload-time = "2022-10-10T10:18:14.928Z" }, + { url = "https://files.pythonhosted.org/packages/2e/42/1a657f9b3540f355f34af735d8643aae1480620689a72423fe69aeeb78e5/cosmology_compat_camb-0.2.0-py3-none-any.whl", hash = "sha256:eb6e74290bb6a6a60a47d1338fc233ba1697058ed7f01a01de1180dbee1bd75d", size = 3903, upload-time = "2025-05-08T10:31:41.997Z" }, ] +[[package]] +name = "cosmosis" +version = "3.25.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "dulwich", marker = "sys_platform == 'linux'" }, + { name = "dynesty", marker = "sys_platform == 'linux'" }, + { name = "emcee", marker = "sys_platform == 'linux'" }, + { name = "h5py", marker = "sys_platform == 'linux'" }, + { name = "matplotlib", marker = "sys_platform == 'linux'" }, + { name = "nautilus-sampler", marker = "sys_platform == 'linux'" }, + { name = "numpy", marker = "sys_platform == 'linux'" }, + { name = "py-bobyqa", marker = "sys_platform == 'linux'" }, + { name = "pybind11", marker = "sys_platform == 'linux'" }, + { name = "pyyaml", marker = "sys_platform == 'linux'" }, + { name = "scikit-learn", marker = "sys_platform == 'linux'" }, + { name = "scipy", marker = "sys_platform == 'linux'" }, + { name = "threadpoolctl", marker = "sys_platform == 'linux'" }, + { name = "zeus-mcmc", marker = "sys_platform == 'linux'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/80/63/83898440e5486294cb17a7ddf01896af66dd153d7f3e77897ed120abd055/cosmosis-3.25.2.tar.gz", hash = "sha256:4a8333395b600a5e8339c637b947a982bac4c88a7b242e4b6de8274be03004a8", size = 416233, upload-time = "2026-03-09T12:01:08.269Z" } + [[package]] name = "coverage" version = "7.15.0" @@ -852,6 +895,44 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/9a/91/428eca0cc8a1142b9952bef02c06bfe54526d730dd070de873424ab8c0bf/ducc0-0.41.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a3a71fd23661ddd6a7f6b44433e778d35a25b3f1492c9239217cdfedbe8178b1", size = 5613706, upload-time = "2026-03-26T18:55:24.971Z" }, ] +[[package]] +name = "dulwich" +version = "1.2.13" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "urllib3", marker = "sys_platform == 'linux'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ab/7d/022515919b5ee3175a70a7e52049c4a2637008a4a49024ad15db1531bda9/dulwich-1.2.13.tar.gz", hash = "sha256:77f0d7012710da1ba0742a197fce215b0e1a05aeb9bc8a03ce94ccd2644317d8", size = 1361288, upload-time = "2026-08-24T12:38:30.179Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6d/42/2383b8eb8e910d18d4e37156d2e191bb5b3b202648636db266065611835b/dulwich-1.2.13-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:8816b7c1e5e527b9c8d8aa322b9b80cb1878c0b4990707ceffa12c2529eb6dc5", size = 1459316, upload-time = "2026-08-24T12:37:18.387Z" }, + { url = "https://files.pythonhosted.org/packages/47/ee/d75a0249a21f22c03db2ab1a8ee9f727ca1fafc6bd7da86ddd0564a05b7b/dulwich-1.2.13-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:1c1ff65f48a81fce6fa78edffc289655556082543f34a4024a0eca5161f2fb85", size = 1490824, upload-time = "2026-08-24T12:37:20.045Z" }, + { url = "https://files.pythonhosted.org/packages/65/7e/52324b023c728e7e0a3397d8388ef85b2980d9f436d34a4b0530f076f5ae/dulwich-1.2.13-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:7805fbcd07a542502e50f7bd63cb6a2ab1b04fa5803bf2ee163866e8c97c6f94", size = 1459564, upload-time = "2026-08-24T12:37:32.602Z" }, + { url = "https://files.pythonhosted.org/packages/5c/2e/f321b0ffd3168d795aad5043bf17190ba8481e293cd09154f0cd3e155134/dulwich-1.2.13-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:d258d59a7af83147d5c888e5ab3a53f2d37ad4ae41728d70969e0bce1ac7ba83", size = 1490259, upload-time = "2026-08-24T12:37:34.311Z" }, + { url = "https://files.pythonhosted.org/packages/47/0d/ab0e5dd1558b15a78fa2d75c03b5af44823b8e1297ffcfef4bb66f24f9d6/dulwich-1.2.13-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:b484a49c166c2e5b2deecdb964d4725a454cfcc87970746f2b2faac252d0bb72", size = 1461868, upload-time = "2026-08-24T12:37:46.208Z" }, + { url = "https://files.pythonhosted.org/packages/45/75/d94f4317f09df37638c81ce90d0988154cb8d537504a9d4c0f5837cf320a/dulwich-1.2.13-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:c11ba8725d8808dee69e3aec1aceed3b716b16da3136c31499b271e0a79913c4", size = 1491986, upload-time = "2026-08-24T12:37:47.977Z" }, + { url = "https://files.pythonhosted.org/packages/46/0f/10fadbbc7efbb103f747af24e690393e6f6c29d85acac5febb34e35cb6e7/dulwich-1.2.13-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:63c26d34f53fe972e64fa6a9f6d7b013fa616a8357893e648710799e2e238db7", size = 1458718, upload-time = "2026-08-24T12:37:56.561Z" }, + { url = "https://files.pythonhosted.org/packages/5d/0e/8b9157e4536e78ab80b2d10759408e026d5f2ea62fc1fe5a386b89f7b4c3/dulwich-1.2.13-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:f0a79903a43491804852333f4e85fd484bc5e35ecf4fc836409f8194e3f1fc0b", size = 1539867, upload-time = "2026-08-24T12:37:58.202Z" }, + { url = "https://files.pythonhosted.org/packages/cc/95/4a840bf858b3d83087de36f5f768ff20cc464f0495a37d7bd8fd97cf1ac1/dulwich-1.2.13-cp315-cp315-manylinux_2_28_aarch64.whl", hash = "sha256:c243c278cb3281c8f6876b3621f2396548041aa316d91f3c78aa2d7a22043ef9", size = 1462669, upload-time = "2026-08-24T12:38:09.971Z" }, + { url = "https://files.pythonhosted.org/packages/e4/de/e3d9e52ce2bf06c63eb360bd467008cdb030655ccf98100bfc72577dd926/dulwich-1.2.13-cp315-cp315-manylinux_2_28_x86_64.whl", hash = "sha256:33ac85b270d2410fa1fc962caeef14a63e04de46dd76c5080f558c5d9752fa04", size = 1540300, upload-time = "2026-08-24T12:38:11.568Z" }, + { url = "https://files.pythonhosted.org/packages/82/52/9c7a9ff65cdd99237220f88018f3b4b2f4cb910d4341b9625c903796ca64/dulwich-1.2.13-cp315-cp315t-manylinux_2_28_aarch64.whl", hash = "sha256:c226635fe7dffdc8718b177a11b904a4bd5ff83dc9b5a443109458603f944489", size = 1460164, upload-time = "2026-08-24T12:38:20.434Z" }, + { url = "https://files.pythonhosted.org/packages/ee/6b/dbeece5fa1dd39416ba7b525e6ae8f9f96f34c1bc9de58f6a398d889cba4/dulwich-1.2.13-cp315-cp315t-manylinux_2_28_x86_64.whl", hash = "sha256:b832922190662896e057b8d84fd28d07d7c8eb9cbe2519348142bb909cd693b4", size = 1493098, upload-time = "2026-08-24T12:38:22.242Z" }, + { url = "https://files.pythonhosted.org/packages/5e/50/838d9c3da15f94b8f36c69e7abe3fea5294569a1dbda3812a07fba429f54/dulwich-1.2.13-py3-none-any.whl", hash = "sha256:edadd2f1019d879f80ab107a8587cb7f3b72d077b23facb8564700e7e3ce1758", size = 730094, upload-time = "2026-08-24T12:38:28.562Z" }, +] + +[[package]] +name = "dynesty" +version = "3.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "matplotlib", marker = "sys_platform == 'linux'" }, + { name = "numpy", marker = "sys_platform == 'linux'" }, + { name = "scipy", marker = "sys_platform == 'linux'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c4/75/db494edc8abfe2273b9cb08c87a2a1165c6ae90d76f746980ef07ec5303a/dynesty-3.1.0.tar.gz", hash = "sha256:851717431f04f749bca45e86704c28fecf9f3b10f85a2a69820b26e42cefab68", size = 35564761, upload-time = "2026-07-17T14:38:31.465Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f1/f9/41c72d0b5fb511fdf23cf62ed50da6c73d4497f03ed1cad04f214dc477c8/dynesty-3.1.0-py3-none-any.whl", hash = "sha256:4e5b77bb261abdcb98e3a538d7498a94d5882a848f0ea46924b404396ef036d5", size = 106027, upload-time = "2026-07-17T14:38:28.523Z" }, +] + [[package]] name = "emcee" version = "3.1.6" @@ -1010,18 +1091,28 @@ wheels = [ [[package]] name = "glass" -version = "2025.1" +version = "2026.2" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "cosmology", marker = "sys_platform == 'linux'" }, + { name = "array-api-compat", marker = "sys_platform == 'linux'" }, + { name = "array-api-extra", marker = "sys_platform == 'linux'" }, { name = "healpix", marker = "sys_platform == 'linux'" }, { name = "healpy", marker = "sys_platform == 'linux'" }, - { name = "numpy", marker = "sys_platform == 'linux'" }, { name = "transformcl", marker = "sys_platform == 'linux'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/a5/d6/26cd58e75834671259f26b9287a584d726e6023da35ddf7ca3b7f2c393fb/glass-2025.1.tar.gz", hash = "sha256:7b1aa2394e16010f7f1b4243f49e7e12d7a4dd28fcbf3e3f7cf25ce4905a8615", size = 48533, upload-time = "2025-02-21T18:43:48.046Z" } +sdist = { url = "https://files.pythonhosted.org/packages/c1/c3/3b7d18cca8d0fa41dbb646ce16935804b380dcf766e7385799f44829b419/glass-2026.2.tar.gz", hash = "sha256:cf5ef6cb76b4738f8dc05ddfc18c359c558bb36e9cd09ca0115fa3ca3deb491f", size = 66236, upload-time = "2026-06-04T15:18:39.155Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/e4/e7/3e89bc5147a3e84245c783098afffbbacbcdf5678ba1aacc2c9c035434da/glass-2025.1-py3-none-any.whl", hash = "sha256:d7919a7d19e05ab8da4e52dbfbcfb5bd1ed2b72e0cc3afde39ec94ced22883c5", size = 47185, upload-time = "2025-02-21T18:43:46.649Z" }, + { url = "https://files.pythonhosted.org/packages/7c/d1/55ea5e63e81db82806e92ba9f9c2b7277c0c2afe57bf8c1d3763e18005b2/glass-2026.2-py3-none-any.whl", hash = "sha256:0af0b5cd8708040925699c307076fb1c79bbb7e6cba4522764bcc21dd3bdadb8", size = 63746, upload-time = "2026-06-04T15:18:37.895Z" }, +] + +[package.optional-dependencies] +examples = [ + { name = "camb", marker = "sys_platform == 'linux'" }, + { name = "cosmology-api", marker = "sys_platform == 'linux'" }, + { name = "cosmology-compat-camb", marker = "sys_platform == 'linux'" }, + { name = "glass-ext-camb", marker = "sys_platform == 'linux'" }, + { name = "jupyter", marker = "sys_platform == 'linux'" }, + { name = "matplotlib", marker = "sys_platform == 'linux'" }, ] [[package]] @@ -2267,6 +2358,30 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/09/dc/f3dfb7488b770f3f67e6545085bf2abea5172e88f57b8ad25ef860ca704c/myst_parser-5.1.0-py3-none-any.whl", hash = "sha256:9c91c52b3cdb4d94a6506e4fab4e2f296c7623a0da0dcbe6de1565c3dad67a8a", size = 85817, upload-time = "2026-05-13T09:38:17.904Z" }, ] +[[package]] +name = "narwhals" +version = "2.25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6f/7b/6248dada39781db1ab3ebf08943080df0796098515a87f6f8696d14ec744/narwhals-2.25.0.tar.gz", hash = "sha256:62c036c810662bf7820b7737077176313bc59350eeeefb808510f388c743e4b2", size = 677076, upload-time = "2026-08-20T18:10:15.454Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/eb/dc/55481808fd70ef1567cf13540ffd4702af3f74b112e35427564b03f79c2d/narwhals-2.25.0-py3-none-any.whl", hash = "sha256:1f0f403e8c7e4463cde9bfe78b12fdd809e3ae3dda6d9b2f802934fb9c7a6a8f", size = 467373, upload-time = "2026-08-20T18:10:13.834Z" }, +] + +[[package]] +name = "nautilus-sampler" +version = "1.0.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy", marker = "sys_platform == 'linux'" }, + { name = "scikit-learn", marker = "sys_platform == 'linux'" }, + { name = "scipy", marker = "sys_platform == 'linux'" }, + { name = "threadpoolctl", marker = "sys_platform == 'linux'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/2c/a50c1a550ba43d86a0b41cf6410986d478d8640139c8e963f4dde8a8bfc2/nautilus_sampler-1.0.6.tar.gz", hash = "sha256:4e90b6d97be742be2e255c35a79e5a388cee043b4d2ce970eec327ed34139b78", size = 43118, upload-time = "2025-12-29T14:08:25.629Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/47/54/372c42bcfe324b650946f6cfee0e5adbd55aafc25cab555460bf5d9e8d36/nautilus_sampler-1.0.6-py3-none-any.whl", hash = "sha256:de6f3b9d249d87f05a673e8597e56621037e84dc712a6c2d3c18ff3672735b9b", size = 35772, upload-time = "2025-12-29T14:08:24.603Z" }, +] + [[package]] name = "nbclient" version = "0.11.0" @@ -2758,6 +2873,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl", hash = "sha256:1db8e35b67b3d218d818ae653e27f06c3aa420901fa7b081ca98cbedc874e0d0", size = 11842, upload-time = "2024-07-21T12:58:20.04Z" }, ] +[[package]] +name = "py-bobyqa" +version = "1.5.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy", marker = "sys_platform == 'linux'" }, + { name = "pandas", marker = "sys_platform == 'linux'" }, + { name = "scipy", marker = "sys_platform == 'linux'" }, + { name = "setuptools", marker = "sys_platform == 'linux'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/6b/41/c4c74daf208ed27e14071e92efb7ea238ffdf77ee93a3a7777ff02d2b0e4/py_bobyqa-1.5.0.tar.gz", hash = "sha256:3c7719b68b28834ea6d538f54603f6a891263f7c21f1a673de79e3a5e0e7e413", size = 51486, upload-time = "2024-09-16T03:52:32.303Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/1e/0d44a4e3a291c009a357fbd1d61511d9306c2c4db9a7ceb6e8104d8d385f/Py_BOBYQA-1.5.0-py3-none-any.whl", hash = "sha256:457afc04d6f2c9f1814934854dc4e542c5e5982a0f80add4b211fcdb0b5811e3", size = 57978, upload-time = "2024-09-16T03:52:30.667Z" }, +] + [[package]] name = "py-cpuinfo" version = "9.0.0" @@ -3354,6 +3484,29 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a1/9f/56288a8aabafa24679d32d01e7db732e8e89e4074cc91aedc0bb7b7c8e46/sacc-2.4-py3-none-any.whl", hash = "sha256:da6b648998738c7c307cbdff644429633726844f8c048642725be5efc1859fa3", size = 50931, upload-time = "2026-07-02T11:29:26.286Z" }, ] +[[package]] +name = "scikit-learn" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "joblib", marker = "sys_platform == 'linux'" }, + { name = "narwhals", marker = "sys_platform == 'linux'" }, + { name = "numpy", marker = "sys_platform == 'linux'" }, + { name = "scipy", marker = "sys_platform == 'linux'" }, + { name = "threadpoolctl", marker = "sys_platform == 'linux'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/fa/6f/37092bdb25f712817231799fc5674d8e704066a8a70c1d2d40517e18b4ab/scikit_learn-1.9.0.tar.gz", hash = "sha256:8833266989d3a5110178a9fae30783675460724d0e1efb13b14901d2c660c557", size = 7750767, upload-time = "2026-06-02T11:54:32.706Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/ee/5adbc77656b71f9456a2f5a7a9fdb4bcf9207a6b962889f1c2f9323afa4e/scikit_learn-1.9.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5e50ed4da51974e86e940690e9a3d82e729b62b5a49f7c9bac534d515d39d86f", size = 8837603, upload-time = "2026-06-02T11:53:30.328Z" }, + { url = "https://files.pythonhosted.org/packages/6c/c2/63fdda36c56437eeb44aaf9493c8bcd62ce230ab1598924fc626ffbfa943/scikit_learn-1.9.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:056c92bb67ad4c28463c2f2653d9701449201e7e7a9e94e321be0f71c4fef2b8", size = 9132097, upload-time = "2026-06-02T11:53:33.456Z" }, + { url = "https://files.pythonhosted.org/packages/0c/39/ffe829a5b8ecb40a518724a997794657fdc354ada5e8fe8e64d998c0bac9/scikit_learn-1.9.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:38c3dcb9a1ffb85505ec53d54c7b4aea0cff70050425a7760c2af661ac85df05", size = 8789690, upload-time = "2026-06-02T11:53:47.461Z" }, + { url = "https://files.pythonhosted.org/packages/1f/88/8dab5de10c638c083772a6be83a3d8106ced492f74a928c8693638e5bb50/scikit_learn-1.9.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:da76d09304a4706db7cc1e3ebaa3b6b98a67365cc11d2996c4f1e58ba47df714", size = 9087723, upload-time = "2026-06-02T11:53:50.702Z" }, + { url = "https://files.pythonhosted.org/packages/7d/79/f4a0c4fe9711154cddabf913471153af79056382ddc612cfe5ee0ff4b72e/scikit_learn-1.9.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5162ad10a418c8a282dde04c9aa06965de3e9a65f33c1440c0ae69bb1a09d913", size = 8847671, upload-time = "2026-06-02T11:54:04.448Z" }, + { url = "https://files.pythonhosted.org/packages/f0/af/4d72d9e475ac83719160c662619e4bf7b95c19507cd582e7d0167a3c3dae/scikit_learn-1.9.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fea2cc5677ab49d6f5bade978c866da44957b712d92e9635e8b4f723013c3cb", size = 9118104, upload-time = "2026-06-02T11:54:07.205Z" }, + { url = "https://files.pythonhosted.org/packages/18/3c/e440e039bb82cd19004edaaad00acbde0fb9b461083c3ecf37941c557312/scikit_learn-1.9.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:147e9329ef0e39f75d4cffa02b2aa48d827832684926cd5210d9a2cb5c57246b", size = 8855050, upload-time = "2026-06-02T11:54:21.699Z" }, + { url = "https://files.pythonhosted.org/packages/43/26/b341b8dab5998da6270a3a42c2152c578501354d36f944b5856757035ef8/scikit_learn-1.9.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5bad8f8b9950321b54c965fdcbac6c6c55e79e16646b49977bcf3668d3870a1a", size = 9097190, upload-time = "2026-06-02T11:54:24.454Z" }, +] + [[package]] name = "scipy" version = "1.17.1" @@ -3737,9 +3890,8 @@ docs = [ { name = "sphinxcontrib-bibtex", marker = "sys_platform == 'linux'" }, ] glass = [ - { name = "cosmology", marker = "sys_platform == 'linux'" }, { name = "fitsio", marker = "sys_platform == 'linux'" }, - { name = "glass", marker = "sys_platform == 'linux'" }, + { name = "glass", extra = ["examples"], marker = "sys_platform == 'linux'" }, { name = "glass-ext-camb", marker = "sys_platform == 'linux'" }, ] test = [ @@ -3748,6 +3900,7 @@ test = [ { name = "ruff", marker = "sys_platform == 'linux'" }, ] workflow = [ + { name = "cosmosis", marker = "sys_platform == 'linux'" }, { name = "mpi4py", marker = "sys_platform == 'linux'" }, { name = "snakemake", marker = "sys_platform == 'linux'" }, ] @@ -3760,13 +3913,13 @@ requires-dist = [ { name = "clmm" }, { name = "colorama" }, { name = "cosmo-numba", git = "https://github.com/aguinot/cosmo-numba.git?rev=main" }, - { name = "cosmology", marker = "extra == 'glass'", specifier = "==2022.10.9" }, + { name = "cosmosis", marker = "extra == 'workflow'", specifier = ">=3.25" }, { name = "cryptography" }, { name = "cs-util", git = "https://github.com/CosmoStat/cs_util.git?rev=develop" }, { name = "emcee" }, { name = "fitsio", marker = "extra == 'glass'" }, { name = "getdist", git = "https://github.com/benabed/getdist.git?rev=113cd22a9a0d013b6f72fe734be81f260f3d3be5" }, - { name = "glass", marker = "extra == 'glass'", specifier = "==2025.1" }, + { name = "glass", extras = ["examples"], marker = "extra == 'glass'", specifier = "==2026.2" }, { name = "glass-ext-camb", marker = "extra == 'glass'", specifier = "==2023.6" }, { name = "h5py" }, { name = "healpy" }, @@ -4421,6 +4574,25 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/88/0a/469e2bd01be1490336e6c8707386845655d59261543315778a3ccc7e8019/zarr-3.2.1-py3-none-any.whl", hash = "sha256:f78cdd3d9687ad0e9f9cba2c5683b64f0c52589c19f685eeabe872e93cc0d2c7", size = 319617, upload-time = "2026-05-05T12:37:20.66Z" }, ] +[[package]] +name = "zeus-mcmc" +version = "2.5.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "matplotlib", marker = "sys_platform == 'linux'" }, + { name = "numpy", marker = "sys_platform == 'linux'" }, + { name = "pytest", marker = "sys_platform == 'linux'" }, + { name = "scikit-learn", marker = "sys_platform == 'linux'" }, + { name = "scipy", marker = "sys_platform == 'linux'" }, + { name = "seaborn", marker = "sys_platform == 'linux'" }, + { name = "setuptools", marker = "sys_platform == 'linux'" }, + { name = "tqdm", marker = "sys_platform == 'linux'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/15/c0/248a600ae0f3d1437612821f7c528c895b6a8408a052c352d4cac46dcf94/zeus-mcmc-2.5.4.tar.gz", hash = "sha256:594baa90de4ad4488c4db5ed6a0446f7103bc4b3de787f4d7d23c91c9aa88769", size = 35051, upload-time = "2023-01-12T04:14:20.688Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/48/cb/99ee8021fa305d65b1fb369fabf96ccc65c60282407390e5355bb9e62f9f/zeus_mcmc-2.5.4-py3-none-any.whl", hash = "sha256:a64a7dae15f413200c6d590a3edfc3b8bc63c6bea3acdf123c0397c8089e123f", size = 24122, upload-time = "2023-01-12T04:14:19.24Z" }, +] + [[package]] name = "zipp" version = "4.1.0" From 64e1ac2cec5f6acf309a5e9ce65055b3f3782cfa Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 18:11:54 +0200 Subject: [PATCH 18/37] containers: pull the CI image by tag, park the MPI rule Snakemake pulls docker://ghcr.io/cosmostat/sp_validation:develop into a shared apptainer prefix on first use and never again, so no digest or path is written down. One constant, CONTAINER_URI in workflow/common.py, is the single source of truth; host-side callers that need a concrete file (the paper shell drivers, interactive use) derive it via workflow/scripts/container_path.py. xi_highres is parked as a comment block: it has never been runnable -- its shell is a bare 'python run_2pcf_highres.py' while the script requires --cat-config and --out -- and parking it leaves covariance_cosmocov as the workflow's only container exception. The MPI reasoning is preserved in the block for whoever revives it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- papers/bmodes/Snakefile | 7 +- papers/bmodes/scripts/run_cov_sweep.sh | 4 +- .../bmodes/scripts/run_pure_eb_ptes_sweep.sh | 4 +- .../scripts/run_pure_eb_semianalytic.sh | 7 +- papers/bmodes/scripts/run_pure_eb_sweep.sh | 4 +- papers/cosmo_val/Snakefile | 7 +- .../tests/data/container_smoke/Snakefile | 5 +- .../tests/test_container_smoke.py | 14 +++ workflow/README.md | 67 ++++++++------ workflow/Snakefile | 10 +- workflow/common.py | 8 ++ workflow/image_sims/config.yaml | 7 +- workflow/profiles/candide/config.yaml | 13 ++- workflow/rules/covariance.smk | 2 +- workflow/rules/twopoint.smk | 91 +++++++++++-------- workflow/scripts/container_path.py | 42 +++++++++ workflow/scripts/im_mbias_config.py | 2 +- 17 files changed, 205 insertions(+), 89 deletions(-) create mode 100755 workflow/scripts/container_path.py diff --git a/papers/bmodes/Snakefile b/papers/bmodes/Snakefile index 165c96f0..273368b7 100644 --- a/papers/bmodes/Snakefile +++ b/papers/bmodes/Snakefile @@ -4,8 +4,6 @@ configfile: "config/config.yaml" configfile: "/n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_val/cat_config.yaml" -container: config.get("container", "/n17data/cdaley/containers/snakemake-sif/current.sif") - envvars: "PYTHONUNBUFFERED", @@ -27,6 +25,11 @@ import common common.configure(config) from common import * +# The image for every rule -- the CI tag, pulled into the profile's +# apptainer-prefix on first use (workflow/README.md). Override with +# `--config container=/path/to/my.sif`. +container: config.get("container", CONTAINER_URI) + # Wildcard constraints — centralized in common.py, not in individual rule files wildcard_constraints: **WILDCARD_CONSTRAINTS diff --git a/papers/bmodes/scripts/run_cov_sweep.sh b/papers/bmodes/scripts/run_cov_sweep.sh index 174504ee..5cf83b36 100755 --- a/papers/bmodes/scripts/run_cov_sweep.sh +++ b/papers/bmodes/scripts/run_cov_sweep.sh @@ -26,8 +26,10 @@ # [--blind A] [--versions "v1 v2 ..."] set -euo pipefail -CONTAINER=/n17data/cdaley/containers/snakemake-sif/current.sif WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra +# The image Snakemake pulled, resolved from the workflow's one declaration +# of it (workflow/common.py CONTAINER_URI + the candide profile's prefix). +CONTAINER=$($WT/workflow/scripts/container_path.py) SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts diff --git a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh index ff46ed8b..b8988902 100644 --- a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh @@ -19,8 +19,10 @@ # --out [--blind A] [--versions "v1 v2 ..."] set -euo pipefail -CONTAINER=/n17data/cdaley/containers/snakemake-sif/current.sif WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra +# The image Snakemake pulled, resolved from the workflow's one declaration +# of it (workflow/common.py CONTAINER_URI + the candide profile's prefix). +CONTAINER=$($WT/workflow/scripts/container_path.py) SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts diff --git a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh index dbb5b212..af2d95bd 100644 --- a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh +++ b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh @@ -14,8 +14,11 @@ # --out [--n-chunks 20] [--n-samples 2000] [--nproc 16] set -euo pipefail -CONTAINER=/n17data/cdaley/containers/snakemake-sif/current.sif -SRC=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra/src +WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra +SRC=$WT/src +# The image Snakemake pulled, resolved from the workflow's one declaration +# of it (workflow/common.py CONTAINER_URI + the candide profile's prefix). +CONTAINER=$($WT/workflow/scripts/container_path.py) SCRIPTS=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra/papers/bmodes/scripts BIND=/home,/scratch,/automnt,/n17data,/n23data1,/n09data diff --git a/papers/bmodes/scripts/run_pure_eb_sweep.sh b/papers/bmodes/scripts/run_pure_eb_sweep.sh index 9f8dc9a1..d588df84 100755 --- a/papers/bmodes/scripts/run_pure_eb_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_sweep.sh @@ -19,8 +19,10 @@ # --out [--blind A] [--versions "v1 v2 ..."] set -euo pipefail -CONTAINER=/n17data/cdaley/containers/snakemake-sif/current.sif WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra +# The image Snakemake pulled, resolved from the workflow's one declaration +# of it (workflow/common.py CONTAINER_URI + the candide profile's prefix). +CONTAINER=$($WT/workflow/scripts/container_path.py) SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts diff --git a/papers/cosmo_val/Snakefile b/papers/cosmo_val/Snakefile index 418a007c..4ad5a7ed 100644 --- a/papers/cosmo_val/Snakefile +++ b/papers/cosmo_val/Snakefile @@ -9,8 +9,6 @@ configfile: "config/config.yaml" configfile: "/n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_val/cat_config.yaml" -container: config.get("container", "/n17data/cdaley/containers/snakemake-sif/current.sif") - envvars: "PYTHONUNBUFFERED", @@ -32,6 +30,11 @@ import common common.configure(config) from common import * +# The image for every rule -- the CI tag, pulled into the profile's +# apptainer-prefix on first use (workflow/README.md). Override with +# `--config container=/path/to/my.sif`. +container: config.get("container", CONTAINER_URI) + # Wildcard constraints — centralized in common.py, not in individual rule files wildcard_constraints: **WILDCARD_CONSTRAINTS diff --git a/src/sp_validation/tests/data/container_smoke/Snakefile b/src/sp_validation/tests/data/container_smoke/Snakefile index 1cdb82b8..42c2023c 100644 --- a/src/sp_validation/tests/data/container_smoke/Snakefile +++ b/src/sp_validation/tests/data/container_smoke/Snakefile @@ -12,7 +12,10 @@ # See container_smoke.py for what the job checks and why. -container: config.get("container", "/n17data/cdaley/containers/snakemake-sif/current.sif") +# Literal rather than workflow/common.py's CONTAINER_URI: this Snakefile is +# test data, deliberately standalone. test_container_smoke.py asserts the two +# agree, so drift fails the test rather than the run. +container: config.get("container", "docker://ghcr.io/cosmostat/sp_validation:develop") rule container_smoke: diff --git a/src/sp_validation/tests/test_container_smoke.py b/src/sp_validation/tests/test_container_smoke.py index 236b5479..bdbc01de 100644 --- a/src/sp_validation/tests/test_container_smoke.py +++ b/src/sp_validation/tests/test_container_smoke.py @@ -41,6 +41,20 @@ def _reference_eigenvalues() -> np.ndarray: return np.linalg.eigh(a + a.T)[0] +def test_smoke_snakefile_names_the_workflow_image(): + """The test Snakefile's literal image must track workflow/common.py's.""" + repo_root = _repo_root() + uri = re.search( + r'^CONTAINER_URI = "(.+)"$', + (repo_root / "workflow/common.py").read_text(), + re.MULTILINE, + ).group(1) + snakefile = ( + repo_root / "src/sp_validation/tests/data/container_smoke/Snakefile" + ).read_text() + assert f'"{uri}"' in snakefile, uri + + @pytest.mark.slow @requires_cluster def test_container_smoke(): diff --git a/workflow/README.md b/workflow/README.md index faaf6128..34302cd0 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -56,10 +56,9 @@ Every rule runs inside the sp_validation container: the profile sets and Snakemake wraps each job's `shell:`/`script:` command in `apptainer exec` itself — no rule writes its own `apptainer exec` call. The image name comes from the `container:` directive in `workflow/Snakefile` (or a rule's own -override, e.g. the image-sims `SIF`). Two rules are -explicit, documented exceptions and keep `container: None` with an inline -`apptainer exec`/host-toolchain call — `xi_highres` (multi-node MPI) and -`covariance_cosmocov` (a host-compiled binary) — see their docstrings in +override, e.g. the image-sims `SIF`). One rule is an explicit, documented +exception and keeps `container: None` with an inline host-toolchain call — +`covariance_cosmocov` (a host-compiled binary) — see its docstring in `workflow/rules/`. `OMP_NUM_THREADS` is not set by the profile either: the slurm executor's `--export=ALL` propagates the driver's env, not a profile flag, so a rule that needs it pinned sets it itself. Per-rule `mem_mb` / @@ -121,63 +120,77 @@ confirm it resolves under `uv`'s tool directory (`uv tool dir`), not ### The container image -Everything runs one image, reached by one path: +Everything runs one image, named once as a registry tag: ``` -/n17data/cdaley/containers/snakemake-sif/current.sif +docker://ghcr.io/cosmostat/sp_validation:develop ``` -That single string is what `workflow/Snakefile`, the paper Snakefiles, the -image-sims `sif:` config key, the `xi_highres` MPI rule's own `apptainer exec`, -and the `papers/bmodes/scripts/run_*.sh` drivers all use. Snakemake treats a -local path as local: it never pulls, never consults a cache, and never touches -the network during a run. +That tag is what `workflow/Snakefile`, the paper Snakefiles and the image-sims +`sif:` config key declare. Nobody writes a `.sif` path: Snakemake pulls the tag +into the profile's `apptainer-prefix` +(`/n17data/cdaley/containers/snakemake-sif`) on first use and reuses the cached +file forever after. No Snakemake rule needs a file. Host-side callers that do — +the `papers/bmodes/scripts/run_*.sh` drivers, and interactive `apptainer exec` — +get it from `workflow/scripts/container_path.py`, which derives the path from +the same tag (Snakemake names a pulled image `{prefix}/{md5(uri)}.simg`; the +script just recomputes that). + +**The first pull is not free.** It happens on the host running `snakemake`, +takes ~15 minutes for ~1.5 GB, and blocks the run — so do the first run of a +new tag from a compute node, not the login node. Every later run finds the +cached file and touches neither cache nor network. **Where the image comes from.** CI (`.github/workflows/deploy-image.yml`) builds it on every push, `FROM ghcr.io/cosmostat/shapepipe:im_sims` with `uv sync --frozen` against `uv.lock`, and publishes to `ghcr.io/cosmostat/sp_validation` tagged by branch — so `:develop` tracks the -tip of `develop`. The package is public; no credentials are needed. Because -`current.sif` is a file rather than a tag, **CI publishing a new image does not -change what your jobs run.** Someone has to refresh it deliberately, which is -the point. +tip of `develop`. The package is public; no credentials are needed. A cached +pull is a *snapshot* of the tag: CI publishing a new image does not change what +your jobs run until someone refreshes. -**Refreshing** — one person does it for everybody: +**Refreshing** — one person does it for everybody. Delete the cached file and +let the next run re-pull it, or pull deliberately: ```bash # From a compute node (~1.5 GB / ~15 min; never on the login node). salloc -p comp -c 4 --time=01:00:00 --no-shell # note the job id export APPTAINER_CACHEDIR=/n17data/cdaley/containers/.apptainer-cache/cache export APPTAINER_TMPDIR=/n17data/cdaley/containers/.apptainer-cache/tmp -srun --jobid= bash -c 'cd /n17data/cdaley/containers/snakemake-sif && \ - apptainer pull --force --name next.sif \ +SIF=$(workflow/scripts/container_path.py) # prints the cached path +srun --jobid= bash -c "cd \$(dirname $SIF) && \ + apptainer pull --force --name next.simg \ docker://ghcr.io/cosmostat/sp_validation:develop && \ - mv -f next.sif current.sif' + mv -f next.simg $SIF" scancel ``` -Pull to `next.sif` and `mv` — never pull straight onto `current.sif`. `mv` +Pull to `next.simg` and `mv` — never pull straight onto the cached name. `mv` within one directory is an atomic rename, so a job either gets the whole old -image or the whole new one. Pulling in place would leave `current.sif` a -half-written file for the ~15 minutes the pull takes, and any job starting in -that window would fail. Jobs already running hold the old inode open and finish -against it unharmed. +image or the whole new one. Pulling in place would leave the file half-written +for the ~15 minutes the pull takes, and any job starting in that window would +fail. Jobs already running hold the old inode open and finish against it +unharmed. + +`snakemake --cleanup-containers` deletes every `*.simg` in the prefix that the +current DAG does not require. With the tag form the cached image *is* required, +so it survives; anything left over from an older tag is what goes. To check what you have: ```bash -apptainer inspect --labels /n17data/cdaley/containers/snakemake-sif/current.sif +apptainer inspect --labels $(workflow/scripts/container_path.py) ``` `org.opencontainers.image.revision` is the sp_validation commit the image was built from. The image-sims workflow records it in `m_bias_config.yaml` as `ghcr_revision`, so a result file says which image produced the number. -**Interactive use** — the same image, the same path: +**Interactive use** — the same image, resolved the same way: ```bash apptainer exec --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data \ - /n17data/cdaley/containers/snakemake-sif/current.sif + $(workflow/scripts/container_path.py) ``` This is what Cail's `app` shell function points at (the function lives in his diff --git a/workflow/Snakefile b/workflow/Snakefile index c517dd66..f29c1889 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -8,11 +8,6 @@ # or run standalone with --configfile pointing at a paper config # (e.g. papers/bmodes/config/config.yaml). -# The one image every call site uses, workflow and interactive alike. Built by -# CI from the GHCR image and refreshed deliberately -- see workflow/README.md. -# Override with `--config container=/path/to/my.sif`. -container: config.get("container", "/n17data/cdaley/containers/snakemake-sif/current.sif") - envvars: "PYTHONUNBUFFERED", @@ -34,6 +29,11 @@ import common common.configure(config) from common import * +# The one image every call site uses, workflow and interactive alike -- the CI +# tag, pulled into the profile's apptainer-prefix on first use (see +# workflow/README.md). Override with `--config container=/path/to/my.sif`. +container: config.get("container", CONTAINER_URI) + # Wildcard constraints — centralized in common.py, not in individual rule files wildcard_constraints: **WILDCARD_CONSTRAINTS diff --git a/workflow/common.py b/workflow/common.py index 3df3413b..a1b7c12d 100644 --- a/workflow/common.py +++ b/workflow/common.py @@ -5,6 +5,14 @@ import re from pathlib import Path +# The one image every entry point declares. A registry tag, not a file: +# Snakemake pulls it into the profile's ``apptainer-prefix`` on first use and +# reuses the cached copy thereafter (see workflow/README.md). Override anywhere +# with ``--config container=/path/to/my.sif``. Host-side callers that need a +# concrete file read this value via workflow/scripts/container_path.py. +CONTAINER_URI = "docker://ghcr.io/cosmostat/sp_validation:develop" + + # Output roots are env-overridable so a reproduction run can write into a # fresh tree without clobbering (or silently reusing) prior products. COSMO_VAL = Path( diff --git a/workflow/image_sims/config.yaml b/workflow/image_sims/config.yaml index fe8faf99..18c08171 100644 --- a/workflow/image_sims/config.yaml +++ b/workflow/image_sims/config.yaml @@ -13,9 +13,10 @@ image_sims: # One image for the whole chain: the sp_validation image is built FROM the # ShapePipe image, so it carries both stacks. CI builds it from the uv lock -- # an unlocked build drifts NumPy past numba's ceiling and the ngmix stage dies - # ("Numba needs NumPy 2.4 or less"). Same path every call site uses; refresh is - # deliberate (workflow/README.md). Override here to run your own image. - sif: /n17data/cdaley/containers/snakemake-sif/current.sif + # ("Numba needs NumPy 2.4 or less"). Same tag every call site uses; Snakemake + # pulls it into the profile's apptainer-prefix (workflow/README.md). Override + # here with a local path to run your own image. + sif: docker://ghcr.io/cosmostat/sp_validation:develop # --- repositories ----------------------------------------------------- # Bound into the image; both repos' src go on PYTHONPATH so this branch's diff --git a/workflow/profiles/candide/config.yaml b/workflow/profiles/candide/config.yaml index 1c96a2b1..979a1f4d 100644 --- a/workflow/profiles/candide/config.yaml +++ b/workflow/profiles/candide/config.yaml @@ -11,9 +11,9 @@ # (``container:`` on the rule or the module-level default in # workflow/Snakefile), never a rule's own ``apptainer exec`` shell call. # ``software-deployment-method: apptainer`` below turns that wrapping on; -# ``apptainer-args`` carries the bind mounts every rule needs. Exceptions -# (``xi_highres``, ``covariance_cosmocov``) are documented at their rule -# definitions in workflow/rules/. +# ``apptainer-args`` carries the bind mounts every rule needs. The one +# exception (``covariance_cosmocov``) is documented at its rule definition in +# workflow/rules/. # # ``snakemake`` itself is a thin host-side tool, pinned via ``uv tool # install`` (see workflow/README.md); run it on the host, never inside an @@ -26,6 +26,13 @@ executor: slurm software-deployment-method: apptainer apptainer-args: "--cleanenv --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data" +# Shared image cache. The entry Snakefiles name a GHCR *tag*; Snakemake pulls it +# here on first use (once, ~15 min, on the host running snakemake) and every +# later run reuses the cached SIF. Shared rather than per-run-directory +# (Snakemake's default is ``.snakemake/singularity``) so one pull serves every +# workflow and every checkout. +apptainer-prefix: /n17data/cdaley/containers/snakemake-sif + # Cluster policy applied to every job unless a rule overrides it. Excludes are # the flaky/no-internet candide nodes (n17 mount issues, n09 no internet, n36). # diff --git a/workflow/rules/covariance.smk b/workflow/rules/covariance.smk index 9a78dd78..791e2a11 100644 --- a/workflow/rules/covariance.smk +++ b/workflow/rules/covariance.smk @@ -146,7 +146,7 @@ EOF rule covariance_cosmocov: """Run the host-compiled CosmoCov binary. - Exception to the profile-driven container model (see + The workflow's only exception to the profile-driven container model (see workflow/profiles/candide/config.yaml): CosmoCov is a host-compiled Fortran/C binary loaded through environment-modules (`module load gcc intelpython openmpi`), not a Python entry point the container ships. diff --git a/workflow/rules/twopoint.smk b/workflow/rules/twopoint.smk index 077f3faa..260468ff 100644 --- a/workflow/rules/twopoint.smk +++ b/workflow/rules/twopoint.smk @@ -24,45 +24,58 @@ rule xi: "../scripts/run_2pcf.py" -rule xi_highres: - """High-resolution xi for COSEBIS integration. - - Exception to the profile-driven container model (see - workflow/profiles/candide/config.yaml): this is multi-node MPI, one - `apptainer exec` per rank. Snakemake's own container wrapping puts the - *whole* shell command -- `mpiexec` included -- inside a single container - instance, so only rank 0's node would run inside it; the other ranks, - spawned by SLURM/PMI on their own nodes, would land bare on the host. - `container: None` plus an explicit `mpiexec -n N apptainer exec ...` - per-rank is therefore required, not a leftover of the old convention. - Because this rule builds its own apptainer call, reaching the source-cache - copy of the script relies on our `--bind /home` rather than on Snakemake's - automatic mount. - """ - container: None - input: - script=workflow.source_path("../scripts/run_2pcf_highres.py"), - output: - txt=str(COSMO_VAL / f"{FIDUCIAL['version']}_xi_minsep={FIDUCIAL['min_sep_int']}_maxsep={FIDUCIAL['max_sep_int']}_nbins=10000_npatch=1.txt"), - xi_plus=str(COSMO_VAL / f"xi_plus_{FIDUCIAL['version']}_minsep={FIDUCIAL['min_sep_int']}_maxsep={FIDUCIAL['max_sep_int']}_nbins=10000_npatch=1.fits"), - xi_minus=str(COSMO_VAL / f"xi_minus_{FIDUCIAL['version']}_minsep={FIDUCIAL['min_sep_int']}_maxsep={FIDUCIAL['max_sep_int']}_nbins=10000_npatch=1.fits"), - resources: - tasks=30, - cpus_per_task=12, - nodes=6, - mem_mb_per_cpu=2000, - runtime=2880, - slurm_extra="'--exclude=n17,n09,n36 --partition=pscomp'", - mpi="/softs/openmpi/5.0.5-slurm-CentOS8/bin/mpiexec", - shell: - # Same image path as the top-level `container:` in workflow/Snakefile; - # this rule cannot inherit it, see docstring. - "{resources.mpi} -n {resources.tasks} " - "apptainer exec " - "--bind /home,/n09data,/n17data,/n23data1,/softs " - "--env LD_LIBRARY_PATH=/softs/openmpi/5.0.5-slurm-CentOS8/lib " - "/n17data/cdaley/containers/snakemake-sif/current.sif " - "python {input.script}" +# PARKED: xi_highres (high-resolution xi for COSEBIS integration). Never +# runnable as written -- the shell invokes run_2pcf_highres.py bare, but the +# script has required --cat-config and --out arguments (true in every version +# since it was introduced). Revive it with those arguments supplied. Keeping it +# parked leaves covariance_cosmocov as the workflow's only container exception. +# +# The MPI reasoning below is hard-won and must survive the revival: +# +# Exception to the profile-driven container model (see +# workflow/profiles/candide/config.yaml): this is multi-node MPI, one +# `apptainer exec` per rank. Snakemake's own container wrapping puts the +# *whole* shell command -- `mpiexec` included -- inside a single container +# instance, so only rank 0's node would run inside it; the other ranks, +# spawned by SLURM/PMI on their own nodes, would land bare on the host. +# `container: None` plus an explicit `mpiexec -n N apptainer exec ...` +# per-rank is therefore required, not a leftover of the old convention. +# Snakemake's slurm-jobstep plugin deliberately does NOT prepend `srun` to a +# job carrying an `mpi` resource, which is what lets the rule's own launcher +# run on the host, outside the container. +# Because this rule builds its own apptainer call, reaching the source-cache +# copy of the script relies on our `--bind /home` rather than on Snakemake's +# automatic mount -- and on a concrete image file, since `apptainer exec` +# takes no `docker://` URI. A revived rule must therefore derive that file +# from CONTAINER_URI (Snakemake pulls to `{apptainer-prefix}/{md5(uri)}.simg`; +# workflow/scripts/container_path.py does exactly this derivation) rather +# than hard-coding a second path that can drift. +# +# rule xi_highres: +# container: None +# params: +# image=, +# input: +# script=workflow.source_path("../scripts/run_2pcf_highres.py"), +# output: +# txt=str(COSMO_VAL / f"{FIDUCIAL['version']}_xi_minsep={FIDUCIAL['min_sep_int']}_maxsep={FIDUCIAL['max_sep_int']}_nbins=10000_npatch=1.txt"), +# xi_plus=str(COSMO_VAL / f"xi_plus_{FIDUCIAL['version']}_minsep={FIDUCIAL['min_sep_int']}_maxsep={FIDUCIAL['max_sep_int']}_nbins=10000_npatch=1.fits"), +# xi_minus=str(COSMO_VAL / f"xi_minus_{FIDUCIAL['version']}_minsep={FIDUCIAL['min_sep_int']}_maxsep={FIDUCIAL['max_sep_int']}_nbins=10000_npatch=1.fits"), +# resources: +# tasks=30, +# cpus_per_task=12, +# nodes=6, +# mem_mb_per_cpu=2000, +# runtime=2880, +# slurm_extra="'--exclude=n17,n09,n36 --partition=pscomp'", +# mpi="/softs/openmpi/5.0.5-slurm-CentOS8/bin/mpiexec", +# shell: +# "{resources.mpi} -n {resources.tasks} " +# "apptainer exec " +# "--bind /home,/n09data,/n17data,/n23data1,/softs " +# "--env LD_LIBRARY_PATH=/softs/openmpi/5.0.5-slurm-CentOS8/lib " +# "{params.image} " +# "python {input.script} --cat-config <...> --out <...>" rule rho_tau_stats: diff --git a/workflow/scripts/container_path.py b/workflow/scripts/container_path.py new file mode 100755 index 00000000..d3e65ea2 --- /dev/null +++ b/workflow/scripts/container_path.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +"""Print the local SIF that Snakemake pulls the workflow image into. + +For host-side callers that are *outside* Snakemake and therefore need a file +rather than a ``docker://`` tag -- the papers/bmodes/scripts drivers, and +interactive ``apptainer exec``. Resolving it here keeps the image in exactly +one place (``CONTAINER_URI`` in workflow/common.py, ``apptainer-prefix`` in the +driving profile) instead of a path anyone has to maintain by hand. + + CONTAINER=$(workflow/scripts/container_path.py) + +Stdlib only, so it runs under bare ``python3`` on the login node with no +environment to activate. Exits non-zero if the image has not been pulled yet; +the message says how to pull it. +""" + +import hashlib +import re +import sys +from pathlib import Path + +WORKFLOW = Path(__file__).resolve().parent.parent +PROFILE = WORKFLOW / "profiles/candide/config.yaml" + + +def _grep(path, pattern, what): + match = re.search(pattern, path.read_text(), re.MULTILINE) + if match is None: + sys.exit(f"{path}: could not find {what}") + return match.group(1).strip().strip("\"'") + + +uri = _grep(WORKFLOW / "common.py", r'^CONTAINER_URI = "(.+)"$', "CONTAINER_URI") +prefix = _grep(PROFILE, r"^apptainer-prefix:\s*(\S+)", "apptainer-prefix") + +sif = Path(prefix) / f"{hashlib.md5(uri.encode()).hexdigest()}.simg" +if not sif.exists(): + sys.exit( + f"{sif} not pulled yet -- run any snakemake target with " + f"--profile {PROFILE.parent} once, or pull it directly (workflow/README.md)." + ) +print(sif) diff --git a/workflow/scripts/im_mbias_config.py b/workflow/scripts/im_mbias_config.py index 20fe6e8c..39073ca1 100644 --- a/workflow/scripts/im_mbias_config.py +++ b/workflow/scripts/im_mbias_config.py @@ -40,7 +40,7 @@ def _sif_revision(): Read the image actually mounted, which Apptainer names in ``APPTAINER_CONTAINER``, rather than the configured ``sif`` -- a run may - override it, and ``current.sif`` is a moving target either way. + override it, and a registry tag says nothing about which build was pulled. Chunked plain-text scan: no exec, no container start, no 1.5 GB in memory. """ sif_path = os.environ.get("APPTAINER_CONTAINER") From b4d2a5e41134863149ecee0da3848b2d88433622 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 18:33:27 +0200 Subject: [PATCH 19/37] tests: drop the glass map-path xfail, its condition is met The xfail asked for a compatible glass+cosmology pair verified in a fresh image. glass 2026.2 with cosmology-compat-camb is that pair: the map path runs end to end (11 shells, 66 spectra, monotonic kappa accumulation), so the marker now only hides regressions. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- src/sp_validation/tests/test_glass_mock.py | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/src/sp_validation/tests/test_glass_mock.py b/src/sp_validation/tests/test_glass_mock.py index bf9e0c2c..8af73092 100644 --- a/src/sp_validation/tests/test_glass_mock.py +++ b/src/sp_validation/tests/test_glass_mock.py @@ -131,20 +131,6 @@ def test_config_change_breaks_reference(): @pytest.mark.skipif(not HAVE_GLASS, reason="GLASS not installed in this image") -@pytest.mark.xfail( - reason=( - "glass_mock map path is incompatible with the installed glass/cosmology " - "API: cosmology.compat.camb is missing entirely in the image, and where " - "it exists cosmology.Cosmology.from_camb returns a CambCosmology lacking " - "comoving_distance, which glass.distance_grid / MultiPlaneConvergence " - "require. The map path was never exercised before GLASS was added to the " - "image. Fix = pin a compatible glass+cosmology pair (or adapt the API " - "calls) and verify in the fresh image; then drop this xfail. " - "See fiber shapepipe/sp_validation glass-cosmology-api-pin." - ), - strict=False, - raises=(AttributeError, ModuleNotFoundError), -) def test_matter_maps_are_seed_deterministic(): """Same config + seed → bit-identical matter/lensing maps. From 368153adb21fd1bae8ea06aad5da509ffd160182 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 19:06:05 +0200 Subject: [PATCH 20/37] docker: install liblapack-dev, pin MPIFC absolutely cosmosis's bundled MultiNest links -llapack and the base image ships only the runtime liblapack.so.3 with no dev symlink, so the build died at 'cannot find -llapack'. It passed on candide only because that sandbox had liblapack-dev installed at some point. MPIFC takes the absolute path: /opt/ompi/bin is not always on PATH, and a miss silently drops the MPI sampler libraries while the install still reports success. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- Dockerfile | 10 +++++++--- pyproject.toml | 2 +- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Dockerfile b/Dockerfile index 35ecf855..e31dc66f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,8 @@ # Development image with more bells and whistles FROM ghcr.io/cosmostat/shapepipe:im_sims +# liblapack-dev: cosmosis's MultiNest links -llapack, and the base image ships +# only the runtime liblapack.so.3 (no dev symlink). RUN apt-get update -y --quiet --fix-missing && \ apt-get dist-upgrade -y --quiet --fix-missing && \ apt-get install -y --quiet \ @@ -10,7 +12,8 @@ RUN apt-get update -y --quiet --fix-missing && \ pkg-config \ htop \ npm \ - tmux + tmux \ + liblapack-dev # The base shapepipe image provides a uv-managed venv at /app/.venv (exported as # VIRTUAL_ENV); install sp_validation's deps into that same venv rather than @@ -38,10 +41,11 @@ COPY pyproject.toml uv.lock /sp_validation/ # cosmosis builds MPI-enabled polychord/multinest only when MPIFC is set: its # setup.py exports MPIFC for conda builds only, and the sampler Makefiles gate on -# `which $(MPIFC)`. Without this the install succeeds but omits libchord_mpi.so, +# `which $(MPIFC)`. Absolute path, not a bare name: /opt/ompi/bin is not always on +# PATH, and a miss silently omits libchord_mpi.so while the install still succeeds, # and `cosmosis --mpi` fails at load time -- which is how the pipeline runs, since # the --smp pool is broken upstream. Must precede the sync that builds cosmosis. -ENV MPIFC=mpif90 +ENV MPIFC=/opt/ompi/bin/mpif90 RUN uv sync --frozen --inexact --no-install-project \ --extra test --extra glass --extra workflow diff --git a/pyproject.toml b/pyproject.toml index ecc4c92e..84475bc8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -155,7 +155,7 @@ glass = [ workflow = [ "snakemake", # run_2pcf_highres.py drives the MPI convergence run; the container ships - # OpenMPI 4.1.4 (mpicc/mpif90 on PATH in /usr/bin) so mpi4py builds against it. + # OpenMPI under /opt/ompi so mpi4py builds against it. "mpi4py", # CosmoSIS drives the cosmo_inference sampling step. Sdist-only, so this is a # source build (~2 min) against the base image's gfortran/GSL/cfitsio/OpenMPI — From eb72b7eae9d337d314ba46151e4adb633c3bd366 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 26 Aug 2026 19:32:22 +0200 Subject: [PATCH 21/37] docker: point UV_PYTHON at the venv, not the base interpreter uv pip honours UV_PYTHON over VIRTUAL_ENV, so naming the system interpreter sent the editable install of sp_validation there instead of /app/.venv -- the image built fine and then failed its own import smoke test. The venv's own python satisfies the original intent (uv can't wander onto a host CPython from the bind-mounted $HOME) while keeping uv pip pointed at the venv. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA --- Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index e31dc66f..14369ffd 100644 --- a/Dockerfile +++ b/Dockerfile @@ -22,7 +22,7 @@ ENV UV_PROJECT_ENVIRONMENT=/app/.venv # $HOME is bind-mounted under apptainer, so uv would otherwise discover the # host's managed CPythons -- including newer ones that satisfy requires-python # -- and build a venv against an interpreter carrying none of this stack. -ENV UV_PYTHON=/usr/local/bin/python3.12 \ +ENV UV_PYTHON=/app/.venv/bin/python \ UV_PYTHON_DOWNLOADS=never WORKDIR /sp_validation From 8de6ba3f715b13228cbd76edb8fc759f3d671ee4 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 01:57:58 +0200 Subject: [PATCH 22/37] workflow: run the launched checkout's sp_validation by default Snakemake's `script:` directive already executes the checkout's script files, while `import sp_validation` resolved to the image's baked copy -- the two halves of one commit, split. `common.inject_checkout_pythonpath()` prepends the checkout's src/ to APPTAINERENV_PYTHONPATH (preserving any user-set value), so the image supplies the frozen dependency stack and the launched tree supplies sp_validation. This is what the image-sims chain has always done for both repos (`_ENV_PREFIX` in rules/image_sims.smk); the main workflow now matches it. Opt out with `--config checkout_pythonpath=false` to reproduce from the image alone. The flag is parsed tolerantly because `--config k=false` can arrive as the string "false". Also drops the hardcoded candide OpenMPI path from workflow/Snakefile: a machine path does not belong in generic workflow code, and it moves to the candide profile in a following commit. Co-Authored-By: Claude Fable 5 --- workflow/Snakefile | 16 +++++++++------ workflow/common.py | 51 +++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 56 insertions(+), 11 deletions(-) diff --git a/workflow/Snakefile b/workflow/Snakefile index f29c1889..980bed66 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -14,11 +14,10 @@ envvars: import os import sys -# Host OpenMPI libs for MPI rules inside the container. -# Apptainer passes APPTAINERENV_* vars into the container as their unprefixed names. -# Only affects rules that import mpi4py; safe for all other rules. -os.environ["APPTAINERENV_LD_LIBRARY_PATH"] = "/softs/openmpi/5.0.5-slurm-CentOS8/lib" - +# Machine-specific env (the host OpenMPI libs MPI rules need inside the +# container) is not set here: it belongs to the machine, so it rides the +# candide profile's `apptainer-args --env`. See workflow/profiles/. +# # Shared helpers live in common.py next to this Snakefile. Snakemake's `module` # imports rules, not Python globals, so helpers travel by plain Python import; # when composed, the paper Snakefile has already imported common and this hits @@ -26,12 +25,17 @@ os.environ["APPTAINERENV_LD_LIBRARY_PATH"] = "/softs/openmpi/5.0.5-slurm-CentOS8 sys.path.insert(0, os.path.realpath(str(workflow.basedir))) import common +# configure() also prepends this checkout's src/ to the container's PYTHONPATH +# (common.inject_checkout_pythonpath), so the launched tree's sp_validation -- +# not the image's baked copy -- is what rules import. `--config +# checkout_pythonpath=false` opts out. common.configure(config) from common import * # The one image every call site uses, workflow and interactive alike -- the CI # tag, pulled into the profile's apptainer-prefix on first use (see -# workflow/README.md). Override with `--config container=/path/to/my.sif`. +# workflow/README.md). Override with `--config container=`, e.g. +# a branch's own CI image `docker://ghcr.io/cosmostat/sp_validation:`. container: config.get("container", CONTAINER_URI) # Wildcard constraints — centralized in common.py, not in individual rule files diff --git a/workflow/common.py b/workflow/common.py index a1b7c12d..e422617f 100644 --- a/workflow/common.py +++ b/workflow/common.py @@ -5,13 +5,19 @@ import re from pathlib import Path -# The one image every entry point declares. A registry tag, not a file: -# Snakemake pulls it into the profile's ``apptainer-prefix`` on first use and -# reuses the cached copy thereafter (see workflow/README.md). Override anywhere -# with ``--config container=/path/to/my.sif``. Host-side callers that need a -# concrete file read this value via workflow/scripts/container_path.py. +# The one image every entry point declares, and the single source of truth for +# it. A registry tag, not a file: Snakemake pulls it into the profile's +# ``apptainer-prefix`` on first use and reuses the cached copy thereafter (see +# workflow/README.md). Override anywhere with ``--config +# container=docker://ghcr.io/cosmostat/sp_validation:`` (to run a +# branch's own CI image) or a local ``.sif`` path. Host-side callers that need a +# concrete file use ``{apptainer-prefix}/current.sif``, the symlink the refresh +# recipe maintains onto whatever Snakemake last pulled. CONTAINER_URI = "docker://ghcr.io/cosmostat/sp_validation:develop" +# This checkout's importable source tree: workflow/common.py -> /src. +REPO_SRC = Path(__file__).resolve().parent.parent / "src" + # Output roots are env-overridable so a reproduction run can write into a # fresh tree without clobbering (or silently reusing) prior products. @@ -58,9 +64,44 @@ PLANCK18 = None +def inject_checkout_pythonpath(workflow_config): + """Make the launched checkout's ``src`` win over the image's baked copy. + + Snakemake's ``script:`` directive already runs the *checkout's* script + files, so without this a rule executes new script code against an old + ``import sp_validation`` -- the two halves of one commit, split. Prepending + ``REPO_SRC`` closes that: the image stays the frozen dependency stack, the + checkout supplies sp_validation. This mirrors what the image-sims chain has + always done for both repos (``_ENV_PREFIX`` in rules/image_sims.smk). + + Apptainer forwards ``APPTAINERENV_``-prefixed host variables into the job as + their unprefixed names, surviving the profile's ``--cleanenv``; setting it + here on the driver reaches every containerized rule. Any value the user + already exported is preserved behind ours. + + Opt out with ``--config checkout_pythonpath=false`` to reproduce a run from + the image alone. Note the caveat: ``rerun-triggers: code`` watches rule and + script files, not ``src/``, so editing a module under ``src/`` does not by + itself mark outputs stale -- force with ``-F``/``--forcerun``. + """ + flag = workflow_config.get("checkout_pythonpath", True) + # `--config key=false` can arrive as the *string* "false" depending on how + # Snakemake parses the value, so don't lean on truthiness alone. + if isinstance(flag, str): + flag = flag.strip().lower() not in ("false", "no", "0", "off", "") + if not flag: + return + if not REPO_SRC.is_dir(): + return + existing = os.environ.get("APPTAINERENV_PYTHONPATH", "") + parts = [str(REPO_SRC)] + [p for p in existing.split(":") if p] + os.environ["APPTAINERENV_PYTHONPATH"] = ":".join(parts) + + def configure(workflow_config): """Install config-derived values after Snakemake has loaded configfiles.""" global CATALOG_CONFIG, DEFAULT_MASK_SUFFIX, FIDUCIAL, PLANCK18 + inject_checkout_pythonpath(workflow_config) CATALOG_CONFIG = workflow_config FIDUCIAL = workflow_config["fiducial"] DEFAULT_MASK_SUFFIX = ( From 236a19a7a333034ca78078b2a23833cd9383acd6 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 01:57:58 +0200 Subject: [PATCH 23/37] profiles: add a machine-independent default, make candide the machine layer workflow/profiles/default carries the container model and nothing else, so the workflow runs off candide with `--profile workflow/profiles/default -j N`. Snakemake cannot compose profiles (one --profile, no `inherits:`), so the small machine-independent set -- software-deployment-method, rerun-triggers, latency-wait -- is duplicated verbatim in both files, marked GENERIC and cross-referenced. That is the least-magic arrangement available. apptainer-args and apptainer-prefix stay out of the shared block: every machine has its own disks and image cache. candide's apptainer-args now carries `--env LD_LIBRARY_PATH=/softs/openmpi/...`, previously an os.environ line in workflow/Snakefile. Co-Authored-By: Claude Fable 5 --- workflow/profiles/candide/config.yaml | 53 +++++++++++++++++++-------- workflow/profiles/default/config.yaml | 46 +++++++++++++++++++++++ 2 files changed, 84 insertions(+), 15 deletions(-) create mode 100644 workflow/profiles/default/config.yaml diff --git a/workflow/profiles/candide/config.yaml b/workflow/profiles/candide/config.yaml index 979a1f4d..e9448ebf 100644 --- a/workflow/profiles/candide/config.yaml +++ b/workflow/profiles/candide/config.yaml @@ -11,28 +11,56 @@ # (``container:`` on the rule or the module-level default in # workflow/Snakefile), never a rule's own ``apptainer exec`` shell call. # ``software-deployment-method: apptainer`` below turns that wrapping on; -# ``apptainer-args`` carries the bind mounts every rule needs. The one -# exception (``covariance_cosmocov``) is documented at its rule definition in -# workflow/rules/. +# ``apptainer-args`` carries the bind mounts every rule needs. Three rules are +# documented exceptions and keep ``container: None`` -- they are enumerated in +# workflow/README.md. # # ``snakemake`` itself is a thin host-side tool, pinned via ``uv tool # install`` (see workflow/README.md); run it on the host, never inside an # ``apptainer shell``. +# +# Off candide, use ``--profile workflow/profiles/default -j N`` instead: that +# profile is the container model with none of the SLURM/machine layer. +# Snakemake cannot compose profiles -- ``--profile`` takes exactly one directory +# and there is no ``inherits:`` key -- so the machine-independent settings +# (marked GENERIC below) are duplicated there verbatim. THE TWO ARE A PAIR: +# change one, change the other. ``apptainer-args`` and ``apptainer-prefix`` are +# NOT generic: each machine has its own disks and its own image cache. executor: slurm -# Kept in sync with the ``app`` bash function's raw ``apptainer exec`` in the -# top-level UNIONS CLAUDE.md -- update both together. +# --- GENERIC: mirrored in workflow/profiles/default/config.yaml ------------- software-deployment-method: apptainer -apptainer-args: "--cleanenv --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data" + +# Rerun a job when its code / params / inputs change, not only on mtime. +# Caveat: "code" watches rule bodies and ``script:`` files, not ``src/`` -- +# editing a module under src/ does not mark outputs stale on its own. +rerun-triggers: ["mtime", "params", "input", "code"] + +# Give an appearing output file a moment on networked filesystems before +# Snakemake calls a job failed for a missing output. +latency-wait: 5 +# --- end GENERIC ------------------------------------------------------------ # Shared image cache. The entry Snakefiles name a GHCR *tag*; Snakemake pulls it # here on first use (once, ~15 min, on the host running snakemake) and every # later run reuses the cached SIF. Shared rather than per-run-directory # (Snakemake's default is ``.snakemake/singularity``) so one pull serves every -# workflow and every checkout. +# workflow and every checkout. ``current.sif`` in this directory is the stable +# name host-side callers use (workflow/README.md). apptainer-prefix: /n17data/cdaley/containers/snakemake-sif +# candide's disks, plus the one machine-specific env var: the host OpenMPI libs +# MPI rules need to find libmpi inside the container (only rules importing +# mpi4py care; harmless for the rest). This was an ``os.environ[...]`` line in +# workflow/Snakefile -- a machine path hard-coded into generic workflow code -- +# and belongs here. Kept in sync with the ``app`` bash function's raw +# ``apptainer exec`` in the top-level UNIONS CLAUDE.md: update both together. +apptainer-args: >- + --cleanenv + --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data + --env LD_LIBRARY_PATH=/softs/openmpi/5.0.5-slurm-CentOS8/lib + # Cluster policy applied to every job unless a rule overrides it. Excludes are # the flaky/no-internet candide nodes (n17 mount issues, n09 no internet, n36). # @@ -59,13 +87,8 @@ default-resources: cpus_per_task: 12 slurm_extra: "'--exclude=n17,n09,n36'" -# Give an appearing output file a moment on candide's automounted filesystems -# before Snakemake calls a job failed for a missing output, and retry a job -# once on transient node failure. -latency-wait: 5 +# Retry a job once on transient node failure, and keep the SLURM logs of +# successful jobs (candide debugging). (``latency-wait`` and ``rerun-triggers`` +# are in the GENERIC block above.) retries: 1 - -# Keep the SLURM logs of successful jobs (candide debugging), and rerun a job -# when its code / params / inputs change, not only on mtime. slurm-keep-successful-logs: true -rerun-triggers: ["mtime", "params", "input", "code"] diff --git a/workflow/profiles/default/config.yaml b/workflow/profiles/default/config.yaml new file mode 100644 index 00000000..49e65375 --- /dev/null +++ b/workflow/profiles/default/config.yaml @@ -0,0 +1,46 @@ +# Machine-independent profile: the container model, and nothing else. +# +# Use it anywhere that is not candide -- a laptop, a workstation, another +# cluster's interactive node: +# +# snakemake --profile workflow/profiles/default -s workflow/Snakefile \ +# --configfile -j 4 +# +# On candide use `--profile workflow/profiles/candide` instead: that profile is +# the GENERIC block below plus the SLURM executor, account/partition, node +# excludes, candide's binds and the host OpenMPI library path. Snakemake cannot +# compose profiles -- `--profile` takes exactly one directory and there is no +# `inherits:` key -- so the GENERIC block is duplicated in both files. THE TWO +# ARE A PAIR: change one, change the other. +# +# Requirements are the same everywhere: `apptainer` on PATH, and `snakemake` +# installed host-side (`uv tool install ...`, see workflow/README.md) -- never +# run from inside an apptainer shell. + +# --- GENERIC: mirrored in workflow/profiles/candide/config.yaml ------------- +# Turn on Snakemake's own container wrapping: it wraps each job's +# `shell:`/`script:` command in `apptainer exec`, using the image named by the +# entry Snakefile's `container:` directive. No rule writes its own +# `apptainer exec` call; the documented exceptions that opt out entirely with +# `container: None` are listed in workflow/README.md. +software-deployment-method: apptainer + +# Rerun a job when its code / params / inputs change, not only on mtime. +# Caveat: "code" watches rule bodies and `script:` files, not `src/` -- editing +# a module under src/ does not mark outputs stale on its own. +rerun-triggers: ["mtime", "params", "input", "code"] + +# Give an appearing output file a moment on networked filesystems before +# Snakemake calls a job failed for a missing output. +latency-wait: 5 +# --- end GENERIC ------------------------------------------------------------ + +# Binds are the one thing you almost certainly need to edit for your machine: +# whatever paths your inputs, outputs and checkout live under. `--cleanenv` so a +# job's environment is the image's, not your shell's. If $HOME and the working +# directory cover everything (apptainer mounts both by default), drop `--bind`. +apptainer-args: "--cleanenv --bind /home" + +# No `apptainer-prefix`: pulled images land in `.snakemake/singularity` under +# the working directory. Set one (an absolute path) if several checkouts or +# users on this machine should share one pull, as candide's profile does. From 78ae806e5fcb73ff63f7faae9cd1277ba8c7bb26 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 02:15:09 +0200 Subject: [PATCH 24/37] presentation: containerize the two rules that ran bare python The Moriond talk-figure rules called `python` with no container, so they ran against whatever interpreter the driver happened to have. Let them inherit the module-level `container:` like every other rule. The two ImageMagick `convert` rules keep `container: None`: `convert` is a host tool, absent from the image. Same for covariance_cosmocov, whose docstring says so directly rather than pointing at a list elsewhere. Co-Authored-By: Claude Fable 5 --- papers/bmodes/rules/presentation.smk | 16 ++++++++++------ workflow/rules/covariance.smk | 8 +++----- 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/papers/bmodes/rules/presentation.smk b/papers/bmodes/rules/presentation.smk index 984172bb..c59f0aa3 100644 --- a/papers/bmodes/rules/presentation.smk +++ b/papers/bmodes/rules/presentation.smk @@ -59,7 +59,11 @@ rule talk_previews: rule talk_figure: - """Convert a single paper PDF to high-res PNG for the talk.""" + """Convert a single paper PDF to high-res PNG for the talk. + + `container: None` on purpose: ImageMagick's `convert` is a host tool and is + not installed in the sp_validation image. + """ input: pdf=lambda w: TALK_FIGURES[w.name], output: @@ -73,7 +77,11 @@ rule talk_figure: rule talk_figure_preview: - """Downscale a talk figure to < 1800px for safe AI reading.""" + """Downscale a talk figure to < 1800px for safe AI reading. + + `container: None` for the same reason as talk_figure: `convert` is a host + tool, absent from the image. + """ input: f"{TALK_DIR}/images/{{name}}.png", output: @@ -120,8 +128,6 @@ rule presentation_omega_m_difference: mock_summary="/n09data/guerrini/glass_mock_chains/summary_parameter_constraints_merged_v6.txt", output: f"{TALK_DIR}/images/omega_m_difference_config_harm.png", - container: - None shell: "python {TALK_DIR}/plot_omega_m_difference.py" @@ -132,8 +138,6 @@ rule presentation_s8_scatter_mocks: mock_summary="/n09data/guerrini/glass_mock_chains/summary_parameter_constraints_merged_v6.txt", output: f"{TALK_DIR}/images/s8_scatter_config_vs_harmonic.png", - container: - None shell: "python {TALK_DIR}/plot_s8_scatter_mocks.py" diff --git a/workflow/rules/covariance.smk b/workflow/rules/covariance.smk index 791e2a11..e9185428 100644 --- a/workflow/rules/covariance.smk +++ b/workflow/rules/covariance.smk @@ -146,11 +146,9 @@ EOF rule covariance_cosmocov: """Run the host-compiled CosmoCov binary. - The workflow's only exception to the profile-driven container model (see - workflow/profiles/candide/config.yaml): CosmoCov is a host-compiled - Fortran/C binary loaded through environment-modules (`module load gcc - intelpython openmpi`), not a Python entry point the container ships. - `container: None` is required here, not a leftover of the old convention. + `container: None` on purpose: CosmoCov is a host-compiled Fortran/C binary + loaded through environment-modules (`module load gcc intelpython openmpi`), + not a Python entry point the container ships. """ input: rules.covariance_ini.output, From 314fda3836933bce2d9ee4875ff89f953f6eb621 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 02:15:09 +0200 Subject: [PATCH 25/37] image_sims: default sif to null and fall back to the workflow's image MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `image_sims: {sif: ...}` was a required structural key, so every run config repeated the image path — a second place for it to drift from what the rest of the workflow runs. Default it to null and resolve it through the same code path as every other entry point; a run config still overrides it to name its own image or a branch tag. Co-Authored-By: Claude Fable 5 --- workflow/image_sims/Snakefile | 11 +++++++++++ workflow/image_sims/config.yaml | 13 +++++++++---- workflow/rules/image_sims.smk | 7 ++++++- 3 files changed, 26 insertions(+), 5 deletions(-) diff --git a/workflow/image_sims/Snakefile b/workflow/image_sims/Snakefile index 457ec750..7420ae68 100644 --- a/workflow/image_sims/Snakefile +++ b/workflow/image_sims/Snakefile @@ -23,6 +23,17 @@ The same rules are also included in the main workflow under ``if "image_sims" in config``. """ +import os +import sys + +# Shared helpers from the generic workflow one directory up -- resolve_container +# in particular, so `sif: null` here falls back to the same image every other +# entry point runs. Snakemake's `include:` shares Python globals, so +# image_sims.smk sees this import. +sys.path.insert(0, os.path.realpath(os.path.join(str(workflow.basedir), ".."))) +import common + + configfile: "workflow/image_sims/config.yaml" diff --git a/workflow/image_sims/config.yaml b/workflow/image_sims/config.yaml index 18c08171..f5f3e3c0 100644 --- a/workflow/image_sims/config.yaml +++ b/workflow/image_sims/config.yaml @@ -13,10 +13,15 @@ image_sims: # One image for the whole chain: the sp_validation image is built FROM the # ShapePipe image, so it carries both stacks. CI builds it from the uv lock -- # an unlocked build drifts NumPy past numba's ceiling and the ngmix stage dies - # ("Numba needs NumPy 2.4 or less"). Same tag every call site uses; Snakemake - # pulls it into the profile's apptainer-prefix (workflow/README.md). Override - # here with a local path to run your own image. - sif: docker://ghcr.io/cosmostat/sp_validation:develop + # ("Numba needs NumPy 2.4 or less"). + # + # `null` means the workflow's one image: your own .sif if you have pulled one + # with `spv-container pull`, else the registry tag for Snakemake to autopull + # (workflow/README.md). Neither is repeated here. + # Override with a local .sif path, or with a branch tag + # (docker://ghcr.io/cosmostat/sp_validation:) to run a + # branch's own CI image. + sif: null # --- repositories ----------------------------------------------------- # Bound into the image; both repos' src go on PYTHONPATH so this branch's diff --git a/workflow/rules/image_sims.smk b/workflow/rules/image_sims.smk index 855987c3..682bc1ba 100644 --- a/workflow/rules/image_sims.smk +++ b/workflow/rules/image_sims.smk @@ -96,7 +96,12 @@ if _missing_structural: # module-level default: these rules are also included from the top-level # workflow/Snakefile, whose module default is the cosmology image (no ShapePipe # stack). Binds come from the driving profile's ``apptainer-args``. -SIF = IMSIM["sif"] +# +# ``sif: null`` (the config default) means "the workflow's one image", resolved +# the same way every other entry point resolves it: this user's own .sif if they +# have pulled one, else the registry tag. Set ``image_sims: {sif: ...}`` in a +# run config to name a different image or a branch tag. +SIF = common.resolve_container({"container": IMSIM["sif"]}) # --- repositories (bound into the image; branch code overrides) ----------- SHAPEPIPE_REPO = IMSIM["shapepipe_repo"] From 260e2e5646c5087d25ce06c40e896927d917b118 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 02:15:23 +0200 Subject: [PATCH 26/37] containers: give every user their own image, driven by spv-container MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The workflow ran out of one shared image directory on candide (/n17data/cdaley/containers/snakemake-sif) with a hand-maintained current.sif symlink pointing at whatever Snakemake last autopulled. That only worked for one person: refreshing the image or repointing the symlink needed write access to another user's directory, and a refresh moved the ground under everyone at once. Everyone now runs their own image file at one canonical per-user path, ~/.cache/sp_validation/sp_validation.sif (SPV_CONTAINER overrides it), owned by a small CLI: spv-container pull # fetch the tag there, atomically spv-container status # revision label vs. this checkout's HEAD spv-container exec # one-off run inside it, candide binds applied sp_validation.container is stdlib-only on purpose: it runs on the host, outside the container, so it must import without the scientific stack — and it works straight from a checkout (`python3 src/sp_validation/container.py status`) with nothing installed. It also holds CONTAINER_URI, which workflow/common.py loads from this checkout by file path, so the CLI and the workflow can never name different images. `container:` now resolves to that local .sif when it exists and to the registry tag otherwise (Snakemake accepts either, and autopulls the tag into .snakemake/singularity). `--config container=...` still overrides both. The candide profile drops apptainer-prefix accordingly, and common.configure() warns — once, never fatally — when the local image predates the checkout. Also drops workflow/scripts/container_path.py, which existed to locate the shared cache. Co-Authored-By: Claude Fable 5 --- papers/bmodes/scripts/run_cov_sweep.sh | 6 +- .../bmodes/scripts/run_pure_eb_ptes_sweep.sh | 6 +- .../scripts/run_pure_eb_semianalytic.sh | 6 +- papers/bmodes/scripts/run_pure_eb_sweep.sh | 6 +- pyproject.toml | 7 + src/sp_validation/container.py | 230 ++++++++++++++++++ .../tests/data/container_smoke/Snakefile | 6 +- .../tests/test_container_smoke.py | 4 +- workflow/Snakefile | 9 +- workflow/common.py | 72 +++++- workflow/profiles/candide/config.yaml | 33 +-- workflow/profiles/default/config.yaml | 22 +- workflow/rules/twopoint.smk | 12 +- workflow/scripts/container_path.py | 42 ---- 14 files changed, 349 insertions(+), 112 deletions(-) create mode 100644 src/sp_validation/container.py delete mode 100755 workflow/scripts/container_path.py diff --git a/papers/bmodes/scripts/run_cov_sweep.sh b/papers/bmodes/scripts/run_cov_sweep.sh index 5cf83b36..c2515a0c 100755 --- a/papers/bmodes/scripts/run_cov_sweep.sh +++ b/papers/bmodes/scripts/run_cov_sweep.sh @@ -27,9 +27,9 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# The image Snakemake pulled, resolved from the workflow's one declaration -# of it (workflow/common.py CONTAINER_URI + the candide profile's prefix). -CONTAINER=$($WT/workflow/scripts/container_path.py) +# This user's own image, at the canonical path `spv-container pull` writes to +# (see workflow/README.md). SPV_CONTAINER overrides it here and everywhere else. +CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts diff --git a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh index b8988902..d37910f2 100644 --- a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh @@ -20,9 +20,9 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# The image Snakemake pulled, resolved from the workflow's one declaration -# of it (workflow/common.py CONTAINER_URI + the candide profile's prefix). -CONTAINER=$($WT/workflow/scripts/container_path.py) +# This user's own image, at the canonical path `spv-container pull` writes to +# (see workflow/README.md). SPV_CONTAINER overrides it here and everywhere else. +CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts diff --git a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh index af2d95bd..a098fbf3 100644 --- a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh +++ b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh @@ -16,9 +16,9 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra SRC=$WT/src -# The image Snakemake pulled, resolved from the workflow's one declaration -# of it (workflow/common.py CONTAINER_URI + the candide profile's prefix). -CONTAINER=$($WT/workflow/scripts/container_path.py) +# This user's own image, at the canonical path `spv-container pull` writes to +# (see workflow/README.md). SPV_CONTAINER overrides it here and everywhere else. +CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} SCRIPTS=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra/papers/bmodes/scripts BIND=/home,/scratch,/automnt,/n17data,/n23data1,/n09data diff --git a/papers/bmodes/scripts/run_pure_eb_sweep.sh b/papers/bmodes/scripts/run_pure_eb_sweep.sh index d588df84..739942e4 100755 --- a/papers/bmodes/scripts/run_pure_eb_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_sweep.sh @@ -20,9 +20,9 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# The image Snakemake pulled, resolved from the workflow's one declaration -# of it (workflow/common.py CONTAINER_URI + the candide profile's prefix). -CONTAINER=$($WT/workflow/scripts/container_path.py) +# This user's own image, at the canonical path `spv-container pull` writes to +# (see workflow/README.md). SPV_CONTAINER overrides it here and everywhere else. +CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts diff --git a/pyproject.toml b/pyproject.toml index 84475bc8..c43499f6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -112,6 +112,13 @@ dependencies = [ [project.urls] Homepage = "https://github.com/CosmoStat/sp_validation" +[project.scripts] +# Host-side management of your own container image (sp_validation.container). +# Stdlib-only by design -- it has to run on the host, outside the container -- +# so it also works straight from a checkout with nothing installed: +# `python3 src/sp_validation/container.py status`. +spv-container = "sp_validation.container:main" + [tool.uv] # The reproducibility target is the Linux container; scope the lock to Linux # (mirrors shapepipe) so `uv lock` resolves the linux-centric stack (pymaster, diff --git a/src/sp_validation/container.py b/src/sp_validation/container.py new file mode 100644 index 00000000..251b2141 --- /dev/null +++ b/src/sp_validation/container.py @@ -0,0 +1,230 @@ +"""Manage this user's local copy of the sp_validation container image. + +Everyone runs their own image file. There is no shared image directory and no +symlink to keep honest: the canonical path is under your own cache +(``~/.cache/sp_validation/sp_validation.sif``), you refresh it when you want to, +and nobody else's refresh moves the ground under a running job. + +Three subcommands, exposed as the ``spv-container`` console script:: + + spv-container pull # fetch the tag to the canonical path + spv-container status # is it there, and which commit is it? + spv-container exec # run something inside it + +This module is deliberately **stdlib-only** (``argparse``/``subprocess``/ +``pathlib``). It runs on the *host*, outside the container, where the science +stack is not installed -- so it must import without it. That also means it works +straight from a checkout with no install at all:: + + python3 src/sp_validation/container.py pull +""" + +import argparse +import os +import shutil +import subprocess +import sys +from pathlib import Path + +# The image every entry point names. CI builds and pushes one per branch, tagged +# by the sanitized branch name, so ``:develop`` tracks the integration branch. +# ``workflow/common.py`` re-exports this as ``CONTAINER_URI``; it is written down +# here, once. +CONTAINER_URI = "docker://ghcr.io/cosmostat/sp_validation:develop" + +# Where this user's image lives. Per-user by construction: one file, one owner, +# no coordination. Override with ``SPV_CONTAINER`` (an absolute path). +DEFAULT_SIF = ( + Path(os.environ.get("XDG_CACHE_HOME", "~/.cache")) + / "sp_validation" + / "sp_validation.sif" +) + +# Bind mounts for interactive `exec`. candide's disks; override wholesale with +# ``SPV_APPTAINER_BINDS`` or per-call with ``--bind``. +DEFAULT_BINDS = "/home,/scratch,/automnt,/n17data,/n23data1,/n09data" + + +def local_sif(): + """Return this user's canonical image path (may not exist yet).""" + override = os.environ.get("SPV_CONTAINER") + path = Path(override) if override else DEFAULT_SIF + return path.expanduser() + + +def image_labels(sif): + """Return the image's OCI labels as a dict, or ``{}`` if unreadable. + + Never raises: a missing file, a missing ``apptainer``, or a corrupt image + all mean "we don't know", which every caller here treats as non-fatal. + """ + sif = Path(sif) + if not sif.exists() or shutil.which("apptainer") is None: + return {} + try: + out = subprocess.run( + ["apptainer", "inspect", "--labels", str(sif)], + capture_output=True, + text=True, + timeout=60, + ) + except (OSError, subprocess.SubprocessError): + return {} + if out.returncode != 0: + return {} + labels = {} + for line in out.stdout.splitlines(): + key, sep, value = line.partition(":") + if sep: + labels[key.strip()] = value.strip() + return labels + + +def image_revision(sif): + """Return the sp_validation commit the image was built from, or ``None``.""" + return image_labels(sif).get("org.opencontainers.image.revision") + + +def _git(*args, cwd=None): + """Run a git command, returning stripped stdout or ``None`` on any failure.""" + try: + out = subprocess.run( + ["git", *args], capture_output=True, text=True, cwd=cwd, timeout=30 + ) + except (OSError, subprocess.SubprocessError): + return None + return out.stdout.strip() if out.returncode == 0 else None + + +def compare_revision(revision, repo=None): + """Place an image revision relative to a checkout's HEAD. + + Returns one of ``"in-sync"``, ``"behind"`` (the image predates HEAD), + ``"ahead"`` (HEAD predates the image), ``"diverged"``, or ``"unknown"`` + (no revision label, no git, or a commit this clone has never fetched). + """ + if not revision: + return "unknown" + repo = repo or Path(__file__).resolve().parents[2] + head = _git("rev-parse", "HEAD", cwd=repo) + if head is None: + return "unknown" + if head == revision: + return "in-sync" + if _git("cat-file", "-e", f"{revision}^{{commit}}", cwd=repo) is None: + return "unknown" + ancestor = subprocess.run( + ["git", "merge-base", "--is-ancestor", revision, head], + capture_output=True, + cwd=repo, + ) + if ancestor.returncode == 0: + return "behind" + reverse = subprocess.run( + ["git", "merge-base", "--is-ancestor", head, revision], + capture_output=True, + cwd=repo, + ) + return "ahead" if reverse.returncode == 0 else "diverged" + + +def cmd_pull(args): + """Pull ``--tag`` to the canonical path, atomically.""" + if shutil.which("apptainer") is None: + sys.exit("apptainer is not on PATH") + sif = local_sif() + sif.parent.mkdir(parents=True, exist_ok=True) + # Pull to a sibling temp name and rename. `mv` within one directory is an + # atomic rename, so a job either gets the whole old image or the whole new + # one; pulling in place would leave the file half-written for the ~15 + # minutes the pull takes, and anything starting in that window would fail. + # Jobs already running hold the old inode open and finish against it. + tmp = sif.with_name(sif.name + f".pull.{os.getpid()}") + print(f"pulling {args.tag}\n -> {sif}") + try: + subprocess.run( + ["apptainer", "pull", "--force", "--name", str(tmp), args.tag], check=True + ) + os.replace(tmp, sif) + except subprocess.CalledProcessError as exc: + tmp.unlink(missing_ok=True) + sys.exit(f"pull failed ({exc.returncode})") + except KeyboardInterrupt: + tmp.unlink(missing_ok=True) + raise + labels = image_labels(sif) + print(f"revision: {labels.get('org.opencontainers.image.revision', 'unknown')}") + print(f"version: {labels.get('org.opencontainers.image.version', 'unknown')}") + return 0 + + +def cmd_status(args): + """Report the image's presence, revision, and standing against the checkout.""" + sif = local_sif() + if not sif.exists(): + print(f"no image at {sif}\nrun: spv-container pull") + return 1 + size_gb = sif.stat().st_size / 1e9 + print(f"image: {sif} ({size_gb:.1f} GB)") + labels = image_labels(sif) + revision = labels.get("org.opencontainers.image.revision") + print(f"revision: {revision or 'unknown'}") + print(f"version: {labels.get('org.opencontainers.image.version', 'unknown')}") + verdict = compare_revision(revision) + explain = { + "in-sync": "matches this checkout's HEAD", + "behind": "older than this checkout's HEAD -- pull to refresh", + "ahead": "newer than this checkout's HEAD", + "diverged": "on a different branch from this checkout", + "unknown": "cannot compare (no label, or a commit this clone lacks)", + }[verdict] + print(f"checkout: {verdict} ({explain})") + return 0 + + +def cmd_exec(args): + """Run a command inside the canonical image -- the one-off testing path.""" + if shutil.which("apptainer") is None: + sys.exit("apptainer is not on PATH") + sif = local_sif() + if not sif.exists(): + sys.exit(f"no image at {sif}; run: spv-container pull") + if not args.command: + sys.exit("nothing to run; pass a command after `exec`") + binds = args.bind or os.environ.get("SPV_APPTAINER_BINDS", DEFAULT_BINDS) + cmd = ["apptainer", "exec", "--cleanenv", "--bind", binds, str(sif), *args.command] + return subprocess.run(cmd).returncode + + +def build_parser(): + parser = argparse.ArgumentParser( + prog="spv-container", description=__doc__.splitlines()[0] + ) + sub = parser.add_subparsers(dest="subcommand", required=True) + + p_pull = sub.add_parser("pull", help="fetch the image to the canonical path") + p_pull.add_argument( + "--tag", + default=CONTAINER_URI, + help=f"image to pull (default: {CONTAINER_URI})", + ) + p_pull.set_defaults(func=cmd_pull) + + p_status = sub.add_parser("status", help="report the local image and its revision") + p_status.set_defaults(func=cmd_status) + + p_exec = sub.add_parser("exec", help="run a command inside the local image") + p_exec.add_argument("--bind", help=f"bind mounts (default: {DEFAULT_BINDS})") + p_exec.add_argument("command", nargs=argparse.REMAINDER) + p_exec.set_defaults(func=cmd_exec) + + return parser + + +def main(argv=None): + args = build_parser().parse_args(argv) + return args.func(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/sp_validation/tests/data/container_smoke/Snakefile b/src/sp_validation/tests/data/container_smoke/Snakefile index 42c2023c..8843333c 100644 --- a/src/sp_validation/tests/data/container_smoke/Snakefile +++ b/src/sp_validation/tests/data/container_smoke/Snakefile @@ -12,9 +12,9 @@ # See container_smoke.py for what the job checks and why. -# Literal rather than workflow/common.py's CONTAINER_URI: this Snakefile is -# test data, deliberately standalone. test_container_smoke.py asserts the two -# agree, so drift fails the test rather than the run. +# Literal rather than the package's CONTAINER_URI: this Snakefile is test data, +# deliberately standalone. test_container_smoke.py asserts the two agree, so +# drift fails the test rather than the run. container: config.get("container", "docker://ghcr.io/cosmostat/sp_validation:develop") diff --git a/src/sp_validation/tests/test_container_smoke.py b/src/sp_validation/tests/test_container_smoke.py index bdbc01de..0bb0068f 100644 --- a/src/sp_validation/tests/test_container_smoke.py +++ b/src/sp_validation/tests/test_container_smoke.py @@ -42,11 +42,11 @@ def _reference_eigenvalues() -> np.ndarray: def test_smoke_snakefile_names_the_workflow_image(): - """The test Snakefile's literal image must track workflow/common.py's.""" + """The test Snakefile's literal image must track the package's CONTAINER_URI.""" repo_root = _repo_root() uri = re.search( r'^CONTAINER_URI = "(.+)"$', - (repo_root / "workflow/common.py").read_text(), + (repo_root / "src/sp_validation/container.py").read_text(), re.MULTILINE, ).group(1) snakefile = ( diff --git a/workflow/Snakefile b/workflow/Snakefile index 980bed66..a24d7dc1 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -32,11 +32,10 @@ import common common.configure(config) from common import * -# The one image every call site uses, workflow and interactive alike -- the CI -# tag, pulled into the profile's apptainer-prefix on first use (see -# workflow/README.md). Override with `--config container=`, e.g. -# a branch's own CI image `docker://ghcr.io/cosmostat/sp_validation:`. -container: config.get("container", CONTAINER_URI) +# The one image every rule runs in: this user's own .sif if they have pulled one +# with `spv-container pull`, else the CI tag for Snakemake to autopull. Override +# with `--config container=`. See common.resolve_container. +container: common.resolve_container(config) # Wildcard constraints — centralized in common.py, not in individual rule files wildcard_constraints: diff --git a/workflow/common.py b/workflow/common.py index e422617f..1e956249 100644 --- a/workflow/common.py +++ b/workflow/common.py @@ -1,23 +1,37 @@ """Shared helpers for the B-modes Snakemake workflow.""" +import importlib.util import json import os import re +import sys from pathlib import Path -# The one image every entry point declares, and the single source of truth for -# it. A registry tag, not a file: Snakemake pulls it into the profile's -# ``apptainer-prefix`` on first use and reuses the cached copy thereafter (see -# workflow/README.md). Override anywhere with ``--config -# container=docker://ghcr.io/cosmostat/sp_validation:`` (to run a -# branch's own CI image) or a local ``.sif`` path. Host-side callers that need a -# concrete file use ``{apptainer-prefix}/current.sif``, the symlink the refresh -# recipe maintains onto whatever Snakemake last pulled. -CONTAINER_URI = "docker://ghcr.io/cosmostat/sp_validation:develop" - # This checkout's importable source tree: workflow/common.py -> /src. REPO_SRC = Path(__file__).resolve().parent.parent / "src" +# The container model lives in the package (``sp_validation/container.py``): the +# registry tag, this user's canonical ``.sif`` path, and the ``spv-container`` +# CLI that fills it. Taken from *this checkout's* src/, so the workflow and the +# CLI can never disagree about either. +# +# Loaded by file path rather than as ``sp_validation.container``: snakemake runs +# on the host, where sp_validation is usually not installed, and importing the +# package would drag in ``__init__`` -> ``version`` -> a metadata warning on +# every launch. The module itself is stdlib-only, so this costs nothing. +_container = importlib.util.module_from_spec( + importlib.util.spec_from_file_location( + "_spv_container", REPO_SRC / "sp_validation" / "container.py" + ) +) +sys.modules["_spv_container"] = _container +_container.__loader__.exec_module(_container) + +CONTAINER_URI = _container.CONTAINER_URI +compare_revision = _container.compare_revision +image_revision = _container.image_revision +local_sif = _container.local_sif + # Output roots are env-overridable so a reproduction run can write into a # fresh tree without clobbering (or silently reusing) prior products. @@ -98,10 +112,48 @@ def inject_checkout_pythonpath(workflow_config): os.environ["APPTAINERENV_PYTHONPATH"] = ":".join(parts) +def resolve_container(workflow_config): + """Return the image every rule should run in. + + Your own ``.sif`` if you have pulled one (``spv-container pull``), else the + registry tag -- which Snakemake autopulls into ``.snakemake/singularity`` + under the working directory. Snakemake's ``container:`` accepts either form. + ``--config container=...`` overrides both and takes either a ``docker://`` + tag (to test a branch's own CI image) or a path to a local ``.sif``. + """ + override = workflow_config.get("container") + if override: + return str(override) + sif = local_sif() + return str(sif) if sif.exists() else CONTAINER_URI + + +def warn_if_image_stale(): + """Print one line if this user's image predates the checkout. + + Advisory only, and never fatal: an older image is usually fine, because the + checkout's ``src/`` is what rules import (inject_checkout_pythonpath). It + matters when the *dependency stack* moved -- a new package, a lockfile bump. + Silent when there is no local image, no apptainer, or no revision label. + """ + sif = local_sif() + if not sif.exists(): + return + revision = image_revision(sif) + if compare_revision(revision) == "behind": + print( + f"[container] {sif.name} was built from {revision[:12]}, which is behind " + "this checkout. Fine unless the dependency stack moved; refresh with " + "`spv-container pull`.", + file=sys.stderr, + ) + + def configure(workflow_config): """Install config-derived values after Snakemake has loaded configfiles.""" global CATALOG_CONFIG, DEFAULT_MASK_SUFFIX, FIDUCIAL, PLANCK18 inject_checkout_pythonpath(workflow_config) + warn_if_image_stale() CATALOG_CONFIG = workflow_config FIDUCIAL = workflow_config["fiducial"] DEFAULT_MASK_SUFFIX = ( diff --git a/workflow/profiles/candide/config.yaml b/workflow/profiles/candide/config.yaml index e9448ebf..3b320b1b 100644 --- a/workflow/profiles/candide/config.yaml +++ b/workflow/profiles/candide/config.yaml @@ -11,21 +11,17 @@ # (``container:`` on the rule or the module-level default in # workflow/Snakefile), never a rule's own ``apptainer exec`` shell call. # ``software-deployment-method: apptainer`` below turns that wrapping on; -# ``apptainer-args`` carries the bind mounts every rule needs. Three rules are -# documented exceptions and keep ``container: None`` -- they are enumerated in -# workflow/README.md. +# ``apptainer-args`` carries the bind mounts every rule needs. A few rules call +# host toolchains and opt out with ``container: None``; each says why in its own +# docstring. # # ``snakemake`` itself is a thin host-side tool, pinned via ``uv tool # install`` (see workflow/README.md); run it on the host, never inside an # ``apptainer shell``. # -# Off candide, use ``--profile workflow/profiles/default -j N`` instead: that -# profile is the container model with none of the SLURM/machine layer. -# Snakemake cannot compose profiles -- ``--profile`` takes exactly one directory -# and there is no ``inherits:`` key -- so the machine-independent settings -# (marked GENERIC below) are duplicated there verbatim. THE TWO ARE A PAIR: -# change one, change the other. ``apptainer-args`` and ``apptainer-prefix`` are -# NOT generic: each machine has its own disks and its own image cache. +# Snakemake cannot compose profiles, so the machine-independent settings (marked +# GENERIC below) are duplicated in workflow/profiles/default/config.yaml, for +# running off candide; change one, change the other. executor: slurm @@ -42,20 +38,17 @@ rerun-triggers: ["mtime", "params", "input", "code"] latency-wait: 5 # --- end GENERIC ------------------------------------------------------------ -# Shared image cache. The entry Snakefiles name a GHCR *tag*; Snakemake pulls it -# here on first use (once, ~15 min, on the host running snakemake) and every -# later run reuses the cached SIF. Shared rather than per-run-directory -# (Snakemake's default is ``.snakemake/singularity``) so one pull serves every -# workflow and every checkout. ``current.sif`` in this directory is the stable -# name host-side callers use (workflow/README.md). -apptainer-prefix: /n17data/cdaley/containers/snakemake-sif - +# No ``apptainer-prefix``: everyone runs their own image, at their own canonical +# path, pulled with ``spv-container pull`` (workflow/README.md). The entry +# Snakefiles resolve ``container:`` to that file when it exists, so there is +# nothing for Snakemake to cache. +# # candide's disks, plus the one machine-specific env var: the host OpenMPI libs # MPI rules need to find libmpi inside the container (only rules importing # mpi4py care; harmless for the rest). This was an ``os.environ[...]`` line in # workflow/Snakefile -- a machine path hard-coded into generic workflow code -- -# and belongs here. Kept in sync with the ``app`` bash function's raw -# ``apptainer exec`` in the top-level UNIONS CLAUDE.md: update both together. +# and belongs here. The bind list matches ``spv-container exec``'s default; keep +# the two in step. apptainer-args: >- --cleanenv --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data diff --git a/workflow/profiles/default/config.yaml b/workflow/profiles/default/config.yaml index 49e65375..3c1042fa 100644 --- a/workflow/profiles/default/config.yaml +++ b/workflow/profiles/default/config.yaml @@ -6,12 +6,11 @@ # snakemake --profile workflow/profiles/default -s workflow/Snakefile \ # --configfile -j 4 # -# On candide use `--profile workflow/profiles/candide` instead: that profile is -# the GENERIC block below plus the SLURM executor, account/partition, node -# excludes, candide's binds and the host OpenMPI library path. Snakemake cannot -# compose profiles -- `--profile` takes exactly one directory and there is no -# `inherits:` key -- so the GENERIC block is duplicated in both files. THE TWO -# ARE A PAIR: change one, change the other. +# On candide -- where the analysis actually runs -- use +# `--profile workflow/profiles/candide` instead: the GENERIC block below plus +# the SLURM executor and candide's machine layer. Snakemake cannot compose +# profiles, so that block is duplicated in both files; change one, change the +# other. # # Requirements are the same everywhere: `apptainer` on PATH, and `snakemake` # installed host-side (`uv tool install ...`, see workflow/README.md) -- never @@ -21,8 +20,8 @@ # Turn on Snakemake's own container wrapping: it wraps each job's # `shell:`/`script:` command in `apptainer exec`, using the image named by the # entry Snakefile's `container:` directive. No rule writes its own -# `apptainer exec` call; the documented exceptions that opt out entirely with -# `container: None` are listed in workflow/README.md. +# `apptainer exec` call; the few that opt out with `container: None` say why in +# their own docstrings. software-deployment-method: apptainer # Rerun a job when its code / params / inputs change, not only on mtime. @@ -41,6 +40,7 @@ latency-wait: 5 # directory cover everything (apptainer mounts both by default), drop `--bind`. apptainer-args: "--cleanenv --bind /home" -# No `apptainer-prefix`: pulled images land in `.snakemake/singularity` under -# the working directory. Set one (an absolute path) if several checkouts or -# users on this machine should share one pull, as candide's profile does. +# No `apptainer-prefix`, here or on candide: `spv-container pull` puts your image +# at one canonical per-user path and the entry Snakefiles resolve `container:` to +# it. Without that file Snakemake autopulls the tag into `.snakemake/singularity` +# under the working directory, which works but re-pulls per run directory. diff --git a/workflow/rules/twopoint.smk b/workflow/rules/twopoint.smk index 260468ff..2b335e31 100644 --- a/workflow/rules/twopoint.smk +++ b/workflow/rules/twopoint.smk @@ -27,8 +27,7 @@ rule xi: # PARKED: xi_highres (high-resolution xi for COSEBIS integration). Never # runnable as written -- the shell invokes run_2pcf_highres.py bare, but the # script has required --cat-config and --out arguments (true in every version -# since it was introduced). Revive it with those arguments supplied. Keeping it -# parked leaves covariance_cosmocov as the workflow's only container exception. +# since it was introduced). Revive it with those arguments supplied. # # The MPI reasoning below is hard-won and must survive the revival: # @@ -46,15 +45,14 @@ rule xi: # Because this rule builds its own apptainer call, reaching the source-cache # copy of the script relies on our `--bind /home` rather than on Snakemake's # automatic mount -- and on a concrete image file, since `apptainer exec` -# takes no `docker://` URI. A revived rule must therefore derive that file -# from CONTAINER_URI (Snakemake pulls to `{apptainer-prefix}/{md5(uri)}.simg`; -# workflow/scripts/container_path.py does exactly this derivation) rather -# than hard-coding a second path that can drift. +# takes no `docker://` URI. A revived rule should take that file from +# `common.local_sif()` (this user's own image, the one `spv-container pull` +# writes) rather than name a second image path that can drift. # # rule xi_highres: # container: None # params: -# image=, +# image=str(local_sif()), # input: # script=workflow.source_path("../scripts/run_2pcf_highres.py"), # output: diff --git a/workflow/scripts/container_path.py b/workflow/scripts/container_path.py deleted file mode 100755 index d3e65ea2..00000000 --- a/workflow/scripts/container_path.py +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env python3 -"""Print the local SIF that Snakemake pulls the workflow image into. - -For host-side callers that are *outside* Snakemake and therefore need a file -rather than a ``docker://`` tag -- the papers/bmodes/scripts drivers, and -interactive ``apptainer exec``. Resolving it here keeps the image in exactly -one place (``CONTAINER_URI`` in workflow/common.py, ``apptainer-prefix`` in the -driving profile) instead of a path anyone has to maintain by hand. - - CONTAINER=$(workflow/scripts/container_path.py) - -Stdlib only, so it runs under bare ``python3`` on the login node with no -environment to activate. Exits non-zero if the image has not been pulled yet; -the message says how to pull it. -""" - -import hashlib -import re -import sys -from pathlib import Path - -WORKFLOW = Path(__file__).resolve().parent.parent -PROFILE = WORKFLOW / "profiles/candide/config.yaml" - - -def _grep(path, pattern, what): - match = re.search(pattern, path.read_text(), re.MULTILINE) - if match is None: - sys.exit(f"{path}: could not find {what}") - return match.group(1).strip().strip("\"'") - - -uri = _grep(WORKFLOW / "common.py", r'^CONTAINER_URI = "(.+)"$', "CONTAINER_URI") -prefix = _grep(PROFILE, r"^apptainer-prefix:\s*(\S+)", "apptainer-prefix") - -sif = Path(prefix) / f"{hashlib.md5(uri.encode()).hexdigest()}.simg" -if not sif.exists(): - sys.exit( - f"{sif} not pulled yet -- run any snakemake target with " - f"--profile {PROFILE.parent} once, or pull it directly (workflow/README.md)." - ) -print(sif) From c4e198cd1035eebcb8ec4c99f8ba6830e48a7535 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 02:19:26 +0200 Subject: [PATCH 27/37] docs: tell the container story once, around the per-user image Rewrites the container sections of workflow/README.md, CLAUDE.md, CONTRIBUTING.md and README.md for the per-user model: one image per person at ~/.cache/sp_validation/sp_validation.sif, `spv-container` to fill and inspect it, and how `container:` resolves to it. The shared-prefix machinery is gone -- current.sif bootstrap, the atomic-mv refresh recipe, the group-writable TODO. Trims the commentary while there. The profile pair says "change one, change the other" once instead of shouting it in three places; off-candide gets a paragraph rather than parallel billing, since candide is where everyone runs; and the enumeration of container exceptions is dropped in favour of the docstring on each rule that opts out. Also repoints the two paper Snakefiles, which resolve `container:` themselves, at common.resolve_container -- and replaces the obsolete `apptainer build --sandbox` recipe in README.md and installation.rst with `apptainer pull`. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 17 ++- CONTRIBUTING.md | 38 ++++++- README.md | 14 ++- docs/source/installation.rst | 7 +- papers/bmodes/Snakefile | 8 +- papers/cosmo_val/Snakefile | 8 +- workflow/README.md | 208 +++++++++++++++++++++-------------- 7 files changed, 191 insertions(+), 109 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e985d6c2..ef0ce671 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -78,10 +78,23 @@ Main configuration in `scripts/calibration/params.py` with parameters: - pyccl for cosmological calculations ## Container Usage -Recommended installation via Apptainer/Docker: +Nothing is hand-built. CI publishes `ghcr.io/cosmostat/sp_validation:` on +every push, and each person keeps their own copy at +`~/.cache/sp_validation/sp_validation.sif`, managed by the `spv-container` CLI: + ```bash -apptainer build --sandbox sp_validation docker://ghcr.io/cosmostat/sp_validation:develop +spv-container pull # fetch :develop there (do it from a compute node) +spv-container status # which commit it was built from, vs. your checkout +spv-container exec # one-off run inside it ``` +Every rule runs inside that image, wrapped by Snakemake itself (`--profile +workflow/profiles/candide` on the cluster, `workflow/profiles/default -j N` +elsewhere). The `sp_validation` a rule imports comes from the *launched +checkout*, not the image: `common.configure()` puts its `src/` on the +container's `PYTHONPATH`. + +`workflow/README.md` is the full story — profiles, image resolution, refresh. + ## Notebook Configuration - The CosmologyValidation class must be initialized in cosmo_val \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 00a8d907..1bf22072 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -13,15 +13,41 @@ inside the project container, which ships the full stack pre-built. ### Container (recommended) +CI builds and pushes an image on **every** push, tagged by the sanitized branch +name (see +[`.github/workflows/deploy-image.yml`](.github/workflows/deploy-image.yml)), so +`:develop` tracks the integration branch and your branch has an image of its +own. Nothing is built by hand. + +You keep your own copy of the image. `spv-container` — a console script installed +with the package, and runnable straight from a checkout as `python3 +src/sp_validation/container.py` — pulls it to +`~/.cache/sp_validation/sp_validation.sif` and runs things inside it: + ```bash -# build a writeable sandbox from the published image -apptainer build --sandbox sp_validation docker://ghcr.io/cosmostat/sp_validation:develop -apptainer shell --writable sp_validation +spv-container pull # fetch :develop; ~1.5 GB, so do it from a compute node +spv-container status # which commit the image was built from +spv-container exec bash # an interactive shell inside it ``` -The image is rebuilt and pushed on every push to `develop` (see -[`.github/workflows/deploy-image.yml`](.github/workflows/deploy-image.yml)), so -`:develop` always tracks the latest integration branch. +Analysis runs through Snakemake, which wraps every job in `apptainer exec` +against that same image for you — see +[`workflow/README.md`](workflow/README.md) for the profiles and the details. + +Two things worth knowing while developing: + +- **Your checkout's code is what runs.** The workflow prepends the launched + checkout's `src/` to the container's `PYTHONPATH`, so the image supplies the + dependency stack and your working tree supplies `sp_validation`. No rebuild + needed to test a change. (Caveat: `rerun-triggers: code` does not watch + `src/`, so force reruns after editing a module.) +- **To test a branch's own image** — when the *stack* changed, not just `src/` — + point the workflow at its CI tag: + + ```bash + snakemake --profile workflow/profiles/candide \ + --config container=docker://ghcr.io/cosmostat/sp_validation:my-branch + ``` ### Local install with `uv` diff --git a/README.md b/README.md index 5da899f3..d1295c4c 100644 --- a/README.md +++ b/README.md @@ -69,16 +69,18 @@ The easiest way to install sp_validation is via a container. Docker images are a We recommend running the image with **Apptainer** (formerly Singularity) which is installed on most HPC clusters. To simply run the image, use the following command: ```bash -# build writeable "sandbox" container in the current directory -# ./sp_validation will be a directory that functions like a vm -apptainer build --sandbox sp_validation docker://ghcr.io/cosmostat/sp_validation:develop +# pull the image to a single .sif file +apptainer pull sp_validation.sif docker://ghcr.io/cosmostat/sp_validation:develop # open a shell in the container -apptainer shell --writable sp_validation +apptainer shell sp_validation.sif # and confirm that the installation was successful python -c "import sp_validation" ``` +CI tags an image by branch, so `:develop` tracks the integration branch and any +branch can be pulled by its (sanitized) name. + You can also run the image with **Docker**: ```bash @@ -90,7 +92,9 @@ We do not currently build images for Apple Silicon/arm64; however the amd64 imag This shell is for interactive development and debugging. To run the analysis workflow (`workflow/`), do not enter this shell — see [`workflow/README.md`](workflow/README.md): Snakemake runs on the host, and -the profile puts each job in the container itself. +the profile puts each job in the container itself. The workflow expects the +image at one canonical per-user path, which the bundled `spv-container` CLI +manages (`spv-container pull` / `status` / `exec`). diff --git a/docs/source/installation.rst b/docs/source/installation.rst index f84acebc..90634fb6 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -15,12 +15,11 @@ The image runs on most systems, including HPC clusters, with no further setup. .. code-block:: bash - # Build a writeable "sandbox" container in the current directory. - # ./sp_validation is a directory that behaves like a small VM. - apptainer build --sandbox sp_validation docker://ghcr.io/cosmostat/sp_validation:develop + # Pull the image to a single .sif file. + apptainer pull sp_validation.sif docker://ghcr.io/cosmostat/sp_validation:develop # Open a shell in the container, then confirm the install works. - apptainer shell --writable sp_validation + apptainer shell sp_validation.sif python -c "import sp_validation" The image also runs under Docker: diff --git a/papers/bmodes/Snakefile b/papers/bmodes/Snakefile index 273368b7..e387181d 100644 --- a/papers/bmodes/Snakefile +++ b/papers/bmodes/Snakefile @@ -25,10 +25,10 @@ import common common.configure(config) from common import * -# The image for every rule -- the CI tag, pulled into the profile's -# apptainer-prefix on first use (workflow/README.md). Override with -# `--config container=/path/to/my.sif`. -container: config.get("container", CONTAINER_URI) +# The image for every rule: this user's own .sif if they have pulled one with +# `spv-container pull`, else the CI tag for Snakemake to autopull. Override with +# `--config container=`. See common.resolve_container. +container: common.resolve_container(config) # Wildcard constraints — centralized in common.py, not in individual rule files wildcard_constraints: diff --git a/papers/cosmo_val/Snakefile b/papers/cosmo_val/Snakefile index 4ad5a7ed..0fb2a305 100644 --- a/papers/cosmo_val/Snakefile +++ b/papers/cosmo_val/Snakefile @@ -30,10 +30,10 @@ import common common.configure(config) from common import * -# The image for every rule -- the CI tag, pulled into the profile's -# apptainer-prefix on first use (workflow/README.md). Override with -# `--config container=/path/to/my.sif`. -container: config.get("container", CONTAINER_URI) +# The image for every rule: this user's own .sif if they have pulled one with +# `spv-container pull`, else the CI tag for Snakemake to autopull. Override with +# `--config container=`. See common.resolve_container. +container: common.resolve_container(config) # Wildcard constraints — centralized in common.py, not in individual rule files wildcard_constraints: diff --git a/workflow/README.md b/workflow/README.md index 34302cd0..bc54370a 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -56,33 +56,80 @@ Every rule runs inside the sp_validation container: the profile sets and Snakemake wraps each job's `shell:`/`script:` command in `apptainer exec` itself — no rule writes its own `apptainer exec` call. The image name comes from the `container:` directive in `workflow/Snakefile` (or a rule's own -override, e.g. the image-sims `SIF`). One rule is an explicit, documented -exception and keeps `container: None` with an inline host-toolchain call — -`covariance_cosmocov` (a host-compiled binary) — see its docstring in -`workflow/rules/`. `OMP_NUM_THREADS` is not set by the profile either: the -slurm executor's `--export=ALL` propagates the driver's env, not a profile -flag, so a rule that needs it pinned sets it itself. Per-rule `mem_mb` / -`runtime` stay on the rules. Off-cluster, drop `--profile` and add `-j N`. See -the profile's own comments for the full rationale. - -### Running your own checkout instead of the image's code - -Rules import the `sp_validation` baked into the image. To run a working copy -instead — testing a branch without rebuilding — prepend it to `PYTHONPATH` at -the container boundary. Apptainer forwards `APPTAINERENV_`-prefixed host -variables into the job (this survives the profile's `--cleanenv`, which strips -everything else), so setting it on the `snakemake` invocation reaches every -rule: +override, e.g. the image-sims `SIF`). + +A few rules shell out to a host toolchain (CosmoCov, ImageMagick) and keep +`container: None`; each says why in its own docstring. + +`OMP_NUM_THREADS` is not set by the profile either: the slurm executor's +`--export=ALL` propagates the driver's env, not a profile flag, so a rule that +needs it pinned sets it itself. Per-rule `mem_mb` / `runtime` stay on the rules. +See the profile's own comments for the full rationale. + +### Off candide — the default profile + +Candide is where the analysis runs, so the candide profile is the one to reach +for. `profiles/default/config.yaml` exists for anywhere else — a laptop, another +cluster — and carries the machine-independent half (container wrapping, rerun +triggers, latency wait) with no SLURM layer: ```bash -APPTAINERENV_PYTHONPATH=/path/to/your/sp_validation/src \ - snakemake --profile workflow/profiles/candide -s workflow/Snakefile +snakemake --profile workflow/profiles/default -s workflow/Snakefile \ + --configfile -j 4 ``` -The checkout has to sit under one of the profile's bind mounts to be visible -inside the job. This is a user-side override on purpose: nothing in the -workflow sets it, so a run reproduces from the image alone unless you ask -otherwise. +Snakemake cannot compose profiles, so both files carry that block (marked +`GENERIC` in each) — change one, change the other. `apptainer-args` is not part +of it: expect to edit the default profile's `--bind` list for your machine. + +### Which `sp_validation` a rule imports: the launched checkout + +The image is the frozen *dependency stack*; the `sp_validation` that runs is +the one in the checkout you launched from. `common.configure()` prepends that +checkout's `src/` to `APPTAINERENV_PYTHONPATH`, which Apptainer forwards into +each job as `PYTHONPATH` (surviving the profile's `--cleanenv`). Any value you +exported yourself is preserved behind it. + +This is the default because the alternative is incoherent: Snakemake's +`script:` directive already runs the checkout's *script files*, so without it a +rule executes new script code against an old `import sp_validation` — the two +halves of one commit, split. The image-sims chain has always worked this way +(`_ENV_PREFIX` in `workflow/rules/image_sims.smk`); the rest of the workflow now +matches it. + +**Caveat:** `rerun-triggers: code` watches rule bodies and `script:` files, not +`src/`. Editing a module under `src/` does not by itself mark outputs stale — +force with `-F` or `--forcerun `. + +To reproduce a run from the image alone, opt out: + +```bash +snakemake --profile workflow/profiles/candide --config checkout_pythonpath=false +``` + +Either way the checkout has to sit under one of the profile's bind mounts to be +visible inside the job. + +### Testing a branch's own image + +CI builds and pushes an image for **every** branch, tagged by the sanitized +branch name (`/` → `-`; see `.github/workflows/deploy-image.yml`). To run a +branch's image rather than `:develop`: + +```bash +snakemake --profile workflow/profiles/candide \ + --config container=docker://ghcr.io/cosmostat/sp_validation:my-branch +``` + +`container` is a config key read by every entry Snakefile +(`common.resolve_container`), so it overrides the default everywhere at once. +Snakemake autopulls the tag, which costs ~15 minutes — do it from a compute node. +To keep that image around instead, `spv-container pull --tag ` puts it at +your canonical path, where it becomes the default. Most of the time you need +none of this: +the checkout-PYTHONPATH default above already runs your branch's Python against +the `:develop` dependency stack. Reach for the branch image when the *stack* +changed (a new dependency, a lockfile bump), not when only `src/` did. ### Never write `/automnt/nXXdataN` in a path @@ -118,94 +165,87 @@ shadow the one `uv tool install` just set up. Run `which snakemake` and confirm it resolves under `uv`'s tool directory (`uv tool dir`), not `~/.local/bin`. -### The container image +### The container image — one per person -Everything runs one image, named once as a registry tag: +Everything runs one image, published by CI as a registry tag: ``` docker://ghcr.io/cosmostat/sp_validation:develop ``` -That tag is what `workflow/Snakefile`, the paper Snakefiles and the image-sims -`sif:` config key declare. Nobody writes a `.sif` path: Snakemake pulls the tag -into the profile's `apptainer-prefix` -(`/n17data/cdaley/containers/snakemake-sif`) on first use and reuses the cached -file forever after. No Snakemake rule needs a file. Host-side callers that do — -the `papers/bmodes/scripts/run_*.sh` drivers, and interactive `apptainer exec` — -get it from `workflow/scripts/container_path.py`, which derives the path from -the same tag (Snakemake names a pulled image `{prefix}/{md5(uri)}.simg`; the -script just recomputes that). - -**The first pull is not free.** It happens on the host running `snakemake`, -takes ~15 minutes for ~1.5 GB, and blocks the run — so do the first run of a -new tag from a compute node, not the login node. Every later run finds the -cached file and touches neither cache nor network. +**Each person keeps their own copy of it.** There is no shared image directory: +you pull your own file, you refresh it when you want to, and nobody else's +refresh moves the ground under your running jobs. The canonical path is -**Where the image comes from.** CI (`.github/workflows/deploy-image.yml`) builds -it on every push, `FROM ghcr.io/cosmostat/shapepipe:im_sims` with `uv sync ---frozen` against `uv.lock`, and publishes to -`ghcr.io/cosmostat/sp_validation` tagged by branch — so `:develop` tracks the -tip of `develop`. The package is public; no credentials are needed. A cached -pull is a *snapshot* of the tag: CI publishing a new image does not change what -your jobs run until someone refreshes. +``` +~/.cache/sp_validation/sp_validation.sif +``` -**Refreshing** — one person does it for everybody. Delete the cached file and -let the next run re-pull it, or pull deliberately: +and a small CLI, `spv-container`, is what puts it there and tells you about it: ```bash -# From a compute node (~1.5 GB / ~15 min; never on the login node). -salloc -p comp -c 4 --time=01:00:00 --no-shell # note the job id -export APPTAINER_CACHEDIR=/n17data/cdaley/containers/.apptainer-cache/cache -export APPTAINER_TMPDIR=/n17data/cdaley/containers/.apptainer-cache/tmp -SIF=$(workflow/scripts/container_path.py) # prints the cached path -srun --jobid= bash -c "cd \$(dirname $SIF) && \ - apptainer pull --force --name next.simg \ - docker://ghcr.io/cosmostat/sp_validation:develop && \ - mv -f next.simg $SIF" -scancel +spv-container pull # fetch :develop to the canonical path (~1.5 GB / ~15 min) +spv-container status # is it there, which commit was it built from, is it current +spv-container exec # run something inside it, candide binds already applied ``` -Pull to `next.simg` and `mv` — never pull straight onto the cached name. `mv` -within one directory is an atomic rename, so a job either gets the whole old -image or the whole new one. Pulling in place would leave the file half-written -for the ~15 minutes the pull takes, and any job starting in that window would -fail. Jobs already running hold the old inode open and finish against it -unharmed. - -`snakemake --cleanup-containers` deletes every `*.simg` in the prefix that the -current DAG does not require. With the tag form the cached image *is* required, -so it survives; anything left over from an older tag is what goes. - -To check what you have: +It ships as a console script with the package, and — being stdlib-only, because +it has to run on the *host* — also works straight from a checkout with nothing +installed: ```bash -apptainer inspect --labels $(workflow/scripts/container_path.py) +python3 src/sp_validation/container.py pull ``` -`org.opencontainers.image.revision` is the sp_validation commit the image was -built from. The image-sims workflow records it in `m_bias_config.yaml` as -`ghcr_revision`, so a result file says which image produced the number. - -**Interactive use** — the same image, resolved the same way: +**Do the pull from a compute node**, not the login node: it moves ~1.5 GB and +takes about fifteen minutes. `pull` writes to a temporary name and renames, so a +job either gets the whole old image or the whole new one; jobs already running +hold the old file open and finish against it unharmed. ```bash -apptainer exec --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data \ - $(workflow/scripts/container_path.py) +salloc -p comp -c 4 --time=01:00:00 --exclude=n17,n09,n36 --no-shell # note the job id +srun --jobid= spv-container pull +scancel ``` -This is what Cail's `app` shell function points at (the function lives in his -shell config, not this repo). There is one image, not two — a refresh moves -interactive use and the workflow together, with nothing to keep in sync. +**How the workflow finds it.** `container:` resolves to your local `.sif` when +that file exists, and to the registry tag when it does not (Snakemake accepts +either, and autopulls a tag into `.snakemake/singularity` under the working +directory — which works, but re-pulls once per run directory, so `spv-container +pull` is the path to prefer). The tag itself is written down once, as +`CONTAINER_URI` in `sp_validation/container.py`, which `workflow/common.py` +re-exports; the image-sims `sif:` config key defaults to `null` and resolves the +same way. Override any of it with `--config container=`, or point +somewhere else entirely with `SPV_CONTAINER`. + +At launch the workflow prints one advisory line if your image was built from a +commit behind your checkout. It never fails the run — an older image is normally +fine, since the checkout's `src/` is what rules import (see above). It matters +when the *dependency stack* moved: a new package, a lockfile bump. + +**Where the image comes from.** CI (`.github/workflows/deploy-image.yml`) builds +it on every push, `FROM ghcr.io/cosmostat/shapepipe:im_sims` with `uv sync +--frozen` against `uv.lock`, and publishes to `ghcr.io/cosmostat/sp_validation` +tagged by branch — so `:develop` tracks the tip of `develop`. The package is +public; no credentials are needed. Your pulled file is a *snapshot*: CI +publishing a new image changes nothing until you pull again. + +`spv-container status` reads `org.opencontainers.image.revision` — the +sp_validation commit the image was built from — and places it against your +checkout's `HEAD`. The image-sims workflow records the same label in +`m_bias_config.yaml` as `ghcr_revision`, so a result file says which image +produced the number. -**Running your own image** instead of the shared one: +**Running an image of your own** instead of the canonical one: ```bash snakemake --profile workflow/profiles/candide --config container=/path/to/my.sif ``` For the image-sims workflow, set `image_sims: {sif: /path/to/my.sif}` in your run -config — it is already a config key. Your image has to sit under one of the -profile's bind mounts to be visible. +config. Your image has to sit under one of the profile's bind mounts to be +visible. To run a *branch's* CI image rather than a local file, see "Testing a +branch's own image" above. One invariant survives from the old hand-built sandbox and still applies: the `script:` directive bind-mounts the host orchestrator's `snakemake` into the job From 7b47b56e10b1e811aa24ded5c41aa8be2a1d10ac Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 02:29:49 +0200 Subject: [PATCH 28/37] containers: add an opt-in writable sandbox, and one resolution order The pristine SIF is read-only, which is what you want almost always -- but it lost the one real advantage of the old hand-built sandbox workflow: `pip install` mid-analysis, when you need a package the image does not carry yet and a CI rebuild is too slow a loop to think in. `spv-container sandbox` unpacks the image into a writable directory at ~/.cache/sp_validation/sandbox/, and `spv-container exec --writable` runs against it so installs persist. Opt-in: nothing builds one for you. Resolution order is now one thing, shared by the CLI, the run_*.sh drivers and the workflow's `container:` -- sandbox if it exists, else SIF if it exists, else the registry tag. Snakemake execs a sandbox directory as happily as a .sif, so a package installed into the sandbox is there for workflow jobs too, with no further wiring. `resolve_image()` in sp_validation.container is the single implementation; common.resolve_container defers to it. The build stages into a sibling directory and swaps it in, as `pull` does, for a sharper reason than pull has: a half-written .sif fails loudly, but a half-unpacked sandbox is still a *directory*, so resolution would elect it and every job would silently run a broken tree. Building before removing also means a `--force` rebuild that fails -- a typo in --source, a network blip -- leaves the sandbox you already had intact, instead of deleting a working environment on the way to not replacing it. The cost of a sandbox is that what runs is no longer fully described by a revision label, so the divergence is made visible rather than left silent: `status` names which layer is live and says the revision only describes what the sandbox was built from (falling back to the SIF's label, marked as inferred, when the sandbox carries none), and the workflow prints one line at launch when a sandbox is in play. `spv-container pull && spv-container sandbox --force` resets. Verified on candide (apptainer 1.5.3): unprivileged `build --sandbox` works through user namespaces with no fakeroot and no subuid mapping; `exec --writable` persists writes while plain `exec` gets a read-only filesystem; `inspect --labels` still reports the source image's OCI labels from a sandbox directory; `--fix-perms` at build time is what keeps the tree removable afterwards (without it apptainer leaves directories that defeat `rm -rf`, which would strand `--force`); and a failed `--force` rebuild leaves the existing sandbox and its contents untouched, with no staging directory left behind. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 7 +- CONTRIBUTING.md | 9 +- papers/bmodes/scripts/run_cov_sweep.sh | 6 +- .../bmodes/scripts/run_pure_eb_ptes_sweep.sh | 6 +- .../scripts/run_pure_eb_semianalytic.sh | 6 +- papers/bmodes/scripts/run_pure_eb_sweep.sh | 6 +- src/sp_validation/container.py | 222 ++++++++++++++++-- workflow/README.md | 46 +++- workflow/common.py | 56 +++-- workflow/rules/twopoint.smk | 8 +- 10 files changed, 306 insertions(+), 66 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ef0ce671..10c75b71 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -84,10 +84,15 @@ every push, and each person keeps their own copy at ```bash spv-container pull # fetch :develop there (do it from a compute node) -spv-container status # which commit it was built from, vs. your checkout +spv-container status # which layer is live, and how current it is spv-container exec # one-off run inside it ``` +Need a package the image lacks mid-analysis? Unpack a writable sandbox once with +`spv-container sandbox`, then `spv-container exec --writable pip install `. +The sandbox then takes precedence over the SIF everywhere, workflow jobs +included; `spv-container pull && spv-container sandbox --force` resets it clean. + Every rule runs inside that image, wrapped by Snakemake itself (`--profile workflow/profiles/candide` on the cluster, `workflow/profiles/default -j N` elsewhere). The `sp_validation` a rule imports comes from the *launched diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1bf22072..09cfe2c1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -26,10 +26,17 @@ src/sp_validation/container.py` — pulls it to ```bash spv-container pull # fetch :develop; ~1.5 GB, so do it from a compute node -spv-container status # which commit the image was built from +spv-container status # which layer is live, and how current it is spv-container exec bash # an interactive shell inside it ``` +That image is read-only. When you need a package it does not carry yet, unpack a +writable sandbox once with `spv-container sandbox` and install into it with +`spv-container exec --writable pip install `; the sandbox then takes +precedence everywhere, workflow jobs included. Treat it as an exploration tool — +the real fix is adding the dependency to `pyproject.toml` — and reset it with +`spv-container pull && spv-container sandbox --force`. + Analysis runs through Snakemake, which wraps every job in `apptainer exec` against that same image for you — see [`workflow/README.md`](workflow/README.md) for the profiles and the details. diff --git a/papers/bmodes/scripts/run_cov_sweep.sh b/papers/bmodes/scripts/run_cov_sweep.sh index c2515a0c..dd0a1330 100755 --- a/papers/bmodes/scripts/run_cov_sweep.sh +++ b/papers/bmodes/scripts/run_cov_sweep.sh @@ -27,9 +27,11 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# This user's own image, at the canonical path `spv-container pull` writes to -# (see workflow/README.md). SPV_CONTAINER overrides it here and everywhere else. +# This user's own image, resolved the way `spv-container` resolves it: the +# writable sandbox if one exists, else the pristine SIF (see workflow/README.md). CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} +[ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ + CONTAINER=$HOME/.cache/sp_validation/sandbox SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts diff --git a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh index d37910f2..203e934d 100644 --- a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh @@ -20,9 +20,11 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# This user's own image, at the canonical path `spv-container pull` writes to -# (see workflow/README.md). SPV_CONTAINER overrides it here and everywhere else. +# This user's own image, resolved the way `spv-container` resolves it: the +# writable sandbox if one exists, else the pristine SIF (see workflow/README.md). CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} +[ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ + CONTAINER=$HOME/.cache/sp_validation/sandbox SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts diff --git a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh index a098fbf3..f7adb6aa 100644 --- a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh +++ b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh @@ -16,9 +16,11 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra SRC=$WT/src -# This user's own image, at the canonical path `spv-container pull` writes to -# (see workflow/README.md). SPV_CONTAINER overrides it here and everywhere else. +# This user's own image, resolved the way `spv-container` resolves it: the +# writable sandbox if one exists, else the pristine SIF (see workflow/README.md). CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} +[ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ + CONTAINER=$HOME/.cache/sp_validation/sandbox SCRIPTS=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra/papers/bmodes/scripts BIND=/home,/scratch,/automnt,/n17data,/n23data1,/n09data diff --git a/papers/bmodes/scripts/run_pure_eb_sweep.sh b/papers/bmodes/scripts/run_pure_eb_sweep.sh index 739942e4..369176bb 100755 --- a/papers/bmodes/scripts/run_pure_eb_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_sweep.sh @@ -20,9 +20,11 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# This user's own image, at the canonical path `spv-container pull` writes to -# (see workflow/README.md). SPV_CONTAINER overrides it here and everywhere else. +# This user's own image, resolved the way `spv-container` resolves it: the +# writable sandbox if one exists, else the pristine SIF (see workflow/README.md). CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} +[ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ + CONTAINER=$HOME/.cache/sp_validation/sandbox SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts diff --git a/src/sp_validation/container.py b/src/sp_validation/container.py index 251b2141..6ef0f2a6 100644 --- a/src/sp_validation/container.py +++ b/src/sp_validation/container.py @@ -1,15 +1,30 @@ """Manage this user's local copy of the sp_validation container image. -Everyone runs their own image file. There is no shared image directory and no -symlink to keep honest: the canonical path is under your own cache -(``~/.cache/sp_validation/sp_validation.sif``), you refresh it when you want to, -and nobody else's refresh moves the ground under a running job. +Everyone runs their own image. There is no shared image directory and no symlink +to keep honest: the canonical paths are under your own cache, you refresh them +when you want to, and nobody else's refresh moves the ground under a running job. -Three subcommands, exposed as the ``spv-container`` console script:: +There are two layers, and you only need the second when you want it: - spv-container pull # fetch the tag to the canonical path - spv-container status # is it there, and which commit is it? - spv-container exec # run something inside it +* the **SIF** (``~/.cache/sp_validation/sp_validation.sif``) -- a pristine, + read-only copy of the published image. This is the default and the normal case. +* an optional **sandbox** (``~/.cache/sp_validation/sandbox/``) -- the same image + unpacked into a writable directory, so ``pip install`` inside it sticks. This + is the escape hatch for exploratory work that needs a package the image does + not carry yet, and it is opt-in: nothing builds one for you. + +Subcommands, exposed as the ``spv-container`` console script:: + + spv-container pull # fetch the tag to the canonical path + spv-container status # what is here, and how current is it + spv-container sandbox # unpack the SIF into a writable dir + spv-container exec # run something inside it + spv-container exec --writable # ... with writes that persist + +Everything resolves the same image in the same order -- **sandbox if it exists, +else the SIF, else the registry tag** -- and that includes the Snakemake +workflow, so a package you installed into your sandbox is there for your +workflow jobs too. This module is deliberately **stdlib-only** (``argparse``/``subprocess``/ ``pathlib``). It runs on the *host*, outside the container, where the science @@ -32,13 +47,14 @@ # here, once. CONTAINER_URI = "docker://ghcr.io/cosmostat/sp_validation:develop" +CACHE_DIR = Path(os.environ.get("XDG_CACHE_HOME", "~/.cache")) / "sp_validation" + # Where this user's image lives. Per-user by construction: one file, one owner, # no coordination. Override with ``SPV_CONTAINER`` (an absolute path). -DEFAULT_SIF = ( - Path(os.environ.get("XDG_CACHE_HOME", "~/.cache")) - / "sp_validation" - / "sp_validation.sif" -) +DEFAULT_SIF = CACHE_DIR / "sp_validation.sif" + +# The optional writable unpacking of that image. Override with ``SPV_SANDBOX``. +DEFAULT_SANDBOX = CACHE_DIR / "sandbox" # Bind mounts for interactive `exec`. candide's disks; override wholesale with # ``SPV_APPTAINER_BINDS`` or per-call with ``--bind``. @@ -52,6 +68,29 @@ def local_sif(): return path.expanduser() +def local_sandbox(): + """Return this user's writable sandbox directory (may not exist).""" + override = os.environ.get("SPV_SANDBOX") + path = Path(override) if override else DEFAULT_SANDBOX + return path.expanduser() + + +def resolve_image(): + """Return ``(path_or_uri, kind)`` for the image everything should run. + + The one resolution order, shared by the CLI and the workflow: the writable + sandbox if it exists, else the pristine SIF if it exists, else the registry + tag for Snakemake to pull. ``kind`` is ``"sandbox"``, ``"sif"`` or ``"tag"``. + """ + sandbox = local_sandbox() + if sandbox.is_dir(): + return str(sandbox), "sandbox" + sif = local_sif() + if sif.exists(): + return str(sif), "sif" + return CONTAINER_URI, "tag" + + def image_labels(sif): """Return the image's OCI labels as a dict, or ``{}`` if unreadable. @@ -158,18 +197,105 @@ def cmd_pull(args): return 0 +def cmd_sandbox(args): + """Unpack the image into a writable directory -- the opt-in escape hatch.""" + if shutil.which("apptainer") is None: + sys.exit("apptainer is not on PATH") + sandbox = local_sandbox() + if sandbox.exists() and not args.force: + sys.exit( + f"sandbox already exists at {sandbox}\n" + "pass --force to discard it and rebuild from a clean image" + ) + source = args.source or ( + str(local_sif()) if local_sif().exists() else CONTAINER_URI + ) + sandbox.parent.mkdir(parents=True, exist_ok=True) + print(f"building sandbox from {source}\n -> {sandbox}") + # Build beside the target and swap it in, as `pull` does -- and for a sharper + # reason here. A half-written .sif fails loudly, but a half-unpacked sandbox + # *directory* is still a directory, so resolve_image() would elect it as the + # live image and every job would silently run a broken tree. + # + # Building first also means a `--force` rebuild that fails (a typo in + # --source, a network blip) leaves the sandbox you already had untouched, + # rather than deleting a working environment on the way to not replacing it. + # + # `--fix-perms` so the tree can be deleted again later (apptainer warns about + # exactly this otherwise). No `--fakeroot`: an unprivileged build from an + # existing image works through user namespaces, which is what candide has. + staging = sandbox.with_name(f"{sandbox.name}.build.{os.getpid()}") + shutil.rmtree(staging, ignore_errors=True) + try: + subprocess.run( + ["apptainer", "build", "--sandbox", "--fix-perms", str(staging), source], + check=True, + ) + except subprocess.CalledProcessError as exc: + shutil.rmtree(staging, ignore_errors=True) + sys.exit(f"sandbox build failed ({exc.returncode}); {sandbox} is unchanged") + except (KeyboardInterrupt, OSError): + shutil.rmtree(staging, ignore_errors=True) + raise + + if sandbox.exists(): + print(f"replacing {sandbox}") + shutil.rmtree(sandbox, ignore_errors=True) + if sandbox.exists(): + shutil.rmtree(staging, ignore_errors=True) + sys.exit(f"could not remove {sandbox}; remove it by hand and retry") + os.replace(staging, sandbox) + print( + "\nthis sandbox now takes precedence over the SIF everywhere, including " + "workflow jobs.\ninstall into it with: spv-container exec --writable pip " + "install \nreset to a clean image with: spv-container pull && " + "spv-container sandbox --force" + ) + return 0 + + def cmd_status(args): - """Report the image's presence, revision, and standing against the checkout.""" + """Report which image layer is live, its revision, and how current it is.""" sif = local_sif() - if not sif.exists(): - print(f"no image at {sif}\nrun: spv-container pull") + sandbox = local_sandbox() + active, kind = resolve_image() + + if sif.exists(): + print(f"SIF: {sif} ({sif.stat().st_size / 1e9:.1f} GB)") + else: + print(f"SIF: absent ({sif})") + if sandbox.is_dir(): + print(f"sandbox: {sandbox} (writable; may carry local modifications)") + else: + print("sandbox: none") + + if kind == "tag": + print(f"\nactive: {active} (registry tag -- nothing pulled locally)") + print("run: spv-container pull") return 1 - size_gb = sif.stat().st_size / 1e9 - print(f"image: {sif} ({size_gb:.1f} GB)") - labels = image_labels(sif) + + print(f"\nactive: {active} ({kind})") + labels = image_labels(active) revision = labels.get("org.opencontainers.image.revision") - print(f"revision: {revision or 'unknown'}") + source = "" + if revision is None and kind == "sandbox" and sif.exists(): + # Some sandbox trees do not carry the original labels through. The SIF + # beside it is the best remaining evidence of what it was built from -- + # a guess, so it is labelled as one rather than printed as fact. + revision = image_revision(sif) + if revision: + source = " (inferred from the SIF beside it, not read from the sandbox)" + print(f"revision: {revision or 'unknown'}{source}") print(f"version: {labels.get('org.opencontainers.image.version', 'unknown')}") + if kind == "sandbox": + # The revision is the image the sandbox was *built from*; anything + # installed into it since is invisible to any label. Say so rather than + # let the revision read as a full description of what is running. + print( + " (the revision above is what the sandbox was built from; " + "anything\n installed into it since is not reflected in " + "any label)" + ) verdict = compare_revision(revision) explain = { "in-sync": "matches this checkout's HEAD", @@ -183,16 +309,39 @@ def cmd_status(args): def cmd_exec(args): - """Run a command inside the canonical image -- the one-off testing path.""" + """Run a command inside the image -- the one-off path for humans and agents.""" if shutil.which("apptainer") is None: sys.exit("apptainer is not on PATH") - sif = local_sif() - if not sif.exists(): - sys.exit(f"no image at {sif}; run: spv-container pull") if not args.command: sys.exit("nothing to run; pass a command after `exec`") binds = args.bind or os.environ.get("SPV_APPTAINER_BINDS", DEFAULT_BINDS) - cmd = ["apptainer", "exec", "--cleanenv", "--bind", binds, str(sif), *args.command] + + if args.writable: + # Writes only persist into a sandbox; a SIF is a read-only filesystem, so + # `--writable` against one fails obscurely. Say what to do instead. + sandbox = local_sandbox() + if not sandbox.is_dir(): + sys.exit( + f"--writable needs a sandbox, and there is none at {sandbox}\n" + "build one with: spv-container sandbox" + ) + image, extra = str(sandbox), ["--writable"] + else: + image, kind = resolve_image() + if kind == "tag": + sys.exit(f"no local image; run: spv-container pull ({image})") + extra = [] + + cmd = [ + "apptainer", + "exec", + *extra, + "--cleanenv", + "--bind", + binds, + image, + *args.command, + ] return subprocess.run(cmd).returncode @@ -210,11 +359,32 @@ def build_parser(): ) p_pull.set_defaults(func=cmd_pull) - p_status = sub.add_parser("status", help="report the local image and its revision") + p_status = sub.add_parser( + "status", help="report which image layer is live and how current it is" + ) p_status.set_defaults(func=cmd_status) + p_sandbox = sub.add_parser( + "sandbox", help="unpack the image into a writable directory (opt-in)" + ) + p_sandbox.add_argument( + "--source", + help="image to unpack (default: the local SIF, or the registry tag)", + ) + p_sandbox.add_argument( + "--force", + action="store_true", + help="discard an existing sandbox and rebuild from a clean image", + ) + p_sandbox.set_defaults(func=cmd_sandbox) + p_exec = sub.add_parser("exec", help="run a command inside the local image") p_exec.add_argument("--bind", help=f"bind mounts (default: {DEFAULT_BINDS})") + p_exec.add_argument( + "--writable", + action="store_true", + help="run against the sandbox so writes (e.g. pip install) persist", + ) p_exec.add_argument("command", nargs=argparse.REMAINDER) p_exec.set_defaults(func=cmd_exec) diff --git a/workflow/README.md b/workflow/README.md index bc54370a..69f0ca16 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -185,7 +185,7 @@ and a small CLI, `spv-container`, is what puts it there and tells you about it: ```bash spv-container pull # fetch :develop to the canonical path (~1.5 GB / ~15 min) -spv-container status # is it there, which commit was it built from, is it current +spv-container status # what is here, which commit it was built from, how current spv-container exec # run something inside it, candide binds already applied ``` @@ -208,11 +208,13 @@ srun --jobid= spv-container pull scancel ``` -**How the workflow finds it.** `container:` resolves to your local `.sif` when -that file exists, and to the registry tag when it does not (Snakemake accepts -either, and autopulls a tag into `.snakemake/singularity` under the working -directory — which works, but re-pulls once per run directory, so `spv-container -pull` is the path to prefer). The tag itself is written down once, as +**How the workflow finds it.** One resolution order, shared by the CLI and the +workflow: your **sandbox** if you have built one (below), else your **`.sif`** if +you have pulled one, else the **registry tag** — which Snakemake autopulls into +`.snakemake/singularity` under the working directory. That works, but re-pulls +once per run directory, so `spv-container pull` is the path to prefer. Snakemake +accepts all three forms, a sandbox directory included. The tag itself is written +down once, as `CONTAINER_URI` in `sp_validation/container.py`, which `workflow/common.py` re-exports; the image-sims `sif:` config key defaults to `null` and resolves the same way. Override any of it with `--config container=`, or point @@ -223,6 +225,36 @@ commit behind your checkout. It never fails the run — an older image is normal fine, since the checkout's `src/` is what rules import (see above). It matters when the *dependency stack* moved: a new package, a lockfile bump. +#### When you need to install something: the sandbox + +The pristine SIF is read-only, which is what you want almost always — it is +exactly the published image, and two people running it run the same thing. But +mid-analysis you sometimes need a package the image does not carry yet, and +rebuilding through CI to find out whether it helps is too slow a loop. + +For that, unpack the image into a writable directory once: + +```bash +spv-container sandbox # ~/.cache/sp_validation/sandbox/ +spv-container exec --writable pip install +``` + +Writes into a sandbox persist. It is opt-in — nothing builds one for you — and +once it exists **it takes precedence over the SIF everywhere, workflow jobs +included**, so a package you install this way is available to your Snakemake runs +without any further wiring. Jobs exec it read-only; only `--writable` writes. + +The cost is that what you are running is no longer fully described by a revision +label. `spv-container status` says which layer is live and flags that, and the +workflow prints one line at launch when a sandbox is in play — the divergence is +visible, never silent. When you are done exploring, either fold the dependency +into `pyproject.toml` (the real fix) or reset to a clean image: + +```bash +spv-container pull # refresh the pristine SIF +spv-container sandbox --force # discard the sandbox, rebuild from it +``` + **Where the image comes from.** CI (`.github/workflows/deploy-image.yml`) builds it on every push, `FROM ghcr.io/cosmostat/shapepipe:im_sims` with `uv sync --frozen` against `uv.lock`, and publishes to `ghcr.io/cosmostat/sp_validation` @@ -232,7 +264,7 @@ publishing a new image changes nothing until you pull again. `spv-container status` reads `org.opencontainers.image.revision` — the sp_validation commit the image was built from — and places it against your -checkout's `HEAD`. The image-sims workflow records the same label in +checkout's `HEAD`, naming which layer (sandbox or SIF) it read. The image-sims workflow records the same label in `m_bias_config.yaml` as `ghcr_revision`, so a result file says which image produced the number. diff --git a/workflow/common.py b/workflow/common.py index 1e956249..c138b533 100644 --- a/workflow/common.py +++ b/workflow/common.py @@ -11,9 +11,10 @@ REPO_SRC = Path(__file__).resolve().parent.parent / "src" # The container model lives in the package (``sp_validation/container.py``): the -# registry tag, this user's canonical ``.sif`` path, and the ``spv-container`` -# CLI that fills it. Taken from *this checkout's* src/, so the workflow and the -# CLI can never disagree about either. +# registry tag, this user's canonical image paths, and the ``spv-container`` CLI +# that fills them. Taken from *this checkout's* src/, so the workflow and the CLI +# can never disagree -- in particular they share one resolution order, which is +# what lets a package installed into a sandbox ride along into workflow jobs. # # Loaded by file path rather than as ``sp_validation.container``: snakemake runs # on the host, where sp_validation is usually not installed, and importing the @@ -30,7 +31,9 @@ CONTAINER_URI = _container.CONTAINER_URI compare_revision = _container.compare_revision image_revision = _container.image_revision +local_sandbox = _container.local_sandbox local_sif = _container.local_sif +resolve_image = _container.resolve_image # Output roots are env-overridable so a reproduction run can write into a @@ -115,35 +118,50 @@ def inject_checkout_pythonpath(workflow_config): def resolve_container(workflow_config): """Return the image every rule should run in. - Your own ``.sif`` if you have pulled one (``spv-container pull``), else the - registry tag -- which Snakemake autopulls into ``.snakemake/singularity`` - under the working directory. Snakemake's ``container:`` accepts either form. - ``--config container=...`` overrides both and takes either a ``docker://`` - tag (to test a branch's own CI image) or a path to a local ``.sif``. + The same order ``spv-container`` uses, so jobs run what interactive work + runs: your writable sandbox if you have built one, else your pristine + ``.sif`` if you have pulled one, else the registry tag -- which Snakemake + autopulls into ``.snakemake/singularity`` under the working directory. + Snakemake's ``container:`` accepts all three (a sandbox directory included). + ``--config container=...`` overrides everything and takes a ``docker://`` + tag, a ``.sif`` path, or a sandbox directory. """ override = workflow_config.get("container") if override: return str(override) - sif = local_sif() - return str(sif) if sif.exists() else CONTAINER_URI + return resolve_image()[0] def warn_if_image_stale(): - """Print one line if this user's image predates the checkout. + """Print one advisory line about a local image that is not pristine or current. + + Never fatal. Two things worth saying at launch: + + * a sandbox is in play, so what jobs run is not fully described by any + revision label -- somebody installed into it on purpose, and that is the + point, but it should not be a silent difference from a clean run; + * the image predates the checkout. Usually fine, because the checkout's + ``src/`` is what rules import (inject_checkout_pythonpath); it matters when + the *dependency stack* moved -- a new package, a lockfile bump. - Advisory only, and never fatal: an older image is usually fine, because the - checkout's ``src/`` is what rules import (inject_checkout_pythonpath). It - matters when the *dependency stack* moved -- a new package, a lockfile bump. Silent when there is no local image, no apptainer, or no revision label. """ - sif = local_sif() - if not sif.exists(): + image, kind = resolve_image() + if kind == "tag": return - revision = image_revision(sif) + revision = image_revision(image) + if kind == "sandbox": + built = f"built from {revision[:12]}" if revision else "revision unknown" + print( + f"[container] running the writable sandbox at {image} ({built}). " + "Anything installed into it is part of this run; " + "`spv-container status` for detail.", + file=sys.stderr, + ) if compare_revision(revision) == "behind": print( - f"[container] {sif.name} was built from {revision[:12]}, which is behind " - "this checkout. Fine unless the dependency stack moved; refresh with " + f"[container] image was built from {revision[:12]}, which is behind this " + "checkout. Fine unless the dependency stack moved; refresh with " "`spv-container pull`.", file=sys.stderr, ) diff --git a/workflow/rules/twopoint.smk b/workflow/rules/twopoint.smk index 2b335e31..5cec5021 100644 --- a/workflow/rules/twopoint.smk +++ b/workflow/rules/twopoint.smk @@ -45,14 +45,14 @@ rule xi: # Because this rule builds its own apptainer call, reaching the source-cache # copy of the script relies on our `--bind /home` rather than on Snakemake's # automatic mount -- and on a concrete image file, since `apptainer exec` -# takes no `docker://` URI. A revived rule should take that file from -# `common.local_sif()` (this user's own image, the one `spv-container pull` -# writes) rather than name a second image path that can drift. +# takes no `docker://` URI. A revived rule should take that path from +# `resolve_image()[0]` -- the same local image everything else resolves -- +# rather than name a second image path that can drift. # # rule xi_highres: # container: None # params: -# image=str(local_sif()), +# image=resolve_image()[0], # input: # script=workflow.source_path("../scripts/run_2pcf_highres.py"), # output: From 5b9c4b9927c8dc1aacd0f034b8c9ada884c25cf7 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 02:21:57 +0200 Subject: [PATCH 29/37] image: build the cosmosis-standard-library fork into the container MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The cosmo_inference .ini templates pointed COSMOSIS_DIR at two different people's home directories (/home/guerrini/... and a scratch path of Lisa's), so running the inference pipeline meant either being one of them or editing the templates by hand. CosmoSIS itself already ships in the image via the `workflow` extra; only the Standard Library — the tree of module files the pipelines name — was missing. Clone and build it at /opt/cosmosis-standard-library, pinned to Sacha Guerrini's fork at b26fa7ff. That fork is 4 commits ahead of upstream and 373 behind; the four are what the UNIONS pipelines need (tau statistics, sample_S8, two z-dependent linear-alignment modules). Carrying them onto current upstream is future work, noted in cosmo_inference/README.md. The templates now read COSMOSIS_DIR from %(CSL_DIR)s, which the image sets — CosmoSIS reads environment variables into an ini's [DEFAULT] section, which is how the existing %(SCRATCH)s references already work. Off-image, export CSL_DIR and the same templates work unchanged. The build follows CSL's documented procedure for a pip-installed cosmosis (`source cosmosis-configure && make`), but targets `shear/` rather than the top-level `make`: the top level also descends into likelihood/, building the Planck, WMAP and ACT likelihoods, which no UNIONS pipeline uses. Of the modules our templates do name, all are pure Python except two under shear/ — `limber`, which project_2d.py links, and cl_to_xi_nicaea's nicaea_interface.so. Co-Authored-By: Claude Fable 5 --- Dockerfile | 45 ++++++++++++++++++- cosmo_inference/README.md | 23 +++++++--- .../templates/cosmosis_pipeline_A_ia.ini | 6 ++- .../templates/cosmosis_pipeline_A_ia_cell.ini | 6 ++- .../templates/cosmosis_pipeline_A_psf.ini | 6 ++- 5 files changed, 75 insertions(+), 11 deletions(-) diff --git a/Dockerfile b/Dockerfile index 14369ffd..b9c72451 100644 --- a/Dockerfile +++ b/Dockerfile @@ -2,7 +2,9 @@ FROM ghcr.io/cosmostat/shapepipe:im_sims # liblapack-dev: cosmosis's MultiNest links -llapack, and the base image ships -# only the runtime liblapack.so.3 (no dev symlink). +# only the runtime liblapack.so.3 (no dev symlink). The gsl/cfitsio/fftw3 dev +# packages are what the CosmoSIS Standard Library's C sources compile against +# (they are the headers CSL's own CI installs); git is for cloning it. RUN apt-get update -y --quiet --fix-missing && \ apt-get dist-upgrade -y --quiet --fix-missing && \ apt-get install -y --quiet \ @@ -10,10 +12,14 @@ RUN apt-get update -y --quiet --fix-missing && \ automake \ libtool \ pkg-config \ + git \ htop \ npm \ tmux \ - liblapack-dev + liblapack-dev \ + libgsl-dev \ + libcfitsio-dev \ + libfftw3-dev # The base shapepipe image provides a uv-managed venv at /app/.venv (exported as # VIRTUAL_ENV); install sp_validation's deps into that same venv rather than @@ -50,6 +56,41 @@ ENV MPIFC=/opt/ompi/bin/mpif90 RUN uv sync --frozen --inexact --no-install-project \ --extra test --extra glass --extra workflow +# The CosmoSIS Standard Library: the module files (camb interface, projection, +# 2pt likelihood, ...) the cosmo_inference pipelines name. The `workflow` extra +# above installs cosmosis itself; CSL is a separate tree of modules that is not +# on PyPI and has to be built against that install, so it is cloned and compiled +# here rather than left to each user (which is what the .ini templates used to +# assume, hard-coding one person's home directory). +# +# Pinned to Sacha Guerrini's fork, which carries the four commits the UNIONS +# pipelines depend on: tau-stats, sample_S8, and two z-dependent linear-alignment +# modules. See cosmo_inference/README.md for the standing of that fork. +ARG CSL_REPO=https://github.com/sachaguer/cosmosis-standard-library.git +ARG CSL_REF=b26fa7ff666ab4d607b2e32e36f799a53bfb1d9c +ENV CSL_DIR=/opt/cosmosis-standard-library + +# `source cosmosis-configure` is CSL's documented way to build against a +# pip-installed cosmosis: the script ships with the cosmosis package and exports +# COSMOSIS_SRC_DIR, which every CSL Makefile includes its compiler config from. +# bash, not sh, because that script is bash. +# +# `make -C shear` rather than a bare `make`: the top-level target also descends +# into likelihood/, which builds the Planck, WMAP and ACT likelihoods -- large, +# data-dependent, and unused by any UNIONS pipeline. Everything our .ini +# templates reference is either pure Python (consistency, sample_S8, camb, +# load_nz_fits, photoz_bias, linear_alignment, add_intrinsic, shear_m_bias, +# xi_sys, 2pt_like -- no Makefile in those trees at all) or lives under shear/: +# `limber`, which project_2d.py links, and `cl_to_xi_nicaea`, whose +# nicaea_interface.so the 2pt_shear stage loads. +RUN bash -c 'set -euo pipefail; \ + export PATH=/app/.venv/bin:$PATH; \ + git clone --filter=blob:none "$CSL_REPO" "$CSL_DIR"; \ + cd "$CSL_DIR"; \ + git checkout --detach "$CSL_REF"; \ + source cosmosis-configure; \ + make -C shear' + # Install sp_validation itself (editable) into the same venv; deps are already # satisfied by the sync above. COPY . /sp_validation diff --git a/cosmo_inference/README.md b/cosmo_inference/README.md index 94998849..de3c40c0 100644 --- a/cosmo_inference/README.md +++ b/cosmo_inference/README.md @@ -4,12 +4,23 @@ by Lisa Goh and Sacha Guerrini, CEA Paris-Saclay This folder contains the files neccessary to run the cosmological inference pipeline on the UNIONS galaxy catalogues. ### Requirements -[CosmoSIS](https://cosmosis.readthedocs.io/en/latest/) ships in the container via -the `workflow` extra, built with MPI support. To sample the PSF leakage -parameters, the fork of -[cosmosis-standard-library](https://github.com/sachaguer/cosmosis-standard-library/) -of Sacha Guerrini has to be used; it is not packaged, so clone and build it -yourself and point `COSMOSIS_DIR` in the pipeline templates at your checkout. +Everything the pipeline needs ships in the container: nothing to install, and no +paths to edit before a run. + +[CosmoSIS](https://cosmosis.readthedocs.io/en/latest/) comes in via the +`workflow` extra, built with MPI support. The CosmoSIS Standard Library — the +tree of modules the `.ini` pipelines name — is built into the image at +`/opt/cosmosis-standard-library`, with `CSL_DIR` pointing there; that is what +`COSMOSIS_DIR` in the templates resolves to. Outside the container, export +`CSL_DIR` at a build of your own and the same templates work unchanged. + +CSL is pinned to **Sacha Guerrini's fork** +([sachaguer/cosmosis-standard-library](https://github.com/sachaguer/cosmosis-standard-library/)) +at `b26fa7ff`, not to upstream: the UNIONS pipelines depend on four commits that +exist only there — tau statistics, `sample_S8`, and two z-dependent +linear-alignment modules. The fork is 4 commits ahead of +`cosmosis-developers/cosmosis-standard-library` and 373 behind it; carrying those +four forward onto current upstream is future work. Launch sampling under MPI (`mpiexec -n N cosmosis --mpi ...`), not `--smp`: CosmoSIS's shared-memory pool is unmaintained and still crashes after sampling diff --git a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia.ini b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia.ini index eb3ab166..d4927cb0 100644 --- a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia.ini +++ b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia.ini @@ -1,6 +1,10 @@ #parameters used elsewhere in this file [DEFAULT] -COSMOSIS_DIR = /n23data1/n06data/lgoh/scratch/cosmosis-standard-library_lisa +# The CosmoSIS Standard Library. CSL_DIR comes from the environment (CosmoSIS +# reads environment variables into [DEFAULT], the same way %(SCRATCH)s below +# works); the container sets it to /opt/cosmosis-standard-library. Outside the +# container, export CSL_DIR to point at your own build. +COSMOSIS_DIR = %(CSL_DIR)s [pipeline] diff --git a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia_cell.ini b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia_cell.ini index 87f06064..7827f4dc 100644 --- a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia_cell.ini +++ b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia_cell.ini @@ -1,6 +1,10 @@ #parameters used elsewhere in this file [DEFAULT] -COSMOSIS_DIR = /home/guerrini/cosmosis-standard-library +# The CosmoSIS Standard Library. CSL_DIR comes from the environment (CosmoSIS +# reads environment variables into [DEFAULT], the same way %(SCRATCH)s below +# works); the container sets it to /opt/cosmosis-standard-library. Outside the +# container, export CSL_DIR to point at your own build. +COSMOSIS_DIR = %(CSL_DIR)s [pipeline] diff --git a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_psf.ini b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_psf.ini index f9f4da51..b15175b5 100644 --- a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_psf.ini +++ b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_psf.ini @@ -1,6 +1,10 @@ #parameters used elsewhere in this file [DEFAULT] -COSMOSIS_DIR = /home/guerrini/cosmosis-standard-library +# The CosmoSIS Standard Library. CSL_DIR comes from the environment (CosmoSIS +# reads environment variables into [DEFAULT], the same way %(SCRATCH)s below +# works); the container sets it to /opt/cosmosis-standard-library. Outside the +# container, export CSL_DIR to point at your own build. +COSMOSIS_DIR = %(CSL_DIR)s [pipeline] From f1325f020975f91d60e4b390947e5884fa9408c1 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 05:00:24 +0200 Subject: [PATCH 30/37] docs: prune duplicated and historical comments The container model, the checkout-PYTHONPATH default, the profile GENERIC mirroring and the CSL_DIR resolution were each explained in three to six places. Give every concept one home -- workflow/README.md for the user-facing story, the docstring of the thing itself for mechanism -- and leave pointers elsewhere. Drop comments narrating what the code used to do; git holds that. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 10 +++--- cosmo_inference/README.md | 5 +-- .../templates/cosmosis_pipeline_A_ia.ini | 6 ++-- .../templates/cosmosis_pipeline_A_ia_cell.ini | 6 ++-- .../templates/cosmosis_pipeline_A_psf.ini | 6 ++-- papers/bmodes/Snakefile | 4 +-- papers/bmodes/scripts/run_cov_sweep.sh | 3 +- .../bmodes/scripts/run_pure_eb_ptes_sweep.sh | 3 +- .../scripts/run_pure_eb_semianalytic.sh | 3 +- papers/bmodes/scripts/run_pure_eb_sweep.sh | 3 +- papers/cosmo_val/Snakefile | 4 +-- src/sp_validation/container.py | 29 +++++++--------- .../tests/data/container_smoke/Snakefile | 12 ++----- .../data/container_smoke/container_smoke.py | 15 +++----- .../tests/test_container_smoke.py | 7 ++-- workflow/README.md | 34 +++++++------------ workflow/Snakefile | 15 +++----- workflow/common.py | 34 +++++++------------ workflow/profiles/candide/config.yaml | 33 ++++-------------- workflow/profiles/default/config.yaml | 17 +++------- workflow/rules/image_sims.smk | 8 ++--- workflow/rules/twopoint.smk | 17 ++++------ 22 files changed, 94 insertions(+), 180 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 10c75b71..c35c20be 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -88,16 +88,14 @@ spv-container status # which layer is live, and how current it is spv-container exec # one-off run inside it ``` -Need a package the image lacks mid-analysis? Unpack a writable sandbox once with -`spv-container sandbox`, then `spv-container exec --writable pip install `. -The sandbox then takes precedence over the SIF everywhere, workflow jobs -included; `spv-container pull && spv-container sandbox --force` resets it clean. +Need a package the image lacks mid-analysis? `spv-container sandbox`, then +`spv-container exec --writable pip install `; the sandbox then takes +precedence over the SIF everywhere, workflow jobs included. Every rule runs inside that image, wrapped by Snakemake itself (`--profile workflow/profiles/candide` on the cluster, `workflow/profiles/default -j N` elsewhere). The `sp_validation` a rule imports comes from the *launched -checkout*, not the image: `common.configure()` puts its `src/` on the -container's `PYTHONPATH`. +checkout*, not the image. `workflow/README.md` is the full story — profiles, image resolution, refresh. diff --git a/cosmo_inference/README.md b/cosmo_inference/README.md index de3c40c0..e6f4ca83 100644 --- a/cosmo_inference/README.md +++ b/cosmo_inference/README.md @@ -10,8 +10,9 @@ paths to edit before a run. [CosmoSIS](https://cosmosis.readthedocs.io/en/latest/) comes in via the `workflow` extra, built with MPI support. The CosmoSIS Standard Library — the tree of modules the `.ini` pipelines name — is built into the image at -`/opt/cosmosis-standard-library`, with `CSL_DIR` pointing there; that is what -`COSMOSIS_DIR` in the templates resolves to. Outside the container, export +`/opt/cosmosis-standard-library`, with `CSL_DIR` pointing there. CosmoSIS reads +environment variables into an `.ini`'s `[DEFAULT]` section, so the templates' +`COSMOSIS_DIR = %(CSL_DIR)s` resolves to it. Outside the container, export `CSL_DIR` at a build of your own and the same templates work unchanged. CSL is pinned to **Sacha Guerrini's fork** diff --git a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia.ini b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia.ini index d4927cb0..656c6e5c 100644 --- a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia.ini +++ b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia.ini @@ -1,9 +1,7 @@ #parameters used elsewhere in this file [DEFAULT] -# The CosmoSIS Standard Library. CSL_DIR comes from the environment (CosmoSIS -# reads environment variables into [DEFAULT], the same way %(SCRATCH)s below -# works); the container sets it to /opt/cosmosis-standard-library. Outside the -# container, export CSL_DIR to point at your own build. +# The CosmoSIS Standard Library; CSL_DIR comes from the environment, set in the +# container (see cosmo_inference/README.md). COSMOSIS_DIR = %(CSL_DIR)s diff --git a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia_cell.ini b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia_cell.ini index 7827f4dc..4a365195 100644 --- a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia_cell.ini +++ b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia_cell.ini @@ -1,9 +1,7 @@ #parameters used elsewhere in this file [DEFAULT] -# The CosmoSIS Standard Library. CSL_DIR comes from the environment (CosmoSIS -# reads environment variables into [DEFAULT], the same way %(SCRATCH)s below -# works); the container sets it to /opt/cosmosis-standard-library. Outside the -# container, export CSL_DIR to point at your own build. +# The CosmoSIS Standard Library; CSL_DIR comes from the environment, set in the +# container (see cosmo_inference/README.md). COSMOSIS_DIR = %(CSL_DIR)s diff --git a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_psf.ini b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_psf.ini index b15175b5..341baf21 100644 --- a/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_psf.ini +++ b/cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_psf.ini @@ -1,9 +1,7 @@ #parameters used elsewhere in this file [DEFAULT] -# The CosmoSIS Standard Library. CSL_DIR comes from the environment (CosmoSIS -# reads environment variables into [DEFAULT], the same way %(SCRATCH)s below -# works); the container sets it to /opt/cosmosis-standard-library. Outside the -# container, export CSL_DIR to point at your own build. +# The CosmoSIS Standard Library; CSL_DIR comes from the environment, set in the +# container (see cosmo_inference/README.md). COSMOSIS_DIR = %(CSL_DIR)s diff --git a/papers/bmodes/Snakefile b/papers/bmodes/Snakefile index e387181d..97bccc90 100644 --- a/papers/bmodes/Snakefile +++ b/papers/bmodes/Snakefile @@ -25,9 +25,7 @@ import common common.configure(config) from common import * -# The image for every rule: this user's own .sif if they have pulled one with -# `spv-container pull`, else the CI tag for Snakemake to autopull. Override with -# `--config container=`. See common.resolve_container. +# The one image every rule runs in; see common.resolve_container. container: common.resolve_container(config) # Wildcard constraints — centralized in common.py, not in individual rule files diff --git a/papers/bmodes/scripts/run_cov_sweep.sh b/papers/bmodes/scripts/run_cov_sweep.sh index dd0a1330..e2eb1e10 100755 --- a/papers/bmodes/scripts/run_cov_sweep.sh +++ b/papers/bmodes/scripts/run_cov_sweep.sh @@ -27,8 +27,7 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# This user's own image, resolved the way `spv-container` resolves it: the -# writable sandbox if one exists, else the pristine SIF (see workflow/README.md). +# This user's own image, resolved as `spv-container` resolves it. CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} [ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ CONTAINER=$HOME/.cache/sp_validation/sandbox diff --git a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh index 203e934d..6926cc3a 100644 --- a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh @@ -20,8 +20,7 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# This user's own image, resolved the way `spv-container` resolves it: the -# writable sandbox if one exists, else the pristine SIF (see workflow/README.md). +# This user's own image, resolved as `spv-container` resolves it. CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} [ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ CONTAINER=$HOME/.cache/sp_validation/sandbox diff --git a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh index f7adb6aa..b193af7f 100644 --- a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh +++ b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh @@ -16,8 +16,7 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra SRC=$WT/src -# This user's own image, resolved the way `spv-container` resolves it: the -# writable sandbox if one exists, else the pristine SIF (see workflow/README.md). +# This user's own image, resolved as `spv-container` resolves it. CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} [ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ CONTAINER=$HOME/.cache/sp_validation/sandbox diff --git a/papers/bmodes/scripts/run_pure_eb_sweep.sh b/papers/bmodes/scripts/run_pure_eb_sweep.sh index 369176bb..1345a94c 100755 --- a/papers/bmodes/scripts/run_pure_eb_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_sweep.sh @@ -20,8 +20,7 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# This user's own image, resolved the way `spv-container` resolves it: the -# writable sandbox if one exists, else the pristine SIF (see workflow/README.md). +# This user's own image, resolved as `spv-container` resolves it. CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} [ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ CONTAINER=$HOME/.cache/sp_validation/sandbox diff --git a/papers/cosmo_val/Snakefile b/papers/cosmo_val/Snakefile index 0fb2a305..49939ae5 100644 --- a/papers/cosmo_val/Snakefile +++ b/papers/cosmo_val/Snakefile @@ -30,9 +30,7 @@ import common common.configure(config) from common import * -# The image for every rule: this user's own .sif if they have pulled one with -# `spv-container pull`, else the CI tag for Snakemake to autopull. Override with -# `--config container=`. See common.resolve_container. +# The one image every rule runs in; see common.resolve_container. container: common.resolve_container(config) # Wildcard constraints — centralized in common.py, not in individual rule files diff --git a/src/sp_validation/container.py b/src/sp_validation/container.py index 6ef0f2a6..3e591b30 100644 --- a/src/sp_validation/container.py +++ b/src/sp_validation/container.py @@ -1,8 +1,8 @@ """Manage this user's local copy of the sp_validation container image. -Everyone runs their own image. There is no shared image directory and no symlink -to keep honest: the canonical paths are under your own cache, you refresh them -when you want to, and nobody else's refresh moves the ground under a running job. +Everyone runs their own image: the canonical paths are under your own cache, you +refresh them when you want to, and nobody else's refresh moves the ground under +a running job. There are two layers, and you only need the second when you want it: @@ -41,10 +41,9 @@ import sys from pathlib import Path -# The image every entry point names. CI builds and pushes one per branch, tagged -# by the sanitized branch name, so ``:develop`` tracks the integration branch. -# ``workflow/common.py`` re-exports this as ``CONTAINER_URI``; it is written down -# here, once. +# The image every entry point names, written down here once (``workflow/ +# common.py`` re-exports it). CI pushes one tag per branch, sanitized, so +# ``:develop`` tracks the integration branch. CONTAINER_URI = "docker://ghcr.io/cosmostat/sp_validation:develop" CACHE_DIR = Path(os.environ.get("XDG_CACHE_HOME", "~/.cache")) / "sp_validation" @@ -173,11 +172,10 @@ def cmd_pull(args): sys.exit("apptainer is not on PATH") sif = local_sif() sif.parent.mkdir(parents=True, exist_ok=True) - # Pull to a sibling temp name and rename. `mv` within one directory is an - # atomic rename, so a job either gets the whole old image or the whole new - # one; pulling in place would leave the file half-written for the ~15 - # minutes the pull takes, and anything starting in that window would fail. - # Jobs already running hold the old inode open and finish against it. + # Pull to a sibling temp name and rename: an atomic rename within one + # directory, so a job gets either the whole old image or the whole new one. + # Pulling in place would leave the file half-written for the ~15 minutes the + # pull takes. Jobs already running hold the old inode open and finish on it. tmp = sif.with_name(sif.name + f".pull.{os.getpid()}") print(f"pulling {args.tag}\n -> {sif}") try: @@ -288,9 +286,6 @@ def cmd_status(args): print(f"revision: {revision or 'unknown'}{source}") print(f"version: {labels.get('org.opencontainers.image.version', 'unknown')}") if kind == "sandbox": - # The revision is the image the sandbox was *built from*; anything - # installed into it since is invisible to any label. Say so rather than - # let the revision read as a full description of what is running. print( " (the revision above is what the sandbox was built from; " "anything\n installed into it since is not reflected in " @@ -317,8 +312,8 @@ def cmd_exec(args): binds = args.bind or os.environ.get("SPV_APPTAINER_BINDS", DEFAULT_BINDS) if args.writable: - # Writes only persist into a sandbox; a SIF is a read-only filesystem, so - # `--writable` against one fails obscurely. Say what to do instead. + # A SIF is a read-only filesystem, so `--writable` against one fails + # obscurely; only a sandbox takes writes. sandbox = local_sandbox() if not sandbox.is_dir(): sys.exit( diff --git a/src/sp_validation/tests/data/container_smoke/Snakefile b/src/sp_validation/tests/data/container_smoke/Snakefile index 8843333c..75527690 100644 --- a/src/sp_validation/tests/data/container_smoke/Snakefile +++ b/src/sp_validation/tests/data/container_smoke/Snakefile @@ -1,15 +1,9 @@ # Standalone workflow exercised by src/sp_validation/tests/test_container_smoke.py. # # The module-level `container:` below mirrors what every real workflow does -# (workflow/Snakefile) -- Snakemake has no way to take a default image -# from a profile. Everything else under test arrives from the driving profile -# (workflow/profiles/candide): the slurm executor, `software-deployment-method: -# apptainer` that turns container wrapping on, and the `apptainer-args` binds. -# No rule-level `container:` and no `apptainer exec` shell call -- Snakemake -# wraps the job itself, and the test asserts APPTAINER_CONTAINER was visible -# inside the job to prove the wrapping actually happened. -# -# See container_smoke.py for what the job checks and why. +# (workflow/Snakefile) -- Snakemake has no way to take a default image from a +# profile. Everything else under test arrives from the driving profile +# (workflow/profiles/candide). See container_smoke.py for what the job checks. # Literal rather than the package's CONTAINER_URI: this Snakefile is test data, diff --git a/src/sp_validation/tests/data/container_smoke/container_smoke.py b/src/sp_validation/tests/data/container_smoke/container_smoke.py index 98e5896f..b36eba11 100644 --- a/src/sp_validation/tests/data/container_smoke/container_smoke.py +++ b/src/sp_validation/tests/data/container_smoke/container_smoke.py @@ -1,19 +1,16 @@ """Rule container_smoke: exercise the containerized-SLURM path end to end. -Cheap sanity check for the profile-driven-container pivot -- same executor +Cheap sanity check of the profile-driven container path -- same executor (slurm), same software-deployment-method (apptainer), same apptainer-args -binds, same container image every real rule uses. No rule-level `container:` -or `apptainer exec` anywhere here; Snakemake wraps the job itself. Four things -it proves, each written to the output YAML: +binds, same container image every real rule uses. Four things it proves, each +written to the output YAML: * the job really ran inside the image (``APPTAINER_CONTAINER``, set by apptainer itself -- without it the rest could all pass on the bare host); * the editable ``sp_validation`` install resolves on the container's PYTHONPATH (import provenance: file + version, not just import success); - * the numeric stack works (numpy eigh on a small fixed matrix). The - ``OMP_NUM_THREADS`` the job sees is recorded but NOT asserted: the profile - deliberately leaves it unset, and rules needing it pinned set it themselves - (see the image_sims rules' env prefix), so "unset" here is correct; + * the numeric stack works (numpy eigh on a small fixed matrix). + ``OMP_NUM_THREADS`` is recorded but not asserted -- see the assertions; * which commit of this checkout is running (git rev-parse from inside the container -- proves /home is bound and usable, not just readable). @@ -30,8 +27,6 @@ # --- the job is actually inside the image --------------------------------- -# apptainer sets APPTAINER_CONTAINER (path of the running image) in every -# process it starts, and it survives --cleanenv. Absent => ran on the bare host. container_info = { "apptainer_container": os.environ.get("APPTAINER_CONTAINER", "unset"), } diff --git a/src/sp_validation/tests/test_container_smoke.py b/src/sp_validation/tests/test_container_smoke.py index 0bb0068f..e10d9ab0 100644 --- a/src/sp_validation/tests/test_container_smoke.py +++ b/src/sp_validation/tests/test_container_smoke.py @@ -109,10 +109,9 @@ def test_container_smoke(): report["numeric"]["eigenvalues"], _reference_eigenvalues(), rtol=1e-10, atol=1e-12 ) - # numeric.omp_num_threads is recorded for observability but deliberately NOT - # asserted: the profile leaves OMP_NUM_THREADS unset by design, and rules - # that need it pinned set it themselves (image_sims' env prefix), so "unset" - # here is the correct state rather than a gap. + # numeric.omp_num_threads is recorded but deliberately NOT asserted: the + # profile leaves OMP_NUM_THREADS unset by design, and rules that need it + # pinned set it themselves, so "unset" here is correct rather than a gap. # git worked inside the container, so /home is bound and usable. assert re.fullmatch(r"[0-9a-f]{40}", report["provenance"]["commit"]), report["provenance"] diff --git a/workflow/README.md b/workflow/README.md index 69f0ca16..81f15277 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -64,7 +64,6 @@ A few rules shell out to a host toolchain (CosmoCov, ImageMagick) and keep `OMP_NUM_THREADS` is not set by the profile either: the slurm executor's `--export=ALL` propagates the driver's env, not a profile flag, so a rule that needs it pinned sets it itself. Per-rule `mem_mb` / `runtime` stay on the rules. -See the profile's own comments for the full rationale. ### Off candide — the default profile @@ -85,17 +84,14 @@ of it: expect to edit the default profile's `--bind` list for your machine. ### Which `sp_validation` a rule imports: the launched checkout The image is the frozen *dependency stack*; the `sp_validation` that runs is -the one in the checkout you launched from. `common.configure()` prepends that -checkout's `src/` to `APPTAINERENV_PYTHONPATH`, which Apptainer forwards into -each job as `PYTHONPATH` (surviving the profile's `--cleanenv`). Any value you -exported yourself is preserved behind it. +the one in the checkout you launched from — `common.configure()` puts that +checkout's `src/` on each job's `PYTHONPATH` +(`common.inject_checkout_pythonpath` has the mechanics). This is the default because the alternative is incoherent: Snakemake's `script:` directive already runs the checkout's *script files*, so without it a rule executes new script code against an old `import sp_validation` — the two -halves of one commit, split. The image-sims chain has always worked this way -(`_ENV_PREFIX` in `workflow/rules/image_sims.smk`); the rest of the workflow now -matches it. +halves of one commit, split. **Caveat:** `rerun-triggers: code` watches rule bodies and `script:` files, not `src/`. Editing a module under `src/` does not by itself mark outputs stale — @@ -157,13 +153,10 @@ wrapping from the profile (see above), so the container is where the science code runs, not where the orchestrator runs — one container per job, never a nested one. -Driving Snakemake from inside a container shell used to be the recommended -path, and is why an old `~/.local/bin/snakemake` (or any host-side `pip -install --user snakemake`) is worth checking for: Apptainer passes your `PATH` -and mounts your `$HOME` by default, so a leftover host install can silently -shadow the one `uv tool install` just set up. Run `which snakemake` and -confirm it resolves under `uv`'s tool directory (`uv tool dir`), not -`~/.local/bin`. +Check for a stray `~/.local/bin/snakemake` (any host-side `pip install --user +snakemake` leaves one): Apptainer passes your `PATH` and mounts your `$HOME` by +default, so it can silently shadow the one `uv tool install` set up. `which +snakemake` should resolve under `uv tool dir`, not `~/.local/bin`. ### The container image — one per person @@ -279,12 +272,11 @@ config. Your image has to sit under one of the profile's bind mounts to be visible. To run a *branch's* CI image rather than a local file, see "Testing a branch's own image" above. -One invariant survives from the old hand-built sandbox and still applies: the -`script:` directive bind-mounts the host orchestrator's `snakemake` into the job -and *appends* it to `sys.path`, so a `snakemake` importable inside the image -wins the lookup. If `script:` rules start failing with `ModuleNotFoundError: No -module named 'snakemake.iocontainers'` or similar, an in-image snakemake older -than the host's is the first thing to check. +One trap to know: the `script:` directive bind-mounts the host orchestrator's +`snakemake` into the job and *appends* it to `sys.path`, so a `snakemake` +importable inside the image wins the lookup. If `script:` rules start failing +with `ModuleNotFoundError: No module named 'snakemake.iocontainers'` or similar, +an in-image snakemake older than the host's is the first thing to check. ### `snakemake` in `script:` files diff --git a/workflow/Snakefile b/workflow/Snakefile index a24d7dc1..908e2c66 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -14,9 +14,8 @@ envvars: import os import sys -# Machine-specific env (the host OpenMPI libs MPI rules need inside the -# container) is not set here: it belongs to the machine, so it rides the -# candide profile's `apptainer-args --env`. See workflow/profiles/. +# Machine-specific env belongs to the machine, so it rides the profile's +# `apptainer-args --env`, not this file. See workflow/profiles/. # # Shared helpers live in common.py next to this Snakefile. Snakemake's `module` # imports rules, not Python globals, so helpers travel by plain Python import; @@ -25,16 +24,12 @@ import sys sys.path.insert(0, os.path.realpath(str(workflow.basedir))) import common -# configure() also prepends this checkout's src/ to the container's PYTHONPATH -# (common.inject_checkout_pythonpath), so the launched tree's sp_validation -- -# not the image's baked copy -- is what rules import. `--config -# checkout_pythonpath=false` opts out. +# configure() also puts this checkout's src/ on the container's PYTHONPATH -- +# see common.inject_checkout_pythonpath. common.configure(config) from common import * -# The one image every rule runs in: this user's own .sif if they have pulled one -# with `spv-container pull`, else the CI tag for Snakemake to autopull. Override -# with `--config container=`. See common.resolve_container. +# The one image every rule runs in; see common.resolve_container. container: common.resolve_container(config) # Wildcard constraints — centralized in common.py, not in individual rule files diff --git a/workflow/common.py b/workflow/common.py index c138b533..145fe2fa 100644 --- a/workflow/common.py +++ b/workflow/common.py @@ -10,11 +10,9 @@ # This checkout's importable source tree: workflow/common.py -> /src. REPO_SRC = Path(__file__).resolve().parent.parent / "src" -# The container model lives in the package (``sp_validation/container.py``): the -# registry tag, this user's canonical image paths, and the ``spv-container`` CLI -# that fills them. Taken from *this checkout's* src/, so the workflow and the CLI -# can never disagree -- in particular they share one resolution order, which is -# what lets a package installed into a sandbox ride along into workflow jobs. +# The container model lives in the package (``sp_validation/container.py``). +# Taken from *this checkout's* src/, so the workflow and the ``spv-container`` +# CLI can never disagree about which image to run. # # Loaded by file path rather than as ``sp_validation.container``: snakemake runs # on the host, where sp_validation is usually not installed, and importing the @@ -88,8 +86,7 @@ def inject_checkout_pythonpath(workflow_config): files, so without this a rule executes new script code against an old ``import sp_validation`` -- the two halves of one commit, split. Prepending ``REPO_SRC`` closes that: the image stays the frozen dependency stack, the - checkout supplies sp_validation. This mirrors what the image-sims chain has - always done for both repos (``_ENV_PREFIX`` in rules/image_sims.smk). + checkout supplies sp_validation. Apptainer forwards ``APPTAINERENV_``-prefixed host variables into the job as their unprefixed names, surviving the profile's ``--cleanenv``; setting it @@ -97,9 +94,7 @@ def inject_checkout_pythonpath(workflow_config): already exported is preserved behind ours. Opt out with ``--config checkout_pythonpath=false`` to reproduce a run from - the image alone. Note the caveat: ``rerun-triggers: code`` watches rule and - script files, not ``src/``, so editing a module under ``src/`` does not by - itself mark outputs stale -- force with ``-F``/``--forcerun``. + the image alone. """ flag = workflow_config.get("checkout_pythonpath", True) # `--config key=false` can arrive as the *string* "false" depending on how @@ -118,13 +113,10 @@ def inject_checkout_pythonpath(workflow_config): def resolve_container(workflow_config): """Return the image every rule should run in. - The same order ``spv-container`` uses, so jobs run what interactive work - runs: your writable sandbox if you have built one, else your pristine - ``.sif`` if you have pulled one, else the registry tag -- which Snakemake - autopulls into ``.snakemake/singularity`` under the working directory. - Snakemake's ``container:`` accepts all three (a sandbox directory included). - ``--config container=...`` overrides everything and takes a ``docker://`` - tag, a ``.sif`` path, or a sandbox directory. + ``--config container=...`` wins if set (a ``docker://`` tag, a ``.sif`` path + or a sandbox directory -- Snakemake's ``container:`` accepts all three); + otherwise ``resolve_image()``, so jobs run what interactive + ``spv-container`` work runs. """ override = workflow_config.get("container") if override: @@ -138,11 +130,11 @@ def warn_if_image_stale(): Never fatal. Two things worth saying at launch: * a sandbox is in play, so what jobs run is not fully described by any - revision label -- somebody installed into it on purpose, and that is the - point, but it should not be a silent difference from a clean run; + revision label -- deliberate, but it should not be a silent difference + from a clean run; * the image predates the checkout. Usually fine, because the checkout's - ``src/`` is what rules import (inject_checkout_pythonpath); it matters when - the *dependency stack* moved -- a new package, a lockfile bump. + ``src/`` is what rules import; it matters when the *dependency stack* + moved -- a new package, a lockfile bump. Silent when there is no local image, no apptainer, or no revision label. """ diff --git a/workflow/profiles/candide/config.yaml b/workflow/profiles/candide/config.yaml index 3b320b1b..19afd89a 100644 --- a/workflow/profiles/candide/config.yaml +++ b/workflow/profiles/candide/config.yaml @@ -6,22 +6,8 @@ # -s workflow/image_sims/Snakefile \ # --configfile # -# Snakemake owns scheduling (one SLURM job per branch x tile) and the -# container: every rule runs through Snakemake's own container wrapping -# (``container:`` on the rule or the module-level default in -# workflow/Snakefile), never a rule's own ``apptainer exec`` shell call. -# ``software-deployment-method: apptainer`` below turns that wrapping on; -# ``apptainer-args`` carries the bind mounts every rule needs. A few rules call -# host toolchains and opt out with ``container: None``; each says why in its own -# docstring. -# -# ``snakemake`` itself is a thin host-side tool, pinned via ``uv tool -# install`` (see workflow/README.md); run it on the host, never inside an -# ``apptainer shell``. -# -# Snakemake cannot compose profiles, so the machine-independent settings (marked -# GENERIC below) are duplicated in workflow/profiles/default/config.yaml, for -# running off candide; change one, change the other. +# Snakemake owns scheduling and the container wrapping; run it host-side, never +# inside an ``apptainer shell``. workflow/README.md is the full story. executor: slurm @@ -38,17 +24,13 @@ rerun-triggers: ["mtime", "params", "input", "code"] latency-wait: 5 # --- end GENERIC ------------------------------------------------------------ -# No ``apptainer-prefix``: everyone runs their own image, at their own canonical -# path, pulled with ``spv-container pull`` (workflow/README.md). The entry -# Snakefiles resolve ``container:`` to that file when it exists, so there is -# nothing for Snakemake to cache. +# No ``apptainer-prefix``: the entry Snakefiles resolve ``container:`` to this +# user's own image path, so there is nothing for Snakemake to cache. # # candide's disks, plus the one machine-specific env var: the host OpenMPI libs # MPI rules need to find libmpi inside the container (only rules importing -# mpi4py care; harmless for the rest). This was an ``os.environ[...]`` line in -# workflow/Snakefile -- a machine path hard-coded into generic workflow code -- -# and belongs here. The bind list matches ``spv-container exec``'s default; keep -# the two in step. +# mpi4py care; harmless for the rest). The bind list matches ``spv-container +# exec``'s default; keep the two in step. apptainer-args: >- --cleanenv --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data @@ -81,7 +63,6 @@ default-resources: slurm_extra: "'--exclude=n17,n09,n36'" # Retry a job once on transient node failure, and keep the SLURM logs of -# successful jobs (candide debugging). (``latency-wait`` and ``rerun-triggers`` -# are in the GENERIC block above.) +# successful jobs (candide debugging). retries: 1 slurm-keep-successful-logs: true diff --git a/workflow/profiles/default/config.yaml b/workflow/profiles/default/config.yaml index 3c1042fa..bf272c03 100644 --- a/workflow/profiles/default/config.yaml +++ b/workflow/profiles/default/config.yaml @@ -8,20 +8,15 @@ # # On candide -- where the analysis actually runs -- use # `--profile workflow/profiles/candide` instead: the GENERIC block below plus -# the SLURM executor and candide's machine layer. Snakemake cannot compose -# profiles, so that block is duplicated in both files; change one, change the -# other. +# the SLURM executor and candide's machine layer. # # Requirements are the same everywhere: `apptainer` on PATH, and `snakemake` # installed host-side (`uv tool install ...`, see workflow/README.md) -- never # run from inside an apptainer shell. # --- GENERIC: mirrored in workflow/profiles/candide/config.yaml ------------- -# Turn on Snakemake's own container wrapping: it wraps each job's -# `shell:`/`script:` command in `apptainer exec`, using the image named by the -# entry Snakefile's `container:` directive. No rule writes its own -# `apptainer exec` call; the few that opt out with `container: None` say why in -# their own docstrings. +# Wrap each job's `shell:`/`script:` command in `apptainer exec`, using the +# image named by the entry Snakefile's `container:` directive. software-deployment-method: apptainer # Rerun a job when its code / params / inputs change, not only on mtime. @@ -40,7 +35,5 @@ latency-wait: 5 # directory cover everything (apptainer mounts both by default), drop `--bind`. apptainer-args: "--cleanenv --bind /home" -# No `apptainer-prefix`, here or on candide: `spv-container pull` puts your image -# at one canonical per-user path and the entry Snakefiles resolve `container:` to -# it. Without that file Snakemake autopulls the tag into `.snakemake/singularity` -# under the working directory, which works but re-pulls per run directory. +# No `apptainer-prefix`, here or on candide: the entry Snakefiles resolve +# `container:` to this user's own image path (workflow/README.md). diff --git a/workflow/rules/image_sims.smk b/workflow/rules/image_sims.smk index 682bc1ba..2b9be8cb 100644 --- a/workflow/rules/image_sims.smk +++ b/workflow/rules/image_sims.smk @@ -95,12 +95,8 @@ if _missing_structural: # Every compute rule carries ``container: SIF`` rather than inheriting a # module-level default: these rules are also included from the top-level # workflow/Snakefile, whose module default is the cosmology image (no ShapePipe -# stack). Binds come from the driving profile's ``apptainer-args``. -# -# ``sif: null`` (the config default) means "the workflow's one image", resolved -# the same way every other entry point resolves it: this user's own .sif if they -# have pulled one, else the registry tag. Set ``image_sims: {sif: ...}`` in a -# run config to name a different image or a branch tag. +# stack). Binds come from the driving profile's ``apptainer-args``. A null +# ``sif`` resolves to the workflow's one image (see workflow/image_sims/config.yaml). SIF = common.resolve_container({"container": IMSIM["sif"]}) # --- repositories (bound into the image; branch code overrides) ----------- diff --git a/workflow/rules/twopoint.smk b/workflow/rules/twopoint.smk index 5cec5021..861c3b05 100644 --- a/workflow/rules/twopoint.smk +++ b/workflow/rules/twopoint.smk @@ -24,30 +24,27 @@ rule xi: "../scripts/run_2pcf.py" -# PARKED: xi_highres (high-resolution xi for COSEBIS integration). Never -# runnable as written -- the shell invokes run_2pcf_highres.py bare, but the -# script has required --cat-config and --out arguments (true in every version -# since it was introduced). Revive it with those arguments supplied. +# PARKED: xi_highres (high-resolution xi for COSEBIS integration). Not runnable +# as written -- the shell invokes run_2pcf_highres.py bare, but the script has +# required --cat-config and --out arguments. Revive it with those supplied. # # The MPI reasoning below is hard-won and must survive the revival: # -# Exception to the profile-driven container model (see -# workflow/profiles/candide/config.yaml): this is multi-node MPI, one +# Exception to the profile-driven container model: this is multi-node MPI, one # `apptainer exec` per rank. Snakemake's own container wrapping puts the # *whole* shell command -- `mpiexec` included -- inside a single container # instance, so only rank 0's node would run inside it; the other ranks, # spawned by SLURM/PMI on their own nodes, would land bare on the host. # `container: None` plus an explicit `mpiexec -n N apptainer exec ...` -# per-rank is therefore required, not a leftover of the old convention. +# per-rank is therefore required. # Snakemake's slurm-jobstep plugin deliberately does NOT prepend `srun` to a # job carrying an `mpi` resource, which is what lets the rule's own launcher # run on the host, outside the container. # Because this rule builds its own apptainer call, reaching the source-cache # copy of the script relies on our `--bind /home` rather than on Snakemake's # automatic mount -- and on a concrete image file, since `apptainer exec` -# takes no `docker://` URI. A revived rule should take that path from -# `resolve_image()[0]` -- the same local image everything else resolves -- -# rather than name a second image path that can drift. +# takes no `docker://` URI. Take that path from `resolve_image()[0]` rather +# than naming a second image path that can drift. # # rule xi_highres: # container: None From bf5722a0de04167eea26f2dfad23b7505dff5d43 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 05:08:01 +0200 Subject: [PATCH 31/37] simplify: collapse duplication added by this branch The container model landed the same few lines in several places; fold each into one home. * the four run_*.sh sweep drivers resolved this user's image (and repeated the bind list) inline -- now one sourced papers/bmodes/scripts/container_env.sh, resolving exactly as sp_validation/container.py does (sandbox first, then SPV_CONTAINER/XDG_CACHE_HOME, binds from SPV_APPTAINER_BINDS) * container.py: one _require_apptainer() instead of three copies of the PATH guard, and compare_revision's merge-base calls go through _git * common.py: resolve_container takes the override value, so image_sims.smk no longer wraps IMSIM["sif"] in a synthetic config dict; drop the CONTAINER_URI / local_sif / local_sandbox re-exports, which have no callers * cosmocov_process.py: only Snakemake runs it, so drop the argv entry point and its main() indirection, matching im_mbias_config.py * xip_xim.py: one catalog() builder for the tomographic and non-tomographic paths instead of two near-identical treecorr.Catalog blocks Co-Authored-By: Claude Fable 5 --- papers/bmodes/Snakefile | 2 +- papers/bmodes/scripts/container_env.sh | 16 +++++++ papers/bmodes/scripts/run_cov_sweep.sh | 6 +-- .../bmodes/scripts/run_pure_eb_ptes_sweep.sh | 6 +-- .../scripts/run_pure_eb_semianalytic.sh | 8 +--- papers/bmodes/scripts/run_pure_eb_sweep.sh | 6 +-- papers/cosmo_val/Snakefile | 2 +- src/sp_validation/container.py | 36 ++++++-------- workflow/Snakefile | 2 +- workflow/common.py | 14 ++---- workflow/rules/image_sims.smk | 2 +- workflow/scripts/cosmocov_process.py | 48 +++++++------------ 12 files changed, 61 insertions(+), 87 deletions(-) create mode 100644 papers/bmodes/scripts/container_env.sh diff --git a/papers/bmodes/Snakefile b/papers/bmodes/Snakefile index 97bccc90..c58663d6 100644 --- a/papers/bmodes/Snakefile +++ b/papers/bmodes/Snakefile @@ -26,7 +26,7 @@ common.configure(config) from common import * # The one image every rule runs in; see common.resolve_container. -container: common.resolve_container(config) +container: common.resolve_container(config.get("container")) # Wildcard constraints — centralized in common.py, not in individual rule files wildcard_constraints: diff --git a/papers/bmodes/scripts/container_env.sh b/papers/bmodes/scripts/container_env.sh new file mode 100644 index 00000000..496f83e2 --- /dev/null +++ b/papers/bmodes/scripts/container_env.sh @@ -0,0 +1,16 @@ +# Shared container settings for the run_*.sh sweep drivers. Source, don't run: +# +# . "$(dirname "${BASH_SOURCE[0]}")/container_env.sh" +# +# Sets CONTAINER (this user's image) and BIND (the mounts to pass to +# `apptainer exec`), resolved exactly as `sp_validation/container.py` does: +# the writable sandbox if there is one, else the SIF. +_spv_cache=${XDG_CACHE_HOME:-$HOME/.cache}/sp_validation +_spv_sandbox=${SPV_SANDBOX:-$_spv_cache/sandbox} +if [ -d "$_spv_sandbox" ]; then + CONTAINER=$_spv_sandbox +else + CONTAINER=${SPV_CONTAINER:-$_spv_cache/sp_validation.sif} +fi +BIND=${SPV_APPTAINER_BINDS:-/home,/scratch,/automnt,/n17data,/n23data1,/n09data} +unset _spv_cache _spv_sandbox diff --git a/papers/bmodes/scripts/run_cov_sweep.sh b/papers/bmodes/scripts/run_cov_sweep.sh index e2eb1e10..ad064408 100755 --- a/papers/bmodes/scripts/run_cov_sweep.sh +++ b/papers/bmodes/scripts/run_cov_sweep.sh @@ -27,14 +27,10 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# This user's own image, resolved as `spv-container` resolves it. -CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} -[ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ - CONTAINER=$HOME/.cache/sp_validation/sandbox +. "$(dirname "${BASH_SOURCE[0]}")/container_env.sh" SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts -BIND=/home,/scratch,/automnt,/n17data,/n23data1,/n09data CONFIG=""; CATCONFIG=""; PLANCK18=""; MASKBASE=""; OUT=""; BLIND="A"; VERSIONS="" MINSEP=0.5; MAXSEP=300.0; NBINS=1000 diff --git a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh index 6926cc3a..09014397 100644 --- a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh @@ -20,14 +20,10 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# This user's own image, resolved as `spv-container` resolves it. -CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} -[ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ - CONTAINER=$HOME/.cache/sp_validation/sandbox +. "$(dirname "${BASH_SOURCE[0]}")/container_env.sh" SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts -BIND=/home,/scratch,/automnt,/n17data,/n23data1,/n09data CONFIG=""; CATCONFIG=""; PUREEBSWEEP=""; COVSWEEP=""; OUT=""; BLIND="A"; VERSIONS="" while [ $# -gt 0 ]; do diff --git a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh index b193af7f..60b85c4e 100644 --- a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh +++ b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh @@ -16,12 +16,8 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra SRC=$WT/src -# This user's own image, resolved as `spv-container` resolves it. -CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} -[ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ - CONTAINER=$HOME/.cache/sp_validation/sandbox -SCRIPTS=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra/papers/bmodes/scripts -BIND=/home,/scratch,/automnt,/n17data,/n23data1,/n09data +. "$(dirname "${BASH_SOURCE[0]}")/container_env.sh" +SCRIPTS=$WT/papers/bmodes/scripts VERSION=""; BLIND="A"; CATCONFIG=""; XIREP=""; XIINT=""; COVINT=""; OUT="" NCHUNKS=20; NSAMPLES=2000; NPROC="${SLURM_CPUS_PER_TASK:-16}" diff --git a/papers/bmodes/scripts/run_pure_eb_sweep.sh b/papers/bmodes/scripts/run_pure_eb_sweep.sh index 1345a94c..742d2cc8 100755 --- a/papers/bmodes/scripts/run_pure_eb_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_sweep.sh @@ -20,14 +20,10 @@ set -euo pipefail WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -# This user's own image, resolved as `spv-container` resolves it. -CONTAINER=${SPV_CONTAINER:-$HOME/.cache/sp_validation/sp_validation.sif} -[ -d "$HOME/.cache/sp_validation/sandbox" ] && [ -z "${SPV_CONTAINER:-}" ] && \ - CONTAINER=$HOME/.cache/sp_validation/sandbox +. "$(dirname "${BASH_SOURCE[0]}")/container_env.sh" SRC=$WT/src WSCRIPTS=$WT/workflow/scripts PSCRIPTS=$WT/papers/bmodes/scripts -BIND=/home,/scratch,/automnt,/n17data,/n23data1,/n09data CONFIG=""; CATCONFIG=""; XISWEEP=""; COVSWEEP=""; OUT=""; BLIND="A"; VERSIONS="" while [ $# -gt 0 ]; do diff --git a/papers/cosmo_val/Snakefile b/papers/cosmo_val/Snakefile index 49939ae5..716a3ac6 100644 --- a/papers/cosmo_val/Snakefile +++ b/papers/cosmo_val/Snakefile @@ -31,7 +31,7 @@ common.configure(config) from common import * # The one image every rule runs in; see common.resolve_container. -container: common.resolve_container(config) +container: common.resolve_container(config.get("container")) # Wildcard constraints — centralized in common.py, not in individual rule files wildcard_constraints: diff --git a/src/sp_validation/container.py b/src/sp_validation/container.py index 3e591b30..3589930f 100644 --- a/src/sp_validation/container.py +++ b/src/sp_validation/container.py @@ -41,9 +41,8 @@ import sys from pathlib import Path -# The image every entry point names, written down here once (``workflow/ -# common.py`` re-exports it). CI pushes one tag per branch, sanitized, so -# ``:develop`` tracks the integration branch. +# The image every entry point names, written down here once. CI pushes one tag +# per branch, sanitized, so ``:develop`` tracks the integration branch. CONTAINER_URI = "docker://ghcr.io/cosmostat/sp_validation:develop" CACHE_DIR = Path(os.environ.get("XDG_CACHE_HOME", "~/.cache")) / "sp_validation" @@ -123,6 +122,12 @@ def image_revision(sif): return image_labels(sif).get("org.opencontainers.image.revision") +def _require_apptainer(): + """Exit unless ``apptainer`` is on PATH.""" + if shutil.which("apptainer") is None: + sys.exit("apptainer is not on PATH") + + def _git(*args, cwd=None): """Run a git command, returning stripped stdout or ``None`` on any failure.""" try: @@ -151,25 +156,16 @@ def compare_revision(revision, repo=None): return "in-sync" if _git("cat-file", "-e", f"{revision}^{{commit}}", cwd=repo) is None: return "unknown" - ancestor = subprocess.run( - ["git", "merge-base", "--is-ancestor", revision, head], - capture_output=True, - cwd=repo, - ) - if ancestor.returncode == 0: + if _git("merge-base", "--is-ancestor", revision, head, cwd=repo) is not None: return "behind" - reverse = subprocess.run( - ["git", "merge-base", "--is-ancestor", head, revision], - capture_output=True, - cwd=repo, - ) - return "ahead" if reverse.returncode == 0 else "diverged" + if _git("merge-base", "--is-ancestor", head, revision, cwd=repo) is not None: + return "ahead" + return "diverged" def cmd_pull(args): """Pull ``--tag`` to the canonical path, atomically.""" - if shutil.which("apptainer") is None: - sys.exit("apptainer is not on PATH") + _require_apptainer() sif = local_sif() sif.parent.mkdir(parents=True, exist_ok=True) # Pull to a sibling temp name and rename: an atomic rename within one @@ -197,8 +193,7 @@ def cmd_pull(args): def cmd_sandbox(args): """Unpack the image into a writable directory -- the opt-in escape hatch.""" - if shutil.which("apptainer") is None: - sys.exit("apptainer is not on PATH") + _require_apptainer() sandbox = local_sandbox() if sandbox.exists() and not args.force: sys.exit( @@ -305,8 +300,7 @@ def cmd_status(args): def cmd_exec(args): """Run a command inside the image -- the one-off path for humans and agents.""" - if shutil.which("apptainer") is None: - sys.exit("apptainer is not on PATH") + _require_apptainer() if not args.command: sys.exit("nothing to run; pass a command after `exec`") binds = args.bind or os.environ.get("SPV_APPTAINER_BINDS", DEFAULT_BINDS) diff --git a/workflow/Snakefile b/workflow/Snakefile index 908e2c66..fe05f02b 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -30,7 +30,7 @@ common.configure(config) from common import * # The one image every rule runs in; see common.resolve_container. -container: common.resolve_container(config) +container: common.resolve_container(config.get("container")) # Wildcard constraints — centralized in common.py, not in individual rule files wildcard_constraints: diff --git a/workflow/common.py b/workflow/common.py index 145fe2fa..506f7cb2 100644 --- a/workflow/common.py +++ b/workflow/common.py @@ -26,11 +26,8 @@ sys.modules["_spv_container"] = _container _container.__loader__.exec_module(_container) -CONTAINER_URI = _container.CONTAINER_URI compare_revision = _container.compare_revision image_revision = _container.image_revision -local_sandbox = _container.local_sandbox -local_sif = _container.local_sif resolve_image = _container.resolve_image @@ -110,15 +107,14 @@ def inject_checkout_pythonpath(workflow_config): os.environ["APPTAINERENV_PYTHONPATH"] = ":".join(parts) -def resolve_container(workflow_config): +def resolve_container(override=None): """Return the image every rule should run in. - ``--config container=...`` wins if set (a ``docker://`` tag, a ``.sif`` path - or a sandbox directory -- Snakemake's ``container:`` accepts all three); - otherwise ``resolve_image()``, so jobs run what interactive - ``spv-container`` work runs. + ``override`` wins if set (a ``docker://`` tag, a ``.sif`` path or a sandbox + directory -- Snakemake's ``container:`` accepts all three); otherwise + ``resolve_image()``, so jobs run what interactive ``spv-container`` work + runs. """ - override = workflow_config.get("container") if override: return str(override) return resolve_image()[0] diff --git a/workflow/rules/image_sims.smk b/workflow/rules/image_sims.smk index 2b9be8cb..cc992391 100644 --- a/workflow/rules/image_sims.smk +++ b/workflow/rules/image_sims.smk @@ -97,7 +97,7 @@ if _missing_structural: # workflow/Snakefile, whose module default is the cosmology image (no ShapePipe # stack). Binds come from the driving profile's ``apptainer-args``. A null # ``sif`` resolves to the workflow's one image (see workflow/image_sims/config.yaml). -SIF = common.resolve_container({"container": IMSIM["sif"]}) +SIF = common.resolve_container(IMSIM["sif"]) # --- repositories (bound into the image; branch code overrides) ----------- SHAPEPIPE_REPO = IMSIM["shapepipe_repo"] diff --git a/workflow/scripts/cosmocov_process.py b/workflow/scripts/cosmocov_process.py index 76ea7e0a..6e6c1723 100644 --- a/workflow/scripts/cosmocov_process.py +++ b/workflow/scripts/cosmocov_process.py @@ -3,6 +3,9 @@ CosmoCov writes one row per (i, j) element with the Gaussian term in column 8 and the non-Gaussian term in column 9; this rebuilds the symmetric matrices, checks positive-definiteness, and plots the correlation matrix. + +Run through Snakemake's ``script:`` directive, which injects ``snakemake`` as a +module global before this file executes. """ import sys @@ -55,35 +58,16 @@ def plot_correlation(cov, ndata, plot_path): plt.close(fig) -def main(covfile, matrix_path, gaussian_path, plot_path): - cov_g, cov_ng, ndata = get_cov(covfile) - print(f"Dimension of cov: {ndata}x{ndata}") - - cov = cov_g + cov_ng - - eigenvalues = np.linalg.eigvalsh(cov) - print(f"min+max eigenvalues cov: {eigenvalues.min():e}, {eigenvalues.max():e}") - if eigenvalues.min() <= 0.0: - sys.exit("non-positive eigenvalue encountered! Covariance invalid!") - - np.savetxt(matrix_path, cov) - np.savetxt(gaussian_path, cov_g) - plot_correlation(cov, ndata, plot_path) - - -if __name__ == "__main__": - try: - snakemake # noqa: F821 - injected by snakemake at runtime - except NameError: - if len(sys.argv) != 3: - print("Usage: python cosmocov_process.py ") - sys.exit(1) - stub = sys.argv[2] - main(sys.argv[1], f"{stub}.txt", f"{stub}_g.txt", f"{stub}_plot.pdf") - else: - main( - snakemake.input[0], # noqa: F821 - snakemake.output.matrix, # noqa: F821 - snakemake.output.gaussian, # noqa: F821 - snakemake.output.plot, # noqa: F821 - ) +cov_g, cov_ng, ndata = get_cov(snakemake.input[0]) # noqa: F821 +print(f"Dimension of cov: {ndata}x{ndata}") + +cov = cov_g + cov_ng + +eigenvalues = np.linalg.eigvalsh(cov) +print(f"min+max eigenvalues cov: {eigenvalues.min():e}, {eigenvalues.max():e}") +if eigenvalues.min() <= 0.0: + sys.exit("non-positive eigenvalue encountered! Covariance invalid!") + +np.savetxt(snakemake.output.matrix, cov) # noqa: F821 +np.savetxt(snakemake.output.gaussian, cov_g) # noqa: F821 +plot_correlation(cov, ndata, snakemake.output.plot) # noqa: F821 From cd34415067d5f0dfa124ef76ce7cd9c3c493702b Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 05:15:08 +0200 Subject: [PATCH 32/37] simplify: give the sweep drivers one shared preamble; fold a duplicated README section The four papers/bmodes sweep drivers each repeated the worktree path, the derived script/source dirs, and a full `apptainer exec ... /usr/local/bin/python` invocation (six sites). container_env.sh now owns all of it and exposes `spv_python` / `sweep_versions`; argv is byte-identical. workflow/README.md explained `--config container=` twice, once for a local .sif and once for a branch tag. One subsection now covers both. Co-Authored-By: Claude Fable 5 --- papers/bmodes/scripts/container_env.sh | 35 ++++++++++++-- papers/bmodes/scripts/run_cov_sweep.sh | 9 +--- .../bmodes/scripts/run_pure_eb_ptes_sweep.sh | 12 +---- .../scripts/run_pure_eb_semianalytic.sh | 13 ++---- papers/bmodes/scripts/run_pure_eb_sweep.sh | 9 +--- workflow/README.md | 46 ++++++++----------- 6 files changed, 59 insertions(+), 65 deletions(-) diff --git a/papers/bmodes/scripts/container_env.sh b/papers/bmodes/scripts/container_env.sh index 496f83e2..a63b40f5 100644 --- a/papers/bmodes/scripts/container_env.sh +++ b/papers/bmodes/scripts/container_env.sh @@ -1,9 +1,15 @@ -# Shared container settings for the run_*.sh sweep drivers. Source, don't run: +# Shared environment for the run_*.sh sweep drivers. Source, don't run: # # . "$(dirname "${BASH_SOURCE[0]}")/container_env.sh" # -# Sets CONTAINER (this user's image) and BIND (the mounts to pass to -# `apptainer exec`), resolved exactly as `sp_validation/container.py` does: +# Sets the checkout the drivers run out of, the container to run in, and +# `spv_python`, which is how every driver invokes python inside it. +WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra +SRC=$WT/src +WSCRIPTS=$WT/workflow/scripts +PSCRIPTS=$WT/papers/bmodes/scripts + +# CONTAINER and BIND are resolved exactly as `sp_validation/container.py` does: # the writable sandbox if there is one, else the SIF. _spv_cache=${XDG_CACHE_HOME:-$HOME/.cache}/sp_validation _spv_sandbox=${SPV_SANDBOX:-$_spv_cache/sandbox} @@ -14,3 +20,26 @@ else fi BIND=${SPV_APPTAINER_BINDS:-/home,/scratch,/automnt,/n17data,/n23data1,/n09data} unset _spv_cache _spv_sandbox + +# Every math library pinned to one thread -- pass as SPV_EXEC_EXTRA where the +# parallelism is by process, not by thread. +SINGLE_THREAD_ENV="--env OMP_NUM_THREADS=1 --env OPENBLAS_NUM_THREADS=1 + --env MKL_NUM_THREADS=1 --env NUMBA_NUM_THREADS=1 --env NUMEXPR_NUM_THREADS=1 + --env VECLIB_MAXIMUM_THREADS=1" + +# Run python inside the container against the checkout's src. Extra +# `apptainer exec` flags go in SPV_EXEC_EXTRA (word-split on purpose). +spv_python() { + apptainer exec --bind "$BIND" --env PYTHONPATH="$SRC" ${SPV_EXEC_EXTRA:-} \ + "$CONTAINER" /usr/local/bin/python "$@" +} + +# Echo the version list a sweep runs over: $VERSIONS if the caller set one, +# else whatever sweep_versions.py resolves from $1 (a config path). +sweep_versions() { + if [ -n "${VERSIONS:-}" ]; then + echo "$VERSIONS" + else + spv_python "$PSCRIPTS/sweep_versions.py" --config "$1" + fi +} diff --git a/papers/bmodes/scripts/run_cov_sweep.sh b/papers/bmodes/scripts/run_cov_sweep.sh index ad064408..f86f20dc 100755 --- a/papers/bmodes/scripts/run_cov_sweep.sh +++ b/papers/bmodes/scripts/run_cov_sweep.sh @@ -26,11 +26,7 @@ # [--blind A] [--versions "v1 v2 ..."] set -euo pipefail -WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra . "$(dirname "${BASH_SOURCE[0]}")/container_env.sh" -SRC=$WT/src -WSCRIPTS=$WT/workflow/scripts -PSCRIPTS=$WT/papers/bmodes/scripts CONFIG=""; CATCONFIG=""; PLANCK18=""; MASKBASE=""; OUT=""; BLIND="A"; VERSIONS="" MINSEP=0.5; MAXSEP=300.0; NBINS=1000 @@ -49,10 +45,7 @@ done mkdir -p "$OUT" -if [ -z "$VERSIONS" ]; then - VERSIONS=$(apptainer exec --bind "$BIND" --env PYTHONPATH="$SRC" "$CONTAINER" \ - /usr/local/bin/python "$PSCRIPTS/sweep_versions.py" --config "$CONFIG") -fi +VERSIONS=$(sweep_versions "$CONFIG") for ver in $VERSIONS; do base="covariance_${ver}_${BLIND}_g_minsep=${MINSEP}_maxsep=${MAXSEP}_nbins=${NBINS}_masked" diff --git a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh index 09014397..0ea3a862 100644 --- a/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_ptes_sweep.sh @@ -19,11 +19,7 @@ # --out [--blind A] [--versions "v1 v2 ..."] set -euo pipefail -WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra . "$(dirname "${BASH_SOURCE[0]}")/container_env.sh" -SRC=$WT/src -WSCRIPTS=$WT/workflow/scripts -PSCRIPTS=$WT/papers/bmodes/scripts CONFIG=""; CATCONFIG=""; PUREEBSWEEP=""; COVSWEEP=""; OUT=""; BLIND="A"; VERSIONS="" while [ $# -gt 0 ]; do @@ -41,10 +37,7 @@ done mkdir -p "$OUT" -if [ -z "$VERSIONS" ]; then - VERSIONS=$(apptainer exec --bind "$BIND" --env PYTHONPATH="$SRC" "$CONTAINER" \ - /usr/local/bin/python "$PSCRIPTS/sweep_versions.py" --config "$CONFIG") -fi +VERSIONS=$(sweep_versions "$CONFIG") for ver in $VERSIONS; do pureeb="$PUREEBSWEEP/${ver}_${BLIND}_pure_eb_semianalytic.npz" @@ -54,8 +47,7 @@ for ver in $VERSIONS; do [ -f "$f" ] || { echo "MISSING upstream input for $ver: $f" >&2; exit 1; } done echo "[pure_eb_ptes_sweep] $ver" - apptainer exec --bind "$BIND" --env PYTHONPATH="$SRC" "$CONTAINER" \ - /usr/local/bin/python "$PSCRIPTS/calculate_pure_eb_ptes.py" \ + spv_python "$PSCRIPTS/calculate_pure_eb_ptes.py" \ --version "$ver" --blind "$BLIND" \ --pure-eb-data "$pureeb" --cov-integration "$covint" \ --npatch 1 --n-samples 2000 --out "$OUT" diff --git a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh index 60b85c4e..472cd81d 100644 --- a/papers/bmodes/scripts/run_pure_eb_semianalytic.sh +++ b/papers/bmodes/scripts/run_pure_eb_semianalytic.sh @@ -14,10 +14,7 @@ # --out [--n-chunks 20] [--n-samples 2000] [--nproc 16] set -euo pipefail -WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra -SRC=$WT/src . "$(dirname "${BASH_SOURCE[0]}")/container_env.sh" -SCRIPTS=$WT/papers/bmodes/scripts VERSION=""; BLIND="A"; CATCONFIG=""; XIREP=""; XIINT=""; COVINT=""; OUT="" NCHUNKS=20; NSAMPLES=2000; NPROC="${SLURM_CPUS_PER_TASK:-16}" @@ -52,11 +49,8 @@ mkdir -p "$OUT/chunks" echo "[pure_eb] $NCHUNKS chunks, $NSAMPLES samples, nproc=$NPROC, version=$VERSION blind=$BLIND" for i in $(seq 0 $((NCHUNKS-1))); do ( - apptainer exec --bind "$BIND" --env PYTHONPATH="$SRC" \ - --env OMP_NUM_THREADS=1 --env OPENBLAS_NUM_THREADS=1 --env MKL_NUM_THREADS=1 \ - --env NUMBA_NUM_THREADS=1 --env NUMEXPR_NUM_THREADS=1 --env VECLIB_MAXIMUM_THREADS=1 \ - "$CONTAINER" \ - /usr/local/bin/python "$SCRIPTS/precompute_pure_eb_chunk.py" \ + SPV_EXEC_EXTRA=$SINGLE_THREAD_ENV + spv_python "$PSCRIPTS/precompute_pure_eb_chunk.py" \ --chunk-id "$i" --n-chunks "$NCHUNKS" --n-samples "$NSAMPLES" \ --version "$VERSION" --blind "$BLIND" --cat-config "$CATCONFIG" \ --xi-reporting "$XIREP" --xi-integration "$XIINT" --cov-integration "$COVINT" \ @@ -76,8 +70,7 @@ done [ "$missing" -eq 0 ] || { echo "[pure_eb] chunk failures — aborting gather" >&2; exit 1; } echo "[pure_eb] all $NCHUNKS chunks done; gathering" -apptainer exec --bind "$BIND" --env PYTHONPATH="$SRC" "$CONTAINER" \ - /usr/local/bin/python "$SCRIPTS/gather_pure_eb_chunks.py" \ +spv_python "$PSCRIPTS/gather_pure_eb_chunks.py" \ --version "$VERSION" --blind "$BLIND" \ --xi-reporting "$XIREP" --xi-integration "$XIINT" \ --chunks-dir "$OUT/chunks" \ diff --git a/papers/bmodes/scripts/run_pure_eb_sweep.sh b/papers/bmodes/scripts/run_pure_eb_sweep.sh index 742d2cc8..8195f150 100755 --- a/papers/bmodes/scripts/run_pure_eb_sweep.sh +++ b/papers/bmodes/scripts/run_pure_eb_sweep.sh @@ -19,11 +19,7 @@ # --out [--blind A] [--versions "v1 v2 ..."] set -euo pipefail -WT=/n17data/cdaley/unions/code/sp_validation.worktrees/repro-paper-ii-astra . "$(dirname "${BASH_SOURCE[0]}")/container_env.sh" -SRC=$WT/src -WSCRIPTS=$WT/workflow/scripts -PSCRIPTS=$WT/papers/bmodes/scripts CONFIG=""; CATCONFIG=""; XISWEEP=""; COVSWEEP=""; OUT=""; BLIND="A"; VERSIONS="" while [ $# -gt 0 ]; do @@ -41,10 +37,7 @@ done mkdir -p "$OUT" -if [ -z "$VERSIONS" ]; then - VERSIONS=$(apptainer exec --bind "$BIND" --env PYTHONPATH="$SRC" "$CONTAINER" \ - /usr/local/bin/python "$PSCRIPTS/sweep_versions.py" --config "$CONFIG") -fi +VERSIONS=$(sweep_versions "$CONFIG") for ver in $VERSIONS; do xirep="$XISWEEP/${ver}_xi_minsep=1.0_maxsep=250.0_nbins=20_npatch=1.txt" diff --git a/workflow/README.md b/workflow/README.md index 81f15277..aed48d86 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -106,26 +106,10 @@ snakemake --profile workflow/profiles/candide --config checkout_pythonpath=false Either way the checkout has to sit under one of the profile's bind mounts to be visible inside the job. -### Testing a branch's own image - -CI builds and pushes an image for **every** branch, tagged by the sanitized -branch name (`/` → `-`; see `.github/workflows/deploy-image.yml`). To run a -branch's image rather than `:develop`: - -```bash -snakemake --profile workflow/profiles/candide \ - --config container=docker://ghcr.io/cosmostat/sp_validation:my-branch -``` - -`container` is a config key read by every entry Snakefile -(`common.resolve_container`), so it overrides the default everywhere at once. -Snakemake autopulls the tag, which costs ~15 minutes — do it from a compute node. -To keep that image around instead, `spv-container pull --tag ` puts it at -your canonical path, where it becomes the default. Most of the time you need -none of this: -the checkout-PYTHONPATH default above already runs your branch's Python against -the `:develop` dependency stack. Reach for the branch image when the *stack* -changed (a new dependency, a lockfile bump), not when only `src/` did. +Most of the time this default is all you need. Reach for a different *image* +only when the dependency stack changed — a new package, a lockfile bump — not +when only `src/` did; "Running an image other than your own" below has the +override. ### Never write `/automnt/nXXdataN` in a path @@ -210,7 +194,7 @@ accepts all three forms, a sandbox directory included. The tag itself is written down once, as `CONTAINER_URI` in `sp_validation/container.py`, which `workflow/common.py` re-exports; the image-sims `sif:` config key defaults to `null` and resolves the -same way. Override any of it with `--config container=`, or point +same way. Override any of it with `--config container=...` (below), or point somewhere else entirely with `SPV_CONTAINER`. At launch the workflow prints one advisory line if your image was built from a @@ -261,16 +245,26 @@ checkout's `HEAD`, naming which layer (sandbox or SIF) it read. The image-sims w `m_bias_config.yaml` as `ghcr_revision`, so a result file says which image produced the number. -**Running an image of your own** instead of the canonical one: +#### Running an image other than your own + +`container` is a config key read by every entry Snakefile +(`common.resolve_container`), so one flag overrides the default everywhere at +once — a local file, or any CI tag: ```bash snakemake --profile workflow/profiles/candide --config container=/path/to/my.sif +snakemake --profile workflow/profiles/candide \ + --config container=docker://ghcr.io/cosmostat/sp_validation:my-branch ``` -For the image-sims workflow, set `image_sims: {sif: /path/to/my.sif}` in your run -config. Your image has to sit under one of the profile's bind mounts to be -visible. To run a *branch's* CI image rather than a local file, see "Testing a -branch's own image" above. +CI tags an image for **every** branch, by sanitized branch name (`/` → `-`), so +the second form is how you test a branch's own stack. Snakemake autopulls a tag +per run directory (~15 minutes — from a compute node); `spv-container pull --tag +` instead puts it at your canonical path, where it becomes your default. + +For the image-sims workflow, set `image_sims: {sif: ...}` in your run config. +Either way the image has to sit under one of the profile's bind mounts to be +visible. One trap to know: the `script:` directive bind-mounts the host orchestrator's `snakemake` into the job and *appends* it to `sys.path`, so a `snakemake` From ba2fd7ce30c502d0d27a585517f4740e60101e55 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 05:28:26 +0200 Subject: [PATCH 33/37] =?UTF-8?q?image:=20actually=20build=20CSL=20?= =?UTF-8?q?=E2=80=94=20cosmosis-configure=20exits=200=20under=20set=20-u?= =?UTF-8?q?=20without=20running=20make?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc --- Dockerfile | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/Dockerfile b/Dockerfile index b9c72451..09db64ff 100644 --- a/Dockerfile +++ b/Dockerfile @@ -70,10 +70,13 @@ ARG CSL_REPO=https://github.com/sachaguer/cosmosis-standard-library.git ARG CSL_REF=b26fa7ff666ab4d607b2e32e36f799a53bfb1d9c ENV CSL_DIR=/opt/cosmosis-standard-library -# `source cosmosis-configure` is CSL's documented way to build against a -# pip-installed cosmosis: the script ships with the cosmosis package and exports -# COSMOSIS_SRC_DIR, which every CSL Makefile includes its compiler config from. -# bash, not sh, because that script is bash. +# `python -m cosmosis.configure` emits the exports (COSMOSIS_SRC_DIR et al.) +# every CSL Makefile includes its compiler config from. Evaluated directly +# rather than through the `cosmosis-configure` wrapper: that wrapper's +# am-I-sourced probe reads unset zsh/ksh variables, which `set -u` turns into +# an error, and its `exit` then ends the sourcing shell with status 0 — make +# never runs and the layer still "succeeds". The trailing `test -f` keeps any +# such silent no-op loud. # # `make -C shear` rather than a bare `make`: the top-level target also descends # into likelihood/, which builds the Planck, WMAP and ACT likelihoods -- large, @@ -88,8 +91,10 @@ RUN bash -c 'set -euo pipefail; \ git clone --filter=blob:none "$CSL_REPO" "$CSL_DIR"; \ cd "$CSL_DIR"; \ git checkout --detach "$CSL_REF"; \ - source cosmosis-configure; \ - make -C shear' + cmds=$(python -m cosmosis.configure); \ + eval "$cmds"; \ + make -C shear; \ + test -f shear/cl_to_xi_nicaea/nicaea_interface.so' # Install sp_validation itself (editable) into the same venv; deps are already # satisfied by the sync above. From 371826c21c9392a2406369175acd3719460c4a3e Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 05:32:24 +0200 Subject: [PATCH 34/37] =?UTF-8?q?image:=20drop=20set=20-u=20in=20the=20CSL?= =?UTF-8?q?=20layer=20=E2=80=94=20the=20configure=20exports=20append=20to?= =?UTF-8?q?=20unset=20paths?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The test -f artifact guard is what keeps a no-op build loud; -u had become the thing breaking the build instead. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc --- Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 09db64ff..a25dc42d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -86,7 +86,7 @@ ENV CSL_DIR=/opt/cosmosis-standard-library # xi_sys, 2pt_like -- no Makefile in those trees at all) or lives under shear/: # `limber`, which project_2d.py links, and `cl_to_xi_nicaea`, whose # nicaea_interface.so the 2pt_shear stage loads. -RUN bash -c 'set -euo pipefail; \ +RUN bash -c 'set -eo pipefail; \ export PATH=/app/.venv/bin:$PATH; \ git clone --filter=blob:none "$CSL_REPO" "$CSL_DIR"; \ cd "$CSL_DIR"; \ From 10bcf28cee64be9fb9323ff8f0548a3c5e20dd4e Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 05:36:31 +0200 Subject: [PATCH 35/37] image: point limber's Makefile at Debian's GSL (GSL_INC/GSL_LIB) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc --- Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/Dockerfile b/Dockerfile index a25dc42d..d24e3040 100644 --- a/Dockerfile +++ b/Dockerfile @@ -93,6 +93,7 @@ RUN bash -c 'set -eo pipefail; \ git checkout --detach "$CSL_REF"; \ cmds=$(python -m cosmosis.configure); \ eval "$cmds"; \ + export GSL_INC=/usr/include GSL_LIB=/usr/lib/x86_64-linux-gnu; \ make -C shear; \ test -f shear/cl_to_xi_nicaea/nicaea_interface.so' From e3ca34fc158d6a22c1a32bd7ab97e298e7ebda67 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 11:10:21 +0200 Subject: [PATCH 36/37] rebase onto develop: shed tomography-branch remnants The container/workflow work is orthogonal to the tomography branch it was accidentally based on. Restore develop's pure_eb docstring, test_cosmo_val call shape, and glass_mock xfail; keep develop's glass==2025.1 pinned set (cosmology 2022.10.9 is load-bearing there, not vestigial) and relock. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0143zcEsfSWr13AfSroMteEC --- pyproject.toml | 4 +- src/sp_validation/cosmo_val/pure_eb.py | 5 +- src/sp_validation/tests/test_cosmo_val.py | 4 +- src/sp_validation/tests/test_glass_mock.py | 13 +++++ uv.lock | 55 +++++----------------- 5 files changed, 30 insertions(+), 51 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index c43499f6..f325121f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -141,8 +141,8 @@ docs = [ "sphinxawesome-theme>=5.3,!=6.0.3" ] # GLASS mock generation (sp_validation.glass_mock). Kept optional: the core -# library and its import guard resolve without GLASS, and the production -# container does not yet ship it. Pinned set (verified 2026-06-13, fiber +# library and its import guard resolve without GLASS (the production container +# ships it via the ``glass`` extra). Pinned set (verified 2026-06-13, fiber # glass-cosmology-api-pin): glass 2025.1 is the unique version with the flat API # the map path uses AND the legacy ``cosmo.dc``/``xm``/``ef`` interface that # ``cosmology`` 2022.10.9 (its newest release) provides — newer glass calls diff --git a/src/sp_validation/cosmo_val/pure_eb.py b/src/sp_validation/cosmo_val/pure_eb.py index 9c075031..71f5d76a 100644 --- a/src/sp_validation/cosmo_val/pure_eb.py +++ b/src/sp_validation/cosmo_val/pure_eb.py @@ -79,10 +79,7 @@ def calculate_pure_eb( Returns ------- dict - One entry per tomographic bin pair, keyed - ``"tomo_bin_{b1}_tomo_bin_{b2}"`` (non-tomographic runs have the - single key ``"tomo_bin_all_tomo_bin_all"``). Each value is a - dictionary containing the following keys: + A dictionary containing the following keys: - "xip_E": Pure E-mode correlation function for xi+. - "xim_E": Pure E-mode correlation function for xi-. diff --git a/src/sp_validation/tests/test_cosmo_val.py b/src/sp_validation/tests/test_cosmo_val.py index 7d49e41a..f50992d4 100644 --- a/src/sp_validation/tests/test_cosmo_val.py +++ b/src/sp_validation/tests/test_cosmo_val.py @@ -622,15 +622,13 @@ def test_calculate_pure_eb_runs_on_synthetic_catalog(self, tmp_path): # mirrors the bmodes workflow's broad-and-fine integration grid; every # reporting bin is well-defined (no edge NaNs). nbins_int~80 here would # NaN the edge bins -- confirmed -- which is the finiteness teeth. - # calculate_pure_eb returns one results dict per tomographic bin pair; - # the non-tomographic run has the single "all x all" key. results = cv.calculate_pure_eb( version, npatch=npatch, min_sep_int=1.0, max_sep_int=300.0, nbins_int=600, - )["tomo_bin_all_tomo_bin_all"] + ) # Reference mode vectors from the seeded synthetic catalog + Schneider # transform. Deterministic (full-sample treecorr, no RNG); regenerate by diff --git a/src/sp_validation/tests/test_glass_mock.py b/src/sp_validation/tests/test_glass_mock.py index 8af73092..a056318b 100644 --- a/src/sp_validation/tests/test_glass_mock.py +++ b/src/sp_validation/tests/test_glass_mock.py @@ -131,6 +131,19 @@ def test_config_change_breaks_reference(): @pytest.mark.skipif(not HAVE_GLASS, reason="GLASS not installed in this image") +@pytest.mark.xfail( + reason=( + "glass_mock map path is incompatible with the installed glass/cosmology " + "API: cosmology.Cosmology.from_camb returns a CambCosmology lacking " + "comoving_distance, which glass.distance_grid / MultiPlaneConvergence " + "require. The map path was never exercised before GLASS was added to the " + "image. Fix = pin a compatible glass+cosmology pair (or adapt the API " + "calls) and verify in the fresh image; then drop this xfail. " + "See fiber shapepipe/sp_validation glass-cosmology-api-pin." + ), + strict=False, + raises=AttributeError, +) def test_matter_maps_are_seed_deterministic(): """Same config + seed → bit-identical matter/lensing maps. diff --git a/uv.lock b/uv.lock index 1b5aca1b..40669e2a 100644 --- a/uv.lock +++ b/uv.lock @@ -112,18 +112,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/86/16/1a8fd2b19544b84575cf84ef7aa3ad4c173b756d5f087c91f85d1b295777/array_api_compat-1.15.0-py3-none-any.whl", hash = "sha256:7b1b9c53269061403fd5f45a8de349f16e7887653328bfa0c5f2d45299ff0a8e", size = 79113, upload-time = "2026-06-07T20:53:23.621Z" }, ] -[[package]] -name = "array-api-extra" -version = "0.11.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "array-api-compat", marker = "sys_platform == 'linux'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/94/40/e2e74294b894b115b05c052364fde283e3684e309c58c8f3e0463270051b/array_api_extra-0.11.1.tar.gz", hash = "sha256:360bc6faf858b1ef2ca0fb3cc86dbac0ed566fa1f78ae515cd234830c8b119f8", size = 102150, upload-time = "2026-08-12T11:29:24.981Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/4f/ee/c0a6a4bb3c5d874c68f57c2471af931f5f4cf2f84fdcc86b8a7467eb3e66/array_api_extra-0.11.1-py3-none-any.whl", hash = "sha256:2da3eed8842ed14cdded9a2a82f11dcafae2aa2c0c1622a3b77e769f72331c64", size = 98046, upload-time = "2026-08-12T11:29:23.571Z" }, -] - [[package]] name = "arrow" version = "1.4.0" @@ -562,24 +550,15 @@ dependencies = [ ] [[package]] -name = "cosmology-api" -version = "0.3.2" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/f0/f4/801cab91dab5b2e8102ed54cc4e2bd257f319c9047db182fcaf53706df6a/cosmology_api-0.3.2.tar.gz", hash = "sha256:7ccdfdf20f91dfc2282aee059adf2530bea6e194a0f8d01487e15a7497cf4694", size = 17338, upload-time = "2025-05-15T13:29:52.355Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/bf/8c/6cc73fafff9f4d75e77a9c3cf5b9c34aa42841cfc7da5c89737daa2c3bd7/cosmology_api-0.3.2-py3-none-any.whl", hash = "sha256:9391ef0b2616bbf4217fefdb94a72370766d59a2f0558b5ade224a92ea093a61", size = 18685, upload-time = "2025-05-15T13:29:50.833Z" }, -] - -[[package]] -name = "cosmology-compat-camb" -version = "0.2.0" +name = "cosmology" +version = "2022.10.9" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "numpy", marker = "sys_platform == 'linux'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/08/2e/a637aae1677d02337be371c9a7b7e9d5051199bb58a2386781230adcee45/cosmology_compat_camb-0.2.0.tar.gz", hash = "sha256:e36fda04a78e16fc1a5c4aa22c30ec1bf64bbe0b092aa1b894ca746ce5e5551b", size = 4562, upload-time = "2025-05-08T10:31:43.226Z" } +sdist = { url = "https://files.pythonhosted.org/packages/c9/38/04099bb2b6a626bd69a3f117afc60f2c47c08899e0730e0e48c898bf4745/cosmology-2022.10.9.tar.gz", hash = "sha256:0c2857c9bf1fdd09f1f11ab5765df0389a4101f3a900fa11251c3f37696f02d4", size = 8488, upload-time = "2022-10-10T10:18:16.229Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/2e/42/1a657f9b3540f355f34af735d8643aae1480620689a72423fe69aeeb78e5/cosmology_compat_camb-0.2.0-py3-none-any.whl", hash = "sha256:eb6e74290bb6a6a60a47d1338fc233ba1697058ed7f01a01de1180dbee1bd75d", size = 3903, upload-time = "2025-05-08T10:31:41.997Z" }, + { url = "https://files.pythonhosted.org/packages/2e/2c/c80db37593ad593e3f6c60705fbdf690b33f22c89edd5507d2731346cd87/cosmology-2022.10.9-py3-none-any.whl", hash = "sha256:3903658c2474177a1a1c75771ae14458d93200c516f8fc6c3f4d776f3287b288", size = 9341, upload-time = "2022-10-10T10:18:14.928Z" }, ] [[package]] @@ -1091,28 +1070,18 @@ wheels = [ [[package]] name = "glass" -version = "2026.2" +version = "2025.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "array-api-compat", marker = "sys_platform == 'linux'" }, - { name = "array-api-extra", marker = "sys_platform == 'linux'" }, + { name = "cosmology", marker = "sys_platform == 'linux'" }, { name = "healpix", marker = "sys_platform == 'linux'" }, { name = "healpy", marker = "sys_platform == 'linux'" }, + { name = "numpy", marker = "sys_platform == 'linux'" }, { name = "transformcl", marker = "sys_platform == 'linux'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/c1/c3/3b7d18cca8d0fa41dbb646ce16935804b380dcf766e7385799f44829b419/glass-2026.2.tar.gz", hash = "sha256:cf5ef6cb76b4738f8dc05ddfc18c359c558bb36e9cd09ca0115fa3ca3deb491f", size = 66236, upload-time = "2026-06-04T15:18:39.155Z" } +sdist = { url = "https://files.pythonhosted.org/packages/a5/d6/26cd58e75834671259f26b9287a584d726e6023da35ddf7ca3b7f2c393fb/glass-2025.1.tar.gz", hash = "sha256:7b1aa2394e16010f7f1b4243f49e7e12d7a4dd28fcbf3e3f7cf25ce4905a8615", size = 48533, upload-time = "2025-02-21T18:43:48.046Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/7c/d1/55ea5e63e81db82806e92ba9f9c2b7277c0c2afe57bf8c1d3763e18005b2/glass-2026.2-py3-none-any.whl", hash = "sha256:0af0b5cd8708040925699c307076fb1c79bbb7e6cba4522764bcc21dd3bdadb8", size = 63746, upload-time = "2026-06-04T15:18:37.895Z" }, -] - -[package.optional-dependencies] -examples = [ - { name = "camb", marker = "sys_platform == 'linux'" }, - { name = "cosmology-api", marker = "sys_platform == 'linux'" }, - { name = "cosmology-compat-camb", marker = "sys_platform == 'linux'" }, - { name = "glass-ext-camb", marker = "sys_platform == 'linux'" }, - { name = "jupyter", marker = "sys_platform == 'linux'" }, - { name = "matplotlib", marker = "sys_platform == 'linux'" }, + { url = "https://files.pythonhosted.org/packages/e4/e7/3e89bc5147a3e84245c783098afffbbacbcdf5678ba1aacc2c9c035434da/glass-2025.1-py3-none-any.whl", hash = "sha256:d7919a7d19e05ab8da4e52dbfbcfb5bd1ed2b72e0cc3afde39ec94ced22883c5", size = 47185, upload-time = "2025-02-21T18:43:46.649Z" }, ] [[package]] @@ -3890,8 +3859,9 @@ docs = [ { name = "sphinxcontrib-bibtex", marker = "sys_platform == 'linux'" }, ] glass = [ + { name = "cosmology", marker = "sys_platform == 'linux'" }, { name = "fitsio", marker = "sys_platform == 'linux'" }, - { name = "glass", extra = ["examples"], marker = "sys_platform == 'linux'" }, + { name = "glass", marker = "sys_platform == 'linux'" }, { name = "glass-ext-camb", marker = "sys_platform == 'linux'" }, ] test = [ @@ -3913,13 +3883,14 @@ requires-dist = [ { name = "clmm" }, { name = "colorama" }, { name = "cosmo-numba", git = "https://github.com/aguinot/cosmo-numba.git?rev=main" }, + { name = "cosmology", marker = "extra == 'glass'", specifier = "==2022.10.9" }, { name = "cosmosis", marker = "extra == 'workflow'", specifier = ">=3.25" }, { name = "cryptography" }, { name = "cs-util", git = "https://github.com/CosmoStat/cs_util.git?rev=develop" }, { name = "emcee" }, { name = "fitsio", marker = "extra == 'glass'" }, { name = "getdist", git = "https://github.com/benabed/getdist.git?rev=113cd22a9a0d013b6f72fe734be81f260f3d3be5" }, - { name = "glass", extras = ["examples"], marker = "extra == 'glass'", specifier = "==2026.2" }, + { name = "glass", marker = "extra == 'glass'", specifier = "==2025.1" }, { name = "glass-ext-camb", marker = "extra == 'glass'", specifier = "==2023.6" }, { name = "h5py" }, { name = "healpy" }, From 06ec1f396b223977bef51a62d53096922dfa2f5d Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Thu, 27 Aug 2026 11:15:28 +0200 Subject: [PATCH 37/37] docs: make spv-container the install story README leads with the four-line install (clone, symlink onto PATH, pull, exec-check); container.py gets a shebang + exec bit so the symlink is a real CLI with no packaging. installation.rst carries the depth (subcommands, per-user model, sandbox, raw apptainer/docker); CONTRIBUTING and workflow/README point at the same symlink step. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0143zcEsfSWr13AfSroMteEC --- CONTRIBUTING.md | 7 +++-- README.md | 46 +++++++++++++---------------- docs/source/installation.rst | 54 +++++++++++++++++++++++++++------- src/sp_validation/container.py | 1 + workflow/README.md | 5 ++-- 5 files changed, 72 insertions(+), 41 deletions(-) mode change 100644 => 100755 src/sp_validation/container.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 09cfe2c1..e385a386 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -19,9 +19,10 @@ name (see `:develop` tracks the integration branch and your branch has an image of its own. Nothing is built by hand. -You keep your own copy of the image. `spv-container` — a console script installed -with the package, and runnable straight from a checkout as `python3 -src/sp_validation/container.py` — pulls it to +You keep your own copy of the image. `spv-container` — stdlib-only, so it runs +straight from a checkout: symlink it onto your PATH (`ln -s +"$PWD/src/sp_validation/container.py" ~/.local/bin/spv-container`, the README's +install step) or call it as `python3 src/sp_validation/container.py` — pulls it to `~/.cache/sp_validation/sp_validation.sif` and runs things inside it: ```bash diff --git a/README.md b/README.md index d1295c4c..455ddd91 100644 --- a/README.md +++ b/README.md @@ -62,39 +62,35 @@ directive imports the shared rules under each run's own config and an output `prefix`, so runs namespace under `results//` without clobbering one another. -## Container Installation (Recommended) +## Installation -The easiest way to install sp_validation is via a container. Docker images are automatically built and pushed to the [GitHub Container Registry (GHCR)](https://github.com/CosmoStat/sp_validation/pkgs/container/sp_validation) on every push to `develop`. This image can be installed and run on most systems (including clusters) with just a few lines of code. - -We recommend running the image with **Apptainer** (formerly Singularity) which is installed on most HPC clusters. To simply run the image, use the following command: +`sp_validation` runs from a pre-built container: CI builds an image carrying +the full scientific stack on every push and publishes it to the +[GitHub Container Registry](https://github.com/CosmoStat/sp_validation/pkgs/container/sp_validation). +The bundled `spv-container` CLI installs and manages your personal copy of it: ```bash -# pull the image to a single .sif file -apptainer pull sp_validation.sif docker://ghcr.io/cosmostat/sp_validation:develop - -# open a shell in the container -apptainer shell sp_validation.sif -# and confirm that the installation was successful -python -c "import sp_validation" -``` +git clone https://github.com/CosmoStat/sp_validation.git +cd sp_validation +ln -s "$PWD/src/sp_validation/container.py" ~/.local/bin/spv-container -CI tags an image by branch, so `:develop` tracks the integration branch and any -branch can be pulled by its (sanitized) name. - -You can also run the image with **Docker**: - -```bash -docker run --rm -it ghcr.io/cosmostat/sp_validation:develop python -c "import sp_validation" +spv-container pull # fetch the image (~1.5 GB) +spv-container exec python -c "import sp_validation" # confirm it works ``` -We do not currently build images for Apple Silicon/arm64; however the amd64 images should work on these systems, albeit with reduced performance. +That is the whole install. `pull` puts the image at its canonical per-user +path (`~/.cache/sp_validation/`), and everything else finds it there — +`spv-container exec` for one-off commands (`spv-container exec bash` for an +interactive shell) and the Snakemake workflow for cluster jobs. +`spv-container status` says what you have and how current it is; +`spv-container sandbox` gives you a writable copy for mid-analysis +`pip install`s. On a cluster, run the pull from a compute node. -This shell is for interactive development and debugging. To run the analysis -workflow (`workflow/`), do not enter this shell — see +To run the analysis workflow (`workflow/`), see [`workflow/README.md`](workflow/README.md): Snakemake runs on the host, and -the profile puts each job in the container itself. The workflow expects the -image at one canonical per-user path, which the bundled `spv-container` CLI -manages (`spv-container pull` / `status` / `exec`). +the profile puts each job in the container itself. For Docker, development +installs, and more depth, see the +[installation docs](https://cosmostat.github.io/sp_validation/installation.html). diff --git a/docs/source/installation.rst b/docs/source/installation.rst index 90634fb6..56de599f 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -2,27 +2,59 @@ Installation ============ ``sp_validation`` is **not** distributed on PyPI. -Install it from a pre-built container, or check out the source with ``uv`` when you need to edit it. +It runs from a pre-built container, managed by the bundled ``spv-container`` CLI; check out the source with ``uv`` only when you need to edit the package itself. -Container (recommended) ------------------------ +Container via ``spv-container`` (recommended) +--------------------------------------------- -Every push to ``develop`` builds an image carrying the full scientific stack and pushes it to the `GitHub Container Registry (GHCR) -`_. +Every push builds an image carrying the full scientific stack and pushes it to the `GitHub Container Registry (GHCR) +`_, tagged by branch — ``:develop`` tracks the integration branch. The image runs on most systems, including HPC clusters, with no further setup. +``spv-container`` installs your personal copy of it and manages it from then on: -`Apptainer `_ (formerly Singularity) is installed on most clusters and is the path we recommend: +.. code-block:: bash + + git clone https://github.com/CosmoStat/sp_validation.git + cd sp_validation + ln -s "$PWD/src/sp_validation/container.py" ~/.local/bin/spv-container + + spv-container pull # fetch the image (~1.5 GB) + spv-container exec python -c "import sp_validation" # confirm it works + +The symlink works because ``container.py`` is deliberately stdlib-only: it runs on the *host*, where the science stack is not installed. +(Inside the container the same CLI is on ``PATH`` as a console script.) +``pull`` requires `Apptainer `_ (formerly Singularity), which is installed on most clusters, and writes the image to one canonical per-user path, ``~/.cache/sp_validation/sp_validation.sif``. +Each user owns their copy: you refresh it when you want to, and nobody else's refresh moves the ground under your running jobs. +On a cluster, run the pull from a compute node — it moves ~1.5 GB. + +The subcommands: .. code-block:: bash - # Pull the image to a single .sif file. - apptainer pull sp_validation.sif docker://ghcr.io/cosmostat/sp_validation:develop + spv-container pull # fetch the published image to the canonical path + spv-container status # what is here, which commit built it, how current + spv-container exec # run a command inside it (exec bash for a shell) + spv-container sandbox # unpack into a writable dir, for pip installs + spv-container exec --writable # ... with writes that persist - # Open a shell in the container, then confirm the install works. +``status`` compares the image's build commit against your checkout's ``HEAD``, so you always know whether a ``pull`` would refresh anything. +The **sandbox** is the escape hatch for exploratory work that needs a package the image does not carry yet: once built, it takes precedence over the SIF everywhere — Snakemake workflow jobs included — until you reset with ``spv-container pull`` + ``spv-container sandbox --force``. + +Everything resolves the image in one order — sandbox if it exists, else your SIF, else the registry tag — and that includes the analysis workflow. +How the workflow uses the image (Snakemake runs on the host; the profile puts each job in the container) is covered in ``workflow/README.md``. + +Other ways to run the image +--------------------------- + +The published image is a normal OCI image; ``spv-container`` is a convenience, not a gatekeeper. +Run it directly with Apptainer: + +.. code-block:: bash + + apptainer pull sp_validation.sif docker://ghcr.io/cosmostat/sp_validation:develop apptainer shell sp_validation.sif - python -c "import sp_validation" -The image also runs under Docker: +or with Docker: .. code-block:: bash diff --git a/src/sp_validation/container.py b/src/sp_validation/container.py old mode 100644 new mode 100755 index 3589930f..7aa5f2ed --- a/src/sp_validation/container.py +++ b/src/sp_validation/container.py @@ -1,3 +1,4 @@ +#!/usr/bin/env python3 """Manage this user's local copy of the sp_validation container image. Everyone runs their own image: the canonical paths are under your own cache, you diff --git a/workflow/README.md b/workflow/README.md index aed48d86..4171074e 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -168,10 +168,11 @@ spv-container exec # run something inside it, candide binds already app It ships as a console script with the package, and — being stdlib-only, because it has to run on the *host* — also works straight from a checkout with nothing -installed: +installed: run `python3 src/sp_validation/container.py`, or put it on your PATH +once (the README's install step): ```bash -python3 src/sp_validation/container.py pull +ln -s "$PWD/src/sp_validation/container.py" ~/.local/bin/spv-container ``` **Do the pull from a compute node**, not the login node: it moves ~1.5 GB and