Improvements to snakemake containerization

.gitignore

@@ -194,4 +194,7 @@ papers/catalog/plots/*.pdf
 papers/cosmo_val/logs/
 
 # Ignore scratch notebooks
-scratch/*/*.ipynb
+scratch/*/*.ipynb
+
+# Snakemake run state
+.snakemake/

CLAUDE.md

@@ -78,10 +78,26 @@ Main configuration in `scripts/calibration/params.py` with parameters:
 - pyccl for cosmological calculations
 
 ## Container Usage
-Recommended installation via Apptainer/Docker:
+Nothing is hand-built. CI publishes `ghcr.io/cosmostat/sp_validation:<branch>` on
+every push, and each person keeps their own copy at
+`~/.cache/sp_validation/sp_validation.sif`, managed by the `spv-container` CLI:
+
 ```bash
-apptainer build --sandbox sp_validation docker://ghcr.io/cosmostat/sp_validation:develop
+spv-container pull            # fetch :develop there (do it from a compute node)
+spv-container status          # which layer is live, and how current it is
+spv-container exec <command>  # one-off run inside it
 ```
 
+Need a package the image lacks mid-analysis? `spv-container sandbox`, then
+`spv-container exec --writable pip install <pkg>`; the sandbox then takes
+precedence over the SIF everywhere, workflow jobs included.
+
+Every rule runs inside that image, wrapped by Snakemake itself (`--profile
+workflow/profiles/candide` on the cluster, `workflow/profiles/default -j N`
+elsewhere). The `sp_validation` a rule imports comes from the *launched
+checkout*, not the image.
+
+`workflow/README.md` is the full story — profiles, image resolution, refresh.
+
 ## Notebook Configuration
 - The CosmologyValidation class must be initialized in cosmo_val

CONTRIBUTING.md

@@ -13,15 +13,49 @@ inside the project container, which ships the full stack pre-built.
 
 ### Container (recommended)
 
+CI builds and pushes an image on **every** push, tagged by the sanitized branch
+name (see
+[`.github/workflows/deploy-image.yml`](.github/workflows/deploy-image.yml)), so
+`:develop` tracks the integration branch and your branch has an image of its
+own. Nothing is built by hand.
+
+You keep your own copy of the image. `spv-container` — stdlib-only, so it runs
+straight from a checkout: symlink it onto your PATH (`ln -s
+"$PWD/src/sp_validation/container.py" ~/.local/bin/spv-container`, the README's
+install step) or call it as `python3 src/sp_validation/container.py` — pulls it to
+`~/.cache/sp_validation/sp_validation.sif` and runs things inside it:
+
 ```bash
-# build a writeable sandbox from the published image
-apptainer build --sandbox sp_validation docker://ghcr.io/cosmostat/sp_validation:develop
-apptainer shell --writable sp_validation
+spv-container pull       # fetch :develop; ~1.5 GB, so do it from a compute node
+spv-container status     # which layer is live, and how current it is
+spv-container exec bash  # an interactive shell inside it
 ```
 
-The image is rebuilt and pushed on every push to `develop` (see
-[`.github/workflows/deploy-image.yml`](.github/workflows/deploy-image.yml)), so
-`:develop` always tracks the latest integration branch.
+That image is read-only. When you need a package it does not carry yet, unpack a
+writable sandbox once with `spv-container sandbox` and install into it with
+`spv-container exec --writable pip install <pkg>`; the sandbox then takes
+precedence everywhere, workflow jobs included. Treat it as an exploration tool —
+the real fix is adding the dependency to `pyproject.toml` — and reset it with
+`spv-container pull && spv-container sandbox --force`.
+
+Analysis runs through Snakemake, which wraps every job in `apptainer exec`
+against that same image for you — see
+[`workflow/README.md`](workflow/README.md) for the profiles and the details.
+
+Two things worth knowing while developing:
+
+- **Your checkout's code is what runs.** The workflow prepends the launched
+  checkout's `src/` to the container's `PYTHONPATH`, so the image supplies the
+  dependency stack and your working tree supplies `sp_validation`. No rebuild
+  needed to test a change. (Caveat: `rerun-triggers: code` does not watch
+  `src/`, so force reruns after editing a module.)
+- **To test a branch's own image** — when the *stack* changed, not just `src/` —
+  point the workflow at its CI tag:
+
+  ```bash
+  snakemake --profile workflow/profiles/candide \
+      --config container=docker://ghcr.io/cosmostat/sp_validation:my-branch <target>
+  ```
 
 ### Local install with `uv`
 

Dockerfile

@@ -1,21 +1,35 @@
 # Development image with more bells and whistles
 FROM ghcr.io/cosmostat/shapepipe:im_sims
 
+# liblapack-dev: cosmosis's MultiNest links -llapack, and the base image ships
+# only the runtime liblapack.so.3 (no dev symlink). The gsl/cfitsio/fftw3 dev
+# packages are what the CosmoSIS Standard Library's C sources compile against
+# (they are the headers CSL's own CI installs); git is for cloning it.
 RUN apt-get update -y --quiet --fix-missing && \
     apt-get dist-upgrade -y --quiet --fix-missing && \
     apt-get install -y --quiet \
         autoconf \
         automake \
         libtool \
         pkg-config \
+        git \
         htop \
         npm \
-        tmux
+        tmux \
+        liblapack-dev \
+        libgsl-dev \
+        libcfitsio-dev \
+        libfftw3-dev
 
 # The base shapepipe image provides a uv-managed venv at /app/.venv (exported as
 # VIRTUAL_ENV); install sp_validation's deps into that same venv rather than
 # spawning a second one under /sp_validation.
 ENV UV_PROJECT_ENVIRONMENT=/app/.venv
+# $HOME is bind-mounted under apptainer, so uv would otherwise discover the
+# host's managed CPythons -- including newer ones that satisfy requires-python
+# -- and build a venv against an interpreter carrying none of this stack.
+ENV UV_PYTHON=/app/.venv/bin/python \
+    UV_PYTHON_DOWNLOADS=never
 
 WORKDIR /sp_validation
 
@@ -30,9 +44,59 @@ WORKDIR /sp_validation
 # numba-safe numpy 2.4.6 come straight from the lock, so the old ad-hoc snakemake
 # and cs_util `--upgrade` layers are gone.
 COPY pyproject.toml uv.lock /sp_validation/
+
+# cosmosis builds MPI-enabled polychord/multinest only when MPIFC is set: its
+# setup.py exports MPIFC for conda builds only, and the sampler Makefiles gate on
+# `which $(MPIFC)`. Absolute path, not a bare name: /opt/ompi/bin is not always on
+# PATH, and a miss silently omits libchord_mpi.so while the install still succeeds,
+# and `cosmosis --mpi` fails at load time -- which is how the pipeline runs, since
+# the --smp pool is broken upstream. Must precede the sync that builds cosmosis.
+ENV MPIFC=/opt/ompi/bin/mpif90
+
 RUN uv sync --frozen --inexact --no-install-project \
     --extra test --extra glass --extra workflow
 
+# The CosmoSIS Standard Library: the module files (camb interface, projection,
+# 2pt likelihood, ...) the cosmo_inference pipelines name. The `workflow` extra
+# above installs cosmosis itself; CSL is a separate tree of modules that is not
+# on PyPI and has to be built against that install, so it is cloned and compiled
+# here rather than left to each user (which is what the .ini templates used to
+# assume, hard-coding one person's home directory).
+#
+# Pinned to Sacha Guerrini's fork, which carries the four commits the UNIONS
+# pipelines depend on: tau-stats, sample_S8, and two z-dependent linear-alignment
+# modules. See cosmo_inference/README.md for the standing of that fork.
+ARG CSL_REPO=https://github.com/sachaguer/cosmosis-standard-library.git
+ARG CSL_REF=b26fa7ff666ab4d607b2e32e36f799a53bfb1d9c
+ENV CSL_DIR=/opt/cosmosis-standard-library
+
+# `python -m cosmosis.configure` emits the exports (COSMOSIS_SRC_DIR et al.)
+# every CSL Makefile includes its compiler config from. Evaluated directly
+# rather than through the `cosmosis-configure` wrapper: that wrapper's
+# am-I-sourced probe reads unset zsh/ksh variables, which `set -u` turns into
+# an error, and its `exit` then ends the sourcing shell with status 0 — make
+# never runs and the layer still "succeeds". The trailing `test -f` keeps any
+# such silent no-op loud.
+#
+# `make -C shear` rather than a bare `make`: the top-level target also descends
+# into likelihood/, which builds the Planck, WMAP and ACT likelihoods -- large,
+# data-dependent, and unused by any UNIONS pipeline. Everything our .ini
+# templates reference is either pure Python (consistency, sample_S8, camb,
+# load_nz_fits, photoz_bias, linear_alignment, add_intrinsic, shear_m_bias,
+# xi_sys, 2pt_like -- no Makefile in those trees at all) or lives under shear/:
+# `limber`, which project_2d.py links, and `cl_to_xi_nicaea`, whose
+# nicaea_interface.so the 2pt_shear stage loads.
+RUN bash -c 'set -eo pipefail; \
+    export PATH=/app/.venv/bin:$PATH; \
+    git clone --filter=blob:none "$CSL_REPO" "$CSL_DIR"; \
+    cd "$CSL_DIR"; \
+    git checkout --detach "$CSL_REF"; \
+    cmds=$(python -m cosmosis.configure); \
+    eval "$cmds"; \
+    export GSL_INC=/usr/include GSL_LIB=/usr/lib/x86_64-linux-gnu; \
+    make -C shear; \
+    test -f shear/cl_to_xi_nicaea/nicaea_interface.so'
+
 # Install sp_validation itself (editable) into the same venv; deps are already
 # satisfied by the sync above.
 COPY . /sp_validation

README.md

@@ -62,30 +62,35 @@ directive imports the shared rules under each run's own config and an output
 `prefix`, so runs namespace under `results/<name>/` without clobbering one
 another.
 
-## Container Installation (Recommended)
+## Installation
 
-The easiest way to install sp_validation is via a container. Docker images are automatically built and pushed to the [GitHub Container Registry (GHCR)](https://github.com/CosmoStat/sp_validation/pkgs/container/sp_validation) on every push to `develop`. This image can be installed and run on most systems (including clusters) with just a few lines of code.
-
-We recommend running the image with **Apptainer** (formerly Singularity) which is installed on most HPC clusters. To simply run the image, use the following command:
+`sp_validation` runs from a pre-built container: CI builds an image carrying
+the full scientific stack on every push and publishes it to the
+[GitHub Container Registry](https://github.com/CosmoStat/sp_validation/pkgs/container/sp_validation).
+The bundled `spv-container` CLI installs and manages your personal copy of it:
 
 ```bash
-# build writeable "sandbox" container in the current directory
-# ./sp_validation will be a directory that functions like a vm
-apptainer build --sandbox sp_validation docker://ghcr.io/cosmostat/sp_validation:develop
-
-# open a shell in the container
-apptainer shell --writable sp_validation 
-# and confirm that the installation was successful
-python -c "import sp_validation"
-```
+git clone https://github.com/CosmoStat/sp_validation.git
+cd sp_validation
+ln -s "$PWD/src/sp_validation/container.py" ~/.local/bin/spv-container
 
-You can also run the image with **Docker**:
-
-```bash
-docker run --rm -it ghcr.io/cosmostat/sp_validation:develop python -c "import sp_validation"
+spv-container pull                                    # fetch the image (~1.5 GB)
+spv-container exec python -c "import sp_validation"   # confirm it works
 ```
 
-We do not currently build images for Apple Silicon/arm64; however the amd64 images should work on these systems, albeit with reduced performance.
+That is the whole install. `pull` puts the image at its canonical per-user
+path (`~/.cache/sp_validation/`), and everything else finds it there —
+`spv-container exec` for one-off commands (`spv-container exec bash` for an
+interactive shell) and the Snakemake workflow for cluster jobs.
+`spv-container status` says what you have and how current it is;
+`spv-container sandbox` gives you a writable copy for mid-analysis
+`pip install`s. On a cluster, run the pull from a compute node.
+
+To run the analysis workflow (`workflow/`), see
+[`workflow/README.md`](workflow/README.md): Snakemake runs on the host, and
+the profile puts each job in the container itself. For Docker, development
+installs, and more depth, see the
+[installation docs](https://cosmostat.github.io/sp_validation/installation.html).
 
 
 

cosmo_inference/README.md

@@ -4,15 +4,46 @@ by Lisa Goh and Sacha Guerrini, CEA Paris-Saclay
 This folder contains the files neccessary to run the cosmological inference pipeline on the UNIONS galaxy catalogues. 
 
 ### Requirements
-To run the pipeline, one would need to have installed [CosmoSIS](https://cosmosis.readthedocs.io/en/latest/). To sample the PSF leakage parameters, the fork of [cosmosis-standard-library](https://github.com/sachaguer/cosmosis-standard-library/) of Sacha Guerrini has to be used.
+Everything the pipeline needs ships in the container: nothing to install, and no
+paths to edit before a run.
+
+[CosmoSIS](https://cosmosis.readthedocs.io/en/latest/) comes in via the
+`workflow` extra, built with MPI support. The CosmoSIS Standard Library — the
+tree of modules the `.ini` pipelines name — is built into the image at
+`/opt/cosmosis-standard-library`, with `CSL_DIR` pointing there. CosmoSIS reads
+environment variables into an `.ini`'s `[DEFAULT]` section, so the templates'
+`COSMOSIS_DIR = %(CSL_DIR)s` resolves to it. Outside the container, export
+`CSL_DIR` at a build of your own and the same templates work unchanged.
+
+CSL is pinned to **Sacha Guerrini's fork**
+([sachaguer/cosmosis-standard-library](https://github.com/sachaguer/cosmosis-standard-library/))
+at `b26fa7ff`, not to upstream: the UNIONS pipelines depend on four commits that
+exist only there — tau statistics, `sample_S8`, and two z-dependent
+linear-alignment modules. The fork is 4 commits ahead of
+`cosmosis-developers/cosmosis-standard-library` and 373 behind it; carrying those
+four forward onto current upstream is future work.
+
+Launch sampling under MPI (`mpiexec -n N cosmosis --mpi ...`), not `--smp`:
+CosmoSIS's shared-memory pool is unmaintained and still crashes after sampling
+completes (`Pool` has no attribute `data`, `runtime/process_pool.py`) as of
+3.25.2.
 
 ### To Run
-The inference pipeline is now orchestrated through Python. Run the main Snakemake workflow from the parent directory:
+The inference pipeline is orchestrated through Snakemake. On the candide
+cluster, drive it with the committed profile — see
+[`workflow/README.md`](../workflow/README.md) for the one-time
+`uv tool install` setup and the full explanation. From the repository root:
 
 ```bash
-snakemake -j<jobs> inference_fiducial
+snakemake --profile workflow/profiles/candide \
+    -s workflow/Snakefile \
+    inference_fiducial --configfile <run config>
 ```
 
+Off-cluster, drop `--profile` and add `-j <jobs>` instead. Each job runs
+inside the sp_validation container automatically — no `apptainer shell` or
+`apptainer exec` needed by hand.
+
 This will automatically execute all steps:
 1. Calculate 2PCF ($\xi_{pm}$) via `cosmo_val.py`
 2. Compute covariance matrices using CosmoCov <!--- LG: now obsolete, making way for OnCovariance cauclation instead --->

cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia.ini

@@ -1,6 +1,8 @@
 #parameters used elsewhere in this file
 [DEFAULT]
-COSMOSIS_DIR = /n23data1/n06data/lgoh/scratch/cosmosis-standard-library_lisa
+# The CosmoSIS Standard Library; CSL_DIR comes from the environment, set in the
+# container (see cosmo_inference/README.md).
+COSMOSIS_DIR = %(CSL_DIR)s
 
 
 [pipeline]

cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia_cell.ini

@@ -1,6 +1,8 @@
 #parameters used elsewhere in this file
 [DEFAULT]
-COSMOSIS_DIR = /home/guerrini/cosmosis-standard-library
+# The CosmoSIS Standard Library; CSL_DIR comes from the environment, set in the
+# container (see cosmo_inference/README.md).
+COSMOSIS_DIR = %(CSL_DIR)s
 
 
 [pipeline]

cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_psf.ini

@@ -1,6 +1,8 @@
 #parameters used elsewhere in this file
 [DEFAULT]
-COSMOSIS_DIR = /home/guerrini/cosmosis-standard-library
+# The CosmoSIS Standard Library; CSL_DIR comes from the environment, set in the
+# container (see cosmo_inference/README.md).
+COSMOSIS_DIR = %(CSL_DIR)s
 
 
 [pipeline]

docs/source/installation.rst

@@ -2,28 +2,59 @@ Installation
 ============
 
 ``sp_validation`` is **not** distributed on PyPI.
-Install it from a pre-built container, or check out the source with ``uv`` when you need to edit it.
+It runs from a pre-built container, managed by the bundled ``spv-container`` CLI; check out the source with ``uv`` only when you need to edit the package itself.
 
-Container (recommended)
------------------------
+Container via ``spv-container`` (recommended)
+---------------------------------------------
 
-Every push to ``develop`` builds an image carrying the full scientific stack and pushes it to the `GitHub Container Registry (GHCR)
-<https://github.com/CosmoStat/sp_validation/pkgs/container/sp_validation>`_.
+Every push builds an image carrying the full scientific stack and pushes it to the `GitHub Container Registry (GHCR)
+<https://github.com/CosmoStat/sp_validation/pkgs/container/sp_validation>`_, tagged by branch — ``:develop`` tracks the integration branch.
 The image runs on most systems, including HPC clusters, with no further setup.
+``spv-container`` installs your personal copy of it and manages it from then on:
 
-`Apptainer <https://apptainer.org>`_ (formerly Singularity) is installed on most clusters and is the path we recommend:
+.. code-block:: bash
+
+   git clone https://github.com/CosmoStat/sp_validation.git
+   cd sp_validation
+   ln -s "$PWD/src/sp_validation/container.py" ~/.local/bin/spv-container
+
+   spv-container pull                                    # fetch the image (~1.5 GB)
+   spv-container exec python -c "import sp_validation"   # confirm it works
+
+The symlink works because ``container.py`` is deliberately stdlib-only: it runs on the *host*, where the science stack is not installed.
+(Inside the container the same CLI is on ``PATH`` as a console script.)
+``pull`` requires `Apptainer <https://apptainer.org>`_ (formerly Singularity), which is installed on most clusters, and writes the image to one canonical per-user path, ``~/.cache/sp_validation/sp_validation.sif``.
+Each user owns their copy: you refresh it when you want to, and nobody else's refresh moves the ground under your running jobs.
+On a cluster, run the pull from a compute node — it moves ~1.5 GB.
+
+The subcommands:
 
 .. code-block:: bash
 
-   # Build a writeable "sandbox" container in the current directory.
-   # ./sp_validation is a directory that behaves like a small VM.
-   apptainer build --sandbox sp_validation docker://ghcr.io/cosmostat/sp_validation:develop
+   spv-container pull                     # fetch the published image to the canonical path
+   spv-container status                   # what is here, which commit built it, how current
+   spv-container exec <cmd...>            # run a command inside it (exec bash for a shell)
+   spv-container sandbox                  # unpack into a writable dir, for pip installs
+   spv-container exec --writable <cmd...> # ... with writes that persist
+
+``status`` compares the image's build commit against your checkout's ``HEAD``, so you always know whether a ``pull`` would refresh anything.
+The **sandbox** is the escape hatch for exploratory work that needs a package the image does not carry yet: once built, it takes precedence over the SIF everywhere — Snakemake workflow jobs included — until you reset with ``spv-container pull`` + ``spv-container sandbox --force``.
+
+Everything resolves the image in one order — sandbox if it exists, else your SIF, else the registry tag — and that includes the analysis workflow.
+How the workflow uses the image (Snakemake runs on the host; the profile puts each job in the container) is covered in ``workflow/README.md``.
+
+Other ways to run the image
+---------------------------
+
+The published image is a normal OCI image; ``spv-container`` is a convenience, not a gatekeeper.
+Run it directly with Apptainer:
+
+.. code-block:: bash
 
-   # Open a shell in the container, then confirm the install works.
-   apptainer shell --writable sp_validation
-   python -c "import sp_validation"
+   apptainer pull sp_validation.sif docker://ghcr.io/cosmostat/sp_validation:develop
+   apptainer shell sp_validation.sif
 
-The image also runs under Docker:
+or with Docker:
 
 .. code-block:: bash