Snakemake orchestration for the real-data pipeline - #852

Draft

cailmdaley wants to merge 31 commits into

feat/snakemake-orchestration

Commits on Jul 16, 2026

feat(orchestration): Snakemake workflow for the real-data pipeline (#848)

cailmdaley
and
claude
committed
f92cb44
fix(orchestration): resolve {threads} at rule level, not inside params.cmd

cailmdaley
and
claude
committed
187950c
fix(orchestration): carry original exposure names through the index; star-cat store naming

cailmdaley
and
claude
committed
ac6bddd
fix(orchestration): exposure-name pass-through, star-cat dirs, P0 run config

cailmdaley
and
claude
committed
9b0456e
fix(orchestration): {output} placeholder inside params.cmd survives literally

cailmdaley
and
claude
committed
6fe141f
fix(orchestration): shard the exposure forest — the module glob's shape is not cosmetic

cailmdaley
and
claude
committed
a110d89
fix(orchestration): carry a wrapper-scripts content hash into every rule's params

cailmdaley
and
claude
committed
28c10d8
fix(orchestration): keep snakemake state and index off /project (hard-full quota)

Cail Daley
and
claude
committed
a273e76

Commits on Jul 17, 2026

fix(orchestration): drop exp_mask's dead per-unit star-cat input

Cail Daley
and
claude
committed
0010def

Commits on Jul 31, 2026

fix(orchestration): nibi-flavor completeness counts; file-level attrition reporting

Cail Daley
and
claude
committed
833aa93
feat(ngmix): env-expanded chunk fields; position-seeded RNG becomes the only mode

Cail Daley
and
claude
committed
20cc506
feat(orchestration): committed workflow configs — no wrapper, no runtime rewriting (#848 D2)

Cail Daley
and
claude
committed
d1c0afb
feat(orchestration): manifests become the DAG's currency — completeness check CLI + manifest-driven report (#848 D2/D3)

cailmdaley
and
claude
committed
1b22b98
feat(orchestration): the rules rewrite — manifests as DAG currency, sharded stores, parse-time index, native scattergather (#848 D1–D5)

cailmdaley
and
claude
committed
83d26f2
fix(orchestration): harden the rewrite against its adversarial review

cailmdaley
and
claude
committed
b149e18
feat(orchestration): clean_exposure becomes an in-DAG rule with tombstones (#848 D5, S5)

Cail Daley
and
claude
committed
12ecfc2
feat(orchestration): nibi profile goes slurm-executor (#848 D4, S6)

Cail Daley
and
claude
committed
e677f67
fix(orchestration): stop the clean_exposure cascade; report reads tombstones (#848 D5)

Cail Daley
and
claude
committed
96c78cd
docs(orchestration): profile comment tells the truth about the clean/trigger gate

Cail Daley
and
claude
committed
4d28897
run: point the quad at a fresh smk-g3 run dir (old smk-p0 tree is unsharded, pre-manifest)

Cail Daley
and
claude
committed
831a127
fix(orchestration): slurm_account is the association name (def-mjhudson_cpu)

Cail Daley
and
claude
committed
f82e7f4

Commits on Aug 15, 2026

fix(orchestration): index build is the main compute parse's alone

Cail Daley
and
claude
committed
5cfdc04
fix(orchestration): sp pins --snakefile; works from any cwd

Cail Daley
and
claude
committed
acfb91a
docs(orchestration): a failed tile pins ~8 exposures, not ~80

Cail Daley
and
claude
committed
dedc1f0
fix(orchestration): manifests are success-only DAG currency

Cail Daley
and
claude
committed
2a24d34
feat(orchestration): star-cat creation moves into the DAG

Cail Daley
and
claude
committed
a1d4dfa

Commits on Aug 16, 2026

fix(orchestration): the verdict composes counts with shapepipe_run's rc

Cail Daley
and
claude
committed
6a8e71a
fix(orchestration): exp_star_cat stops trusting what it cannot see

Cail Daley
and
claude
committed
7341012
fix(orchestration): reclaim the star-cat farm, stop the home-quota leak

Cail Daley
and
claude
committed
cba14de

Commits on Aug 24, 2026

run(smk-g3): flip clean on — quad validated, batch1 append needs the room

Cail Daley
and
claude
committed
30567ac
run(smk-g3): the campaign appends the first 30 batch1 tiles

Cail Daley
and
claude
committed
fadc277