From 91013eab87fb5dac8e1f611aeef320cf69382889 Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Fri, 11 Sep 2026 11:44:39 -0400 Subject: [PATCH] =?UTF-8?q?cortex(euclid=5Fsersics):=20birth=20=E2=80=94?= =?UTF-8?q?=20100=20unanimous=20lenses=20on=20sep1=20data,=20core=20two-st?= =?UTF-8?q?age=20run=20342696?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01RUgGHe76cvtCT95bEJr1QJ --- dashboard.html | 9 ++ dashboard.md | 19 ++++ projects.yaml | 14 +++ tasks/euclid_sersics/core_two_stage_100.md | 109 +++++++++++++++++++++ 4 files changed, 151 insertions(+) create mode 100644 tasks/euclid_sersics/core_two_stage_100.md diff --git a/dashboard.html b/dashboard.html index 3b69906..138ab6d 100644 --- a/dashboard.html +++ b/dashboard.html @@ -241,6 +241,7 @@ ep_toy_gaussianDo three EP fits agree with NUTS at N=5001- euclid_dr1Does the sep1 DR1 delivery reproduce the prelim ten lenses01- euclid_dr1_prelimDo ordered Nautilus reruns reproduce the euclid catalogue values?102026-09-10 +euclid_sersicsCore two-stage run of the 100 sersic lenses01- ic50_workspaceDoes EP match the graphical joint fit end to end01- slope_hierarchy_scaleDoes hierarchical slope recovery hold at N=25 to 5001- subhalo_validationDoes rectangular_adapt false-detect a subhalo on the clumpy source052026-09-07 @@ -330,6 +331,14 @@

euclid_dr1_prelim

tasks/euclid_dr1_prelim/

+

euclid_sersics

+

Local /mnt/c/Users/Jammy/Science/euclid_sersics

+

RAL /mnt/ral/jnightin/euclid_sersics

+

active ยท ral partition ยท tasks: submitted 1

+

Core two-stage run of the 100 sersic lensessubmitted โ€” runs 342696 ยท wall 0:00 of 36:00 (0%)

+

tasks/euclid_sersics/

+
+

ic50_workspace

Local /mnt/c/Users/Jammy/Science/ic50_workspace

RAL /mnt/ral/jnightin/ic50_workspace

diff --git a/dashboard.md b/dashboard.md index 21d5ec0..b8bfcc6 100644 --- a/dashboard.md +++ b/dashboard.md @@ -34,6 +34,7 @@ One row per active project โ€” the next thing, what it is holding, and when it l | ep_toy_gaussian | Do three EP fits agree with NUTS at N=50 | 0 | 1 | - | | euclid_dr1 | Does the sep1 DR1 delivery reproduce the prelim ten lenses | 0 | 1 | - | | euclid_dr1_prelim | Do ordered Nautilus reruns reproduce the euclid catalogue values? | 1 | 0 | 2026-09-10 | +| euclid_sersics | Core two-stage run of the 100 sersic lenses | 0 | 1 | - | | ic50_workspace | Does EP match the graphical joint fit end to end | 0 | 1 | - | | slope_hierarchy_scale | Does hierarchical slope recovery hold at N=25 to 50 | 0 | 1 | - | | subhalo_validation | Does rectangular_adapt false-detect a subhalo on the clumpy source | 0 | 5 | 2026-09-07 | @@ -235,6 +236,24 @@ Finish: write today's journal entry, rewrite wiki/project/state.md, and return ( [tasks/euclid_dr1_prelim/](https://github.com/PyAutoLabs/PyAutoCortex/tree/main/tasks/euclid_dr1_prelim/) +### euclid_sersics + +**Local** `/mnt/c/Users/Jammy/Science/euclid_sersics` + +**RAL** `/mnt/ral/jnightin/euclid_sersics` + +active ยท ral partition ยท tasks: submitted 1 + +
๐Ÿ“‹ Core two-stage run of the 100 sersic lenses โ€” submitted โ€” runs 342696 ยท wall 0:00 of 36:00 (0%) + +``` +cd /mnt/c/Users/Jammy/Science/euclid_sersics && hpc/sync jobs +``` + +
+ +[tasks/euclid_sersics/](https://github.com/PyAutoLabs/PyAutoCortex/tree/main/tasks/euclid_sersics/) + ### ic50_workspace **Local** `/mnt/c/Users/Jammy/Science/ic50_workspace` diff --git a/projects.yaml b/projects.yaml index f54ef71..5552b0e 100644 --- a/projects.yaml +++ b/projects.yaml @@ -115,6 +115,20 @@ euclid_dr1: status: active note: "born 2026-09-11: the actual DR1 run on the sep1 segmentation delivery (Segmentation.zip = 3 nested batch zips, 15032 tiles, every tile already in the pipeline dataset layout); born by cloning euclid_strong_lens_modeling_pipeline main with config/visualize colormap back to autoarray; first task reruns the ten euclid_dr1_prelim lenses on the new data; remote none until the human creates one" +euclid_sersics: + remote: none + local_path: /mnt/c/Users/Jammy/Science/euclid_sersics + ral_root: /mnt/ral/jnightin/euclid_sersics + mirror: none + sync_cli: hpc/sync + sync_verbs: [push, pull, logs, sync, push-data-init, pull-full, status, submit, push-submit, jobs, sacct, cancel, wait-and-pull, tail, du, check, clear-logs] + ledger: wiki/project/state.md + assistant: autolens_assistant + witness_file: results/**/*.json + partition: ral + status: active + note: "born 2026-09-11: Sersic-index pile-up investigation โ€” 100 unanimous-success lenses (euclid inspection) on the sep1 segmentation data; core two-stage CPU run first, then four sersic_lens_model variants (baseline / nโˆˆ[0.5,10] / central noise inflation / Sersic+point MGE) reuse its vis_lp; remote none until the human creates one" + analytic_gaussian: remote: none local_path: /mnt/c/Users/Jammy/Science/analytic_gaussian diff --git a/tasks/euclid_sersics/core_two_stage_100.md b/tasks/euclid_sersics/core_two_stage_100.md new file mode 100644 index 0000000..7a1733e --- /dev/null +++ b/tasks/euclid_sersics/core_two_stage_100.md @@ -0,0 +1,109 @@ +# Euclid_sersics โ€” the core two-stage run of the 100 Sersic-sample lenses + +Project: euclid_sersics +Summary: Core two-stage run of the 100 sersic lenses +State: submitted +Gates: +Witness: 100/100 array tasks complete vis_lp and vis_pix within 36 h; aggregator yields 100 lens_mass rows; no task aborted +Budget: 36:00 +Runs: 342696 +Ruling: +Review-minutes: 25 +Epic: euclid-sersics +Filed: 2026-09-11 + +## Question + +The June `euclid` catalogue fits one `lp_linear.Sersic` per lens under `Uniform(0.8, 5.0)` with +hard limits, and the returned Sersic index **piles up against the top of the prior**: over 2983 +lenses the median n is 4.80, 42 % sit above 4.9, and **nothing** sits at the lower edge. In the +unanimously-inspected subset the pile-up is *worse* (median 4.84), which is the opposite of what +a bad-fit signature does. `scripts/simulator.py` in the pipeline already names it: "Real Euclid +fits routinely return sersic_index = 5 โ€ฆ an artefact". + +The project that settles it (`euclid_sersics`) will run **four Sersic-stage variants** โ€” +`baseline` (today's prior), `wide_n` (n โˆˆ [0.5, 10]), `central_noise` (noise inflated inside the +seeing disk) and `sersic_point` (Sersic + point MGE) โ€” on one fixed sample, to separate "the +prior is the bound" from "something biases n upward" (a too-broad model PSF or an unmodelled +nuclear point source both push n up; correct-PSF seeing pushes it *down*). + +**This task is not that comparison.** It is the run every one of those variants stands on: the +**core two-stage fit of all 100 lenses**, `vis_lp` then `vis_pix`, on the route `euclid_dr1` is +already using. It produces the `vis_lp` result the variants reuse rather than re-derive, and the +`baseline` lens model the other three are read against. Its question is therefore an +*engineering* one, and it is asked first because a variant comparison built on a sample that +half-completed is worthless: + +**Does the 100-lens sample run clean, end to end, on the sep1 delivery?** + +D1 is resolved: **both stages run.** `vis_pix` is not skipped. The pixelized source is what makes +the lens-light Sersic honest โ€” a variant that moves n by absorbing source flux into the lens +would be invisible on a `vis_lp`-only run. + +## The sample + +100 lenses, chosen by a rule written down so it cannot drift, because every variant must run on +**these exact lenses**: from `euclid`'s inspection set (2990 lenses, 1โ€“7 human votes each, 1293 +unanimous Success) โ€” **unanimous Success, present in both `lens_sersic.csv` and `lens_mass.csv`, +sorted by `total_valid_votes` descending then `object_id` ascending, first 100**. That is 16 +lenses at 7/7 votes and 84 at 6/6, median June n = 4.917, 22 of them interior (n < 4.5), min +0.859, max 4.999. + +The data is the September-1 segmentation delivery, streamed out of `euclid_dr1`'s +`Segmentation.zip` into `dataset/dr1_sep1_sersics/` โ€” 100 tile folders, 11 files each, all +verified present, two spot-checked through `util.load_vis_dataset`. + +June `object_id` โ†’ sep1 tile is a **coordinate join, never a name match**: tile names were parsed +(`RA`/`DEC` as 3 integer + 10 decimal digits of degrees) and nearest-neighboured against all +15 032 sep1 names, a unique match required inside 3". **100/100 matched, 95 names identical, the +5 that moved by 0.68โ€“0.92" (max 0.921")**. `sample/lens_map.csv` in the project is that table and +is the only legitimate join back to the June catalogue. + +## Witness + +100/100 array tasks complete vis_lp and vis_pix within 36 h; aggregator yields 100 lens_mass rows; no task aborted + +## Where to look + +- `euclid_sersics` (project row `euclid_sersics`): `output/dr1_sep1_sersics//` once pulled + โ€” one subdirectory per stage per lens; the data is `dataset/dr1_sep1_sersics/` and the route is + `hpc/batch_cpu/submit_initial_lens_model_two_stage` (array `0-99`, partition `ral`, 8 CPUs, + 64 GB, 36 h) +- `euclid_sersics` `sample/lens_map.csv` โ€” the June `object_id` โ†” sep1 tile provenance table, and + `sample/lenses_100_june.csv` โ€” the June-side list with votes, ฮธ_E, n and R_eff +- `euclid_sersics` `wiki/project/state.md` โ€” this project's own ledger (the commentary; the + ruling of record is the Cortex's) +- `euclid_dr1` (project row `euclid_dr1`): the same two-stage route on the same delivery, run + 342650 โ€” the reference for what a healthy run of this script looks like +- `autolens_assistant/skills/euclid_prepare_data.md`, `euclid_setup_pipeline.md`, + `euclid_model_lens.md`, `euclid_hpc_runs.md` โ€” how this pipeline is driven + +## Notes + +- 2026-09-11 โ€” **the project was born for this task.** `euclid_sersics` is a fresh clone of + `euclid_strong_lens_modeling_pipeline` at **`dbbb2a5`** โ€” deliberately the *same base commit* as + `euclid_dr1`, plus that project's two local deltas (`activate.sh`'s `AUTOLENS_ASSISTANT` export; + `config/visualize/general.yaml` back on the library default `colormap: autoarray`). Any + difference between the two projects' fits is therefore the sample or the model, never the + pipeline. The clone's `origin` is the pipeline repo and it has local commits that are never + pushed. +- 2026-09-11 โ€” **the submit script differs from `euclid_dr1`'s in five things and nothing else:** + the job name, `PROJECT_PATH`, `sample=dr1_sep1_sersics`, `--array=0-99`, and the dataset list, + which is read from the sample directory + (`mapfile -t datasets < <(ls $PROJECT_PATH/dataset/$sample/ | grep '^Tile' | sort)`) instead of + 100 inline tile names, behind a guard that exits non-zero unless exactly 100 are found. The + guard matters: a partial `hpc/sync push` would otherwise shift every array index and fit the + wrong lens under each one, silently. +- 2026-09-11 โ€” **the pipeline work the variants need does not exist yet**: a `--variant` flag in + `scripts/sersic_lens_model.py`, a central-noise-inflation hook in `util.load_vis_dataset`, and + an `mge_point_model_from` for the Sersic+point variant. That is a separate PR against the + pipeline repo, and no variant can be submitted before it merges. This task is unblocked by none + of it. + +## Runs + +- 342696: submitted โ€” ral โ€” submitted 2026-09-11 โ€” wall 0:00 + +## Ruling + +(none)