Skip to content

docs: design HST MGE pilot correctness qualification #417

Closed

Description

@Jammy2211

Overview

Design an independent scientific correctness witness for the shipped HST/MGE CPU pilot. The human approved this bounded audit and documentation phase on 2026-10-13: "I approve, go".

Plan

  • Trace the exact objective, resolved settings and reconstruction semantics.
  • Assess independent reference candidates and shared-code limitations.
  • Specify comparisons, tolerance decisions and a bounded implementation follow-up.

Tier: judge — merge mode: human /prm.

Branch survey

Only autolens_profiling is affected. Main at e4266a1; pre-existing untracked dataset/abell_1201/ must be preserved. No competing active claim. Use feature/hst-mge-pilot-qualification in .worktrees/hst-mge-pilot-qualification. Pulse's concurrent check-in owns campaign updates.

Detailed plan and original prompt

HST/MGE pilot: independent correctness qualification design

Filed: 2026-10-13
Type: research
Consequence: judge
Target: @autolens_profiling
Status: draft; awaiting approval of the bounded plan

Original request

ok continue

Context: continue after merging autolens_profiling#416 and PyAutoPulse#55 and
closing implementation issue #415. The raw measurement tooling is shipped;
independent scientific correctness and collection approval remain pending.
Canonical intent: https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/hst_mge_cpu_pilot.md
Previous phase: complete/2026/10/hst-mge-pilot-observations.md.

Proposed bounded phase

Audit the exact imaging-mge-runtime-v1 objective and design its independent
correctness witness before implementing or executing that witness. This phase
produces a source-grounded qualification design, not numerical acceptance.

  1. Trace current.from_dataset/build_workload and cell_builders.imaging_mge through
    the actual AnalysisImaging/FitImaging objective and inversion settings. Pin
    inspected source revisions; distinguish configured defaults from values only
    knowable on the chosen execution host. Enumerate input, mask, PSF,
    oversampling, parameter, coefficient-ordering, solver and normalization identity.
  2. Audit candidate references and existing regression fixtures. Preserve
    non-negativity, actual objective terms and reconstruction semantics. Explicitly
    assess shared-code weaknesses; same-code eager/JIT agreement is insufficient.
    Recommend an independent construction/solve route or record why unavailable.
  3. Specify the witness artifact and comparisons: reference hashes/provenance,
    likelihood terms, coefficients and reconstructed image, absolute/relative
    errors, near-zero normalization, and conditioning/non-uniqueness handling.
    Propose justified tolerance choices for human review, never silently accept
    inherited constants or fabricate numerical evidence.
  4. Document remaining collection choices (host, immutable stack, production
    dataset, resolved settings, load method and resource ceiling) as unresolved
    where no verified evidence exists. Define a separately bounded implementation
    and synthetic validation follow-up, routed to the correctness-owning regression
    workspace if required; do not expand this research phase into library fixes.

Planned files and validation

Add scripts/misc/jax_compile/QUALIFICATION.md with source links and the decision
table; add a short link from PILOT.md. No producer, library, test, default or
timing-record changes. Review every source citation and objective term, check
relative links, and run the applicable project documentation/generated checks.
Do not regenerate measurement timestamps or claim accepted baseline evidence.

No inference, production profiling, scientific witness execution, HPC submission,
baseline freeze/promotion or release. Production inputs and scientific tolerance
approval remain a later human decision. Pulse retains campaign state; this phase
does not duplicate its campaign or interfere with a concurrent Pulse check-in.

Tier: judge — merge mode: human /prm.

Activity

  1. Jammy2211 commented on Oct 13, 2026

    @Jammy2211
    ContributorAuthor

    Shipped design — 2026-10-13

    PR: #418, head 3d6f83f73b84d81ed708129b09a97b4c873e5d3c. CI was running at publication; tier judge, human /prm.

    Added the source-pinned qualification design and a link from PILOT.md. It distinguishes the stabilized non-negative reconstruction from the reported data likelihood, assesses reference independence, proposes an independent operator/augmented NNLS approach, and specifies artifacts, comparisons and candidate error budgets. The small-dataset count/provenance limitation is documented for a later guard; runner code is unchanged.

    Passed: ruff check/format, README/catalogue (including Pulse reader)/wiki/dashboard checks, 23 pinned-source links plus local links, and the existing MGE import-only smoke. Heart readiness is STALE only for release validation incomplete: no rehearsal for current source, permitting development publication.

    No inference, scientific reference solve or production profiling ran. Numerical criteria remain proposals; coefficient tolerances, reference validation and production collection approval remain pending. The next bounded implementation belongs in the correctness regression workspace plus a project adapter, with separately planned synthetic execution caps. Pulse retains campaign choices and scientific acceptance.

    Mind handoff: active/hst_mge_pilot_qualification.md; task worktree and branch remain for review.

  2. Jammy2211 commented on Oct 17, 2026

    @Jammy2211
    ContributorAuthor

    Shipped

    PR #418 merged 2026-10-17 (human /prm after an independent review, verdict FINDINGS, none blocking): scripts/misc/jax_compile/QUALIFICATION.md designs the independent HST/MGE pilot correctness qualification (what the pilot computes, an independent operator + scipy NNLS witness, identities to capture, candidate tolerances marked unaccepted). Still open, carried to the next bounded pilot step: background-load method (none proposed), the 1e-8 relative image L2 budget (likely unattainable; calibration on the synthetic harness must be allowed to revise it), and the correctness-process feasibility Pulse asked for before the 32-process budget is finalized. No compute, freeze or acceptance authorized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions