Skip to content

bug: EP counts zero-iteration Laplace convergence (NO_CHANGE) as a stale/skipped update #1711

Closed

Description

@Jammy2211

Overview

Criterion 6 of the analytic_gaussian wave-1 witness (PyAutoCortex ruling R-20260910-04) missed on 19/400 seed-legs: a factor ended the EP run with zero SUCCESS updates. On the exactly-Gaussian leg A (seed 140) the factor is PriorFactor5 {NO_CHANGE: 2}; on 18 leg-B seeds it is a PriorFactor with only NO_CHANGE, in six of them together with HierarchicalFactor1 with only BAD_PROJECTION. NO_CHANGE is produced only by autofit/graphical/laplace/newton.py (StatusFlag.get_flag(success, i) with i == 0, updated=False), so a factor whose tilted density is already at its mode on first visit is reported exactly like a factor whose line search failed every sweep, and both the library's STALE warning and the ensemble's criterion 6 inherit that conflation. This task diagnoses the two shapes, fixes the reporting, re-checks the BAD_PROJECTION shape under the shipped projection="moments" (#1656), pins leg A seed 140 in autofit_workspace_test and amends criterion 6.

Campaign context: PyAutoInsight tasks/ep_campaign.md (graphical-ep epic); the Insight campaigns.yaml row asked to reconcile shipped fixes and select one bounded phase — this is that phase.

Plan

Tier: glance — merge mode: auto-merge on green if the Witness passes, post-merge summary.

Detailed implementation plan

Affected Repositories

  • PyAutoFit (primary)
  • autofit_workspace_test (regression script)
  • PyAutoInsight (ledger), PyAutoCortex (criterion text; temp worktree off origin/main)

Branch Survey

Repository Current Branch Dirty?
./fit/PyAutoFit main (d3a7721) clean
./fit/autofit_workspace_test main (763d79d) clean

Suggested branch: feature/ep-zero-success-factors — worktree ~/Code/PyAutoLabs-wt/ep-zero-success-factors/

Implementation Steps

  1. Repro (worktree PyAutoFit on PYTHONPATH): cd /mnt/c/Users/Jammy/Science/active/analytic_gaussian && NUMBA_CACHE_DIR=/tmp/numba_cache MPLCONFIGDIR=/tmp/matplotlib python3 scripts/ensemble_seed.py --seed 140 --legs a --columns ep and seeds 29/38/71 --legs b; write to a scratch output dir so results/ens_n5/ (committed b44390d) is never overwritten. Read ep_history.csv / ep_diagnostics.results; capture the "..., iter=0" status messages. Add a projection="moments" leg (local patch of analytic_autofit.py if the script has no knob; nothing committed to the science repo).
  2. Library: autofit/graphical/laplace/newton.py ~290–380 — a stop condition met at i == 0 with success=True is convergence: report updated=True, keep flag=NO_CHANGE. autofit/graphical/expectation_propagation/optimiser.py _track_failure / _stale_factor_warnings (~380–520) — a NO_CHANGE with an accepted projection counts as updated and its variables as seen; the per-(factor, variable) pass must not list a variable already at the fixed point. Check mean_field.py:511–620 (update_factor_mean_field, check_valid/update_invalid) so an accepted identical projection is not misclassified. Docstrings: optimiser.py NO_CHANGE paragraph, StatusFlag in utils.py.
  3. Unit test in test_autofit/graphical/ (next to the feat(ep): report staleness per (factor, variable) and record reverted variables in ep_history.csv #1576 stale-tracking tests): a small Gaussian graph with one PriorFactor at its mode from the start → reported updated, no stale warning, history shows NO_CHANGE. Red before, green after.
  4. Re-check seeds 29/71 leg B under projection="moments": HierarchicalFactor1 SUCCESS ≥ 1 expected (cured by feat: LaplaceOptimiser(projection="moments") — moment-matching projection for the hierarchical scatter (#1654) #1656). Record the outcome on this issue.
  5. Workspace: autofit_workspace_test/scripts/graphical/analytic_gaussian_seed140.py (or a --seed leg of analytic_gaussian.py if it stays within the CI cap noted at line 62): leg A, seed 140, moments projection; asserts closed-form parity rows including widths, every factor SUCCESS ≥ 1 or accepted NO_CHANGE, STALE warning absent. Not in config/build/no_run.yaml.
  6. Criterion 6: PyAutoCortex archive/tasks/analytic_gaussian/ensemble_parity.md row 6 + newest-first Log note in projects/analytic_gaussian.md (cortex.py check). Hand the human a diff for ensemble_seed.py:160 any_factor_never_succeeded.
  7. Insight: tasks/ep_campaign.md phase-table 2026-10-23 entries + findings follow-up rows; campaigns.yaml graphical-ep.next/updated.

Key Files

  • autofit/graphical/laplace/newton.py — zero-iteration convergence → updated semantics
  • autofit/graphical/expectation_propagation/optimiser.py — stale tracking (_factors_skipped, _stale_factor_warnings)
  • autofit/graphical/mean_field.py — projection acceptance
  • autofit/graphical/utils.py — StatusFlag
  • autofit_workspace_test/scripts/graphical/analytic_gaussian.py — pattern for the regression
  • PyAutoInsight/tasks/ep_campaign.md, campaigns.yaml — ledger

Witness

python3 scripts/ensemble_seed.py --seed 140 --legs a --columns ep in analytic_gaussian writes a sidecar whose legs.A.info["ep stale"] is False and every factor has SUCCESS ≥ 1 — or, if the zero-SUCCESS factor is ruled legitimate, criterion 6 is amended with the reason; plus a regression in autofit_workspace_test/scripts/graphical/ pinning the leg-A case.

Lineage

PyAutoFit#1405, #1562, #1574, #1576, #1580, #1656; PyAutoCortex ruling R-20260910-04; Mind prompt draft/bug/autofit/ep_factors_end_the_run_with_zero.md; Insight tasks/ep_campaign.md.

Original Prompt

Click to expand starting prompt

EP factors end the run with zero SUCCESS updates on the exactly-Gaussian leg A of analytic_gaussian (seed 140) and on 18/200 leg-B seeds

Type: bug
Target: PyAutoFit
Repos:

  • PyAutoFit
  • autofit_workspace_test
    Themes:
  • graphical-ep
    Difficulty: medium
    Autonomy: supervised
    Priority: high
    Status: formalised
    Consequence: glance
    Witness: python3 scripts/ensemble_seed.py --seed 140 --legs a --columns ep in analytic_gaussian writes a sidecar whose legs.A.info["ep stale"] is False and in whose legs.A.info["ep flags"] every factor has SUCCESS >= 1 — or, if a zero-SUCCESS factor is ruled legitimate, criterion 6 is amended with the reason; plus a regression in autofit_workspace_test/scripts/graphical/ that pins the leg-A case.
    Review-minutes: 3
    Unattended: ready
    Epic: graphical-ep
    Filed: 2026-09-10

Why

Criterion 6 of the pre-registered witness of the science task
PyAutoCortex/tasks/analytic_gaussian/ensemble_parity.md ("no seed in which any factor ends the
run with zero SUCCESS updates") missed: 19 of 400 seed-legs recorded at least one factor that
finished the EP run having never once been updated successfully — the STALE state PyAutoFit
#1562 / #1574 / #1576 / #1580 taught the library to report.

Ruled 2026-09-10 as PyAutoCortex/rulings/2026/09/R-20260910-04.md (accept with findings).

Affected seeds, from the 200-seed N=5 ensemble (RAL job 342413_[0-199], autofit 2026.8.17.1):

  • leg B (sigma ~ TruncatedGaussian(10, 5, 0, 100)): seeds 29, 38, 44, 52, 62, 71, 75, 83,
    90, 106, 133, 138, 147, 149, 152, 160, 178, 193
  • leg A (sigma = 10 fixed): seed 140

Leg A is the damning one. It is a linear-Gaussian graph — mu ~ N(50, 10^2),
x_i | mu ~ N(mu, 10^2), y_ij | x_i ~ N(x_i, s_i^2) — on which Laplace EP is exact by
construction, and criterion 3 confirms it: autofit EP recovers leg A 200/200 with worst
a = 0.001. A factor that never takes an update on a graph where every projection is available
in closed form is a library defect, not a model limitation.

The sidecars show two distinct shapes:

  • a PriorFactor* that ends with only NO_CHANGE (leg A seed 140: PriorFactor5 {NO_CHANGE: 2};
    most leg-B seeds, e.g. 38/44/52/62 PriorFactor17 {NO_CHANGE: 3})
  • a HierarchicalFactor1 that ends with only BAD_PROJECTION and never a SUCCESS
    (leg B seeds 29, 71, 106, 152, 178, 193 — e.g. {BAD_PROJECTION: 15}, {BAD_PROJECTION: 20})

Whether those are one defect or two is part of the question.

Repro

Science project analytic_gaussian at /mnt/c/Users/Jammy/Science/analytic_gaussian
(git main, no remote), results committed at b44390d:

cd /mnt/c/Users/Jammy/Science/analytic_gaussian
python3 scripts/ensemble_seed.py --seed 140 --legs a --columns ep

and read the EP flag fields of the sidecar it writes — in
results/ens_n5/seed_0140.json they are legs.A.info["ep flags"]
(a {factor_name: {FLAG: count}} map) and the derived boolean legs.A.info["ep stale"].
The witness scripts are borrowed verbatim from
autofit_workspace_test/scripts/graphical/analytic_{reference,ep_minimal,autofit}.py @ 165a19e.

Ask

  • Which factor is it, on which update, and why does it never succeed — is the projection
    rejected, is the update a no-op because the message is already the cavity, or is the factor
    never visited?
  • Are the NO_CHANGE-only PriorFactor case and the BAD_PROJECTION-only HierarchicalFactor
    case the same defect or two?
  • Fix it, or explain why a zero-SUCCESS factor is legitimate on an exactly-Gaussian graph — and
    if it is legitimate, criterion 6 is what has to change, with the reason recorded.
  • Add a regression to autofit_workspace_test/scripts/graphical/ that pins the leg-A seed-140
    case: on the exactly-Gaussian leg every factor ends with at least one SUCCESS update.

Lineage

  • PyAutoFit#1405 — still open, the Laplace-on-scatter collapse basin
  • PyAutoFit#1580 — merged, the most recent STALE-reporting fix; this evidence is from a mirror
    that contains it (RAL PyAutoFit 66f9f8d5d contains 08207bad0)
  • PyAutoCortex/rulings/2026/09/R-20260910-04.md — the ruling
  • 2026-09-30: LaplaceOptimiser(projection="moments") (PyAutoFit#1656, record complete/2026/09/ep-moment-projection.md) gives 3/3 HierarchicalFactor SUCCESS on the tight toy (50, 50.5, 49.5) where the default mode path gives 0 (all BAD_PROJECTION); the BAD_PROJECTION-only HierarchicalFactor case may be cured under projection="moments" — re-check seed 140 / the 18 leg-B seeds with it before fixing the mode path

🤖 Generated with Claude Code

Activity

  1. Jammy2211 commented on Oct 23, 2026

    @Jammy2211
    CollaboratorAuthor

    Library PR Created

    PyAutoFit PR #1712 (feature/ep-zero-success-factors, commit bcaf6f7, label pending-release).

    Diagnosis verdict: the NO_CHANGE-only PriorFactor is a legitimate converged update, not a stale factor — Newton met its gradient tolerance at the start point (iter=0), the projection was accepted and the message updated (N(50, 22.4) → N(49.9994, 10.0)), and the final posterior matches the closed form to a = 5e-5, b = 2e-5. The library's STALE tracking never flagged it; the conflation is the ensemble witness's criterion 6 (SUCCESS-only count). The BAD_PROJECTION-only HierarchicalFactor shape (leg B seeds 29/71/106/152/178/193) is the #1405 Laplace-on-scatter state, cured by projection="moments" (#1656): seeds 29 and 71 re-run under moments give HierarchicalFactor0 {SUCCESS: 28} / {SUCCESS: 23}, no STALE. Two findings, two causes; only the first needed a (bookkeeping-only) library change. No EP update maths touched; repro numerics bit-identical before/after.

    Tests: 2 new (one red→green), test_autofit/graphical 322 passed, full suite 3561 passed / 1 skipped / 1 xfailed. Heart YELLOW at ship (manifest drift; PyAutoLens release validation stale) acknowledged by the human.

    Next: autofit_workspace_test regression leg (seed 140 leg A inside scripts/graphical/analytic_gaussian.py, +2 s) via ship_workspace; PyAutoCortex criterion-6 amendment; ensemble_seed.py:160 diff handed to the human.

    Possible new scope (not touched): seed 29 leg B under moments still misses the sigma cell (a 0.225 / b 0.416 vs 0.15 / 0.25) while sigma stays inside [q05, q95].

  2. added a commit that references this issue on Oct 23, 2026
  3. Jammy2211 commented on Oct 23, 2026

    @Jammy2211
    CollaboratorAuthor

    Shipped — 2026-10-23

    PRs

    Upstream PR

    #1712 (merged abb8bfa, pending-release)

    Summary

    The NO_CHANGE-only PriorFactor is a converged update (Newton stop condition met at the start point, projection accepted, closed-form match a 5e-5 / b 2e-5), so optimise_quasi_newton now reports it updated=True with the flag kept as NO_CHANGE; the EP stale tracking already handled it. The BAD_PROJECTION-only HierarchicalFactor shape is the #1405 scatter state cured by projection="moments" (#1656), verified on seeds 29 and 71. Witness criterion 6 now counts SUCCESS or NO_CHANGE as a completed update; the seed-140 leg pins it in autofit_workspace_test.

    Session Notes

    • No EP update maths, projection or Laplace numerics changed; repro numerics bit-identical before and after.
    • Owed to the human: apply the ensemble_seed.py:160 any_factor_never_succeeded diff (count SUCCESS + NO_CHANGE) in the science checkout /mnt/c/Users/Jammy/Science/active/analytic_gaussian; merge Cortex#75 and Insight#38; decide whether Insight's byte-pinned tasks/ep_campaign.md gets re-pinned with the in-place edit.
    • Side finding, unfiled: seed 29 leg B under moments still misses the sigma cell (a 0.225 / b 0.416 vs 0.15 / 0.25) while sigma stays inside [q05, q95].
    • Glance-tier merge by the shipping session under the approved plan; revert = git revert abb8bfacb on PyAutoFit main and git revert 97cb781 on autofit_workspace_test main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions