Repository navigation
perf: back off the fnnls warm-start memo on scattered evaluation streams #613
Description
Overview
The fnnls warm-start memo (on by default since #498) seeds each positive-only solve from the previous evaluation's final passive set. On a scattered evaluation stream (iid draws, e.g. Nautilus early live points) the seed is bad, the post-solve fallback guard drops it, the next solve restarts dense and re-seeds -- so every other solve pays for a bad seed. autolens_profiling#332 measured the alma interferometer Delaunay solve 2.17x slower memo-on vs memo-off on an iid stream (rect 1.08x), while a local walk gains 0.68x / 0.13x. This adds a per-key back-off so the memo stops re-seeding a key that keeps falling back, and re-probes it on an exponential schedule so a stream that turns local recovers the gain.
Source task: organs/PyAutoPulse/tasks/interferometer_nnls_memo_scattered_stream_guard.md (related autolens_profiling#332).
Plan
- Reproduce locally with a small CPU witness (scattered vs local-walk stream, solver-only timing) before changing anything.
- Track, per memo key, how many seeded solves in a row fell back.
- After 2 consecutive fallbacks, skip the memo seed (solve dense, still refreshing the entry) for an exponentially growing number of solves (1, 2, 4, ... capped at 32), then probe the seed again; an accepted seed resets the streak.
- A local-walk stream never reaches two consecutive fallbacks, so its behaviour (and every reconstruction) is unchanged; no default tolerance or config key changes.
- Unit tests for both streams over several seeds; re-time the witness.
Tier: glance — merge mode: human /prm (scope of this session ends at a local commit; Heart RED, shipping is the human's call)
Detailed implementation plan
Affected Repositories
- PyAutoArray (primary)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./PyAutoArray | main | clean |
Suggested branch: feature/nnls-memo-scattered-backoff
Worktree root: ~/Code/PyAutoLabs-wt/nnls-memo-scattered-backoff/
Implementation Steps
autoarray/inversion/inversion/nnls_memo.py: add a bounded per-key back-off table (fallback streak,skip remaining), withbackoff_should_skip(key),backoff_record_fallback(key),backoff_record_accept(key); constants_NNLS_BACKOFF_AFTER_FALLBACKS = 2,_NNLS_BACKOFF_MAX_SKIP = 32.autoarray/inversion/inversion/inversion_util.py::reconstruction_positive_only_from: consult the back-off before reading the memo entry; record fallback / accept after the existing ratio guard; dense solves still refresh the entry.test_autoarray/inversion/inversion/test_nnls_memo.py: state-machine tests; iid stream (several seeds) — seeded solves bounded, reconstructions equal memo-off to round-off; local walk (several seeds) — back-off never engages and reconstructions are bit-identical with the back-off disabled.
Key Files
autoarray/inversion/inversion/nnls_memo.pyautoarray/inversion/inversion/inversion_util.pytest_autoarray/inversion/inversion/test_nnls_memo.py
Original Prompt
Click to expand
See the Pulse task file above; Witness: on a recorded real-sampler evaluation sequence the memo-on fnnls solve is no slower than memo-off on every phase, keeping >= 80 % of the local-walk gain; figure of merit unchanged.
Activity
Jammy2211 commentedon Oct 4, 2026
Status: implemented locally, ship pending human (Heart RED)
Local commit
0d9bbecdonfeature/nnls-memo-scattered-backoff(worktree~/Code/PyAutoLabs-wt/nnls-memo-scattered-backoff). Not pushed, no PR.Guard. Per-key back-off in
nnls_memo: after 2 consecutive seeded-solve fallbacks, the next 1, 2, 4, ... (cap 32) would-be-seeded solves start dense and still refresh the entry. Each backed-off dense solve also checks for free whether the seed it skipped would have passed the guard, using the symmetric difference with the final passive set, which equalswarm_start_errors. If it would have passed, the remaining skips are cancelled. An accepted real seed resets the streak. A local walk never falls back twice in a row, so it is untouched. There is a new stats keywarm_start_backoff, and a new helpermemo_clear()that clears entries and back-off state together. No config or tolerance changes.Witness (local, synthetic SIS-like rect n=576, solver-only, min of 3 replays, 3 seeds, 64 solves, 8 threads CPU):
Seeded solves on iid fall from 32/64 to 7-13/64. The remaining iid overhead is the exponential ramp (probes at solves 1, 3, 6, 10, 16, 26, 44, ...). Over long streams it tends to about 1 bad probe per 33 solves. Reconstructions agree with memo-off to <= 3.4e-15 in every arm.
Tests:
test_nnls_memo.pyhas 4 scattered + 4 walk + 3 scattered-then-walk seeds, plus state-machine tests. On the walk stream the reconstructions are bit-identical with the back-off disabled. The fulltest_autoarraysuite gives 1975 passed, 4 xfailed.Not done / follow-ups: the Pulse-task witness (Nautilus replay on alma Delaunay-1500 / rect 39², plus the HST imaging Delaunay control) needs the autolens_profiling harness and real data. autolens_profiling harnesses that reset the memo with
nnls_memo._nnls_passive_set_memo.clear()(fixed_light_numba.py, delaunay_numba_nnls_iterations.py, fixed_light_numpy_solvers.py, fixed_light_s4b_checks.py) should switch tonnls_memo.memo_clear(), or back-off state can carry from one arm into the next.Jammy2211 commentedon Oct 7, 2026
Library PR Created
PR: #615 (
pending-release, Closes #613). Branch rebased onto main; full PyAutoArray suite 1975 passed, 4 xfailed.Heart readiness was YELLOW, acknowledged by the human for this ship:
Workspace impact: none (internal solver behaviour; new
warm_start_backoffstats key, newnnls_memo.memo_clear()).Follow-ups (not merge gates)
nnls_memo.memo_clear()instead of clearingnnls_memo._nnls_passive_set_memodirectly (fixed_light_numba.py,delaunay_numba_nnls_iterations.py,fixed_light_numpy_solvers.py,fixed_light_s4b_checks.py), otherwise back-off state leaks across rows._production_config.pyalso citesnnls_memo.py:63, which this PR shifts (prose only).Next: CI on #615, then
/prm(human).