docs(search): state the opt-in rule for _test_mode_samples_info #1448

The mechanism

Under PYAUTO_TEST_MODE=2 / =3, AbstractSearch._fit_bypass_test_mode
(abstract_search.py:864) skips the sampler and hand-builds a result: fake
samples at the prior median, plus a samples_info of only
{total_iterations: 1, time: 0.0, log_evidence: <ll>}, merged at line 947 with
self._test_mode_samples_info(). The base hook (line 979) returns {}.

Nine search modules write samples_info in their real path — nautilus, dynesty,
emcee, zeus, bfgs, drawer, blackjax nuts, multi_start_gradient, abstract. Exactly
one overrides the hook: BlackJAXNUTS
(.../mcmc/blackjax/nuts/search.py:386).

Why the asymmetry is correct

Reachability. No library path reads these diagnostic keys under bypass. The
properties that read them — SamplesMCMC.total_steps (samples/mcmc.py:200),
SamplesNest.number_live_points / total_samples / log_evidence
(samples/nest.py:77-92) — live on Samples subclasses the bypass never
constructs; _fit_bypass_test_mode always builds a SamplesPDF. The only
possible consumers are workspace scripts reading samples_info[...] directly.

Consumer sweep across all eleven workspace/tutorial repos finds exactly two
such sites:

The first is precisely why the NUTS override exists: 5d175ebcc (#1260, May
2026) added both the hook and that override to stop this tutorial crashing under
smoke. The second was fixed on the workspace side (wst#83 / PR#84, merged
f4c45c1).

Nautilus, Dynesty, Emcee and Zeus have no bypassed consumer either — which is why
none of them override the hook. autofit_workspace/scripts/searches/mle.py uses
MultiStartAdam but never touches samples_info, so MultiStartGradient has no
bypassed consumer at all.

The rule both decisions actually followed

Prints → placeholders. Asserts → real search. A tutorial that only
displays diagnostics may legitimately run bypassed, so its search needs the
override with honest empties. A script that asserts on diagnostics must not
run bypassed at all — it declares ENV: real_search, and no override is wanted.

Rejected alternatives

• Add an AbstractMultiStartGradient override (for surface consistency with
  NUTS): rejected. It serves no existing consumer, and a placeholder
  total_steps would let a future assert total_steps < n_steps silently pass
  on a stub 0 — the exact failure mode avoided in wst#83.
• A unit test pinning the bypass keys: rejected. A test can pin
  BlackJAXNUTS's keys but cannot express "and the others deliberately have
  none" without freezing the sampler roster.

Affected Repositories

• PyAutoFit (primary)

Branch Survey

Two active.md entries claimed PyAutoFit and both were stale: the
point-source-defaults-campaign line read "PR #1441 OPEN" (merged
2026-08-01T12:47:32Z) and the nautilus-1core-serial-pool line read "PR OPEN …
restore main after merge" (#1443 merged 2026-08-01T19:12:47Z, merge commit
5bf32dab, contained in origin/main). Both released; neither is a live
conflict.

Suggested branch: feature/test-mode-samples-info-hook-contract

Worktree root: ~/Code/PyAutoLabs-wt/test-mode-samples-info-hook-contract/

Work Classification: Library

Implementation Steps

1. autofit/non_linear/search/abstract_search.py — rewrite the
   _test_mode_samples_info docstring (line 979) per the rule above, naming both
   precedents and the "do NOT add placeholders for an asserting reader" hazard.
2. Run the search unit tests; confirm docstring-only diff.

Key Files

• autofit/non_linear/search/abstract_search.py:979 — the hook to document.
• autofit/non_linear/search/mcmc/blackjax/nuts/search.py:386 — the one override
  (unchanged; cited as precedent).
• autofit/non_linear/search/mle/multi_start_gradient/search.py — the deliberate
  non-override (unchanged; cited as counter-precedent).

Acceptance

The base hook's docstring states the opt-in rule and the print-vs-assert test
explicitly. Docstring-only — no behaviour change, no new override, no new test.

Filed 2026-08-04 as PyAutoMind/draft/bug/autofit/multi_start_test_mode_samples_info_gap.md,
split out of autofit_workspace_test#83:

AbstractMultiStartGradient has no _test_mode_samples_info override (NUTS does).
Decide whether the hook contract is "every sampler with diagnostic samples_info
keys must override it" (→ add the MultiStartGradient override with NaN/0/None
placeholders, per the NUTS precedent) or "the hook is opt-in for keys downstream
code genuinely reads under bypass" (→ leave it, and say so in the hook docstring
so the asymmetry stops looking like an oversight). Weigh against the workspace
doctrine that a bypassed run should fail loudly rather than silently produce
placeholder diagnostics.