Repository navigation
test: search conformance suite layer (ii), backend execution (search-extensibility A0a(ii)) #1671
Description
Overview
Phase A0a(ii) of the search-extensibility epic: the backend-execution layer of the search conformance suite, running every one of the 15 searches on the full-extras CI legs at PYAUTO_TEST_MODE=1 with real DirectoryPaths, skipped by declared capability (never importorskip) on the no-jax leg. Tests only. Blocked by A0b (same repo, and A0b's Drawer/NullPaths and samples_from repairs are asserted here); implemented stacked on A0b's branch so the PR exists before A0b merges. Launched --auto by the human on 2026-10-08 (Autonomy: safe, test cap safe; Consequence: judge).
Plan
- New
test_autofit/non_linear/search/test_conformance_backend.pyoverconformance_roster.searches_under_test(). - Per search: output file set with real paths, completed-path resume,
NullPathsfit,samples_from(model, None)fallback,samples_infokey sets. - Capability-matched fixtures (numpy vs JAX
MockAnalysis); module docstring states the coverage contract. - No library source; a search that cannot pass gets a strict xfail naming A2/A3, listed on this issue.
Tier: judge — merge mode: human /prm (after A0b)
Detailed implementation plan
Affected Repositories
- PyAutoFit (tests only,
pending-release)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| fit/PyAutoFit | main | clean (A0b worktree holds the claim) |
Suggested branch: feature/search-ext-a0a2-backend-conformance, stacked on feature/search-ext-a0b-hygiene in the A0b worktree; PR base = the A0b branch until it merges.
Implementation Steps
- Read
test_conformance.pyandconformance_roster.pyfor the roster, capability attributes and golden-table conventions. - Write the backend module and fixtures; budgets via
apply_test_modeonly. - Emulate the no-jax leg with an import hook; confirm exactly NSS, BlackJAXNUTS, SMC, MultiStart×4 skip by capability.
- Full
pytest test_autofit; runtime of the new module under 90 s.
Validation checklist (--auto run — plan was not pre-approved)
- Effective level: safe (header: safe, cap: test → safe)
- Gate legs recorded in the PR body at ship
Original Prompt
Click to expand starting prompt
Search conformance suite, layer (ii): backend execution on the full-extras legs (epic search-extensibility, phase A0a(ii))
Type: test
Target: autofit
Repos:
- PyAutoFit
Themes: - searches
- jax
- ci
Difficulty: medium
Autonomy: safe
Consequence: judge
Witness: layer (ii) green on the[optional]CI legs for all 15 searches; on an emulated no-jax leg NSS, BlackJAXNUTS, SMC and the MultiStart×4 are skipped by declared capability (never by importorskip) and the rest pass
Unattended: ready
Priority: high
Epic: search-extensibility
Blocked-by: search-ext-a0b-hygiene (PyAutoFit, A0b must merge first; implemented stacked on its branch)
Status: draft
Filed: 2026-10-08
Phase A0a(ii) of the search-extensibility epic (draft/research/autofit/search_extensibility_epic.md; plan
search_extensibility_epic_report.md §4 A0a layer (ii)). Tests only. Layer (i) is merged (PyAutoFit#1667,
test_autofit/non_linear/search/{test_conformance,conformance_roster}.py). Lands after A0b so Drawer/NullPaths
is not an xfail. Human launch 2026-10-08: --auto for every A0 phase.
Original request (verbatim from the epic plan)
"Layer (ii), backend execution, Heart full-extras legs only (lib-tests.yml installs [optional], :114),
explicitly skipped by declared capability on unittest-nojax: capability-matched fixtures (numpy MockAnalysis for
jax_use none/optional, a JAX MockAnalysis for required) at PYAUTO_TEST_MODE=1 with real DirectoryPaths; it
asserts the output file set, the completed-path resume, a NullPaths fit, the samples_from(model, None) fallback
and the samples_info key sets. The module docstring states the coverage contract: every backend executes on the
full-extras leg, and importorskip may not hide a required backend there. Layer (ii) lands after A0b, so
Drawer/NullPaths is not an xfail. Verify: pytest test_autofit/non_linear on a full-extras env and with
jax/blackjax/optax uninstalled. Witness: layer (i) green on all legs; layer (ii) green on [optional] legs, with
NSS/NUTS/SMC/MultiStart skipped on the no-jax leg by declared capability."
Scope
- New
test_autofit/non_linear/search/test_conformance_backend.pyreusingconformance_roster.searches_under_test().
Per search, atPYAUTO_TEST_MODE=1: (a) a fit with realDirectoryPathsintotmp_pathasserting the expected
output file set per search (search.json,model.json,samples.csv/samples_info.json, the backend's own
checkpoint name from survey 01 §8), (b) a secondfiton the completed path resumes without rerunning (assert via
the backend's call count or the timer file), (c) aNullPathsfit returns aResult, (d)samples_from(model, None)
falls back (A0b's narrowed except), (e)samples_infokey set per family. - Capability gate: skip by the declared capability attributes A0a(i) introduced (
jax_useor equivalent), never
importorskip; the no-jax emulation (import-hook block) must show exactly NSS, BlackJAXNUTS, SMC and MultiStart×4
skipped. - Fixtures: numpy
MockAnalysisforjax_usenone/optional; a JAXMockAnalysis(jnp log-likelihood) for required. - Mark the whole module to run only when
[optional]extras are installed on CI (theunittestlegs), skipping
per-search onunittest-nojaxby capability. - Not in scope: any library source. If a search cannot satisfy (a)-(e) without a library change, add a strict xfail
naming the later phase (A2/A3) and list it on the issue.
Verification
pytest test_autofit/non_linear/search -x; full pytest test_autofit; no-jax leg emulated; runtime of the new
module under 90 s on a laptop (reduce budgets via apply_test_mode, never by editing searches).
Shape
One PyAutoFit PR (pending-release). Implemented on a branch stacked on feature/search-ext-a0b-hygiene with the PR
based on that branch; after A0b merges the human retargets to main (or it retargets automatically) and merges.
Tier judge.
Activity
Jammy2211 commentedon Oct 8, 2026
Shipped to PR-open under
--auto(2026-10-08): PyAutoFit#1673, stacked on #1672 (A0b) with that branch as its base; merge #1672 first. Gate: 3207 passed / 2 skipped / 9 xfailed; nojax emulation 71 passed with exactly NSS, BlackJAXNUTS, SMC and MultiStart×4 skipped by declared capability; review CLEAN; Heart STALE. Six new strict xfails record real bugs for later phases: Dynesty×2 (A3:savestate.savedeleted after the fit, so a restored folder always falls back to samples.csv) and Emcee/Zeus/BlackJAXNUTS/SMC (A4: import-timeAutoCorrelationsSettingsdefault ignores run-time test mode). Minor findings: Drawersamples_info["time"]is None with real paths; NSS writestimeas a float.Jammy2211 commentedon Oct 8, 2026
Merged 2026-10-08 by human
/prmafter #1672: PyAutoFit#1673 (cf1acd7), pending-release. Record: PyAutoMindcomplete/2026/10/search-ext-a0a2-backend-conformance.md. The six strict xfails (Dynesty×2 → A3; Emcee/Zeus/BlackJAXNUTS/SMC → A4) are the next phases' witnesses.