Skip to content

test: search conformance suite layer (ii), backend execution of the 15 searches (search-extensibility A0a(ii)) - #1673

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/search-ext-a0a2-backend-conformance
Oct 8, 2026
Merged

Jammy2211 merged 1 commit into
mainfrom
feature/search-ext-a0a2-backend-conformance

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

Phase A0a(ii) of the search-extensibility epic (PyAutoFit#1671): the backend-execution layer of the search conformance suite. Tests only. Stacked on #1672 (A0b): this PR's base is feature/search-ext-a0b-hygiene because it asserts A0b's Drawer/NullPaths and samples_from repairs; merge #1672 first, then this one (GitHub retargets it to main when the base branch goes). Shipped under the human's 2026-10-08 --auto launch; tier judge.

  • New test_autofit/non_linear/search/test_conformance_backend.py over the 15-search roster at PYAUTO_TEST_MODE=1, fits cached once per module: (a) real DirectoryPaths fit asserting the exact output file set inside the zip, (b) completed-path resume (_fit call counter 1 then 0), (c) NullPaths fit returns a Result with samples, (d) samples_from(model, None) raises its real error and falls back only on FileNotFoundError/NotImplementedError with the WARNING, (e) frozen samples_info key sets per search plus a shared family core.
  • conformance_roster.py gains the declared capability attributes layer (i) left out: jax_use (none/optional/required) and jax_modules. Golden table unchanged.
  • Coverage contract in the module docstring: every backend executes on the full-extras legs; on unittest-nojax searches skip by declared capability only, never importorskip.

Six strict xfails for real library bugs, each naming its repair phase:

  1. DynestyStatic, DynestyDynamic → A3: after a completed fit samples_via_internal_from(None) reads savestate.save, which output_search_internal deletes, so a restored folder always falls back to samples.csv.
  2. Emcee, Zeus, BlackJAXNUTS, SMC → A4: the auto_correlation_settings=AutoCorrelationsSettings() default is built at import time, so test mode's check_size=1 only applies when PYAUTO_TEST_MODE is set before import.

One test-side workaround follows from item 2: Emcee gets a fresh AutoCorrelationsSettings() built under test mode, or a 10-step chain crashes in emcee.autocorr. Documented in the module. Minor findings not tested: Drawer's samples_info["time"] is None even with real paths; NSS writes "time" as a float where every other search writes a string.

API Changes

None (tests only).

Test Plan

  • New module alone: 134 passed, 6 xfailed, ~55 s
  • pytest test_autofit/non_linear/search -x: 604 passed, 1 skipped, 9 xfailed
  • Full pytest test_autofit -x: 3207 passed, 2 skipped, 9 xfailed
  • unittest-nojax emulated: 71 passed, 65 skipped (exactly NSS, BlackJAXNUTS, SMC, MultiStart×4, all by declared capability), 4 xfailed (re-run by the reviewing session)
  • CI green on unittest 3.12 / 3.13 / nojax and docs
Full API Changes (for automation & release notes)

Removed

  • none

Added

  • none (test roster attributes only)

Migration

  • none

Validation checklist (--auto run — plan was not pre-approved)

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

…ches

- test_conformance_backend.py runs every search's real backend at
  PYAUTO_TEST_MODE=1 with real DirectoryPaths: the archived output file set
  (generic files, samples.csv/samples_info.json, covariance.csv for PDF
  families, the backend's own search_internal file), completed-path resume
  without a second _fit call, a NullPaths fit, the A0b narrowed samples_from
  fallback, and frozen samples_info key sets per search and family.
- Capability-matched likelihoods: numpy for jax_use none/optional, jax.numpy
  for required (NSS, BlackJAXNUTS, SMC, MultiStart x4).
- The roster declares jax_use and jax_modules; on unittest-nojax a search
  skips only by that declared capability, never by importorskip.
- Strict xfails: Dynesty x2 restored-output samples_from falls back to
  samples.csv (A3); MCMC x4 import-time auto_correlation_settings default
  ignores a run-time PYAUTO_TEST_MODE (A4).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the pending-release PR queued for the next release build label Oct 8, 2026
Base automatically changed from feature/search-ext-a0b-hygiene to main October 8, 2026 09:20
@Jammy2211
Jammy2211 merged commit cf1acd7 into main Oct 8, 2026
3 checks passed
@Jammy2211
Jammy2211 deleted the feature/search-ext-a0a2-backend-conformance branch October 8, 2026 09:21
@Jammy2211 Jammy2211 removed the pending-release PR queued for the next release build label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant