Skip to content

test: search conformance suite layer (ii), backend execution (search-extensibility A0a(ii)) #1671

Closed

Description

@Jammy2211

Overview

Phase A0a(ii) of the search-extensibility epic: the backend-execution layer of the search conformance suite, running every one of the 15 searches on the full-extras CI legs at PYAUTO_TEST_MODE=1 with real DirectoryPaths, skipped by declared capability (never importorskip) on the no-jax leg. Tests only. Blocked by A0b (same repo, and A0b's Drawer/NullPaths and samples_from repairs are asserted here); implemented stacked on A0b's branch so the PR exists before A0b merges. Launched --auto by the human on 2026-10-08 (Autonomy: safe, test cap safe; Consequence: judge).

Plan

  • New test_autofit/non_linear/search/test_conformance_backend.py over conformance_roster.searches_under_test().
  • Per search: output file set with real paths, completed-path resume, NullPaths fit, samples_from(model, None) fallback, samples_info key sets.
  • Capability-matched fixtures (numpy vs JAX MockAnalysis); module docstring states the coverage contract.
  • No library source; a search that cannot pass gets a strict xfail naming A2/A3, listed on this issue.

Tier: judge — merge mode: human /prm (after A0b)

Detailed implementation plan

Affected Repositories

  • PyAutoFit (tests only, pending-release)

Branch Survey

Repository Current Branch Dirty?
fit/PyAutoFit main clean (A0b worktree holds the claim)

Suggested branch: feature/search-ext-a0a2-backend-conformance, stacked on feature/search-ext-a0b-hygiene in the A0b worktree; PR base = the A0b branch until it merges.

Implementation Steps

  1. Read test_conformance.py and conformance_roster.py for the roster, capability attributes and golden-table conventions.
  2. Write the backend module and fixtures; budgets via apply_test_mode only.
  3. Emulate the no-jax leg with an import hook; confirm exactly NSS, BlackJAXNUTS, SMC, MultiStart×4 skip by capability.
  4. Full pytest test_autofit; runtime of the new module under 90 s.

Validation checklist (--auto run — plan was not pre-approved)

  • Effective level: safe (header: safe, cap: test → safe)
  • Gate legs recorded in the PR body at ship

Original Prompt

Click to expand starting prompt

Search conformance suite, layer (ii): backend execution on the full-extras legs (epic search-extensibility, phase A0a(ii))

Type: test
Target: autofit
Repos:

  • PyAutoFit
    Themes:
  • searches
  • jax
  • ci
    Difficulty: medium
    Autonomy: safe
    Consequence: judge
    Witness: layer (ii) green on the [optional] CI legs for all 15 searches; on an emulated no-jax leg NSS, BlackJAXNUTS, SMC and the MultiStart×4 are skipped by declared capability (never by importorskip) and the rest pass
    Unattended: ready
    Priority: high
    Epic: search-extensibility
    Blocked-by: search-ext-a0b-hygiene (PyAutoFit, A0b must merge first; implemented stacked on its branch)
    Status: draft
    Filed: 2026-10-08

Phase A0a(ii) of the search-extensibility epic (draft/research/autofit/search_extensibility_epic.md; plan
search_extensibility_epic_report.md §4 A0a layer (ii)). Tests only. Layer (i) is merged (PyAutoFit#1667,
test_autofit/non_linear/search/{test_conformance,conformance_roster}.py). Lands after A0b so Drawer/NullPaths
is not an xfail. Human launch 2026-10-08: --auto for every A0 phase.

Original request (verbatim from the epic plan)

"Layer (ii), backend execution, Heart full-extras legs only (lib-tests.yml installs [optional], :114),
explicitly skipped by declared capability on unittest-nojax: capability-matched fixtures (numpy MockAnalysis for
jax_use none/optional, a JAX MockAnalysis for required) at PYAUTO_TEST_MODE=1 with real DirectoryPaths; it
asserts the output file set, the completed-path resume, a NullPaths fit, the samples_from(model, None) fallback
and the samples_info key sets. The module docstring states the coverage contract: every backend executes on the
full-extras leg, and importorskip may not hide a required backend there. Layer (ii) lands after A0b, so
Drawer/NullPaths is not an xfail. Verify: pytest test_autofit/non_linear on a full-extras env and with
jax/blackjax/optax uninstalled. Witness: layer (i) green on all legs; layer (ii) green on [optional] legs, with
NSS/NUTS/SMC/MultiStart skipped on the no-jax leg by declared capability."

Scope

  • New test_autofit/non_linear/search/test_conformance_backend.py reusing conformance_roster.searches_under_test().
    Per search, at PYAUTO_TEST_MODE=1: (a) a fit with real DirectoryPaths into tmp_path asserting the expected
    output file set per search (search.json, model.json, samples.csv/samples_info.json, the backend's own
    checkpoint name from survey 01 §8), (b) a second fit on the completed path resumes without rerunning (assert via
    the backend's call count or the timer file), (c) a NullPaths fit returns a Result, (d) samples_from(model, None)
    falls back (A0b's narrowed except), (e) samples_info key set per family.
  • Capability gate: skip by the declared capability attributes A0a(i) introduced (jax_use or equivalent), never
    importorskip; the no-jax emulation (import-hook block) must show exactly NSS, BlackJAXNUTS, SMC and MultiStart×4
    skipped.
  • Fixtures: numpy MockAnalysis for jax_use none/optional; a JAX MockAnalysis (jnp log-likelihood) for required.
  • Mark the whole module to run only when [optional] extras are installed on CI (the unittest legs), skipping
    per-search on unittest-nojax by capability.
  • Not in scope: any library source. If a search cannot satisfy (a)-(e) without a library change, add a strict xfail
    naming the later phase (A2/A3) and list it on the issue.

Verification

pytest test_autofit/non_linear/search -x; full pytest test_autofit; no-jax leg emulated; runtime of the new
module under 90 s on a laptop (reduce budgets via apply_test_mode, never by editing searches).

Shape

One PyAutoFit PR (pending-release). Implemented on a branch stacked on feature/search-ext-a0b-hygiene with the PR
based on that branch; after A0b merges the human retargets to main (or it retargets automatically) and merges.
Tier judge.

Activity

  1. Jammy2211 commented on Oct 8, 2026

    @Jammy2211
    CollaboratorAuthor

    Shipped to PR-open under --auto (2026-10-08): PyAutoFit#1673, stacked on #1672 (A0b) with that branch as its base; merge #1672 first. Gate: 3207 passed / 2 skipped / 9 xfailed; nojax emulation 71 passed with exactly NSS, BlackJAXNUTS, SMC and MultiStart×4 skipped by declared capability; review CLEAN; Heart STALE. Six new strict xfails record real bugs for later phases: Dynesty×2 (A3: savestate.save deleted after the fit, so a restored folder always falls back to samples.csv) and Emcee/Zeus/BlackJAXNUTS/SMC (A4: import-time AutoCorrelationsSettings default ignores run-time test mode). Minor findings: Drawer samples_info["time"] is None with real paths; NSS writes time as a float.

  2. Jammy2211 commented on Oct 8, 2026

    @Jammy2211
    CollaboratorAuthor

    Merged 2026-10-08 by human /prm after #1672: PyAutoFit#1673 (cf1acd7), pending-release. Record: PyAutoMind complete/2026/10/search-ext-a0a2-backend-conformance.md. The six strict xfails (Dynesty×2 → A3; Emcee/Zeus/BlackJAXNUTS/SMC → A4) are the next phases' witnesses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions