[Jammy2211]

Summary

hpc-campaign phase 1 (#1735). Every fit that outputs to disk now keeps a small status record beside its output folder, so a campaign of thousands of fits can be monitored and diagnosed without opening results:

• <output_path>.status.json: current state, attempt, resources (peak memory, CPU seconds, GPU when allocated, CPU model, Slurm job fields) and, for Nautilus / Dynesty / Emcee / BFGS / LBFGS, progress and ETA. Written atomically, capped at 2 KB, refreshed every hpc.status_interval_s (60 s)
• <output_path>.status.jsonl: append-only history; a resumed fit shows attempt: 2
• failure record: an exception in fit() writes state: failed, a failure_class and <output_path>.traceback.txt, then re-raises unchanged
• SIGUSR1 (Slurm --signal=B:USR1@<s>) flushes the checkpoint and writes timeout_pending; SIGTERM does the same then exits 143. Installed only in the main thread of the top-level process; previous handlers are restored
• autofit/non_linear/status/states.py is the single run-state / failure-class / product vocabulary; results-library imports it
• hpc.status: false writes nothing and installs no handlers

The files are siblings of the output folder, so they survive output.remove_files.

Merge order: after #1736 (results-library PR 0a). Both touch abstract_search.py and the Nautilus/dynesty search files; a test merge of this branch onto #1736 has no conflicts. Companion PRs: PyAutoLabs/PyAutoNerves#207 (site layer) and PyAutoLabs/autofit_workspace#179 (docs).

API Changes

Additive. New package autofit.non_linear.status (RunState, FailureClass, Product, StatusWriter, FitStatus, ResourceProbe). New config keys hpc.status (default true) and hpc.status_interval_s (default 60). progress() added to Nautilus, Dynesty, Emcee and BFGS/LBFGS. Behaviour change: fits that output to disk now write the status files above; set hpc.status: false to opt out.
See full details below.

Test Plan

• test_autofit/non_linear/status/test_fit_status.py + test_units.py (new): status written at start / cadence / completion; killed-and-resumed → attempt: 2; ≤2 KB; failure record + re-raise; SIGUSR1 → timeout_pending; hpc.status: false changes nothing
• Full test_autofit suite: 3,952 passed + the 3 test_aggregate_csv tests that had clashed with a parallel run (/tmp/summary.csv) pass when run on their own (17/17)
• Smoke: pyauto-heart smoke autofit autolens_test on the phase 1 worktree (PyAutoFit + PyAutoNerves + autofit_workspace branches together): 47 passed, 0 failed

Full API Changes (for automation & release notes)

Added

• autofit.non_linear.status.states — RunState, FailureClass, Product enums (shared vocabulary)
• autofit.non_linear.status.writer.StatusWriter(output_path) — atomic .status.json, append-only .status.jsonl, .traceback.txt
• autofit.non_linear.status.probe — ResourceProbe, job_from_env, gpu_allocated, peak_rss, cpu_seconds, cpu_model, gpu
• autofit.non_linear.status.record — FitStatus lifecycle (start, update, completed, skipped, failed, timeout_pending, close), status_recorded(search), status_notify(search, transition, *args), status_enabled(), status_interval_s()
• autofit.non_linear.status.signals — SIGUSR1 / SIGTERM handlers
• progress(search_internal) on Nautilus, Dynesty, Emcee, BFGS / LBFGS
• config/general.yaml: hpc.status: true, hpc.status_interval_s: 60

Changed behaviour

• fit() runs inside status_recorded; hooks at resume (running), FitContext.update (progress), _complete_fit (completed) and result_via_completed_fit (skipped, not a new attempt)

Migration

• None. Opt out with hpc: status: false in general.yaml.

Refs #1735

🤖 Generated with Claude Code