[Jammy2211]

Summary

At D = 100, Nautilus failed before sampling in two ways (found by the correlated_nd pilot, autofit_inference#18). This PR adds a fail-fast live-point guard, a run-scoped workaround for nautilus's HDF5 checkpoint format, and a checkpoint=False opt-out.

Root causes (nautilus-sampler 1.0.5, h5py 3.13.0 / HDF5 1.14.6)

1. Live-point floor. Ellipsoid.compute (nautilus/bounds/basic.py:298-300) raises ValueError: Number of points must be larger than number dimensions. Nothing checks this up front, so n_live <= n_dim runs start and then die at the first bound.
2. Checkpoint header limit. Ellipsoid.write (nautilus/bounds/basic.py:404-407) stores A, B and B_inv (n_dim**2 float64 each) as HDF5 attributes. Sampler.write creates the file with h5py.File(filepath, 'x') (nautilus/sampler.py:1270), which uses HDF5's earliest file format, where an object-header message is capped at 64 KiB. From n_dim = 91 the first write after an ellipsoid bound (sampler.py:431-439) raises OSError: Unable to synchronously create attribute (object header message is too large).

What changed (autofit/non_linear/search/nest/nautilus/search.py)

• Guard: NAUTILUS_N_LIVE_MESSAGE plus a Nautilus.check_model override, run by fit and again at the top of run(ctx). It raises SearchException before any likelihood call unless n_live > prior_count and, when set, n_points_min >= prior_count + 1.
• Workaround: _large_attribute_hdf5() swaps nautilus.sampler.h5py for a thin proxy whose File defaults to libver=('v108', 'latest') when the caller passes none. Everything else delegates to the real h5py. It wraps Sampler(...) construction and call_search in both fit_x1_cpu and fit_multiprocessing, and restores the real module in finally. Checkpoint and resume work at any D. It is commented for removal once nautilus fixes this upstream.
• Opt-out: a new checkpoint: bool = True constructor argument. False makes checkpoint_file return None, so nautilus writes nothing. This is a resume-less mode. It is not an identifier field, because only __identifier_fields__ enter the identifier. The conformance constructor golden is updated. run(ctx) no longer calls Path(None) when there is no checkpoint file.
• Docs: a new "Nautilus at 91 or more parameters" section in docs/design/checkpointing.md (with an Upstream issue: <link pending> line), the Nautilus docstring (n_live/n_points_min floor, checkpoint), and docs/cookbooks/search.md.

Tests (test_autofit/non_linear/search/nest/test_nautilus.py)

• A D = 100 Ellipsoid.compute(rng.random((300, 100))) is written through the proxied File, reopened 'r+' and read with Ellipsoid.read. B and B_inv are unchanged.
• A strict xfail control (raises=OSError) writes the same bound in nautilus's default format. It will XPASS, and so fail, once nautilus is fixed, which is the cue to remove the workaround.
• A test checks that the proxy restores nautilus.sampler.h5py when an exception is raised inside it.
• Integration: a D = 100 Gaussian through the real search.fit → run(ctx) → fit_x1_cpu path, with Nautilus(n_live=101, n_networks=1, n_batch=101, n_like_max=404, seed=1) and directory output. The fit stops at its first full update (after the checkpoint write that follows an ellipsoid bound). A fresh nautilus Sampler on the same file then resumes with a matching n_like and at least 2 bounds. It takes about 2 s. Without the workaround it fails with the upstream OSError (checked by hand).
  • Deviation: the test monkeypatches nautilus.sampler.NautilusBound with a fast stand-in. It is a real Ellipsoid fitted to the live set, written with the same Ellipsoid.write attributes, but it samples from the unit cube instead of rejection-sampling. nautilus's real bound at D = 100 with about 100 live points rejection-samples an enlarged ellipsoid against the unit cube at near-zero acceptance. It did not finish within 80 s, even with neural_network_kwargs={'max_iter': 5} or n_live=200. The real-sampler witness (corr_gaussian_d100 n_live_200) stays with autofit_inference.
• Guard: n_live=100 on a 100-parameter model raises the named error with the analysis likelihood counter at 0 and the backend never entered. n_points_min=50 raises. n_live=101 (and n_points_min=101) passes. run(ctx) repeats the guard.
• checkpoint=False → checkpoint_file is None, and checkpoint is not an identifier field.

Local runs: test_nautilus.py gives 25 passed and 1 xfailed. test_autofit/non_linear/search plus test_autofit/database/identifier gives 1114 passed and 1 xfailed under -n 4. The 7 tests that failed under xdist all race on shared test_autofit/output directories (FileNotFoundError on output PNGs and dills in emcee, zeus and dynesty). They pass when rerun serially (131 passed). The full suite was not run locally (it takes more than 10 min); CI runs it.

Closes #1697

Upstream issue draft (human posts)

Title: Checkpoint write fails for n_dim ≥ 91: object header message is too large

Body:

Versions: nautilus-sampler 1.0.5, h5py 3.13.0 / HDF5 1.14.6.

With filepath set and n_dim=100, the first Sampler.write after an ellipsoid bound is added raises:

OSError: Unable to synchronously create attribute (object header message is too large)

Cause: Ellipsoid.write (nautilus/bounds/basic.py:404-407) stores A, B and B_inv (each an n_dim × n_dim float64 array) as HDF5 attributes, and h5py.File(filepath, 'x') (nautilus/sampler.py:1270) uses the earliest HDF5 file format, which limits an object-header message to 64 KiB. n_dim**2 * 8 bytes exceeds that from n_dim = 91.

Possible fixes:

• store these arrays with create_dataset (reading old attribute-based files for backwards compatibility), or
• open the file with libver=('v108', 'latest'). We verified this round-trips a D = 100 ellipsoid (Ellipsoid.write → Ellipsoid.read, matrices unchanged) and resumes a sampler from the file.

Seen with PyAutoFit in a correlated-Gaussian D = 100 benchmark.

🤖 Generated with Claude Code