PyAutoFit: add n_effective to DynestyDynamic, and fix truncated dynamic runs under a finite iterations_per_full_update

[samlange04]

Package: PyAutoFit (autofit), current main (710f4b343), dynesty 2.1.5. Branch: https://github.com/samlange04/PyAutoFit/tree/feature/dynesty-dynamic-n-effective

Proposal

Expose dynesty's n_effective on DynestyDynamic, and fix a bug it exposed: with a finite iterations_per_full_update, DynestyDynamic returns a truncated baseline run instead of a complete dynamic run.

1. n_effective on DynestyDynamic

dynesty's dynamic sampler keeps adding batches of live points until its stopping function judges the estimated effective sample size (ESS) to have reached n_effective. The default in dynesty 2.x is max(10000, ndim ** 2). PyAutoFit currently has no way to set it, so the only ways to make a dynamic run cheaper are maxcall / maxbatch, which cut the run instead of lowering its target.

For the lens models I run (10 to 20 parameters) a target of a few thousand effective samples gives the same posterior at a fraction of the batch phase. The branch adds n_effective: Optional[int] = None to DynestyDynamic.__init__, forwarded to run_nested only when not None, so the default behaviour is unchanged. It is not an identifier field. Following #1202, there is no yaml default; it is an explicit __init__ default like the other dynamic arguments.

DynestyStatic is deliberately left alone: dynesty accepts n_effective on NestedSampler.run_nested but emits a DeprecationWarning saying it will be removed.

2. Bug: a finite iterations_per_full_update truncates a dynamic run

Setting iterations_per_full_update (so output is written on the fly) makes DynestyDynamic stop after a cut-short baseline run, with an ESS of about 1 and no batches. Three things in dynesty combine:

• DynamicNestedSampler.run_nested compares maxcall with its cumulative call count self.ncall, carried over between calls. The static sampler's counter resets per call, which is what run_search_internal assumes. So the second chunk is handed a budget it has already spent and returns without sampling; the "no new calls" criterion then finishes the search.
• A baseline run that maxcall cut short cannot be continued: the next run_nested call restarts it (sample_initial calls reset()), and resume=True refuses because the previous call ended in RUN_DONE.
• results.ncall undercounts the sampler's own ncall (it omits each batch's live-point initialisation), so comparing it against a cumulative budget misjudges whether a chunk converged.

Proposed fix

Per-sampler hooks on AbstractDynesty (total_calls_from, maxcall_from, chunk_kwargs, chunk_is_finished) whose defaults reproduce the current static behaviour exactly. DynestyDynamic overrides them: the first chunk of a chunked run completes the whole baseline (maxbatch=0, no budget, since it cannot be resumed), then the batch phase is chunked by cumulative budget total_calls + iterations_per_full_update, and a batch chunk counts as finished only if it stopped strictly inside its budget. Intermediate output for dynamic runs therefore starts after the baseline.

Measured on a 3-parameter Gaussian with iterations_per_full_update=1500: baseline + 3 batch chunks + 1 no-op chunk, ESS 3587 against 3944 for the same run in a single chunk, same logZ and parameters. DynestyStatic verified unaffected (4 chunks of ~1500 calls, same result as before).

Happy to split the two into separate PRs if preferred, but the bug only matters for users who would also want n_effective (anyone trying to make a dynamic run both observable and affordable), and the fix is what makes n_effective testable with on-the-fly output.

[samlange04]

PR: PyAutoLabs/PyAutoFit#1665

[Jammy2211]

Accepted and delivered. Your PR (PyAutoLabs/PyAutoFit#1665) merged on 2026-10-15 and shipped in autofit 2026.10.16.1 (current release 2026.10.18.1): DynestyDynamic(n_effective=...) is forwarded to run_nested only when set, and a finite iterations_per_full_update now completes the baseline before chunking the batch phase. Keeping the two changes in one PR was the right call for exactly the reason you gave. Thanks @samlange04, this one was a pleasure to merge. Marking this as the answer and closing the thread as resolved.