PyAutoFit: DynestyDynamic raises OverflowError (int too large to convert to C long) when run without an output path

[samlange04]

Package: PyAutoFit (autofit), current main (891fe64b9), dynesty 2.1.5. Platform: macOS, Python 3.13.7 (the overflow is not platform specific).

What happens

DynestyDynamic crashes when the search has no output path, i.e. it was created without a name so its paths is NullPaths. DynestyStatic works in the same situation.

import numpy as np
import autofit as af

class Gaussian:
    def __init__(self, centre=0.0, normalization=1.0, sigma=1.0):
        self.centre, self.normalization, self.sigma = centre, normalization, sigma

class Analysis(af.Analysis):
    def __init__(self, data, x):
        super().__init__()
        self.data, self.x = data, x
    def log_likelihood_function(self, instance):
        m = instance.normalization * np.exp(-0.5 * ((self.x - instance.centre) / instance.sigma) ** 2)
        return -0.5 * np.sum((self.data - m) ** 2 / 0.1 ** 2)

x = np.linspace(-5, 5, 50)
data = 2.0 * np.exp(-0.5 * (x / 1.5) ** 2)
model = af.Model(Gaussian)
model.centre = af.UniformPrior(-2, 2)
model.normalization = af.UniformPrior(0, 5)
model.sigma = af.UniformPrior(0.1, 4)

search = af.DynestyDynamic(nlive_init=50)   # no name -> NullPaths
search.fit(model=model, analysis=Analysis(data, x))

  File ".../autofit/non_linear/search/nest/dynesty/search/abstract.py", line 457, in run_search_internal
    search_internal.run_nested(
  File ".../dynesty/dynamicsampler.py", line 2084, in run_nested
    mcall = min(maxcall - ncall, maxcall_batch)
                ~~~~~~~~^~~~~~~
OverflowError: Python int too large to convert to C long

Giving the search a name (so it writes to an output directory) avoids it.

Cause

AbstractDynesty.iterations_from returns the per-chunk call budget passed to dynesty as maxcall. For NullPaths with no maxcall set it returns int(1e99) as an "unbounded" sentinel. dynesty's dynamic sampler evaluates maxcall - ncall where ncall is a numpy int64, and a 99-digit Python int cannot be converted to a C long for that subtraction. The static sampler only compares ncall >= maxcall in pure Python, so it is unaffected.

Proposed fix

Return sys.maxsize instead of int(1e99) in the NullPaths branch. sys.maxsize is the sentinel dynesty itself substitutes when maxcall=None, and sys.maxsize - np.int64(n) is well defined. The maxcall-set branch is unchanged.

Branch: https://github.com/samlange04/PyAutoFit/tree/fix/dynesty-dynamic-nullpaths-overflow. With it applied the script above runs to completion, and the fix comes with a regression test parametrised over both dynesty searches.

PR: PyAutoLabs/PyAutoFit#1658

[xop01]

Give the search a name until the patch is merged:

search = af.DynestyDynamic(name="gauss", nlive_init=50)

With no name, the call budget is a huge number (int(1e99)). Dynesty's dynamic sampler subtracts from that with a numpy integer and crashes, because the number does not fit in a C long. DynestyStatic never does that subtraction, so it still runs. A name makes the search write an output folder and skip that budget. The change that returns sys.maxsize instead is the real fix.

@samlange04

[Jammy2211]

Hi @samlange04, thank you for another excellent report, with a reproducer that runs as written. The diagnosis is spot on: our int(1e99) sentinel overflows dynesty's maxcall - ncall against a numpy int64 in the dynamic sampler, and sys.maxsize matches dynesty's own unbounded sentinel. PR #1658 (with its regression test) is merged into main and will ship in the next release. Until then, giving the search a name= works around it, as @xop01 kindly pointed out (thanks!).