Skip to content

Report a non-finite EP evidence term by name instead of warn-and-sum (#1713) - #1714

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/ep-evidence-nonfinite
Oct 23, 2026
Merged

Jammy2211 merged 1 commit into
mainfrom
feature/ep-evidence-nonfinite

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

Closes #1713. On analytic_gaussian ensemble seeds 25 and 172 (ruling R-20260910-04), EPMeanField.log_evidence emitted RuntimeWarning: invalid value encountered in add repeatedly and wrote NaN into ep_history.csv from sweep 4 onwards. Diagnosis (reproduced on main, mode projection only): one HierarchicalFactor member sends the scatter sigma an accepted TruncatedNormalMessage with mean −7374 and sigma 384 on [0, 100], so its support sits ~19 sigma into the tail; TruncatedNormalMessage.log_partition computes log(Φ(b) − Φ(a)) directly, which underflows to −inf (true value ≈ −187.9), the variable's evidence becomes +inf, every adjacent factor term −inf, and the sum NaN. The posterior is bit-identical to the committed sidecars; evidence_tol is None by default so stopping is KL-only; only the evidence readout, its plot and the factor log line were affected.

This PR is reporting only: the non-finite term is detected before summing, named once per fit in a logger.warning and in ep_diagnostics.results, and the evidence is returned as an explicit NaN with no numpy warnings; evidence_divergence returns inf on a NaN so an evidence-based stopping rule can never claim convergence on it. No message arithmetic, projection or optimiser changes. The cure of the term itself (a tail-stable log_partition) is message maths and is filed separately as PyAutoMind/draft/bug/autofit/truncated_normal_log_partition_tail_underflow.md for the human's decision.

Workspace regression: autofit_workspace_test (seed-0 legs assert no ep_mean_field RuntimeWarning and finite evidence; new seed-25 mode-projection leg asserts the term is named).

API Changes

No public API change. EPMeanField.log_evidence now returns float("nan") (instead of a NaN produced with numpy warnings) when any term is non-finite and logs the offending variables/factors once per fit; new read-only property EPMeanField.nonfinite_evidence_terms ({label: kind}); EPHistory.evidence_divergence returns inf instead of NaN when an evidence is NaN; ep_diagnostics.results gains an evidence_nonfinite: line when it happens.
See full details below.

Test Plan

  • New test_autofit/graphical/functionality/test_evidence_nonfinite.py (7 tests; 6 fail on the previous code, 7 pass now): no RuntimeWarning, one named warning, NaN evidence, healthy evidence unchanged from the old formula, divergence inf on NaN / 0 when healthy, diagnostics line
  • pytest test_autofit/graphical -q -x: 329 passed
  • Full pytest test_autofit serially: 3890 passed, 1 skipped, 1 xfailed (under -n 4 the suite shows pre-existing xdist races on the shared test_autofit/output/ — unrelated to this change)
  • Science repro (scratch copy of analytic_gaussian scripts): seeds 25 and 172 under -W error::RuntimeWarning exit 0; ep_diagnostics.results names variable 'truncatedgaussianprior_387' (+inf, first at step 39) (step 40 on seed 172); posteriors bit-identical to the committed sidecars
Full API Changes (for automation & release notes)

Removed

  • none

Added

  • autofit.graphical.expectation_propagation.ep_mean_field.EPMeanField.nonfinite_evidence_terms — property, {label: kind} of non-finite evidence terms (empty when finite)
  • autofit.graphical.expectation_propagation.diagnostics.check_evidence_nonfinite() — the evidence_nonfinite: results line; EPDiagnostics.evidence_nonfinite records first step and kind per term

Changed

  • EPMeanField.log_evidence — explicit float("nan") on any non-finite term, one logger.warning per fit naming it, no numpy RuntimeWarnings; finite path unchanged
  • EPHistory.evidence_divergence — inf when the difference is NaN
  • EPOptimiser — appends check_evidence_nonfinite to the ep_diagnostics.results warnings

Migration

  • none

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

…1713)

On analytic_gaussian seeds 25 and 172 a HierarchicalFactor member sends the
scatter sigma an accepted TruncatedNormalMessage whose support lies ~19 sigma
into its tail; its log_partition underflows to -inf, the variable's evidence
goes +inf, every adjacent factor term -inf, and log_evidence summed them to
NaN with a numpy RuntimeWarning on every EP update. The posterior is
unaffected (bit-identical sidecars); only the evidence readout was.

log_evidence now detects a non-finite term before summing, names it once per
fit (weakly keyed on the factor graph) in a logger warning and in
ep_diagnostics.results, and returns an explicit NaN; evidence_divergence
returns inf on a NaN so no evidence stopping rule can claim convergence.
No message arithmetic, projection or optimiser changes; the tail-stable
log_partition cure is filed separately.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Jammy2211

Copy link
Copy Markdown
Collaborator Author

Workspace regression: PyAutoLabs/autofit_workspace_test#116 · new-scope prompt (message maths, human decision): PyAutoMind draft/bug/autofit/truncated_normal_log_partition_tail_underflow.md

@Jammy2211
Jammy2211 merged commit 919ebd4 into main Oct 23, 2026
7 of 8 checks passed
@Jammy2211
Jammy2211 deleted the feature/ep-evidence-nonfinite branch October 23, 2026 15:25
@pyautolabs-bot pyautolabs-bot Bot removed the pending-release PR queued for the next release build label Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: EP log_evidence sums a non-finite term silently (NaN evidence on analytic_gaussian seeds 25/172)

1 participant