[fbonc]

Adds EnbPI from Xu and Xie, "Conformal prediction interval for dynamic time-series", ICML 2021, as the first time series conformal method in pyhealth.calib. It builds prediction intervals for time series regression and updates them online as labels are revealed, without assuming exchangeability. ACI and AgACI will follow on the same interface.

Changes

• pyhealth.calib.predictionset.EnbPI trains a bootstrap ensemble from a model_fn factory, fills a sliding window with out-of-bag absolute residuals, and returns the center plus or minus the conformal (1 - alpha) quantile of the window. reset() restores the calibration window, and state_dict() carries the full calibration state for checkpoints.
• IntervalPredictor in pyhealth/calib/base_classes.py declares calibrate, forward and update, like SetPredictor.
• pyhealth/metrics/prediction_interval.py adds miscoverage_pi, interval_width and rolling_miscoverage_pi@W, available through regression_metrics_fn(..., y_lower=..., y_upper=...).
• API docs, a README section, examples/conformal_timeseries/enbpi_synthetic.py and tests.

Key decisions

• The default center is phi over the out-of-bag predictions, as in the paper's experiments and MAPIE. center="quantile" reproduces Algorithm 1 line 12, whose (1 - alpha) quantile biases the point prediction upward.
• EnbPI trains its own ensemble from a model_fn factory, following MAPIE's clone-per-bootstrap convention, so the bootstrap bags always match the models. No calibrator in PyHealth trained models before, so there was no in-repo convention.
• IntervalPredictor fixes only the public interface. Each method implements its own online loop, because ACI and AgACI update in ways a shared loop would constrain.
• forward is stateless and works inside Trainer.inference, and update slides the window. predict_sequential is a convenience method for evaluation that runs the paper's loop over a labeled test stream.
• Every calibrate call trains a new ensemble on the data passed in, like other calibrators recompute from their data. reset() restores the calibration window without retraining, so evaluations can be rerun from the same start.
• state_dict() saves the whole predictor, including the bootstrap indices and the residual window, through PyTorch's extra state hooks, so load_state_dict and Trainer checkpoints restore calibrated inference and resume online updates. The hooks sit in a separate checkpointing section at the end of EnbPI, apart from the algorithm.
• calibrate raises when no training point has an out-of-bag model, since every interval would be NaN. The paper does not address this case, and with the recommended B = 20 to 30 it essentially never happens.
• Scope is regression mode with one stream in time order and absolute residuals, as in the ICML paper. Many series at once (timeseries_regression) and signed residuals are follow-ups.

[github-actions]

Test Coverage

Total: 74.27% (20594 / 27729 lines)

Modified file	Coverage	Lines
pyhealth/calib/base_classes.py	75.00%	18 / 24
pyhealth/calib/predictionset/__init__.py	100.00%	10 / 10
pyhealth/calib/predictionset/timeseries/__init__.py	100.00%	3 / 3
pyhealth/calib/predictionset/timeseries/enbpi.py	100.00%	188 / 188
pyhealth/calib/predictionset/timeseries/utils.py	100.00%	22 / 22
pyhealth/metrics/prediction_interval.py	100.00%	20 / 20
pyhealth/metrics/regression.py	90.20%	46 / 51

Workflow run · d68b819

[Logiquo]

/review

[github-actions]

_{Reviewed commit d68b819}

[github-actions]

[medium] conformal_quantile duplicates existing _query_quantile

The split-conformal quantile (ceil((1-alpha)(n+1))-th smallest score, +inf when the rank exceeds n) is already implemented in pyhealth/calib/predictionset/base_conformal/init.py:43 as _query_quantile, which the label, cluster and other conformal methods use. This new helper repeats the same formula. Its differences (presorted fast path, alpha outside [0,1] returning -inf, a 1e-9 float tolerance) are not used by the existing helper, but they are not offered as a reason for a separate function either. Fix: extend _query_quantile with the presorted and out-of-range alpha behavior, or call it from EnbPI and document why the new edge cases differ from the set predictors.

[github-actions]

[low] Subset deep-copies the full dataset once per bootstrap model

_train_models calls train_dataset.subset(...) for each of the n_bootstrap models. SampleDataset.subset (pyhealth/datasets/sample_dataset.py:857-868) runs copy.deepcopy(self) before replacing _data, so each bag copies the entire dataset. Calibration cost therefore grows with B times the dataset size before any training. Fix: build the bag datasets from one shared copy of the data, or pass the index list to a loader without copying.

[github-actions]

[low] Interval metrics silently skipped when only one bound is given

has_intervals requires both y_lower and y_upper. If a caller passes only one bound, miscoverage_pi and interval_width are dropped without an error, and the result simply lacks those keys. The docstring documents this, but the mistake is easy to miss. Fix: raise ValueError when exactly one of y_lower or y_upper is given.

[github-actions]

[low] Return annotation no longer matches rolling metric output

The annotation is Dict[str, float], but rolling_miscoverage_pi@​W returns a numpy array, as the docstring says. Fix: change the annotation to Dict[str, float | np.ndarray] or a similar union.

[github-actions]

[low] Trainer.inference compatibility is claimed but not tested

The forward docstring says it works inside Trainer.inference. The tests call forward directly and never run it through Trainer.inference, which would check the loss and y_true handling and the output keys. Fix: add one test that runs Trainer.inference on a small calibrated EnbPI and checks the returned keys and shapes.

[github-actions]

AI Review

suggestion: approve
decision: human review

Thank you for this contribution. It adds EnbPI as the first time series conformal interval method in pyhealth.calib, with an IntervalPredictor interface, interval metrics reachable through regression_metrics_fn, docs, an example and thorough tests. The implementation looks correct and matches its description. The tests use hand-computed values, checkpoint round trips and an end-to-end coverage check.

Coverage meets the bar. The whole repository is at 74.27%, above the 60% bar. The new files are at 100%. regression.py is at 90.2%. base_classes.py is at 75%, slightly under 80%. Its uncovered lines are the abstract-style ... bodies of the new interface, so there is little to test there.

A maintainer will take a look before this is decided. The change adds IntervalPredictor, a new base class in pyhealth/calib/base_classes.py that ACI and AgACI are planned to subclass. It also extends the shared regression_metrics_fn with optional parameters. EnbPI's checkpoint format relies on PyTorch's extra state hooks. Existing calls look unaffected. The maintainer will still want to check the new base-class contract, since it is the core part of the codebase.

Optional suggestions:

• conformal_quantile overlaps with the existing _query_quantile helper in base_conformal. You could reuse or extend that helper, or note briefly why a separate one is needed. The differences are the presorted fast path, alpha outside [0, 1] and the float tolerance.
• SampleDataset.subset may deep-copy the dataset for each bootstrap model. Check whether that is costly for large datasets.
• regression_metrics_fn silently skips interval metrics when only one of y_lower or y_upper is passed. Raising an error would be clearer. Its return annotation also no longer matches the array-valued rolling metric.
• A test that runs forward through Trainer.inference would back up the claim in the docs.

_{Reviewed commit d68b819}