From fadf867a37fa44895635dc1c80c8182dbf1c2cff Mon Sep 17 00:00:00 2001 From: Florent Pollet Date: Mon, 16 Nov 2026 00:23:08 -0500 Subject: [PATCH] Document the shared temporal-leakage checks in the run_e2e contract and testing guide run_e2e is rerun on a copy with every evaluated future rewritten (bit-identical predictions required) and called with a null_world dataset (chance-level AUROC required); both come from meds_algorithm.conformance.temporal. Co-Authored-By: Claude Opus 5.5 --- template/docs/TESTING.md | 5 ++++- template/tests/e2e.py | 7 +++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/template/docs/TESTING.md b/template/docs/TESTING.md index 2f672e9..0f62133 100644 --- a/template/docs/TESTING.md +++ b/template/docs/TESTING.md @@ -58,7 +58,10 @@ also requires a committed lockfile and a tag on main history. See the generated ## Algorithm-specific tests to add -Add source-backed tests for numerical behavior, temporal cutoffs, feature selection, frozen pretrained +The shared suite checks temporal cutoffs from outside: a rerun with every evaluated future rewritten must +reproduce predictions bit for bit, and a null world (future independent of the past) must give chance-level +held-out AUROC. These see only predictions on the exercised fixture. Add source-backed tests for numerical +behavior, internal temporal cutoffs (for example, that inputs and states at T are unchanged by events after T), feature selection, frozen pretrained weights where applicable, payload compatibility, configuration effects and fresh-process reconstruction. Neural configuration tests should prove that capacity, learning rate, batch size, weight decay and dropout control the intended code. The generated config round-trip test alone does not prove that. diff --git a/template/tests/e2e.py b/template/tests/e2e.py index 85d30d7..b140fef 100644 --- a/template/tests/e2e.py +++ b/template/tests/e2e.py @@ -14,6 +14,13 @@ def run_e2e(tmp_path, *, profile: str, **inputs): Conformance reruns this hook with evaluated outcomes removed. Reuse supplied histories, not an old prepared artifact; emit fresh artifacts below tmp_path. Training/context splits retain their permitted labels. + + Temporal cutoff: conformance also reruns this hook with ``external_meds_dir`` replaced by a copy in which + every evaluated subject's events after its latest prediction time are rewritten + (meds_algorithm.conformance.temporal.perturb_future); predictions must be bit-identical. It also calls + this hook with a null-world dataset (``metadata/dataset.json`` names it ``null_world``; codes + ``NULL//...``) whose future is independent of its past, and requires held-out AUROC at chance. Run that + dataset with the same small, offline test settings as the fixture. """ raise NotImplementedError( "Prepare a small MEDS input/task, execute the selected algorithm profile's real commands, and return "