Repository navigation
Temporal-leakage conformance: future-blind rerun and null-world chance test - #21
Draft
florian6973 wants to merge 1 commit into
Draft
florian6973 wants to merge 1 commit into
florian6973 wants to merge 1 commit into
Conversation
…hance test
The shared suite checked label leakage and transductive fitting but had no
check that information from after a prediction time cannot reach the
prediction (e.g. NOAH's forward time gap to the next event).
- conformance/temporal.py:
- perturb_future: rewrite every evaluated subject's events after its latest
indexed prediction time (log-uniform offsets, plain rows only, tie order kept)
- assert_future_blind_pipeline: rerun the lifecycle on the perturbed data;
predictions must be bit-identical (no statistical threshold)
- build_null_world_dataset: homogeneous Poisson world where the label
"any event in (T, T + ln2/rate]" is independent of the history
- null_auroc_test / assert_chance_level: one-sided Mann-Whitney, one key per
held-out subject, exact false-positive rate alpha
- assert_profile_run now runs the future-blind rerun (test_end_to_end)
- new inherited test_null_world_predictions_are_at_chance (non-zero-shot)
- CONTRACT/TESTING/README updated; suite test counts 38 -> 44
Validated against NOAH (rvandewater/Multi_MEDS) on its own fixture: clean
passes; an injected forward-gap leak and a vitals-bin leak both fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The shared suite checks label leakage and transductive fitting, but nothing checks that information from after a prediction time cannot reach the prediction. One example is NOAH's forward time gap to the next event, if gaps are derived before cutting the history at T.
What
New
conformance/temporal.pywith two black-box checks:Future-blind rerun (deterministic gate).
perturb_futurerewrites every evaluated subject's events after its latest indexed prediction time:assert_future_blind_pipelinereruns the lifecycle on that copy and requires bit-identical predictions. It now runs insideassert_profile_run(test_end_to_end).Null world (statistical alarm).
build_null_world_datasetuses a homogeneous Poisson process, so the label "any event in (T, T + ln2/rate]" is independent of the history.assert_chance_levelapplies a one-sided Mann-Whitney test with one key per held-out subject at alpha = 1e-3. It runs as the new inherited testtest_null_world_predictions_are_at_chance, skipped for zero-shot profiles.Also in this PR:
assert_same_predictions.Evidence (NOAH, rvandewater/Multi_MEDS, supervised profile)
The leaks were injected into NOAH for this experiment. Null world, 600 held-out keys, 5 seeds:
The shared future-blind check on NOAH's own multimodal fixture:
Downstream impact
After bumping the pin,
run_e2emust accept thenull_worlddataset (dataset.jsonname) with small offline test settings. Companion template PR: McDermottHealthAI/MEDS_algorithm_template, branchtemporal-leakage-conformance.Tests
tests/test_temporal.py.test_registration_is_per_profile_and_rejects_escapefails on Windows, also onmain(afile://path format issue).🤖 Generated with Claude Code