Skip to content

Temporal-leakage conformance: future-blind rerun and null-world chance test - #21

Draft
florian6973 wants to merge 1 commit into
mainfrom
temporal-leakage-conformance
Draft

florian6973 wants to merge 1 commit into
mainfrom
temporal-leakage-conformance

Conversation

@florian6973

Copy link
Copy Markdown
Collaborator

Why

The shared suite checks label leakage and transductive fitting, but nothing checks that information from after a prediction time cannot reach the prediction. One example is NOAH's forward time gap to the next event, if gaps are derived before cutting the history at T.

What

New conformance/temporal.py with two black-box checks:

  • Future-blind rerun (deterministic gate). perturb_future rewrites every evaluated subject's events after its latest indexed prediction time:

    • new events are placed at log-uniform offsets, many within seconds of T;
    • only plain rows are copied (no object links);
    • the row order of tied events is preserved.

    assert_future_blind_pipeline reruns the lifecycle on that copy and requires bit-identical predictions. It now runs inside assert_profile_run (test_end_to_end).

  • Null world (statistical alarm). build_null_world_dataset uses a homogeneous Poisson process, so the label "any event in (T, T + ln2/rate]" is independent of the history. assert_chance_level applies a one-sided Mann-Whitney test with one key per held-out subject at alpha = 1e-3. It runs as the new inherited test test_null_world_predictions_are_at_chance, skipped for zero-shot profiles.

Also in this PR:

  • The label-blind comparison is factored into assert_same_predictions.
  • CONTRACT, TESTING and README are updated.
  • The suite's collected-test count goes from 38 to 44.

Evidence (NOAH, rvandewater/Multi_MEDS, supervised profile)

The leaks were injected into NOAH for this experiment. Null world, 600 held-out keys, 5 seeds:

variant AUROC, 8 epochs AUROC, 30 epochs keys changed by the future rewrite
clean 0.49–0.55 0.49–0.54 0/600 (bit-identical)
next-event-gap leak 0.49–0.67 0.74–0.81 (p ≤ 1e-24) 600/600
vitals-bin leak 0.53–0.62 0.53–0.64 ~120/600

The shared future-blind check on NOAH's own multimodal fixture:

  • the clean code passes;
  • the gap leak fails (7 of 12 keys move);
  • the bin leak fails (8 of 12 keys move).

Downstream impact

After bumping the pin, run_e2e must accept the null_world dataset (dataset.json name) with small offline test settings. Companion template PR: McDermottHealthAI/MEDS_algorithm_template, branch temporal-leakage-conformance.

Tests

  • 6 new tests in tests/test_temporal.py.
  • Full suite: 154 passed.
  • test_registration_is_per_profile_and_rejects_escape fails on Windows, also on main (a file:// path format issue).

🤖 Generated with Claude Code

…hance test

The shared suite checked label leakage and transductive fitting but had no
check that information from after a prediction time cannot reach the
prediction (e.g. NOAH's forward time gap to the next event).

- conformance/temporal.py:
  - perturb_future: rewrite every evaluated subject's events after its latest
    indexed prediction time (log-uniform offsets, plain rows only, tie order kept)
  - assert_future_blind_pipeline: rerun the lifecycle on the perturbed data;
    predictions must be bit-identical (no statistical threshold)
  - build_null_world_dataset: homogeneous Poisson world where the label
    "any event in (T, T + ln2/rate]" is independent of the history
  - null_auroc_test / assert_chance_level: one-sided Mann-Whitney, one key per
    held-out subject, exact false-positive rate alpha
- assert_profile_run now runs the future-blind rerun (test_end_to_end)
- new inherited test_null_world_predictions_are_at_chance (non-zero-shot)
- CONTRACT/TESTING/README updated; suite test counts 38 -> 44

Validated against NOAH (rvandewater/Multi_MEDS) on its own fixture: clean
passes; an injected forward-gap leak and a vitals-bin leak both fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant