Skip to content

Document the shared temporal-leakage checks for run_e2e - #52

Draft
florian6973 wants to merge 1 commit into
mainfrom
temporal-leakage-conformance
Draft

florian6973 wants to merge 1 commit into
mainfrom
temporal-leakage-conformance

Conversation

@florian6973

Copy link
Copy Markdown
Collaborator

Companion to McDermottHealthAI/meds-algorithm#21, which adds conformance/temporal.py. That PR adds two checks:

  • a future-blind rerun that requires bit-identical predictions;
  • a null-world test that requires chance-level AUROC.

Changes:

  • The template/tests/e2e.py run_e2e docstring now states both obligations: honour a perturbed external_meds_dir, and run the null_world dataset with small offline test settings.
  • template/docs/TESTING.md explains what the shared checks cover and what algorithm-specific tests are still needed (internal invariance of inputs and states at T).

This PR changes documentation only. The 18 Windows failures in tests/test_render.py also occur on main.

🤖 Generated with Claude Code

…nd testing guide

run_e2e is rerun on a copy with every evaluated future rewritten (bit-identical
predictions required) and called with a null_world dataset (chance-level AUROC
required); both come from meds_algorithm.conformance.temporal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant