Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion template/docs/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,10 @@ also requires a committed lockfile and a tag on main history. See the generated

## Algorithm-specific tests to add

Add source-backed tests for numerical behavior, temporal cutoffs, feature selection, frozen pretrained
The shared suite checks temporal cutoffs from outside: a rerun with every evaluated future rewritten must
reproduce predictions bit for bit, and a null world (future independent of the past) must give chance-level
held-out AUROC. These see only predictions on the exercised fixture. Add source-backed tests for numerical
behavior, internal temporal cutoffs (for example, that inputs and states at T are unchanged by events after T), feature selection, frozen pretrained
weights where applicable, payload compatibility, configuration effects and fresh-process reconstruction.
Neural configuration tests should prove that capacity, learning rate, batch size, weight decay and dropout
control the intended code. The generated config round-trip test alone does not prove that.
Expand Down
7 changes: 7 additions & 0 deletions template/tests/e2e.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ def run_e2e(tmp_path, *, profile: str, **inputs):
Conformance reruns
this hook with evaluated outcomes removed. Reuse supplied histories, not an old prepared artifact;
emit fresh artifacts below tmp_path. Training/context splits retain their permitted labels.

Temporal cutoff: conformance also reruns this hook with ``external_meds_dir`` replaced by a copy in which
every evaluated subject's events after its latest prediction time are rewritten
(meds_algorithm.conformance.temporal.perturb_future); predictions must be bit-identical. It also calls
this hook with a null-world dataset (``metadata/dataset.json`` names it ``null_world``; codes
``NULL//...``) whose future is independent of its past, and requires held-out AUROC at chance. Run that
dataset with the same small, offline test settings as the fixture.
"""
raise NotImplementedError(
"Prepare a small MEDS input/task, execute the selected algorithm profile's real commands, and return "
Expand Down
Loading