Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,8 @@ uv build
Tests inject publication/restoration failures, launch a child from a worker thread, check explicit
algorithm provenance, reject missing/mixed/overlapping label inputs, and exercise all six lifecycles with
synthetic command implementations. Full-lifecycle conformance reruns algorithm adapters with forbidden
outcomes removed and fresh artifacts; algorithm-specific cutoff, frozen-weight, compatibility and primary
outcomes removed, and again with every evaluated future rewritten, with fresh artifacts; a null-world run checks
chance-level discrimination of the future; algorithm-specific cutoff, frozen-weight, compatibility and primary
behavior tests remain required. Package tests do not establish scientific validity of a downstream algorithm.

MEDS-DEV provisioning requires explicit `-m meds_dev` or `-m real_data` selection; setting
Expand Down
2 changes: 1 addition & 1 deletion docs/testing.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Tests and acceptance

The following catalogues separate template checks from algorithm acceptance checks.
The [shared acceptance inventory](reference/tests.md#downstream-acceptance-suite) describes the eight
The [shared acceptance inventory](reference/tests.md#downstream-acceptance-suite) describes the nine
inherited checks. Generated repositories keep only fixture wiring, execution adapters and scientific
tests. See also [Shared-package tests](reference/tests.md) for tests of the runtime and suite itself.

Expand Down
20 changes: 17 additions & 3 deletions src/meds_algorithm/CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,19 @@ dry-run, and its real end-to-end chain. The rendered `tests/test_end_to_end.py`
the integer `seed` override, seed every RNG/framework/subprocess and use deterministic fixture settings.
`attach_ground_truth` returns predictions unchanged on an unlabelled index. This tests outcome
independence on the exercised fixture, not every possible code path.
- **Temporal cutoff (future perturbation):** the complete lifecycle runs again on a copy of the MEDS
dataset in which every evaluated subject's events after its latest indexed prediction time are deleted
and replaced by plain events copied from the same shard at log-uniform offsets, many within seconds of
`T` (`conformance.temporal.perturb_future`). Histories at every indexed time, including the row order
of ties, are unchanged, so probabilities MUST be bit-identical by key. Any path from the future to a
prediction fails this check, even one the fitted model barely uses; no statistical threshold is
involved.
- **Temporal cutoff (null world):** for non-zero-shot profiles, the full lifecycle runs on
`conformance.temporal.build_null_world_dataset`, where events follow one homogeneous Poisson process and
the label ("any event in `(T, T + ln 2 / rate]`") is independent of everything recorded up to `T`. A
one-sided Mann-Whitney test on one key per held-out subject MUST NOT reject chance at `alpha = 1e-3`.
This is a smoke alarm for leaks that training learned to exploit; the perturbation check is the gate.
`run_e2e` MUST accept this dataset (`dataset.json` names it `null_world`) with its test settings.
- **Prediction validity:** validate every Parquet shard in both runs, including unique/non-null keys and
finite probabilities in [0, 1]. No particular shard filename is required.
- **Transductive fitting:** every tuning and held-out subject carries a canary code that occurs in no
Expand Down Expand Up @@ -566,6 +579,7 @@ payloads. Implement these algorithm-specific assertions rather than relying only
The shared `assert_profile_run` conformance helper reruns the full lifecycle through `run_e2e` with
forbidden label columns removed. It requires fresh artifacts below a new run directory and preserves
permitted training/context labels. Include `seed`, `external_meds_dir` and all task inputs in `E2EResult.inputs`
so the check uses the same histories and task. This detects forbidden reads on the exercised path;
it does not prove all possible code paths, temporal cutoffs, or absence of labels encoded in other
features. Implement the corresponding algorithm-specific checks as well.
so the check uses the same histories and task. The same helper reruns the lifecycle with every evaluated
subject's future rewritten (`assert_future_blind_pipeline`). These detect forbidden reads and future
information on the exercised path; they do not prove all possible code paths or absence of labels encoded
in other features. Implement the corresponding algorithm-specific checks as well.
18 changes: 16 additions & 2 deletions src/meds_algorithm/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ local processes and temporary files. The default suite does not opt in to MEDS-D
Passing establishes the checked software boundaries, not clinical validity, numerical equivalence to a
paper, GPU correctness or absence of all leakage. Algorithm repositories must supply scientific tests,
cutoff/frozen-weight checks and end-to-end adapters for their concrete implementations. The label-blind
rerun is an adversarial check, not a filesystem sandbox or proof against malicious code.
and future-blind reruns are adversarial checks, not a filesystem sandbox or proof against malicious code.

## Downstream acceptance suite

Expand All @@ -49,7 +49,8 @@ in the generated `tests/e2e.py` and the shared `E2EResult`/`MEDSDevRun` types.
|---|---|---|
| `test_profile_implementations_match_contract` | Compares each registered profile's commands with the core lifecycle. | Detect incorrect registration or dispatch wiring. |
| `test_algorithm_configuration_roundtrips` | Saves and reloads each supplied concrete configuration. | Preserve settings across independent processes, including required constructor fields. |
| `test_end_to_end` | Runs each implemented profile and checks artifacts, manifests, coverage, fresh label-blind reruns, split canaries and zero-shot task binding. | Detect broken persistence, label leakage and transductive fitting through the real lifecycle. |
| `test_end_to_end` | Runs each implemented profile and checks artifacts, manifests, coverage, fresh label-blind reruns, a fresh rerun with every evaluated subject's events after its latest prediction time rewritten (predictions must be bit-identical), split canaries and zero-shot task binding. | Detect broken persistence, label leakage, information from after the prediction time and transductive fitting through the real lifecycle. |
| `test_null_world_predictions_are_at_chance` | Runs each completed non-zero-shot profile on a null world (homogeneous Poisson events; the label, any event in `(T, T + ln 2 / rate]`, is independent of the history) and applies a one-sided Mann-Whitney test, one key per held-out subject, at `alpha = 1e-3`. | End-to-end smoke alarm for temporal leaks that training learned to exploit, such as a forward time gap to the next real event. `run_e2e` must accept the `null_world` dataset with its test settings. |
| `test_completed_algorithm_has_a_completed_report` | Validates report evidence for every completed profile. | Prevent incomplete profiles from masking unsupported completion claims. |
| `test_algorithm_yaml_is_a_meds_dev_descriptor` | Checks each profile descriptor's required structure. | Catch packaging errors before integration. |
| `test_slurm_dry_run_is_non_mutating` | Runs every profile's launcher with example inputs, checks the file inventory is unchanged and all commands are listed. | Inspect execution plans without submitting work; this checks file creation/removal, not changes to existing file contents. |
Expand Down Expand Up @@ -123,6 +124,19 @@ canonical and is rendered in the template documentation rather than copied into
| `test_nn_config_requires_explicit_training_settings` | Rejects missing learning rate/batch size and checks classical config independence. | Do not invent universal training defaults or require neural settings for classical algorithms. |
| `test_repository_records_nn_settings_from_config_file` | Dispatches with a file override and checks resolved saved config and manifest. | Ensure experiment settings actually reach the algorithm and persisted provenance. |

### test_temporal.py

[Source](https://github.com/McDermottHealthAI/meds-algorithm/blob/main/tests/test_temporal.py)

| Test | What it checks | Why it exists |
|---|---|---|
| `test_null_world_shape` | Builds a null world and checks one key per held-out subject, two per training subject, prevalence near 0.5 and records that run past the label horizon. | Keep held-out keys independent and record ends uninformative, so the chance-level test is exact. |
| `test_causal_scores_are_at_chance_and_leaky_scores_are_caught` | Scores keys by hours since the last event (causal) and by hours to the next event (a NOAH-style forward-gap leak). | Show that the chance-level check passes a causal score and fails a leaky one. |
| `test_perturbation_keeps_history_and_rewrites_only_indexed_futures` | Compares histories up to every indexed time, the rewritten future, the vocabulary and untouched training subjects. | Guarantee the future-blind rerun changes nothing an honest algorithm may read. |
| `test_future_blind_rerun_passes_causal_and_fails_leaky` | Runs the future-blind rerun on the causal and the leaky toy algorithm. | Show that any dependence on the future fails without a statistical threshold. |
| `test_mann_whitney_is_calibrated_and_handles_ties` | Checks the rejection rate under the null with heavy ties, constant scores, perfect separation and one-class input. | Keep the chance-level test's false-positive rate at `alpha`. |
| `test_perturbation_keeps_tie_order_and_never_copies_objects` | Rewrites a future next to ties at one time and an object-link row. | MEDS orders ties by row and objects are single acquisitions; reordering or duplicating either would fail honest algorithms. |

### test_conformance.py

[Source](https://github.com/McDermottHealthAI/meds-algorithm/blob/main/tests/test_conformance.py)
Expand Down
31 changes: 9 additions & 22 deletions src/meds_algorithm/conformance/end_to_end.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
)

from .assertions import assert_prediction_keys, read_predictions
from .temporal import assert_future_blind_pipeline, assert_same_predictions


def _strip_labels(labels_dir: Path, destination: Path, splits) -> Path:
Expand Down Expand Up @@ -64,6 +65,9 @@ def assert_profile_run(result, *, profile, run_e2e, fitted_codes, tmp_path):
assert_prediction_keys(artifacts[prediction_step], result.labels_dir, result.splits)

assert_label_blind_pipeline(result, profile=profile, run_e2e=run_e2e, tmp_path=tmp_path)
# Temporal cutoff (CONTRACT section 8): rewriting every evaluated subject's events after its latest
# prediction time must leave every prediction unchanged.
assert_future_blind_pipeline(result, profile=profile, run_e2e=run_e2e, tmp_path=tmp_path)

# Absence of transductive fitting (SPEC 2.1): codes that occur only in tuning or held-out subjects must
# not be part of anything the algorithm fitted.
Expand Down Expand Up @@ -124,31 +128,14 @@ def assert_label_blind_pipeline(result, *, profile, run_e2e, tmp_path):
prediction = blind.artifacts[expected_steps[-1]]
assert_prediction_keys(prediction, result.labels_dir, result.splits)
# Do not require the evaluation-only boolean_value on a deliberately unlabelled run.
import polars as pl

from ..data.index import read_index
from ..evaluation.predictions import check_decoded

blind_predictions = read_predictions(prediction, require_truth=False)
check_decoded(read_index(result.labels_dir, result.splits), blind_predictions)
keys = ["subject_id", "prediction_time"]
probability = "predicted_boolean_probability"
decisions = "predicted_boolean_value" in original.columns
columns = [*keys, probability, *(["predicted_boolean_value"] if decisions else [])]
# Explicit comparison instead of polars.testing.assert_frame_equal, whose tolerance keywords were
# renamed (rtol/atol -> rel_tol/abs_tol) within the supported polars>=1 range.
try:
a = original.select(columns).sort(keys)
b = blind_predictions.select(columns).sort(keys)
except pl.exceptions.ColumnNotFoundError as error:
raise AssertionError("Label-blind predictions lack a column of the original run") from error
same = (
a.select(keys).equals(b.select(keys))
and ((a[probability] - b[probability]).abs() <= 1e-8 + 1e-6 * b[probability].abs()).all()
and (not decisions or a["predicted_boolean_value"].equals(b["predicted_boolean_value"]))
assert_same_predictions(
original,
blind_predictions,
message="Predictions changed when forbidden labels were removed; fix leakage or make the "
"conformance fixture reproducible with the recorded seed",
)
if not same:
raise AssertionError(
"Predictions changed when forbidden labels were removed; fix leakage or make the "
"conformance fixture reproducible with the recorded seed"
)
20 changes: 19 additions & 1 deletion src/meds_algorithm/conformance/suite.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,11 @@
from .cases import AlgorithmCase
from .end_to_end import assert_profile_run
from .reports import assert_implementation_report
from .temporal import assert_chance_level, build_null_world_dataset


class AlgorithmConformanceTests:
"""Eight shared acceptance checks, executed against the downstream algorithm."""
"""Nine shared acceptance checks, executed against the downstream algorithm."""

@pytest.fixture
def completed_profile(self, algorithm_case: AlgorithmCase, profile):
Expand Down Expand Up @@ -61,6 +62,23 @@ def test_end_to_end(self, algorithm_case: AlgorithmCase, completed_profile, tmp_
tmp_path=tmp_path,
)

def test_null_world_predictions_are_at_chance(
self, algorithm_case: AlgorithmCase, completed_profile, tmp_path
):
"""Temporal cutoff: with a future independent of the past, held-out AUROC must be at chance."""
if completed_profile.startswith("zero_shot_"):
pytest.skip("null world: zero-shot profiles need a task; the future-perturbation rerun covers it")
world = build_null_world_dataset(tmp_path / "null-world", seed=0)
result = algorithm_case.run_e2e(
tmp_path / "null-world-run",
profile=completed_profile,
seed=0,
external_meds_dir=world.meds_dir,
external_labels_dir=world.labels_dir,
splits=["held_out"],
)
assert_chance_level(result, alpha=1e-3)

def test_completed_algorithm_has_a_completed_report(self, algorithm_case: AlgorithmCase):
report = (algorithm_case.root / "IMPLEMENTATION_REPORT.md").read_text(encoding="utf-8")
assert_implementation_report(report, algorithm_case.profiles, algorithm_case.stub_profiles)
Expand Down
Loading
Loading