Prediction reports: training-set membership, novelty and conformal intervals (v4.4.0)

[RobbinBouwmeester]

Stacked on #117 (the report reuses MultiHeadRidgeCalibration and the uses_all_heads flag); the diff shown here is the report feature only. Retarget to main once #117 is merged.

What

prediction_report(psm_list, psm_list_reference=..., coverage=0.90, training_index=...) returns predictions together with what a bare number cannot say, one row per PSM:

column	meaning
predicted_rt	calibrated prediction, as predict_and_calibrate
ci_lower, ci_upper	conformal interval at the requested coverage
in_reference	exact peptidoform match in the calibration reference
dist_to_reference	Levenshtein distance to the closest reference sequence
in_training *	exact match anywhere in the 10,105,640-peptidoform corpus behind the bundled multitask model
in_selected_heads_training *	match within the training sets of the setups the calibration selected (all 80 for the multi-head calibration, the single best head otherwise)
dist_to_training *	distance to the closest training sequence, exact to 10 edits, capped beyond

* with a TrainingIndex, a ~400 MB memory-mapped artifact (sorted key hashes, per-setup membership CSR, 6.16 M unique sequences) built offline from the training cache by build_training_index.py in the research repo. The report works without it. Hosting to be decided (HF hub?).

The interval

Cross-fitted split-conformal on the reference: five folds, each predicted by a calibration fitted on the other four, half-width = finite-sample quantile of those honest |residuals| per predicted-RT bin (peak width varies along a gradient). No retraining, works for any model and calibration, finite-sample marginal coverage guarantee. Chosen over quantile regression (pinball) for exactly those reasons.

Validated on the eight held-out PRIDE setups (Figure 1b corpus), nominal 90 %:

• coverage per setup 0.88-0.97 (median 0.91); at nominal 80 %: median 0.80; at 95 %: median 0.96;
• width 4-10 % of the gradient on well-behaved setups, honestly wide where predictions are bad (35 % on the wheat-gluten out-of-domain setup, 79 % on the run that pools fractions);
• distance-scaled widths were tested and rejected: error is flat in edit distance (Figure 2), so they only overcover.

End-to-end through this API on three held-out setups: coverage@90 = 0.914 / 0.898 / 0.974, and the membership columns behave (PXD079927: 60 % of test peptidoforms in the corpus but only 1.2 % in the 80 selected setups; PXD081880: 100 % / 95 %).

Canonical keys

canonical_peptidoform_key reproduces the corpus key format (peprec positions, U:<unimod id>, lowercased unmapped names, position-sorted, charge ignored) — verified against real corpus keys, including modified ones.

Verification

• 155 tests pass; 14 new in tests/test_report.py (key format, interval coverage / adaptivity / thin-bin fallback on synthetic data, index membership global and per-setup, capped distances, non-index directory refused, report end-to-end with and without index, pre-fitted calibration refused, multi-head head listing).
• ruff check / format clean on the new files.
• New dependency: rapidfuzz (pure wheel).
• Version 4.4.0, CHANGELOG and usage docs updated.

🤖 Generated with Claude Code

[RobbinBouwmeester]

Index size addressed in 39b621e: the artifact is now a single 105 MB .dlcidx file instead of a 400 MB directory. Stdlib LZMA zip; the interesting part is the hashes — random bits compress 1.4x at best, so they are truncated to 40 bits in a bucketed layout instead (2^24 bucket counts + 16-bit remainders, ~26 MB), costing a false positive about once per 100,000 membership queries, which a provenance flag can afford. Setup lists 2.8x, sorted sequences 3.1x, CSR pointers stored as row lengths 5.3x. Loads in about a second, answers bit-identical to the raw form, and every index test now runs against both formats.