From 2aa041ba382a79ca682a9a81ca871f6124c67c6b Mon Sep 17 00:00:00 2001 From: Elmehdi Aitbrahim Date: Mon, 7 Sep 2026 06:43:54 -0400 Subject: [PATCH] feat(web): the Evidence Matrix, read rather than computed (#708 view 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last of #708's four views, and the one that waited for an engine record instead of being built on a guess. WHY IT IS A READ Building the matrix on request costs 11.9 / 12.9 / 14.3 seconds per session on the real ledger -- ~39 s of CPU for three sessions, on a page the console re-polls every 15 seconds. And over the ledger as a whole it does not run at all: `build_matrix` requires synchronous columns, so a PBO is only ever defined WITHIN a session whose trials share a bar count, and a page cannot pick that scope without inventing an operator's decision. So #726 made `trials pbo` record every field of its `PBOResult`, and this reads them. The console displays results an operator ran and never runs one on their behalf -- pinned by a test that makes `build_matrix` raise if it is called. AN UNRUN MATRIX IS NOT AN EMPTY ONE Three states, and the middle one is why the report carries `candidate_sessions` at all: no ledger, a ledger with columns and no recorded run, and a ledger with runs. The middle one is the state this ships in, and it is the only one that can be acted on. THE GUIDANCE NAMES A SESSION THAT ACTUALLY HAS COLUMNS `keel trials pbo --session all` is the obvious thing to print and would filter to a session literally called "all", find nothing, and print a refusal -- teaching an operator that the page does not know what it is talking about. So the report counts the trials `build_matrix` would ACCEPT (a per-bar series, not `series_missing`) and names one of those. On the tracked ledger that is `pbo-grid-entry-lookback-2026-07-20`, one of three genuine candidates. It deliberately does not check SYNCHRONICITY, which would mean reading every series -- most of the cost this module exists to avoid. Suggesting a session that might turn out ragged, and getting a clear refusal from the command, is a much smaller harm than a page that costs twelve seconds to render. PBO CARRIES NO JUDGEMENT It is the one figure a reader will want graded and grading it is exactly what the rail refuses: a high PBO beside a flat, positive OOS scatter is the GOOD outcome -- a broad plateau of near-identical configurations produces high PBO by construction. `trials pbo`'s own closing sentence says to read it alongside the degradation slope, never alone, and both cross plainly so a reader can. The dominance flags DO carry a state, because they are already verdicts, and they are three-valued: `False` says the distribution did not dominate, which is not what an absent field says. ⛔ No sortable column on the route and no sort key in the view. A matrix ordered by PBO is a leaderboard of overfitting scores, and `cscv.py` forbids PBO as a ranking key in its own source. TWO TESTS THAT WERE NOT TESTING ANYTHING `test_a_recorded_run_stops_telling_the_operator_to_run_one` seeded a ledger with no columns, so `suggested_session` was empty and the invocation was empty whatever the rule said -- it passed against a page that prints the command forever. And the bool-is-an-int guard was pinned through a fixture no writer can produce, so it asserted nothing; it is checked at the reader now, which is the honest way to pin a defensive check. Both found by mutation. So was a stale `__pycache__`: the mutant `MIN_COLUMNS_FOR_A_RUN = 2` -> `1` is the same byte length as the original, so restoring the source within one mtime tick left Python reusing the compiled mutant. Worth knowing for any same-length mutation. Eleven mutants killed: the matrix built on request, pre-#726 gauntlet rows read as matrices, a dominance flag read as a column count, an absent flag read as a denial, a backfilled session suggested, a one-column session suggested, the command still printed after a run, PBO gaining a judgement, the section dropped from the view, the table gaining a sort control, and the route becoming sortable. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01KZZxmspQXe5qJ9FAsG13s6 --- keel/commands/evidence_matrix.py | 202 ++++++++++++++++++++ keel/web/api.py | 23 +++ keel/web/payload.py | 128 +++++++++++++ keel/web/static/js/main.js | 4 +- keel/web/static/js/render.js | 82 ++++++++- tests/commands/test_evidence_matrix.py | 243 +++++++++++++++++++++++++ tests/web/test_api.py | 1 + tests/web/test_client_assets.py | 33 ++++ tests/web/test_research_view.py | 138 +++++++++++++- 9 files changed, 850 insertions(+), 4 deletions(-) create mode 100644 keel/commands/evidence_matrix.py create mode 100644 tests/commands/test_evidence_matrix.py diff --git a/keel/commands/evidence_matrix.py b/keel/commands/evidence_matrix.py new file mode 100644 index 0000000..54b36ff --- /dev/null +++ b/keel/commands/evidence_matrix.py @@ -0,0 +1,202 @@ +"""The Evidence Matrix: every recorded CSCV run, read rather than computed (#708 view 2). + +── WHY THIS IS A READ AND NOT A COMPUTATION ───────────────────────────────────────────────────── + +The obvious implementation is to build the matrix on request. Measured on the real ledger, that +costs 11.9 / 12.9 / 14.3 seconds per session -- roughly 39 seconds of CPU for three sessions, on a +page the console re-polls every 15 seconds. And over the ledger as a WHOLE it does not run at all: + + ValueError: columns are not synchronous: found lengths [1819, 1828]; + §78.6 requires a true matrix with the same rows for every column + +`matrix.build_matrix` requires synchronous columns, so a PBO is only ever defined WITHIN a session +whose trials share a bar count. A page cannot pick that scope for the operator without inventing +their decision. + +So #726 made `trials pbo` record every field of its `PBOResult`, and this reads them. The +distinction is the whole design: **the console displays results an operator ran, and never runs +one on their behalf.** + +── AN UNRUN MATRIX IS NOT AN EMPTY ONE ────────────────────────────────────────────────────────── + +Three states, and the middle one is the reason this module has a `candidate_sessions` field at +all: + +* **no ledger** -- a deployment without the research repository beside it. +* **a ledger with columns and no recorded run** -- the gauntlet has simply not been run here yet, + and the page can name the exact command that would change that. +* **a ledger with recorded runs** -- the matrix. + +The guidance names a session that ACTUALLY HAS COLUMNS. `keel trials pbo --session all` looks like +the obvious thing to suggest and would filter to a session literally named "all", find nothing, and +print a refusal -- teaching an operator that the page does not know what it is talking about. + +⛔ THE STRATHERN RAIL. Every figure here is a diagnostic and none of them is sortable, on the route +or in the view. A matrix ordered by PBO is a leaderboard of overfitting scores, and PBO's own +module carries the warning: it "evaluates the quality of a selection process and must never become +the objective that selection relies on". +""" + +from __future__ import annotations + +from dataclasses import dataclass +from decimal import Decimal +from pathlib import Path +from typing import Any + +from keel.research.ledger import read_trials + +#: The kind #726 writes a recorded CSCV run under. +CSCV_KIND = "cscv" + +#: The command that would populate this page. Composed in Python and placed by the client (the +#: rule #707's cancel modal follows), and it names a REAL session: `--session all` filters to a +#: session literally called "all", finds nothing, and prints a refusal. +MATRIX_INVOCATION = "keel trials pbo --session {session}" + +#: How many synchronous columns a session needs before `trials pbo` can say anything about it. +#: `matrix.build_matrix` warns below `MIN_RECOMMENDED_COLUMNS` and refuses at zero; two is the +#: floor at which a combinatorial split exists at all. +MIN_COLUMNS_FOR_A_RUN = 2 + + +@dataclass(frozen=True) +class MatrixRow: + """One recorded CSCV run -- every field `PBOResult` carries, as it was recorded. + + Absent figures are `None`, never zero. A `pbo` of `0` is the strongest possible statement + about a selection process and a missing one is no statement at all; the six pre-#726 gauntlet + rows carry neither, and this page says so rather than rendering them as perfect. + """ + + trial_id: str + timestamp: int + session: str + pbo: Decimal | None + degradation_slope: Decimal | None + degradation_intercept: Decimal | None + prob_loss: Decimal | None + dominance_1st: bool | None + dominance_2nd: bool | None + n_columns: int | None + n_blocks: int | None + n_combinations: int | None + rows_used: int | None + rows_dropped: int | None + columns_refused: int | None + + +@dataclass(frozen=True) +class MatrixReport: + now_ts: int + ledger_present: bool + rows: tuple[MatrixRow, ...] + #: Sessions whose trials could form a matrix, whether or not one has been run over them. + #: What the empty state names, so its command is one that would actually work. + candidate_sessions: tuple[str, ...] + + @property + def recorded_count(self) -> int: + """Held on the report because `keel/web/payload.py` may not call `len()` (Rule 6e).""" + return len(self.rows) + + @property + def any_recorded(self) -> bool: + return bool(self.rows) + + @property + def suggested_session(self) -> str: + """The session the empty state tells an operator to run against, or `""` when none could. + + The FIRST candidate rather than a chosen one: choosing would be this page ranking sessions + by something, and there is nothing here it may rank by. + """ + return self.candidate_sessions[0] if self.candidate_sessions else "" + + +def _decimal_or_none(summary: dict[str, Any], key: str) -> Decimal | None: + value = summary.get(key) + return value if isinstance(value, Decimal) else None + + +def _int_or_none(summary: dict[str, Any], key: str) -> int | None: + value = summary.get(key) + # `bool` is an `int` in Python and is never one of these counts. Checked first, because + # `isinstance(True, int)` would otherwise render a dominance flag as a column count. + if isinstance(value, bool): + return None + return value if isinstance(value, int) else None + + +def _flag_or_none(summary: dict[str, Any], key: str) -> bool | None: + """THREE-VALUED. `bool(None)` is `False`, and `False` on a dominance flag is a positive claim + -- "the in-sample distribution did not dominate" -- which is not what an absent field says.""" + value = summary.get(key) + if isinstance(value, bool): + return value + if isinstance(value, int): + return bool(value) + return None + + +def _candidate_sessions(trials: list[Any]) -> tuple[str, ...]: + """Sessions holding enough usable columns for `trials pbo` to run over them. + + Counts the trials `matrix.build_matrix` would ACCEPT -- a per-bar series, not `series_missing` + -- rather than every trial with the session label, because a session of six backfilled rows + would otherwise be suggested and the suggested command would refuse. + + It does NOT check synchronicity. Doing so means reading every series, which is most of the + cost this module exists to avoid, and a session whose columns turn out to be ragged gets a + clear refusal from the command itself. Suggesting a session that might not work is a much + smaller harm than a page that costs 12 seconds to render. + """ + usable: dict[str, int] = {} + for trial in trials: + if trial.series_missing or not trial.per_bar_pnl: + continue + usable[trial.session] = usable.get(trial.session, 0) + 1 + return tuple( + session for session, count in usable.items() if count >= MIN_COLUMNS_FOR_A_RUN + ) + + +def gather_matrix(path: Path | str, *, now_ts: int) -> MatrixReport: + """Every recorded CSCV run in the ledger at `path`, oldest first. No computation.""" + ledger = Path(path) + if not ledger.exists(): + return MatrixReport( + now_ts=now_ts, ledger_present=False, rows=(), candidate_sessions=() + ) + + trials = list(read_trials(ledger)) + rows = tuple( + MatrixRow( + trial_id=trial.trial_id, + timestamp=trial.timestamp, + session=trial.session, + pbo=_decimal_or_none(trial.summary, "pbo"), + degradation_slope=_decimal_or_none(trial.summary, "degradation_slope"), + degradation_intercept=_decimal_or_none(trial.summary, "degradation_intercept"), + prob_loss=_decimal_or_none(trial.summary, "prob_loss"), + dominance_1st=_flag_or_none(trial.summary, "dominance_1st"), + dominance_2nd=_flag_or_none(trial.summary, "dominance_2nd"), + n_columns=_int_or_none(trial.summary, "n_columns"), + n_blocks=_int_or_none(trial.summary, "n_blocks"), + n_combinations=_int_or_none(trial.summary, "n_combinations"), + rows_used=_int_or_none(trial.summary, "rows_used"), + rows_dropped=_int_or_none(trial.summary, "rows_dropped"), + columns_refused=_int_or_none(trial.summary, "columns_refused"), + ) + for trial in trials + # The KIND, not the presence of a `pbo` key: the six pre-#726 gauntlet rows carry a `pbo` + # and are not CSCV runs -- they are per-trial gauntlet outcomes, which #708's view 3 shows. + # Reading them here would put two different measurements in one table under one heading. + if trial.kind == CSCV_KIND + ) + return MatrixReport( + now_ts=now_ts, + ledger_present=True, + rows=rows, + candidate_sessions=_candidate_sessions(trials), + ) diff --git a/keel/web/api.py b/keel/web/api.py index b0401cf..a586530 100644 --- a/keel/web/api.py +++ b/keel/web/api.py @@ -465,6 +465,19 @@ def read_gauntlet(cfg: ServeConfig, _query: Query, _state: Any, now_ts: int) -> return payload.gauntlet_payload(gather_gauntlet(_ledger_path(cfg), now_ts=now_ts)) +def read_matrix(cfg: ServeConfig, _query: Query, _state: Any, now_ts: int) -> dict[str, Any]: + """The Evidence Matrix (#708 view 2) -- recorded CSCV runs, READ. + + No database and no computation. `matrix.build_matrix` costs 11.9-14.3 s per session on the + real ledger and raises over the ledger as a whole (columns are only synchronous within a + session), and this route answers a page that polls every 15 s. #726 made `trials pbo` record + its whole result; this reads it. + """ + from keel.commands.evidence_matrix import gather_matrix + + return payload.matrix_payload(gather_matrix(_ledger_path(cfg), now_ts=now_ts)) + + def read_slippage(cfg: ServeConfig, _query: Query, _state: Any, now_ts: int) -> dict[str, Any]: """What a fill is assumed to cost, per product (#708, view 4). @@ -1000,6 +1013,16 @@ class ApiRoute: collection="", sortable=(), ), + # #708 view 2. The rail again: no `collection`, no `sortable`. A matrix ordered by PBO is a + # leaderboard of overfitting scores, and `cscv.py` forbids PBO as a ranking key in its own + # source. + "/api/research/matrix": ApiRoute( + html_route="/research", + read=read_matrix, + needs_database=False, + collection="", + sortable=(), + ), "/api/research/gauntlet": ApiRoute( html_route="/research", read=read_gauntlet, diff --git a/keel/web/payload.py b/keel/web/payload.py index c81603e..338a89e 100644 --- a/keel/web/payload.py +++ b/keel/web/payload.py @@ -121,6 +121,7 @@ if TYPE_CHECKING: # pragma: no cover - typing only from keel.commands.activity import ActivityCycle, ActivityEvent, ActivityFeed from keel.commands.balances import AssetBalanceRow, BalancesReport + from keel.commands.evidence_matrix import MatrixReport, MatrixRow from keel.commands.gauntlet import GauntletReport, GauntletRow from keel.commands.insights import ( AccountSummary, @@ -1394,6 +1395,133 @@ def _rules_followed_state(value: bool | None) -> str: return NEUTRAL if value else WARN +# -- the evidence matrix (#708 view 2) ------------------------------------------------------------- +# +# READ, never computed. `build_matrix` costs 11.9-14.3 s per session on the real ledger and raises +# over the ledger as a whole; this page polls every 15 s. #726 made `trials pbo` record its whole +# `PBOResult` and this serialises what it recorded. +# +# ⛔ THE STRATHERN RAIL, on the wire. No sortable column on the route and no sort key in the view. +# A matrix ordered by PBO is a leaderboard of overfitting scores, and `cscv.py` carries the +# warning: PBO "evaluates the quality of a selection process and must never become the objective +# that selection relies on". + + +def _matrix_row_payload(row: MatrixRow) -> dict[str, Any]: + """One recorded CSCV run. + + **`pbo` carries NO state.** It is the one figure a reader will want graded, and grading it is + exactly what the rail refuses: a high PBO beside a flat, positive OOS scatter is the GOOD + outcome -- a broad plateau of near-identical configurations produces high PBO by construction + -- so a colour here would be a verdict the number does not support. `trials pbo`'s own closing + sentence says to read it alongside the degradation slope, never alone, and both cross plainly + so a reader does exactly that. + + The dominance flags DO carry a state, because they are already verdicts: stochastic dominance + either held or it did not. Three-valued, so an unrecorded flag is not read as a denial. + """ + return { + "at": moment(row.timestamp), + "trial_id": row.trial_id, + "session": row.session, + "pbo": ratio(row.pbo, places=4), + "degradation_slope": ratio(row.degradation_slope, places=4), + "degradation_intercept": ratio(row.degradation_intercept, places=4), + "prob_loss": ratio(row.prob_loss, places=4), + "dominance_1st": _dominance_payload(row.dominance_1st), + "dominance_2nd": _dominance_payload(row.dominance_2nd), + "n_columns": count(row.n_columns), + "n_blocks": count(row.n_blocks), + "n_combinations": count(row.n_combinations), + "rows_used": count(row.rows_used), + "rows_dropped": count(row.rows_dropped), + "columns_refused": count(row.columns_refused), + } + + +def _dominance_payload(value: bool | None) -> Field: + """A verdict that already happened, in three readings. + + `flag()` would collapse the third: `False` says the in-sample distribution did NOT dominate, + and `None` says nobody recorded whether it did. + """ + if value is None: + return label("", display="not recorded", state=UNKNOWN) + return label( + "yes" if value else "no", + display="dominated" if value else "did not dominate", + state=NEUTRAL, + ) + + +def _matrix_state_payload(report: MatrixReport) -> Field: + """Which of the three states this deployment is in, as the one sentence the page leads with. + + An unrun matrix is not an empty one, and the middle state is why this is not a `flag`: + + * **no ledger** -- a deployment without the research repository beside it. Nothing to run. + * **columns, no run** -- the honest common case, and the one that can be acted on. + * **recorded runs** -- the matrix. + """ + if not report.ledger_present: + return label( + "no-ledger", + display="No research ledger beside this deployment — there is nothing to compile.", + state=UNKNOWN, + ) + if report.any_recorded: + return label( + "recorded", + display=( + "Compiled from recorded gauntlet runs — nothing here was computed " + "for this page." + ), + state=NEUTRAL, + ) + if report.suggested_session: + return label( + "unrun", + display=( + "No recorded evidence matrix. Matrix data is compiled from combinatorial gauntlet " + "runs, which are never computed for this page — run one in your terminal." + ), + state=UNKNOWN, + ) + return label( + "no-columns", + display=( + "No recorded evidence matrix, and no session holds enough usable columns to run one: " + "a matrix needs trials with a per-bar P&L series, and every recorded trial is " + "series_missing." + ), + state=UNKNOWN, + ) + + +def matrix_payload(report: MatrixReport) -> dict[str, Any]: + """The Evidence Matrix (#708 view 2). + + `invocation` is composed HERE and placed by the client, the same rule #707's cancel modal + follows: a client concatenating `keel trials pbo --session ` and a name could print a command + that does not exist, and the session it names has to be one that ACTUALLY HAS COLUMNS -- + `--session all` would filter to a session literally named "all" and refuse. + """ + from keel.commands.evidence_matrix import MATRIX_INVOCATION + + return { + "as_of": iso(report.now_ts), + "generated_at": moment(report.now_ts), + "state": _matrix_state_payload(report), + "recorded_count": count(report.recorded_count), + "invocation": ( + MATRIX_INVOCATION.format(session=report.suggested_session) + if report.suggested_session and not report.any_recorded + else "" + ), + "rows": [_matrix_row_payload(row) for row in report.rows], + } + + # -- plans, inverted (#706) ------------------------------------------------------------------------ # # THE ONE PAGE IN THIS APPLICATION WHOSE SUBJECT IS THE PROJECT RATHER THAN THE DEPLOYMENT, and diff --git a/keel/web/static/js/main.js b/keel/web/static/js/main.js index b0b440a..fd7b6c3 100644 --- a/keel/web/static/js/main.js +++ b/keel/web/static/js/main.js @@ -108,7 +108,7 @@ const ROUTES = [ { name: "balances", label: "Balances", endpoints: ["balances"] }, { name: "timeline", label: "Timeline", endpoints: ["timeline"] }, { name: "insights", label: "Insights", endpoints: ["insights", "journal"] }, - { name: "research", label: "Research", endpoints: ["research/trials", "research/gauntlet", "research/slippage"] }, + { name: "research", label: "Research", endpoints: ["research/trials", "research/gauntlet", "research/slippage", "research/matrix"] }, { name: "rules", label: "Rules", endpoints: ["rules"] }, { name: "venues", label: "Venues", endpoints: ["venues"] }, { name: "gates", label: "Gates", endpoints: ["gates"] }, @@ -463,10 +463,12 @@ function mount(route, readings) { // rather than taking the whole research record down. const gauntlet = readings[1]; const slippage = readings[2]; + const matrix = readings[3]; return researchView( data, gauntlet ? gauntlet.data : null, slippage ? slippage.data : null, + matrix ? matrix.data : null, ); } if (route.name === "rules") return rulesView(data, primary.sort, onSort); diff --git a/keel/web/static/js/render.js b/keel/web/static/js/render.js index b315e38..233411c 100644 --- a/keel/web/static/js/render.js +++ b/keel/web/static/js/render.js @@ -1347,7 +1347,7 @@ function kindSwitch(current, kinds, onKind) { * @param {any} slippage the `/api/research/slippage` payload, or `null` if that read failed. * @returns {DocumentFragment} */ -export function researchView(data, gauntlet, slippage) { +export function researchView(data, gauntlet, slippage, matrix) { const fragment = document.createDocumentFragment(); fragment.append(el("h1", undefined, "Research")); @@ -1445,12 +1445,92 @@ export function researchView(data, gauntlet, slippage) { ), ); + fragment.append(matrixSection(matrix)); fragment.append(gauntletSection(gauntlet)); fragment.append(slippageSection(slippage)); return fragment; } +/** + * The Evidence Matrix (#708 view 2): recorded CSCV runs, never a computed one. + * + * **The empty state is the point of this section, and it is the state it ships in.** Building a + * matrix costs 11.9-14.3 seconds per session on the real ledger and raises over the ledger as a + * whole, on a page that re-polls every 15 seconds — so the console shows what an operator RAN and + * hands them the command when they have not run one. + * + * That command comes off the payload and names a session that actually has columns. + * `--session all` is the obvious thing to print and would filter to a session literally called + * "all", find nothing, and refuse — teaching the reader that the page does not know what it is + * talking about. + * + * ⛔ No sort control on any column. A matrix ordered by PBO is a leaderboard of overfitting + * scores, and `cscv.py` forbids PBO as a ranking key in its own source. + * + * @param {any} matrix `/api/research/matrix`'s `data`, or `null` if that read failed. + * @returns {DocumentFragment} + */ +function matrixSection(matrix) { + const fragment = document.createDocumentFragment(); + fragment.append(heading("h-matrix", "Evidence matrix")); + if (!matrix) { + fragment.append(el("p", "empty", "The evidence matrix could not be read.")); + return fragment; + } + + const state = el("p", "sub"); + state.append(field(matrix.state)); + fragment.append(state); + + // The command, only where there is one to give: the payload sends an empty string once a run + // has been recorded, and a page still telling an operator to run something they have run would + // be reading its own table wrong. + if (plain(matrix.invocation)) { + fragment.append(el("pre", "invocation", plain(matrix.invocation))); + } + + fragment.append( + table( + "h-matrix", + [ + { label: "when (UTC)", numeric: false }, + { label: "session", numeric: false }, + { label: "columns", numeric: true }, + { label: "blocks", numeric: true }, + { label: "combinations", numeric: true }, + { label: "rows used", numeric: true }, + { label: "rows dropped", numeric: true }, + { label: "PBO", numeric: true }, + { label: "degradation slope", numeric: true }, + { label: "P[OOS < 0]", numeric: true }, + { label: "1st-order dominance", numeric: false }, + { label: "2nd-order dominance", numeric: false }, + ], + (matrix.rows || []).map(/** @param {any} row */ (row) => [ + row.at, + plain(row.session) || "—", + row.n_columns, + row.n_blocks, + row.n_combinations, + row.rows_used, + row.rows_dropped, + // PBO carries no state, deliberately: a high PBO beside a flat, positive OOS scatter is + // the GOOD outcome, so a colour here would be a verdict the number does not support. + row.pbo, + row.degradation_slope, + row.prob_loss, + row.dominance_1st, + row.dominance_2nd, + ]), + // From the payload: "no ledger", "no run yet" and "no session with columns" are three + // different sentences and choosing between them is a judgement (Rule 2). + plain(matrix.state.display), + ), + ); + return fragment; +} + /** * The Promotion Gauntlet scorecard (#708, view 3) -- as RECORDED, never as computed. diff --git a/tests/commands/test_evidence_matrix.py b/tests/commands/test_evidence_matrix.py new file mode 100644 index 0000000..0c310c8 --- /dev/null +++ b/tests/commands/test_evidence_matrix.py @@ -0,0 +1,243 @@ +"""The Evidence Matrix reads recorded CSCV runs; it never computes one (#708 view 2). + +Building the matrix on request costs 11.9 / 12.9 / 14.3 seconds per session on the real ledger -- +~39 s of CPU for three sessions, on a page the console re-polls every 15 seconds -- and over the +ledger as a whole it raises, because `build_matrix` requires synchronous columns and a PBO is only +defined within one session. So #726 made `trials pbo` record its whole `PBOResult`, and this reads +it. +""" + +from __future__ import annotations + +from decimal import Decimal +from pathlib import Path + +from keel.commands.evidence_matrix import gather_matrix +from keel.research import ledger + +NOW = 1_800_000_000 + + +def _cscv(path: Path, *, session: str = "s1", trial_id: str = "cscv-s1-s4", **summary): + fields = { + "pbo": Decimal("0.8810"), + "degradation_slope": Decimal("-0.42"), + "degradation_intercept": Decimal("0.05"), + "prob_loss": Decimal("0.71"), + "dominance_1st": False, + "dominance_2nd": True, + "n_columns": 12, + "n_blocks": 16, + "n_combinations": 12870, + "rows_used": 1819, + "rows_dropped": 9, + "columns_refused": 2, + } + fields.update(summary) + return ledger.append_trial( + path, + trial_id=trial_id, + session=session, + rule="(cscv over the recorded columns)", + provenance="a_priori", + kind="cscv", + decision="diagnostic_only", + series_missing=True, + summary=fields, + ) + + +def _column(path: Path, *, session: str = "s1", trial_id: str = "col-1"): + return ledger.append_trial( + path, + trial_id=trial_id, + session=session, + rule="turtle_breakout", + params={"n": 20}, + provenance="fitted", + kind="sweep_node", + decision="rejected", + per_bar_pnl=[Decimal("1"), Decimal("-1")], + ) + + +# -- what it reads --------------------------------------------------------------------------------- + + +def test_every_recorded_field_reaches_the_row(tmp_path: Path) -> None: + path = tmp_path / "ledger.jsonl" + _cscv(path) + (row,) = gather_matrix(path, now_ts=NOW).rows + + assert row.pbo == Decimal("0.8810") + assert row.degradation_slope == Decimal("-0.42") + assert row.prob_loss == Decimal("0.71") + assert row.dominance_1st is False + assert row.dominance_2nd is True + assert (row.n_columns, row.n_blocks, row.n_combinations) == (12, 16, 12870) + assert (row.rows_used, row.rows_dropped, row.columns_refused) == (1819, 9, 2) + + +def test_a_dominance_flag_is_three_valued(tmp_path: Path) -> None: + """`bool(None)` is `False`, and `False` here is a positive claim -- "the in-sample + distribution did not dominate" -- which is not what an absent field says.""" + path = tmp_path / "ledger.jsonl" + _cscv(path, dominance_1st=None, dominance_2nd=None) + (row,) = gather_matrix(path, now_ts=NOW).rows + + assert row.dominance_1st is None + assert row.dominance_2nd is None + + +def test_a_count_reader_never_returns_a_bool(tmp_path: Path) -> None: + """`bool` is a subclass of `int` in Python, so an unguarded `isinstance(value, int)` reads + `true` as the number 1. + + Tested at the READER rather than through a fixture: no writer stores a bool under a count key + today, so a round-trip test would pass with or without the guard and would be asserting + nothing. This is a defensive check and the honest way to pin one is directly. + """ + from keel.commands.evidence_matrix import _int_or_none + + assert _int_or_none({"n_columns": True}, "n_columns") is None + assert _int_or_none({"n_columns": False}, "n_columns") is None + assert _int_or_none({"n_columns": 12}, "n_columns") == 12 + assert _int_or_none({"n_columns": None}, "n_columns") is None + + +def test_a_dominance_flag_survives_beside_an_absent_count(tmp_path: Path) -> None: + path = tmp_path / "ledger.jsonl" + _cscv(path, n_columns=None, dominance_1st=True) + (row,) = gather_matrix(path, now_ts=NOW).rows + + assert row.n_columns is None + assert row.dominance_1st is True + + +def test_an_absent_figure_is_none_and_never_zero(tmp_path: Path) -> None: + """A `pbo` of 0 is the strongest possible statement about a selection process; a missing one + is no statement at all.""" + path = tmp_path / "ledger.jsonl" + _cscv(path, pbo=None, prob_loss=None) + (row,) = gather_matrix(path, now_ts=NOW).rows + + assert row.pbo is None + assert row.prob_loss is None + + +def test_the_six_pre_726_gauntlet_rows_are_not_read_as_matrices(tmp_path: Path) -> None: + """They carry a `pbo` and are NOT CSCV runs -- they are per-trial gauntlet outcomes, which + #708's view 3 shows. Selecting on the presence of a `pbo` key would put two different + measurements in one table under one heading.""" + path = tmp_path / "ledger.jsonl" + ledger.append_trial( + path, + trial_id="476-optuna-turtle_breakout", + session="optuna", + rule="turtle_breakout", + provenance="fitted", + kind="sweep_node", + decision="rejected", + series_missing=True, + summary={"pbo": Decimal("0.7"), "pbo_available": 1, "gate_passed": 0}, + ) + report = gather_matrix(path, now_ts=NOW) + + assert report.rows == () + assert report.ledger_present is True + + +def test_rows_read_in_ledger_order(tmp_path: Path) -> None: + path = tmp_path / "ledger.jsonl" + _cscv(path, trial_id="first") + _cscv(path, trial_id="second") + assert [row.trial_id for row in gather_matrix(path, now_ts=NOW).rows] == ["first", "second"] + + +# -- the three empty states ------------------------------------------------------------------------ + + +def test_a_missing_ledger_is_distinct_from_an_empty_one(tmp_path: Path) -> None: + report = gather_matrix(tmp_path / "absent.jsonl", now_ts=NOW) + assert report.ledger_present is False + assert report.any_recorded is False + assert report.candidate_sessions == () + + +def test_a_ledger_with_columns_and_no_run_names_a_session_that_would_work(tmp_path: Path) -> None: + """The guidance has to name a session that ACTUALLY HAS COLUMNS. + + `keel trials pbo --session all` is the obvious thing to suggest and would filter to a session + literally named "all", find nothing, and print a refusal -- teaching an operator that the page + does not know what it is talking about. + """ + path = tmp_path / "ledger.jsonl" + _column(path, session="sweep-a", trial_id="a1") + _column(path, session="sweep-a", trial_id="a2") + + report = gather_matrix(path, now_ts=NOW) + assert report.any_recorded is False + assert report.ledger_present is True + assert report.suggested_session == "sweep-a" + + +def test_a_session_of_backfilled_rows_is_not_suggested(tmp_path: Path) -> None: + """`build_matrix` refuses `series_missing` trials, so suggesting that session would hand the + operator a command that refuses.""" + path = tmp_path / "ledger.jsonl" + for index in range(4): + ledger.append_trial( + path, + trial_id=f"backfilled-{index}", + session="historic", + rule="turtle_breakout", + provenance="fitted", + kind="sweep_node", + decision="rejected", + series_missing=True, + ) + + report = gather_matrix(path, now_ts=NOW) + assert report.candidate_sessions == () + assert report.suggested_session == "" + + +def test_a_session_with_one_column_is_not_suggested(tmp_path: Path) -> None: + """One column is no combinatorial split. `MIN_COLUMNS_FOR_A_RUN` is the floor at which the + question is even well-formed.""" + path = tmp_path / "ledger.jsonl" + _column(path, session="lonely", trial_id="only-one") + + assert gather_matrix(path, now_ts=NOW).candidate_sessions == () + + +def test_reading_the_matrix_never_builds_one(tmp_path: Path, monkeypatch) -> None: + """The property this whole module exists for. `build_matrix` is where the 12 seconds live, + and a page that polls every 15 of them must not call it.""" + from keel.research import matrix as matrix_mod + + def _boom(*_args: object, **_kwargs: object) -> None: + raise AssertionError("gather_matrix called build_matrix -- it must only READ") + + monkeypatch.setattr(matrix_mod, "build_matrix", _boom) + + path = tmp_path / "ledger.jsonl" + _column(path, trial_id="c1") + _column(path, trial_id="c2") + _cscv(path) + + report = gather_matrix(path, now_ts=NOW) + assert report.recorded_count == 1 + + +def test_it_reads_the_real_tracked_ledger_and_finds_no_run_yet(tmp_path: Path) -> None: + """The state this ships in, stated rather than assumed: nobody has run `trials pbo` since + #726 taught it to record, so the page shows its guidance -- naming one of the three sessions + that genuinely hold columns.""" + tracked = Path(__file__).resolve().parents[2] / "docs/experiments/trials-ledger.jsonl" + report = gather_matrix(tracked, now_ts=NOW) + + assert report.ledger_present is True + assert report.any_recorded is False + assert report.suggested_session in report.candidate_sessions + assert report.candidate_sessions, "the tracked ledger holds sessions a run could work on" diff --git a/tests/web/test_api.py b/tests/web/test_api.py index 3c6d363..eb90ff8 100644 --- a/tests/web/test_api.py +++ b/tests/web/test_api.py @@ -66,6 +66,7 @@ "/api/research/trials", "/api/research/slippage", "/api/research/gauntlet", + "/api/research/matrix", "/api/rules", "/api/venues", "/api/gates", diff --git a/tests/web/test_client_assets.py b/tests/web/test_client_assets.py index 02c3c27..785e264 100644 --- a/tests/web/test_client_assets.py +++ b/tests/web/test_client_assets.py @@ -1762,6 +1762,8 @@ def _row_reads(view: str) -> dict[tuple[str, str], set[str]]: ("rulesView", "data", "rules", "/api/rules", "rules"), ("researchView", "slippage", "rows", "/api/research/slippage", "none"), ("researchView", "gauntlet", "rows", "/api/research/gauntlet", "gauntlet"), + # #708 view 2. Registered WITH the code that reads it -- the rule `_ROW_ENDPOINTS` states. + ("researchView", "matrix", "rows", "/api/research/matrix", "matrix"), # #705's discretionary journal, registered WITH the code that reads it. It hangs off # `/api/journal`'s `notes`, and the seeder writes one entry through the repository because the # CLI writer refuses to run without a terminal. @@ -1839,6 +1841,37 @@ def _seed_for(kind: str, db_path: str) -> None: _seed_orders(db_path, (("BTC-USD", "buy", "50000"),)) elif kind == "rules": _seed_rules(db_path, ("breakout",)) + elif kind == "matrix": + # A recorded CSCV run, appended through the real ledger writer -- the page has no way to + # produce one and must not: it READS what `trials pbo` recorded. + from decimal import Decimal + + from keel.research import ledger as _ledger + + _ledger.append_trial( + _ledger.DEFAULT_LEDGER_PATH, + trial_id="cscv-s1-s16", + session="s1", + rule="(cscv over the recorded columns)", + provenance="a_priori", + kind="cscv", + decision="diagnostic_only", + series_missing=True, + summary={ + "pbo": Decimal("0.88"), + "degradation_slope": Decimal("-0.4"), + "degradation_intercept": Decimal("0.05"), + "prob_loss": Decimal("0.7"), + "dominance_1st": False, + "dominance_2nd": True, + "n_columns": 12, + "n_blocks": 16, + "n_combinations": 12870, + "rows_used": 1819, + "rows_dropped": 9, + "columns_refused": 0, + }, + ) elif kind == "journal": # Through the REPOSITORY, not the CLI: `keel journal add` refuses to run without a # terminal (#705), which is the property that makes the record an attestation and is diff --git a/tests/web/test_research_view.py b/tests/web/test_research_view.py index c4b447a..5c8be97 100644 --- a/tests/web/test_research_view.py +++ b/tests/web/test_research_view.py @@ -611,7 +611,7 @@ def test_the_slippage_section_is_wired_into_the_research_view() -> None: assert '"research/slippage"' in main_code assert "function slippageSection" in _code("render.js") - assert "export function researchView(data, gauntlet, slippage)" in _code("render.js") + assert "export function researchView(data, gauntlet, slippage, matrix)" in _code("render.js") def test_the_slippage_section_states_the_basis_and_declares_no_sort_key() -> None: @@ -824,7 +824,7 @@ def test_the_gauntlet_section_is_wired_into_the_research_view() -> None: assert '"research/gauntlet"' in main_code assert "function gauntletSection" in _code("render.js") - assert "export function researchView(data, gauntlet, slippage)" in _code("render.js") + assert "export function researchView(data, gauntlet, slippage, matrix)" in _code("render.js") def test_the_gauntlet_section_shows_the_seed_and_both_expectancies() -> None: @@ -860,3 +860,137 @@ def _section_body(name: str) -> str: if exported != -1 and (end == -1 or exported < end): end = exported return after if end == -1 else after[:end] + + +# -- the evidence matrix (#708 view 2) ------------------------------------------------------------- + + +def test_the_matrix_section_is_wired_into_the_research_view() -> None: + source = _code("render.js") + start = source.index("export function researchView(") + end = source.index("\nfunction ", start) + view = source[start:end] + assert "matrixSection(matrix)" in view + + +def test_the_matrix_section_offers_no_sort_control() -> None: + """A matrix ordered by PBO is a leaderboard of overfitting scores, and `cscv.py` forbids PBO + as a ranking key in its own source. `table()` draws a control only when handed a sort pair.""" + source = _code("render.js") + start = source.index("function matrixSection(") + end = ( + source.index("\nexport function ", start) + if "\nexport function " in source[start:] + else len(source) + ) + section = source[start:end] + + assert "onSort" not in section + assert "sort:" not in section + + +def test_the_matrix_route_declares_no_sortable_column_either() -> None: + from keel.web.api import API_ROUTES + + route = API_ROUTES["/api/research/matrix"] + assert route.sortable == () + assert route.collection == "" + + +def test_the_matrix_view_places_the_command_and_never_builds_it() -> None: + """Rule 2, and the session matters: `--session all` would filter to a session literally + called "all", find nothing, and refuse.""" + source = _code("render.js") + start = source.index("function matrixSection(") + section = source[start : start + 4000] + + assert "matrix.invocation" in section + assert "keel trials pbo" not in section + + +def test_the_unrun_state_names_a_session_that_actually_has_columns(tmp_path: Path) -> None: + """The state this ships in. The page must hand over a command that WORKS, not one shaped + like the answer.""" + from keel.commands.evidence_matrix import gather_matrix + from keel.web import payload as payload_mod + + tracked = Path(__file__).resolve().parents[2] / "docs/experiments/trials-ledger.jsonl" + body = payload_mod.matrix_payload(gather_matrix(tracked, now_ts=1_800_000_000)) + + assert body["state"]["value"] == "unrun" + assert body["invocation"].startswith("keel trials pbo --session ") + named = body["invocation"].rsplit(" ", 1)[-1] + assert named in gather_matrix(tracked, now_ts=0).candidate_sessions + + +def test_a_recorded_run_stops_telling_the_operator_to_run_one(tmp_path: Path) -> None: + """A page still printing the command after a run would be reading its own table wrong.""" + from decimal import Decimal + + from keel.commands.evidence_matrix import gather_matrix + from keel.research import ledger + from keel.web import payload as payload_mod + + path = tmp_path / "ledger.jsonl" + # COLUMNS TOO, so `suggested_session` is non-empty. Without them the ledger has no candidate + # session at all, the invocation is empty whatever the rule says, and this test passes against + # a page that goes on printing the command forever. + for index in range(2): + ledger.append_trial( + path, + trial_id=f"col-{index}", + session="s1", + rule="turtle_breakout", + params={"n": 20 + index}, + provenance="fitted", + kind="sweep_node", + decision="rejected", + per_bar_pnl=[Decimal("1"), Decimal("-1")], + ) + ledger.append_trial( + path, + trial_id="cscv-s1-s16", + session="s1", + rule="(cscv)", + provenance="a_priori", + kind="cscv", + decision="diagnostic_only", + series_missing=True, + summary={"pbo": Decimal("0.5"), "n_columns": 12}, + ) + report = gather_matrix(path, now_ts=1_800_000_000) + assert report.suggested_session, "the premise: a session a run COULD be suggested for" + + body = payload_mod.matrix_payload(report) + assert body["state"]["value"] == "recorded" + assert body["invocation"] == "" + + +def test_pbo_crosses_without_a_judgement(tmp_path: Path) -> None: + """The one figure a reader wants graded, and grading it is what the rail refuses: a high PBO + beside a flat, positive OOS scatter is the GOOD outcome, so a colour would be a verdict the + number does not support. `trials pbo`'s own closing sentence says to read it alongside the + degradation slope, never alone.""" + from decimal import Decimal + + from keel.commands.evidence_matrix import gather_matrix + from keel.research import ledger + from keel.web import payload as payload_mod + + path = tmp_path / "ledger.jsonl" + for label_id, pbo in (("low", "0.05"), ("high", "0.95")): + ledger.append_trial( + path, + trial_id=f"cscv-{label_id}", + session="s1", + rule="(cscv)", + provenance="a_priori", + kind="cscv", + decision="diagnostic_only", + series_missing=True, + summary={"pbo": Decimal(pbo)}, + ) + body = payload_mod.matrix_payload(gather_matrix(path, now_ts=1_800_000_000)) + + states = {row["pbo"]["state"] for row in body["rows"]} + assert states == {"neutral"}, f"PBO carries a judgement: {states}"