From a6c40765f0b46d654a0121d9c7ef082cc8553650 Mon Sep 17 00:00:00 2001 From: Elmehdi Aitbrahim Date: Mon, 7 Sep 2026 02:04:17 -0400 Subject: [PATCH] =?UTF-8?q?feat(web):=20close=20#701=20and=20#702=20?= =?UTF-8?q?=E2=80=94=20the=20mark=20reconciliation=20pinned,=20the=20attes?= =?UTF-8?q?tation=20chip,=20and=20a=20staleness=20verdict?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The three gaps the acceptance audit found. Each was a criterion nothing in the suite could fail on. #701: THE RECONCILIATION WAS DOCUMENTED AND UNPINNED `commands/positions.py` has claimed since it shipped that "THE MARK IS THE RAILS' MARK, AND THAT IS THE POINT", and nothing asserted it. Now both real paths run against one seeded book and the numbers are compared. Writing it found why it needed writing: THE TWO SIDES READ DIFFERENT TABLES. `agent._held_position` derives the holding from FILLED LIVE ORDERS -- the audit log, the same source `guards.py` uses -- while the page reads the `positions` table's tranches. A real cycle writes both. The first cut of the test seeded only the tranche, so the rails saw an empty order log and reported `unrealized` as zero, which is a true statement about nothing. A coarser series is seeded alongside the finest and holds a different close, so the test is about the GRANULARITY CHOICE rather than about there being one candle in the database. The absent case is pinned too, because that is where two implementations of one idea usually part company: the rails contribute zero for a holding valued at cost, the page reports `None` for a figure it has no mark for, and the total is what rail 11 read. #701: THE ATTESTATION CHIP, ON #718's WINDOW The page had a freshness chip and no attestation chip -- the issue asked for both. #718 recorded `attest_due_ts`, `keel doctor` reads it, and the page did not. FOUR states, not a boolean, because the middle two are different facts: absent is `unattested` (which `screen_asset` already REJECTS, so a quiet blank would say the opposite of the gate that governs the holding), a recorded window that has passed is `expired`, one about to is `due`, and `attested` covers both "in date" and "no window recorded" -- which #718 made a legitimate state rather than a missing one. The threshold is IMPORTED from doctor rather than restated: a page warning at one horizon beside a `keel doctor` warning at another would have an operator believing whichever they read last. And every state is WARN or UNKNOWN, never BAD, which is doctor's own choice for this column: an expired window vetoes nothing, and a red chip would tell an operator the opposite. Beside the entry gate, never merged with it. One is a claim about the WORLD and the other about DATA, they disagree in both directions, and a flagged row must say which. #702: A STALENESS VERDICT THAT DOES NOT CRY WOLF ON THIS DEPLOYMENT Every stamp crossed as `moment(...)`, unconditionally NEUTRAL. Nothing compared an as-of against now, so a reading from last week rendered exactly like one from a minute ago -- the page told you WHEN and left you to do the subtraction, which Rule 2 says is Python's job. The obvious threshold is wrong here, and the deployment proves it. The config ships `interval_sec: 900` and the live profile is driven by a wrapper that runs the agent ONCE PER UTC DAY -- the scheduler fires more often and the wrapper decides. A window scaled off the config value alone would mark that deployment stale for about twenty-three and a half hours out of every twenty-four, while it worked perfectly. `agent._finest_granularity` records the identical hazard for the identical reason: a slow series "would spuriously flag a perfectly healthy feed as stale". So the window is three intervals of the LARGER of the configured cadence and the deployment's own observed one -- the MEDIAN gap between recent readings, because a mean would let one week-long outage widen the window permanently. Three rather than two: a single missed cycle is a restart, not a stop. Fewer than two readings is `None`, not "fast": one reading is no cadence, and the honest answer with nothing to compare against is "not known". A deployment that has never cycled is UNKNOWN and never stale -- a reading that does not exist is not an old one. `current` is NEUTRAL and never GOOD. A fresh reading is the ordinary state, and grading it green would make the absence of green read as a fault on every page that has just started. Twelve mutants killed, including: the page marking at a coarser series than the rails, an expired window reading as attested, an absent attestation reading as attested, a NULL window treated as expired, the page inventing its own approaching threshold, the attestation read going per row, an expired attestation graded bad, the stale window ignoring the observed cadence, the cadence using a mean, a never-cycled deployment called stale, a fresh reading graded good, and the badge dropping the age. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01KZZxmspQXe5qJ9FAsG13s6 --- keel/commands/balances.py | 87 ++++++++++- keel/commands/positions.py | 84 ++++++++++- keel/web/payload.py | 84 +++++++++++ keel/web/static/js/render.js | 11 ++ tests/commands/test_balances.py | 81 ++++++++++ tests/commands/test_positions.py | 252 +++++++++++++++++++++++++++++++ tests/web/test_balances_view.py | 53 +++++++ tests/web/test_positions_view.py | 96 ++++++++++++ 8 files changed, 744 insertions(+), 4 deletions(-) diff --git a/keel/commands/balances.py b/keel/commands/balances.py index 5f827ca..63acb1d 100644 --- a/keel/commands/balances.py +++ b/keel/commands/balances.py @@ -41,6 +41,7 @@ from keel.commands.positions import PositionRow, gather_positions from keel.config import Config from keel.data.repository import Repository +from keel.types import EquityReading @dataclass(frozen=True) @@ -154,6 +155,22 @@ class BalancesReport: assets: tuple[AssetBalanceRow, ...] + #: Whether the newest reading has fallen behind this deployment's own cadence (#702). + #: + #: Decided HERE, not by a client counting seconds (Rule 2): "is 3 days old a problem" depends + #: on how often this deployment cycles, and that is a judgement. FALSE when there is no + #: reading at all -- a reading that does not exist is not an old one. + cash_stale: bool = False + + #: The gap this deployment's recent readings actually arrive at, or `None` when there are + #: fewer than two to measure. `None` is not "fast": it is nothing to compare against, which is + #: why `_stale_window_sec` falls back to the configured interval rather than to zero. + observed_interval_sec: int | None = None + + #: How far behind a reading may fall before `cash_stale`. Carried so the page can SAY the + #: threshold rather than leave a reader guessing why a stamp is or is not flagged. + stale_window_sec: int = 0 + @property def asset_count(self) -> int: """How many products this report holds. Derived, and held here rather than measured by a @@ -161,6 +178,57 @@ def asset_count(self) -> int: return len(self.assets) +#: How many cycle intervals a reading may fall behind before the page calls it stale. +#: +#: Three, not two: a deployment that misses ONE cycle -- a restart, a slow venue, a laptop asleep +#: for a beat -- is not a deployment that has stopped, and a badge that fires on a single skipped +#: run is a badge an operator learns to ignore. +STALE_INTERVALS = 3 + +#: How many recent readings the observed cadence is measured over. +STALE_SAMPLE = 8 + + +def _observed_interval_sec(readings: Sequence[EquityReading]) -> int | None: + """The deployment's own cadence, as the MEDIAN gap between its recent readings. + + `None` when there are fewer than two readings, because one reading is no cadence -- and the + honest answer to "is this old?" with nothing to compare against is "not known", never "fine". + + The MEDIAN rather than the mean or the last gap: a deployment that was off for a week has one + enormous gap in its history, and a mean would let that single outage widen the window + permanently. A median describes what this deployment normally does. + """ + stamps = sorted(reading.ts for reading in readings) + if len(stamps) < 2: + return None + gaps = sorted(later - earlier for earlier, later in zip(stamps, stamps[1:], strict=False)) + middle = len(gaps) // 2 + if len(gaps) % 2: + return int(gaps[middle]) + return int((gaps[middle - 1] + gaps[middle]) // 2) + + +def _stale_window_sec(config: Config, observed: int | None) -> int: + """How far behind a reading may fall before it is stale. + + **The LARGER of the configured interval and the deployment's own observed cadence, and the + second half is not a refinement -- it is what stops this badge crying wolf on the live + profile.** + + `auto_trade.interval_sec` ships as 900, and the live deployment is driven by a wrapper that + runs the agent once per UTC DAY: the scheduler fires far more often and the wrapper decides. + Scaling the window off the config value alone would mark that deployment stale for about + twenty-three and a half hours out of every twenty-four, while it worked perfectly. + + `agent._finest_granularity` records exactly this hazard for exactly this reason -- a slow + series "would spuriously flag a perfectly healthy feed as stale" -- and the answer here is the + same: judge a deployment against what it actually does. + """ + configured = int(getattr(getattr(config, "auto_trade", None), "interval_sec", 0) or 0) + return STALE_INTERVALS * max(configured, observed or 0) + + def gather_balances(repo: Repository, config: Config, *, now_ts: int) -> BalancesReport: """Everything the account holds, from what a cycle wrote down. No broker, no network. @@ -172,10 +240,15 @@ def gather_balances(repo: Repository, config: Config, *, now_ts: int) -> Balance reading = None balance = None + # Empty when no mode is stamped -- a deployment before its first cycle. The staleness read + # below runs over it either way and answers "nothing to judge", which is the honest answer. + recorded: list[EquityReading] = [] if mode: - recorded = repo.get_equity_points(mode=mode, limit=1) - # `limit=1` keeps the MOST RECENT reading (`get_equity_points`' own contract), so this is - # one row off an index rather than the whole series read to take its last element. + # `STALE_SAMPLE`, not 1. The newest reading is still `recorded[-1]` (`get_equity_points` + # keeps the most recent and returns them oldest-first), and the ones behind it are what + # the observed cadence is measured over -- see `_stale_window_sec` for why a cadence read + # off the deployment matters more than the one the config declares. + recorded = repo.get_equity_points(mode=mode, limit=STALE_SAMPLE) reading = recorded[-1] if recorded else None # Same `limit=1`-keeps-the-newest contract, narrowed to the SETTLEMENT currency (#719) -- @@ -189,6 +262,8 @@ def gather_balances(repo: Repository, config: Config, *, now_ts: int) -> Balance # gate, so computing one would be three of every four candle reads plus a rules read and # a rule construction, per request, on a view the console re-polls every 15 seconds. positions = gather_positions(repo, config, now_ts=now_ts, with_readiness=False) + observed = _observed_interval_sec(recorded) + window = _stale_window_sec(config, observed) return BalancesReport( now_ts=now_ts, mode=mode, @@ -198,6 +273,12 @@ def gather_balances(repo: Repository, config: Config, *, now_ts: int) -> Balance unrealized=None if reading is None else reading.unrealized, hwm=None if reading is None else reading.hwm, has_recorded_cash=reading is not None, + observed_interval_sec=observed, + stale_window_sec=window, + # A reading that does not exist is not an OLD one. `has_recorded_cash` says there is none, + # and a stale badge over a deployment that has never run would be a false alarm about a + # non-event. + cash_stale=reading is not None and (now_ts - reading.ts) > window, paper_cash=repo.get_state("paper_cash_usdc") if mode == "paper" else None, settled_cash=None if balance is None else balance.available, total_cash=None if balance is None else balance.total, diff --git a/keel/commands/positions.py b/keel/commands/positions.py index bee59c3..922dce3 100644 --- a/keel/commands/positions.py +++ b/keel/commands/positions.py @@ -40,10 +40,48 @@ from keel_core.types import Granularity from keel import agent as agent_mod +from keel.commands.doctor import ATTEST_WINDOW_APPROACHING_SEC as _DOCTOR_APPROACHING_SEC from keel.config import Config from keel.data import freshness as freshness_mod from keel.data.repository import Repository +#: What a holding's asset attestation says, as a closed vocabulary (#701). +#: +#: FOUR words for four different facts, and the middle two are why this is not a boolean: +#: `unattested` is the screen's own rejection, `expired` is a claim that has run out, `due` is one +#: about to, and `attested` covers both "in date" and "no window recorded" -- which #718 made a +#: legitimate state rather than a missing one. +ATTESTATION_STATES: tuple[str, ...] = ("attested", "due", "expired", "unattested") + +#: How near a window has to be before the page says so. `doctor`'s own threshold, IMPORTED rather +#: than restated: a page warning at one horizon beside a `keel doctor` warning at another would +#: have an operator believing whichever they read last. +ATTEST_APPROACHING_SEC = _DOCTOR_APPROACHING_SEC + + +def _attestation_state(row: dict[str, Any] | None, *, now_ts: int) -> tuple[str, int | None]: + """`(state, due_ts)` for one asset's attestation row. + + The window rules are `doctor._attestation_window_findings`', deliberately: `due <= now_ts` is + CLOSED (with `<`, a window landing exactly on the second falls into no group at all, which is + a false all-clear), and a NULL window is not judged at all. + + **Reporting only, like doctor's.** `screen_asset` never reads `attest_due_ts`, so an expired + window does not veto anything -- #718 left that decision to a human. This page says what the + record holds and does not imply a block that is not there. + """ + if row is None: + return "unattested", None + raw = row.get("attest_due_ts") if hasattr(row, "get") else row["attest_due_ts"] + if raw is None: + return "attested", None + due = int(raw) + if due <= now_ts: + return "expired", due + if due - now_ts <= ATTEST_APPROACHING_SEC: + return "due", due + return "attested", due + @dataclass(frozen=True) class PositionRow: @@ -103,6 +141,26 @@ class PositionRow: ready: bool ready_reason: str | None + #: What the operator has sworn about this holding's ASSET, and whether that claim is in date + #: (#701, on #718's recorded window). One of `ATTESTATION_STATES`. + #: + #: A holding is a claim about the world -- what the token is, what backs it -- and + #: `keel/compliance/screen.py`'s rule is that an asset nobody has classified is unknown and + #: unknown is a REJECTION. So absent reads `unattested`, never a quiet blank: a page rendering + #: it as fine would say the opposite of the gate that governs it. + #: + #: Distinct from `ready` above, which is the entry-gate verdict about DATA. The two disagree in + #: both directions and a reader needs to know which is which: an asset can be perfectly + #: attested with a cold series, or freshly priced with a lapsed claim. + #: + #: Defaulted and trailing, so every existing `PositionRow(...)` construction is untouched. + attestation: str = "unattested" + + #: The recorded window's close, or `None` when the operator recorded none. NULL is NOT judged + #: -- `doctor._attestation_window_findings` states that convention and #718 made the column + #: optional deliberately, so no window is not an expired one. + attest_due_ts: int | None = None + @dataclass(frozen=True) class PositionsReport: @@ -286,6 +344,12 @@ def gather_positions( gates = _gate_granularities(repo, config) if with_readiness else {} fallback = _fallback_granularity(config) if with_readiness else None + # ONE read for the whole book, not one per tranche: this page re-polls every 15 seconds, and + # `get_asset_attestation` per row would be a query per row. Same shape as #707's + # `open_bracket_order_ids`. + attestations = { + str(row.get("asset") or "").upper(): row for row in repo.get_asset_attestations() + } marks: dict[str, tuple[Decimal | None, int | None]] = {} # Keyed on (product, gate granularity), NOT on product alone: one product can hold tranches # opened by rules on different timeframes, and a per-product cache would hand the second @@ -307,16 +371,32 @@ def gather_positions( else: ready, ready_reason = False, None mark, mark_ts = marks[product_id] - rows.append(_row_from_dict(raw, mark, mark_ts, ready, ready_reason)) + attestation, attest_due = _attestation_state( + attestations.get(_base_asset(product_id)), now_ts=now_ts + ) + rows.append( + _row_from_dict( + raw, mark, mark_ts, ready, ready_reason, attestation, attest_due + ) + ) return PositionsReport(now_ts=now_ts, rows=tuple(rows)) +def _base_asset(product_id: str) -> str: + """`BTC-USD` -> `BTC`, the same split `compliance/screen.py` and `execution/guards.py` use to + key an asset attestation off a product. Upper-cased because `asset_attestations` is keyed on + the operator's own spelling and a page must not miss a claim over a letter case.""" + return str(product_id).split("-")[0].upper() + + def _row_from_dict( raw: dict[str, Any], mark: Decimal | None, mark_ts: int | None, ready: bool, ready_reason: str | None, + attestation: str = "unattested", + attest_due_ts: int | None = None, ) -> PositionRow: """One repository dict, projected. Every judgement this report makes is made here, once, so no renderer has to make it twice.""" @@ -358,4 +438,6 @@ def _row_from_dict( realized_fees=raw.get("realized_fees") or Decimal("0"), ready=ready, ready_reason=ready_reason, + attestation=attestation, + attest_due_ts=attest_due_ts, ) diff --git a/keel/web/payload.py b/keel/web/payload.py index 09caa99..c81603e 100644 --- a/keel/web/payload.py +++ b/keel/web/payload.py @@ -775,6 +775,32 @@ def _bracket_field(position: OpenPositionStatus, mode: str) -> Field: return label("n/a", display="n/a -- paper resolves stop/target on candle touch", state=NEUTRAL) +#: What each attestation state MEANS and how it reads (#701). +#: +#: `unattested` is UNKNOWN, not BAD: `screen_asset` rejects an unclassified asset, so the holding +#: is already gated -- the page reports a missing claim rather than grading the operator for it. +#: `expired` and `due` WARN and never FAIL, which is doctor's own choice for the same column and +#: for the same reason: an expired window vetoes nothing (#718 left that decision to a human), and +#: a FAIL would tell an operator the opposite. +_ATTESTATION_CHIP: Mapping[str, tuple[str, str]] = { + "attested": ("attested", NEUTRAL), + "due": ("attestation window closing", WARN), + "expired": ("ATTESTATION EXPIRED", WARN), + "unattested": ("no attestation on record", UNKNOWN), +} + + +def _attestation_chip(row: PositionRow) -> Field: + """The chip, and the window behind it. + + A `label` rather than a `flag`, because there are FOUR readings and two of them are not the + negation of the others: "no window recorded" and "window passed" are different facts about an + attestation that exists, and "never attested" is a different fact again. + """ + display, state = _ATTESTATION_CHIP.get(row.attestation, ("unknown", UNKNOWN)) + return label(row.attestation, display=display, state=state) + + def _position_payload(position: OpenPositionStatus, mode: str) -> dict[str, Any]: """One open position. @@ -1985,6 +2011,12 @@ def _position_row_payload(row: PositionRow) -> dict[str, Any]: "realized_proceeds": money(row.realized_proceeds), "realized_fees": money(row.realized_fees), "freshness": _readiness_field(row.ready, row.ready_reason), + # #701. BESIDE the freshness verdict, never merged with it: one is a claim about the + # WORLD (what this token is, and whether the claim is in date) and the other is a + # claim about DATA. They disagree in both directions, and a reader needs to know + # which of the two is the reason a row is flagged. + "attestation": _attestation_chip(row), + "attest_due_at": moment(row.attest_due_ts), } @@ -2030,6 +2062,55 @@ def _asset_balance_payload(row: AssetBalanceRow) -> dict[str, Any]: } +def _human_span(seconds: int) -> str: + """A duration a reader can act on -- "3 days", "4 hours". No arithmetic in the client (Rule 2), + and no bare epoch difference on a page whose whole subject is when things were recorded.""" + if seconds >= 172_800: + return f"{seconds // 86_400} days" + if seconds >= 7_200: + return f"{seconds // 3_600} hours" + if seconds >= 120: + return f"{seconds // 60} minutes" + return f"{seconds} seconds" + + +def _staleness_payload(report: BalancesReport) -> Field: + """Whether the figures on this page describe now, and how confidently that can be said (#702). + + THREE readings, and the middle one is why this is not a `flag`: + + * **nothing recorded** -- a deployment before its first cycle. UNKNOWN: a reading that does + not exist is not an old one, and a stale badge here would be a false alarm about a + non-event. + * **behind this deployment's own cadence** -- WARN, and it names the age. A venue outage holds + the last row for as long as it lasts, so this is the difference between a current figure and + one that merely looks current. + * **current** -- NEUTRAL and never GOOD. A fresh reading is the ordinary state, and grading it + green would make the absence of green read as a fault on every page that has just started. + + The JUDGEMENT is the report's (`cash_stale`), made against the cadence this deployment + actually cycles at rather than the one its config declares -- see `_stale_window_sec` for the + live profile that would otherwise be marked stale twenty-three hours a day. + """ + if not report.has_recorded_cash: + return label( + "unrecorded", + display="no cycle has recorded a reading yet", + state=UNKNOWN, + ) + age = report.now_ts - (report.cash_as_of or report.now_ts) + if report.cash_stale: + return label( + "stale", + display=( + f"STALE — the last reading is {_human_span(age)} old, past this deployment's " + f"own {_human_span(report.stale_window_sec)} window" + ), + state=WARN, + ) + return label("current", display=f"recorded {_human_span(age)} ago", state=NEUTRAL) + + def balances_payload(report: BalancesReport) -> dict[str, Any]: """`gather_balances`'s `BalancesReport`, as JSON (#702). @@ -2089,6 +2170,9 @@ def balances_payload(report: BalancesReport) -> dict[str, Any]: ), "asset_count": count(report.asset_count), "assets": [_asset_balance_payload(row) for row in report.assets], + # #702. The judgement, not the subtraction: "is 3 days old a problem" depends on how + # often this deployment cycles, and Rule 2 keeps that in Python. + "freshness": _staleness_payload(report), } diff --git a/keel/web/static/js/render.js b/keel/web/static/js/render.js index b0850a4..b315e38 100644 --- a/keel/web/static/js/render.js +++ b/keel/web/static/js/render.js @@ -1646,6 +1646,10 @@ export function balancesView(data, sort, onSort) { const sub = el("p", "sub"); sub.append(field(data.generated_at), " · ", field(data.recorded)); + // #702. Every tile on this page is a recorded figure with a stamp, and a stamp only separates a + // current reading from a stale one if somebody does the subtraction. The payload does it, judged + // against the cadence this deployment actually cycles at, and this places the verdict. + if (data.freshness) sub.append(" · ", field(data.freshness)); fragment.append(sub); fragment.append( @@ -1789,6 +1793,12 @@ export function positionsView(data, sort, onSort) { // on different timeframes has two verdicts -- a single chip above the table would // state one of them over the other. { label: "entry gate", numeric: false, key: "freshness" }, + // #701. BESIDE the entry gate, never merged with it. One is a claim about the WORLD -- + // what this token is, and whether the operator's claim is still in date -- and the + // other is a claim about DATA. They disagree in both directions, and a row flagged for + // one reason must not read as flagged for the other. No sort key: an attestation state + // is not a figure to rank tranches by. + { label: "attestation", numeric: false }, ], held.map(/** @param {any} row */ (row) => [ row.opened_at, @@ -1803,6 +1813,7 @@ export function positionsView(data, sort, onSort) { row.stop_distance, row.stop_distance_pct, row.freshness, + row.attestation, ]), "No open tranches for this product.", { sort: sort, onSort: onSort }, diff --git a/tests/commands/test_balances.py b/tests/commands/test_balances.py index 6929952..27578d6 100644 --- a/tests/commands/test_balances.py +++ b/tests/commands/test_balances.py @@ -551,3 +551,84 @@ def test_the_recorded_split_carries_the_instant_it_was_observed( assert report.settled_as_of == NOW_TS - 3600 + + +# -- staleness, judged in Python (#702) ------------------------------------------------------------ + + +NOW = NOW_TS + + +def _record_reading(repo: Repository, *, ts: int) -> None: + """One live reading at `ts`, and the mode stamp every balances read goes through.""" + repo.set_state("equity_state_mode", "live") + repo.record_equity_point(_reading(ts, "live", "250")) + + +def test_a_fresh_reading_is_not_stale(repo, tmp_path) -> None: + _record_reading(repo, ts=NOW - 60) + report = gather_balances(repo, _config(tmp_path), now_ts=NOW) + assert report.cash_stale is False + + +def test_a_reading_older_than_the_threshold_is_stale(repo, tmp_path) -> None: + _record_reading(repo, ts=NOW - (10 * 86_400)) + report = gather_balances(repo, _config(tmp_path), now_ts=NOW) + assert report.cash_stale is True + + +def test_a_deployment_that_cycles_DAILY_is_not_called_stale_all_day(repo, tmp_path) -> None: + """The threshold cannot be `2 × auto_trade.interval_sec` alone, and this is the deployment + that proves it. + + The config ships `interval_sec: 900`, and the live profile is driven by a wrapper that runs + the agent ONCE PER UTC DAY -- the scheduler fires more often and the wrapper decides. Scaling + a staleness window off the config value would mark that deployment stale for roughly + twenty-three and a half hours out of every twenty-four, on a page that is working perfectly. + + `agent._finest_granularity` records the identical hazard for the identical reason: a slow + series "would spuriously flag a perfectly healthy feed as stale". A badge that cries wolf + daily is a badge an operator learns to ignore, which costs more than never having shipped it. + + So the window is the LARGER of the configured interval and the deployment's own observed + cadence, and a book whose readings arrive a day apart is judged against a day. + """ + day = 86_400 + for index in range(4, 0, -1): + _record_reading(repo, ts=NOW - (index * day)) + + # TWELVE HOURS after the last reading, which is the interesting moment. Sixty seconds after it + # nothing would call the page stale and the test would pass against a config-only window too; + # half a day in, a window scaled off `interval_sec: 900` says STALE and the deployment is + # perfectly healthy. + report = gather_balances(repo, _config(tmp_path), now_ts=NOW - day + (12 * 3_600)) + assert report.observed_interval_sec == day + assert report.cash_stale is False, "a healthy daily deployment was called stale mid-cycle" + + +def test_a_daily_deployment_that_actually_stops_IS_stale(repo, tmp_path) -> None: + """The other half. Widening the window for a slow cadence must not widen it to useless: a + deployment that cycled daily and then stopped for a week is exactly what this badge is for.""" + day = 86_400 + for index in range(4, 0, -1): + _record_reading(repo, ts=NOW - (7 * day) - (index * day)) + + report = gather_balances(repo, _config(tmp_path), now_ts=NOW) + assert report.cash_stale is True + + +def test_a_single_reading_cannot_be_judged_and_says_so(repo, tmp_path) -> None: + """One reading is no cadence. There is nothing to compare against, so the honest answer is + "not known", never "fine" -- the same refusal `_chain_payload` and the session chip make.""" + _record_reading(repo, ts=NOW - 60) + report = gather_balances(repo, _config(tmp_path), now_ts=NOW) + assert report.observed_interval_sec is None + + +def test_a_deployment_with_no_reading_at_all_is_not_stale_it_is_unrecorded(repo, tmp_path) -> None: + """Nothing was read, so nothing is old. `has_recorded_cash` is what says there is no reading, + and a stale badge over a deployment that has never run would be a false alarm about a + non-event.""" + report = gather_balances(repo, _config(tmp_path), now_ts=NOW) + assert report.has_recorded_cash is False + assert report.cash_stale is False diff --git a/tests/commands/test_positions.py b/tests/commands/test_positions.py index 8614eee..996d6f2 100644 --- a/tests/commands/test_positions.py +++ b/tests/commands/test_positions.py @@ -19,6 +19,7 @@ from typing import Any import pytest +from keel_broker_api.orders import Side from keel_core.types import Candle, Granularity from keel import agent as agent_mod @@ -501,3 +502,254 @@ def test_the_products_list_is_in_first_seen_order(repo: Repository, tmp_path: Pa "SOL-USD", "BTC-USD", ) + + +# -- the mark this page quotes IS the mark the rails moved on (#701) ------------------------------ +# +# The module docstring has claimed this since the page shipped -- "THE MARK IS THE RAILS' MARK, AND +# THAT IS THE POINT" -- and nothing asserted it. A documented invariant with no test is the shape +# every finding in this milestone has taken, so here it is driven through BOTH real paths rather +# than reasoned about from the shared helper. +# +# It matters because the two answers are not equal in status. `agent._mark_to_market_parts`' +# number is the one that moved rail 11's drawdown scalars and got written to `equity_points`. If +# this page ever quoted a different current price, the page would be the wrong one. + + +def _fill_the_entry(repo: Repository, qty: str = "2", price: str = "100") -> None: + """The filled live BUY behind the tranche. + + **The two sides read different tables, and that is the whole reason this needs a pin rather + than an argument.** `agent._held_position` derives the holding from FILLED LIVE ORDERS -- the + audit log, the same source `guards.py` and `executor._held_position` use -- while the page + reads the `positions` table's tranches. A real cycle writes both: `executor` fills the order + and `run_once` opens the tranche from it. + + A test that seeded only the tranche made the rails see nothing at all and report `unrealized` + as zero -- which is a true statement about an empty order log and tells you nothing about + whether the two agree. + """ + repo.insert_order( + { + "mode": "live", + "product_id": PRODUCT, + # `Side.BUY.value`, which is UPPERCASE. `_held_position` compares against it exactly, + # so a lowercase "buy" here reads as a side it does not recognise and the holding + # silently weighs nothing -- which is how the first cut of this test "passed" the + # rails with an empty order log. + "side": Side.BUY.value, + "qty": Decimal(qty), + "status": "filled", + "actual_fill": Decimal(price), + "created_at": NOW_TS - DAY, + } + ) + + +def _rails_price_map(repo: Repository, config: Config) -> dict[str, Decimal]: + """The price map exactly as `agent.run_once` builds it. + + Not `{PRODUCT: Decimal("150")}` handed in by the test: the point of the pin is that the two + paths agree about WHICH candle is the mark, and a literal would assume the answer. + """ + from keel import agent as agent_mod + + finest = agent_mod._finest_granularity(list(config.market_data.granularities)) + assert finest is not None + prices: dict[str, Decimal] = {} + for product in (PRODUCT,): + candles = repo.get_candles(product, finest) + if candles: + prices[product] = candles[-1].close + return prices + + +def test_the_page_and_the_rails_mark_the_same_holding_at_the_same_price( + repo: Repository, tmp_path: Path +) -> None: + """One number, two callers. + + A COARSER series is seeded too, and holds a different close. That is what makes this a test of + the granularity choice rather than of "there is only one candle in the database" -- if either + path stopped agreeing about which timeframe is finest, they would quote 150 and 999. + """ + config = _config(tmp_path) + _open_tranche(repo) + _mark(repo, "150") + repo.upsert_candles(PRODUCT, Granularity.ONE_DAY, [_candle(NOW_TS - 86_400, "999")]) + + prices = _rails_price_map(repo, config) + assert prices == {PRODUCT: Decimal("150")}, "the rails' own map did not take the finest close" + + row = gather_positions(repo, config, now_ts=NOW_TS).rows[0] + assert row.mark == prices[PRODUCT] + + +def test_the_pages_unrealized_reconciles_with_the_leg_recorded_against_the_rails( + repo: Repository, tmp_path: Path +) -> None: + """The figure, not just the price. + + `_mark_to_market_parts` is driven for real, with the price map `run_once` would have handed + it, and its `unrealized` leg is the one written to `equity_points` and read by rail 11. The + page sums its own per-tranche `unrealized_pnl` from the same mark, so the two must be equal -- + and a page whose P&L disagreed with the number the drawdown rail acted on would be describing + a different account. + """ + from keel import agent as agent_mod + from tests.test_agent import FakeBroker + + config = _config(tmp_path) + _open_tranche(repo) + _fill_the_entry(repo) + _mark(repo, "150") + + parts = agent_mod._mark_to_market_parts( + repo, FakeBroker(), [PRODUCT], _rails_price_map(repo, config), config.quote_currency + ) + assert parts is not None + + report = gather_positions(repo, config, now_ts=NOW_TS) + page_unrealized = sum( + (row.unrealized_pnl for row in report.rows if row.unrealized_pnl is not None), + Decimal("0"), + ) + assert page_unrealized == parts.unrealized + + +def test_an_unmarked_holding_reconciles_at_ZERO_on_both_sides( + repo: Repository, tmp_path: Path +) -> None: + """The agreement has to survive the absent case, which is where two implementations of one + idea usually part company: `equity.unrealized_on_marks` contributes ZERO for a holding valued + at cost, and this page reports `None` for a figure it has no mark for. Zero and "not known" + are different words for the reader and the same number in the total, and the total is what + rail 11 read.""" + from keel import agent as agent_mod + from tests.test_agent import FakeBroker + + config = _config(tmp_path) + _open_tranche(repo) + _fill_the_entry(repo) # the holding exists in both records; no candles seeded at all + + parts = agent_mod._mark_to_market_parts( + repo, FakeBroker(), [PRODUCT], _rails_price_map(repo, config), config.quote_currency + ) + assert parts is not None + assert parts.unrealized == Decimal("0") + + row = gather_positions(repo, config, now_ts=NOW_TS).rows[0] + assert row.mark is None + assert row.unrealized_pnl is None + + +# -- the attestation chip (#701, on #718's window) ------------------------------------------------ + + +def _attest( + repo: Repository, *, asset: str = "BTC", due: int | None = None, at: int = NOW_TS - DAY +): + repo.upsert_asset_attestation( + asset=asset, + sector="tech", + backing="native", + pays_yield=False, + source="prospectus", + attested_by="operator", + attested_at=at, + attest_due_ts=due, + ) + + +def test_an_unattested_holding_says_so_rather_than_reading_as_fine( + repo: Repository, tmp_path: Path +) -> None: + """Absent is UNKNOWN, never OK. `keel/compliance/screen.py`'s rule is that an asset nobody has + classified is unknown and unknown is a rejection -- a page rendering it as a quiet blank would + say the opposite of the gate that governs it.""" + _open_tranche(repo) + _mark(repo, "150") + row = gather_positions(repo, _config(tmp_path), now_ts=NOW_TS).rows[0] + + assert row.attestation == "unattested" + + +def test_an_attestation_with_no_window_is_attested_and_not_judged_on_time( + repo: Repository, tmp_path: Path +) -> None: + """`attest_due_ts` is OPTIONAL (#718): NULL means the operator recorded no window, which is + not the same as an expired one. Doctor's `_attestation_window_findings` states the convention + -- a row with no window is in neither the passed nor the approaching group -- and this follows + it rather than inventing a deadline.""" + _open_tranche(repo) + _mark(repo, "150") + _attest(repo, due=None) + row = gather_positions(repo, _config(tmp_path), now_ts=NOW_TS).rows[0] + + assert row.attestation == "attested" + assert row.attest_due_ts is None + + +def test_a_window_that_has_passed_reads_as_expired(repo: Repository, tmp_path: Path) -> None: + """`due <= now` is closed, the same boundary doctor uses -- and for the reason stated there: + with `<`, a window landing exactly on the second falls into no group at all, which is a false + all-clear.""" + _open_tranche(repo) + _mark(repo, "150") + _attest(repo, due=NOW_TS) + row = gather_positions(repo, _config(tmp_path), now_ts=NOW_TS).rows[0] + + assert row.attestation == "expired" + + +def test_a_window_closing_soon_reads_as_due(repo: Repository, tmp_path: Path) -> None: + _open_tranche(repo) + _mark(repo, "150") + _attest(repo, due=NOW_TS + DAY) + row = gather_positions(repo, _config(tmp_path), now_ts=NOW_TS).rows[0] + + assert row.attestation == "due" + assert row.attest_due_ts == NOW_TS + DAY + + +def test_a_window_far_out_is_simply_attested(repo: Repository, tmp_path: Path) -> None: + from keel.commands.doctor import ATTEST_WINDOW_APPROACHING_SEC + + _open_tranche(repo) + _mark(repo, "150") + _attest(repo, due=NOW_TS + ATTEST_WINDOW_APPROACHING_SEC + DAY) + row = gather_positions(repo, _config(tmp_path), now_ts=NOW_TS).rows[0] + + assert row.attestation == "attested" + + +def test_the_approaching_threshold_is_doctors_and_not_a_second_number( + repo: Repository, tmp_path: Path +) -> None: + """One threshold, two surfaces. A page warning at 30 days beside a `keel doctor` warning at 14 + would have an operator believing whichever they looked at last.""" + from keel.commands import positions as positions_mod + from keel.commands.doctor import ATTEST_WINDOW_APPROACHING_SEC + + assert positions_mod.ATTEST_APPROACHING_SEC is ATTEST_WINDOW_APPROACHING_SEC + + +def test_reading_the_attestations_costs_one_query_however_many_tranches( + repo: Repository, tmp_path: Path +) -> None: + """A per-row lookup on a page that re-polls every 15 seconds. `get_asset_attestations()` + answers the whole book in one read, the same shape #707's `open_bracket_order_ids` uses.""" + for _ in range(12): + _open_tranche(repo) + _mark(repo, "150") + _attest(repo) + + seen: list[str] = [] + repo._conn.set_trace_callback(seen.append) # noqa: SLF001 + try: + gather_positions(repo, _config(tmp_path), now_ts=NOW_TS) + finally: + repo._conn.set_trace_callback(None) # noqa: SLF001 + + reads = [sql for sql in seen if "asset_attestations" in sql] + assert len(reads) == 1, f"{len(reads)} attestation queries for 12 tranches" diff --git a/tests/web/test_balances_view.py b/tests/web/test_balances_view.py index 8a0f8cf..285f3dd 100644 --- a/tests/web/test_balances_view.py +++ b/tests/web/test_balances_view.py @@ -295,3 +295,56 @@ def test_the_view_stamps_the_recorded_figures() -> None: def test_the_view_names_the_settled_split_as_unrecorded() -> None: """Omitting the tiles would let a reader take the available figure for the settled one.""" assert "settled_breakdown" in _view_body() + + +# -- the staleness verdict on the wire (#702) ----------------------------------------------------- + + +def _stale_report(tmp_path: Path, *, ages: tuple[int, ...]): + """A book whose live readings sit at `ages` seconds before now, oldest first.""" + from tests.commands.test_balances import NOW_TS as B_NOW + from tests.commands.test_balances import _config, _reading + + conn = connect(str(tmp_path / "stale.db")) + migrate(conn) + repo = Repository(conn) + repo.set_state("equity_state_mode", "live") + for age in ages: + repo.record_equity_point(_reading(B_NOW - age, "live", "250")) + return gather_balances(repo, _config(tmp_path), now_ts=B_NOW) + + +def test_a_current_reading_is_neutral_and_never_green(tmp_path: Path) -> None: + """A fresh reading is the ORDINARY state. Grading it green would make the absence of green + read as a fault on every page that has only just started.""" + body = web_payload.balances_payload(_stale_report(tmp_path, ages=(900, 60))) + assert body["freshness"]["state"] == "neutral" + assert body["freshness"]["value"] == "current" + + +def test_a_stale_reading_warns_and_says_how_old(tmp_path: Path) -> None: + body = web_payload.balances_payload(_stale_report(tmp_path, ages=(900_000, 864_000))) + chip = body["freshness"] + + assert chip["state"] == "warn" + assert chip["value"] == "stale" + assert "STALE" in chip["display"] + assert "days" in chip["display"], "an operator needs the age, not just the word" + + +def test_a_deployment_that_has_never_cycled_is_unknown_not_stale(tmp_path: Path) -> None: + body = web_payload.balances_payload(_stale_report(tmp_path, ages=())) + assert body["freshness"]["state"] == "unknown" + assert "STALE" not in body["freshness"]["display"] + + +def test_the_client_places_the_verdict_and_computes_no_age(tmp_path: Path) -> None: + """Rule 2 and the arithmetic ban together: `render.js` may not subtract two timestamps, so the + sentence has to arrive composed.""" + source = _source("render.js") + start = source.index("export function balancesView(") + end = source.index("\nexport function ", start) + view = source[start:end] + + assert "data.freshness" in view + assert "STALE" not in view, "the wording belongs to payload._staleness_payload" diff --git a/tests/web/test_positions_view.py b/tests/web/test_positions_view.py index b3ab75e..b22f003 100644 --- a/tests/web/test_positions_view.py +++ b/tests/web/test_positions_view.py @@ -242,3 +242,99 @@ def test_the_positions_view_names_the_stop_distance_both_ways() -> None: view = _function_body("render.js", "positionsView") assert "stop_distance" in view assert "stop_distance_pct" in view + + +# -- the attestation chip on the wire (#701) ------------------------------------------------------ + + +def _payload_mod(): + from keel.web import payload as payload_mod + + return payload_mod + + +def _attested_report(tmp_path: Path, *, due, **overrides): + from tests.commands.test_positions import NOW_TS as POS_NOW + from tests.commands.test_positions import _config, _mark, _open_tranche + + conn = connect(str(tmp_path / "attest.db")) + migrate(conn) + repo = Repository(conn) + _open_tranche(repo) + _mark(repo, "150") + if due is not ...: + repo.upsert_asset_attestation( + asset="BTC", + sector="tech", + backing="native", + pays_yield=False, + source="prospectus", + attested_by="operator", + attested_at=POS_NOW - 86_400, + attest_due_ts=due, + ) + return gather_positions(repo, _config(tmp_path), now_ts=POS_NOW) + + +def test_an_expired_attestation_warns_on_the_wire(tmp_path: Path) -> None: + from tests.commands.test_positions import NOW_TS as POS_NOW + + body = _payload_mod().positions_payload(_attested_report(tmp_path, due=POS_NOW - 10)) + chip = body["rows"][0]["attestation"] + + assert chip["value"] == "expired" + assert chip["state"] == "warn" + assert "EXPIRED" in chip["display"] + + +def test_an_absent_attestation_is_unknown_and_not_a_failure(tmp_path: Path) -> None: + """`screen_asset` already rejects an unclassified asset, so the holding is gated. The page + reports a missing claim rather than grading the operator for it.""" + body = _payload_mod().positions_payload(_attested_report(tmp_path, due=...)) + chip = body["rows"][0]["attestation"] + + assert chip["value"] == "unattested" + assert chip["state"] == "unknown" + + +def test_no_attestation_state_is_ever_bad(tmp_path: Path) -> None: + """WARN and never FAIL, which is doctor's own choice for this column and for the same reason: + an expired window vetoes nothing (#718 left that to a human), and a `bad` chip would tell an + operator the opposite.""" + from tests.commands.test_positions import NOW_TS as POS_NOW + + for due in (..., None, POS_NOW - 10, POS_NOW + 86_400, POS_NOW + 400 * 86_400): + body = _payload_mod().positions_payload(_attested_report(tmp_path, due=due)) + assert body["rows"][0]["attestation"]["state"] != "bad", due + + +def test_the_window_crosses_as_a_moment_not_an_epoch(tmp_path: Path) -> None: + from tests.commands.test_positions import NOW_TS as POS_NOW + + body = _payload_mod().positions_payload(_attested_report(tmp_path, due=POS_NOW + 86_400)) + assert body["rows"][0]["attest_due_at"]["display"] + assert isinstance(body["rows"][0]["attest_due_at"]["value"], str) + + +def test_the_attestation_chip_is_separate_from_the_freshness_chip(tmp_path: Path) -> None: + """The two disagree in both directions -- a perfectly attested asset with a cold series, or a + freshly priced one with a lapsed claim -- and a reader has to be able to tell which is which.""" + from tests.commands.test_positions import NOW_TS as POS_NOW + + row = _payload_mod().positions_payload(_attested_report(tmp_path, due=POS_NOW - 10))["rows"][0] + assert row["attestation"]["value"] == "expired" + assert "ready" in row or "freshness" in row + + +def test_the_positions_view_shows_the_attestation_chip_beside_the_entry_gate() -> None: + """Two columns, not one merged verdict. A reader has to be able to tell a lapsed claim about + the asset from a cold price series.""" + source = _source("render.js") + start = source.index("export function positionsView(") + end = source.index("\nfunction ", start) + view = source[start:end] + + assert 'label: "attestation"' in view + assert "row.attestation" in view + assert 'label: "entry gate"' in view + assert "row.freshness" in view