Skip to content

Fix #1471: replace the render-gate perf wall-clock bound with a deterministic op-count check - #1487

Open
mohidmakhdoomi wants to merge 10 commits into
mainfrom
builder/bugfix-1471
Open

Fix #1471: replace the render-gate perf wall-clock bound with a deterministic op-count check#1487
mohidmakhdoomi wants to merge 10 commits into
mainfrom
builder/bugfix-1471

Conversation

@mohidmakhdoomi

Copy link
Copy Markdown
Collaborator

Summary

The render-gate perf test asserted a best-of-5 wall-clock measurement against a CI-aware budget, so it flaked on loaded runners. This replaces it with a deterministic op-count check that asserts the classifier's algorithmic cost — one classify is O(viewport), not O(ring size) — in integers that cannot flake.

Test-only change; no production file is touched.

Fixes #1471

Root Cause

render-gate.test.ts measured performance.now() deltas around classifyScreen over a 4 MB replay and asserted best < (process.env.CI ? 800 : 250) ms. Two problems:

  1. It measures the machine, not the algorithm. Identical code that passes comfortably on an idle box measured 391.7ms when pinned to one contended core (taskset -c 0 + busy loops) — reproduced locally, AssertionError: expected 391.73237999999947 to be less than 250. Best-of-5-min narrows the spread but cannot remove it: a fully contended core has no lucky run. So the bound either flakes or gets loosened until it no longer catches the regression it exists for (its history: 75ms → 250 → a CI-aware 800).
  2. It measures the wrong path. It timed classifyScreen, the transient whole-ring entry that parses the entire replay into a throwaway Terminal per call — genuinely O(ring size). Production (round 2) classifies the persistent bounded SessionScreen mirror via classifyBuffer, which only reads an already-parsed viewport.

Fix

New suite — "deterministic op count: one classify is O(viewport), not O(ring size) (#1471)" — exercising the mirror path (SessionScreen.feed chunked as PtySession.onPtyData does → read()classifyBuffer).

classifyBuffer takes the terminal as a parameter and only reads it, so the test hands it a Proxy facade counting getLine / getCell calls and any bytes written. The real classifier runs; nothing is stubbed and no production instrumentation is added.

Test Asserts
4 MB vs ~200 B of history, same repainted final screen op counts are byte-identical — 20000× the history, identical work
one classify lineReads ≤ 2·rows, cellReads ≤ cols·rows, bytesParsed === 0
three classifies of a static screen flat per-classify cost (the backstop re-checks on a timer)
negative control the retired whole-ring path re-parses ~4 MB per classify — so the counter demonstrably discriminates the two cost models

The existing 4 MB whole-ring test keeps its correctness half (renders whole, no slice/size cap, classifies its busy tail); only the timing loop and the CI-aware budget are removed.

Verified against simulated regressions (both reverted afterward):

Simulated regression in render-gate.ts Result
screenLines scans from history start instead of viewportY 3 op-count tests fail
pure cost regression — walk all history, verdict unchanged 2 fail: lineReads 1131 vs the ≤130 viewport bound, 1131 vs 66 across history sizes

The second matters most: the verdict stays correct, so only the op count catches it — exactly what the wall-clock bound was proxying for.

Test Plan

  • Regression test added (fails under a simulated O(history) regression, passes on main's classifier)
  • Build passes (pnpm --filter @cluesmith/codev build)
  • All tests pass (4860 passed / 48 skipped / 0 failed)
  • Deterministic under load: all 46 tests in the file pass pinned to the contended core that broke the old assertion
  • Test file typechecks clean (the package build excludes __tests__, so this was checked under a temporary tsconfig)

🤖 Generated with Claude Code

mohidmakhdoomi and others added 6 commits August 17, 2026 19:26
…ministic op-count check

The whole-ring perf test asserted a best-of-5 wall-clock measurement against a
CI-aware budget (`process.env.CI ? 800 : 250` ms). Wall clock measures the
machine, not the algorithm: the identical code that passed comfortably on an
idle box measured 391.7ms pinned to one contended core. So the bound either
flakes on a loaded runner or gets loosened until it stops catching the
regression it exists for (75ms -> 250 -> a CI-aware 800).

Assert the cost property directly instead, on the path production actually
uses. Post round-2 the gate classifies the persistent bounded SessionScreen
mirror, so one classify is O(viewport): a Proxy facade over the mirror's live
terminal counts the classifier's getLine/getCell calls and any bytes it parses,
and the real `classifyBuffer` is handed that facade. Four tests pin it — 4 MB
of history and ~200 bytes of history ending in the same repainted screen cost
byte-identical work; one classify reads at most one viewport and parses zero
bytes; repeated classifies of a static screen cost the same each time; and a
negative control shows the retired whole-ring path re-parsing ~4 MB per
classify, so the counter demonstrably discriminates the two cost models.

Verified against simulated regressions: a pure cost regression (walk all
history, verdict unchanged) fails on lineReads 1131 vs the <=130 viewport
bound — the wall clock's job, now done in integers that cannot flake. The file
passes pinned to the contended core that broke the old assertion.

The 4 MB whole-ring test keeps its correctness half (renders whole, classifies
its busy tail); only the timing loop and the budget are removed. Test-only
change; no production file touched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All three lanes APPROVE with no blocking issues. Claude's three non-blocking
notes, addressed:

- The negative control's comment claimed the retired path re-parses "per
  classify" and does the "same cell reads", but the test performed one write +
  one classify and never compared cell reads against the mirror. It now
  classifies TWICE through throwaway terminals and asserts
  cellReads === mirror.cellReads * ROUNDS and
  bytesParsed === replay.length * ROUNDS, so both claims are exercised rather
  than inferred from classifyScreen's shape.
- Documented that cellReads <= cols*rows is the EXACT worst case, not a loose
  ceiling: a future second per-cell look-ahead trips it with no O(history)
  regression. That is deliberate — such a change doubles the gate's per-check
  work and deserves a decision, not a silent pass.
- Noted that classifyCounted inlines classifyAgentScreen's body (the facade has
  to sit between read() and classifyBuffer), so this suite pins the algorithm's
  cost, not the call-site wiring — with a pointer to the "PRODUCTION data path"
  suite that covers the wiring.

Re-verified after the change: the injected pure cost regression still fails
exactly the op-count tests (lineReads 1131 vs the <=130 bound); 46/46 in the
file; 4860 passed / 48 skipped / 0 failed for the package; test file typechecks
clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mohidmakhdoomi

Copy link
Copy Markdown
Collaborator Author

Architect integration review (risk tier: Low — test-only, one file)

Verdict: APPROVE (parked for maintainer review + merge; we are not maintainers).

  • Root cause is deeper than the issue framed, and the fix is aimed correctly. The flaky assertion didn't just measure wall-clock — it measured the retired whole-ring path (classifyScreen, genuinely O(ring size)) rather than production's persistent-mirror path (classifyBuffer, O(viewport)). The new op-count suite pins the invariant the wall-clock bound was proxying for, on the path production actually runs — exactly what Tracking: mailbox-first send follow-ups — gate correctness, interrupt semantics, attribution, visibility (post-1313) #1483's guidance asked for.
  • The counting facade measures the real classifier. A Proxy over the mirror's live terminal counts getLine/getCell/bytes; nothing is stubbed. Integer comparisons cannot flake on a loaded runner.
  • Regression evidence is the strongest part: an injected pure-cost regression (verdict unchanged, cost O(history)) is caught only by the op counts (1131 line reads vs the ≤130 geometric bound) — demonstrating the new test detects what the old one existed for but couldn't reliably catch. Claude's CMAP lane independently re-ran the experiment and reproduced the numbers.
  • The 4 MB test keeps its correctness half; only the timing loop and CI-aware budget are removed. Negative control shows the counter distinguishes the two cost models.
  • Builder CMAP: 3× APPROVE (HIGH). 7/7 GitHub CI checks green, including the shared runners the old assertion flaked on.

Deliberately scoped tight so the gate-hardening issues (#1473, #1474) touching this file later rebase cleanly.

mohidmakhdoomi and others added 4 commits August 17, 2026 19:48
PR #1487 parked for the maintainer to merge (we are not maintainers on this
project); issue #1471 auto-closes on merge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Render gate: replace the perf wall-clock assertion with a deterministic op-count check

1 participant