Fix #1471: replace the render-gate perf wall-clock bound with a deterministic op-count check - #1487
Open
mohidmakhdoomi wants to merge 10 commits into
Open
Fix #1471: replace the render-gate perf wall-clock bound with a deterministic op-count check#1487mohidmakhdoomi wants to merge 10 commits into
mohidmakhdoomi wants to merge 10 commits into
Conversation
…ministic op-count check The whole-ring perf test asserted a best-of-5 wall-clock measurement against a CI-aware budget (`process.env.CI ? 800 : 250` ms). Wall clock measures the machine, not the algorithm: the identical code that passed comfortably on an idle box measured 391.7ms pinned to one contended core. So the bound either flakes on a loaded runner or gets loosened until it stops catching the regression it exists for (75ms -> 250 -> a CI-aware 800). Assert the cost property directly instead, on the path production actually uses. Post round-2 the gate classifies the persistent bounded SessionScreen mirror, so one classify is O(viewport): a Proxy facade over the mirror's live terminal counts the classifier's getLine/getCell calls and any bytes it parses, and the real `classifyBuffer` is handed that facade. Four tests pin it — 4 MB of history and ~200 bytes of history ending in the same repainted screen cost byte-identical work; one classify reads at most one viewport and parses zero bytes; repeated classifies of a static screen cost the same each time; and a negative control shows the retired whole-ring path re-parsing ~4 MB per classify, so the counter demonstrably discriminates the two cost models. Verified against simulated regressions: a pure cost regression (walk all history, verdict unchanged) fails on lineReads 1131 vs the <=130 viewport bound — the wall clock's job, now done in integers that cannot flake. The file passes pinned to the contended core that broke the old assertion. The 4 MB whole-ring test keeps its correctness half (renders whole, classifies its busy tail); only the timing loop and the budget are removed. Test-only change; no production file touched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All three lanes APPROVE with no blocking issues. Claude's three non-blocking notes, addressed: - The negative control's comment claimed the retired path re-parses "per classify" and does the "same cell reads", but the test performed one write + one classify and never compared cell reads against the mirror. It now classifies TWICE through throwaway terminals and asserts cellReads === mirror.cellReads * ROUNDS and bytesParsed === replay.length * ROUNDS, so both claims are exercised rather than inferred from classifyScreen's shape. - Documented that cellReads <= cols*rows is the EXACT worst case, not a loose ceiling: a future second per-cell look-ahead trips it with no O(history) regression. That is deliberate — such a change doubles the gate's per-check work and deserves a decision, not a silent pass. - Noted that classifyCounted inlines classifyAgentScreen's body (the facade has to sit between read() and classifyBuffer), so this suite pins the algorithm's cost, not the call-site wiring — with a pointer to the "PRODUCTION data path" suite that covers the wiring. Re-verified after the change: the injected pure cost regression still fails exactly the op-count tests (lineReads 1131 vs the <=130 bound); 46/46 in the file; 4860 passed / 48 skipped / 0 failed for the package; test file typechecks clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
Architect integration review (risk tier: Low — test-only, one file)Verdict: APPROVE (parked for maintainer review + merge; we are not maintainers).
Deliberately scoped tight so the gate-hardening issues (#1473, #1474) touching this file later rebase cleanly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The render-gate perf test asserted a best-of-5 wall-clock measurement against a CI-aware budget, so it flaked on loaded runners. This replaces it with a deterministic op-count check that asserts the classifier's algorithmic cost — one classify is O(viewport), not O(ring size) — in integers that cannot flake.
Test-only change; no production file is touched.
Fixes #1471
Root Cause
render-gate.test.tsmeasuredperformance.now()deltas aroundclassifyScreenover a 4 MB replay and assertedbest < (process.env.CI ? 800 : 250)ms. Two problems:taskset -c 0+ busy loops) — reproduced locally,AssertionError: expected 391.73237999999947 to be less than 250. Best-of-5-min narrows the spread but cannot remove it: a fully contended core has no lucky run. So the bound either flakes or gets loosened until it no longer catches the regression it exists for (its history: 75ms → 250 → a CI-aware 800).classifyScreen, the transient whole-ring entry that parses the entire replay into a throwawayTerminalper call — genuinely O(ring size). Production (round 2) classifies the persistent boundedSessionScreenmirror viaclassifyBuffer, which only reads an already-parsed viewport.Fix
New suite — "deterministic op count: one classify is O(viewport), not O(ring size) (#1471)" — exercising the mirror path (
SessionScreen.feedchunked asPtySession.onPtyDatadoes →read()→classifyBuffer).classifyBuffertakes the terminal as a parameter and only reads it, so the test hands it aProxyfacade countinggetLine/getCellcalls and any bytes written. The real classifier runs; nothing is stubbed and no production instrumentation is added.lineReads ≤ 2·rows,cellReads ≤ cols·rows,bytesParsed === 0The existing 4 MB whole-ring test keeps its correctness half (renders whole, no slice/size cap, classifies its busy tail); only the timing loop and the CI-aware budget are removed.
Verified against simulated regressions (both reverted afterward):
render-gate.tsscreenLinesscans from history start instead ofviewportYlineReads1131 vs the ≤130 viewport bound, 1131 vs 66 across history sizesThe second matters most: the verdict stays correct, so only the op count catches it — exactly what the wall-clock bound was proxying for.
Test Plan
main's classifier)pnpm --filter @cluesmith/codev build)__tests__, so this was checked under a temporary tsconfig)🤖 Generated with Claude Code