[PIR #1365] Serializer convergence: one lock at the terminal write edge - #1492
[PIR #1365] Serializer convergence: one lock at the terminal write edge#1492mohidmakhdoomi wants to merge 27 commits into
Conversation
…ence design Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rupt-latency ceiling Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e counter Adds the three pieces the convergence needs from the per-terminal lock: - trySubmitToSession / isSubmissionInFlight: a non-blocking acquisition for the gated delivery path, which must never queue. MailboxDrainer.tick walks agents sequentially, so one delivery parked on a terminal lock would stall every other agent's mail plus that tick's escalation and prune passes. - OPERATOR_SUBMIT_WAIT_CEILING_MS (2s) + SubmitOptions: operator submissions block, but boundedly. A paced write runs (lines-1)*10+80 ms and a body is capped only by parseJsonBody's 1 MiB, so an unbounded wait could stall afx interrupt -- the human's escape hatch -- for minutes. Past the ceiling it proceeds unserialized, which is exactly the pre-#1365 behaviour, so it is never worse than the old status quo. - unserializedWriteCount: degraded writes are counted per session so a concurrent delivery can detect that it was raced, rather than the ceiling opening a second silent-loss route. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mission lock The gated delivery path and the escape/interrupt paths held disjoint locks, so they could interleave on one terminal. The failure that mattered was not a garbled composer but a false 'delivered': a ^C landing inside a delivery's own text->Enter window cleared the composer, the delivery's Enter submitted nothing, every byte still reached the PTY so the write reported success, and the row was marked delivered for a message the agent never saw. --escape produced the truncated variant, and is the more likely trigger for a long body, whose exposed window is longest. submitMessagePaced replaces writeMessagePaced as the delivery's write edge: the same paced write, performed as one submission on the session's per-terminal lock, taken as a LEAF inside the per-agent serializer. Lock order is always per-agent -> per-terminal, and PtySession.write() emits no submit signal, so there is no cycle in either direction. The lock wraps the write only, never the gate classify -- --interrupt must not queue behind a screen classification. The port now reports written | dropped | contended | preempted | aborted instead of a bare boolean, so every not-delivered outcome holds the row rather than collapsing into 'no-live-pty'. Its precheck runs inside the lock and re-checks writability, the gate's ringToken, and the row's own status -- the last of which stops a dismiss landing during the write edge from putting bytes on the wire. Operator call sites (interrupt, escape, delayed ^C) pass the wait ceiling and log loudly when they degrade. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… key hygiene 23 new tests. The corruption cases are each paired with a bypass-the-lock CONTROL that reproduces the original bug (body lost to a ^C, body truncated by an ESC, both with the write still reporting success), so they assert the fix rather than merely exercising it. Also covered: the drainer keeps serving other agents while one terminal is held; a delivery declines contention immediately instead of waiting; the ceiling degrades and announces itself; a raced delivery reports preempted and HOLDS its row; deliveries to different terminals do not serialize; a session with no id throws. gateSession in tower-routes.test.ts gains a real id -- it is the hazard the review predicted: an un-annotated fake reaching the live wiring would have keyed every lock on undefined, collapsing per-terminal serialization into one global lock with nothing failing. The runtime guard turns that into a loud throw, and 13 tests duly failed until the fake was fixed. spec-1313-paced-write-drop is re-pointed from the retired writeMessagePaced onto submitMessagePaced, so the silent-loss fix stays guarded at the live write edge rather than at a function nothing calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rewrites session-submit.ts's 'Exactly what it covers' boundary comment (in the code commits) and arch.md section 7 item 5 so the model lives in one place rather than as a patchwork of separately-reasoned decisions: which writers take the lock and which stay deliberately uncovered, the per-agent -> per-terminal order, the deliveries-decline / operators-block asymmetry and why each side differs, the ceiling and its degradation, and the delayed-interrupt sequencing (^C on the timer, body through the gate, not atomic by design). States the guarantee honestly, per the plan review: serialization is the structural guarantee; the in-lock precheck NARROWS the echo-lag residual but cannot close it, since ringToken counts output and un-echoed input from an uncovered writer still reads as unchanged. That residual is #1473. The precheck's structural value is that it keeps the acquisition policy a free choice -- switching the delivery from declining to waiting (e.g. #1481 ordering an interrupt ahead of its body) stays safe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ve Tower afx dev from this worktree would bind the live Tower's port (4100 is shared by design) and restarting the live Tower kills every builder session, so the running-worktree evidence is produced the way send-integration.e2e.test.ts does: this worktree's built Tower spawned on port 14650, real shellper-backed PTY sessions, real HTTP endpoints. Nothing stubbed -- routes -> mailbox -> render gate -> locks -> PTY is the wire path. The oracle is the existing echo-terminal fixture (stty raw -echo; exec cat): the PTY re-emits every byte written to it, in order, so GET /api/terminals/:id/output is a faithful ordered transcript of what every writer actually put on the terminal. The profile resolves through the real wrapped-launch fallback (.builder-start.sh), which is how a live builder's profile resolves -- without it the gate holds everything no-profile and there is nothing to assert about. 66/66 checks. Scenario 3 is the sharpest: the interrupt returned in 2156ms rather than waiting out a ~4.1s paced write, Tower logged the degradation at WARN, and the raced delivery reported preempted and HELD its row -- the degraded path declining to claim a delivery it could not youch for, end to end. Two honest limits, stated in the transcript rather than approximated: the 503 TERMINAL_NOT_WRITABLE branch needs a shellper socket that died while the session still reports running, which cannot be produced from the public API without staging it (covered by tower-routes.test.ts:1560, and untouched by this change); and this fixture's agent exists only as a live terminal, never as a registry-known builder, so a send after its death correctly 404s rather than exercising the hold-instead-of-404 seam. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Routes the governance updates by tier: the arch.md section 7 rewrite landed with the code; the hot-tier lock-order invariant is APPENDED to the existing mailbox-first fact rather than added as an eleventh, so the 10-fact cap holds with no displacement. Three cold lessons. Both hot lessons candidates were considered and deliberately not promoted -- promoting either needs a displacement, which is the maintainer's call, not a builder's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Architect integration review — 3-way CMAP (risk tier: High — core write edge, session-submit/mailbox-delivery)Verdicts: gemini APPROVE · codex REQUEST_CHANGES · claude REQUEST_CHANGES — all HIGH confidence, and the two RC lanes independently converged on the same finding and the same fix. The blocking finding (verified by the architect against the branch)The 2s operator wait ceiling narrows Spec 1273's operator-vs-operator guarantee: pre-#1365, two operator submissions to one terminal were serialized unboundedly; with the ceiling, a second operator action can degrade past a long body-bearing first one and land inside its text→Enter window — and since the interrupt path claims Directed fix (both lanes agree): record the lock holder's kind in the chain and arm the ceiling only when a delivery write holds the lock — that bounds exactly the new exposure D3 was aimed at (the escape hatch stalling behind a long delivery) while leaving Spec 1273's operator serialization intact and making the "never worse" claim provably true per-pair. Plus: surface a degraded/raced indicator on the operator response alongside the existing WARN; correct the four claim-sites ( Non-blocking (in-PR if cheap, else flagged)
What all three lanes and the architect agree is rightThe convergence design itself: leaf per-terminal lock inside the per-agent serializer (cycle-free, all PTY writers accounted for), delivery fail-fast preserving drainer liveness, in-lock precheck with row-status re-check, the preempted counter sampled around the delivery write, honest narrowing (not closing) of the #1473 echo-lag residual, 23 paired tests each with a bypass-the-lock control, and 66/66 live-wire dev-approval evidence ( Awaiting the fix push; the pr gate follows. Parked for maintainer approval + merge; we are not maintainers. |
…raded writes Two real defects found by the PR #1492 consultation (codex and claude converged on the same core finding independently), both verified against the code before fixing. 1. OPERATOR-VS-OPERATOR REGRESSION. Before #1365 submitToSession had no ceiling, so two operator submissions to one terminal were ALWAYS fully serialized (Spec 1273). My ceiling let a second operator degrade past a long body-bearing first operator, making that one pair strictly WORSE than the status quo -- so the 'never worse' claim was false in that corner. Chain entries now carry a SubmissionKind and the ceiling arms only when nothing ahead is an operator (queued counts, not just in-flight). Restores exact pre-#1365 op-op serialization while keeping the escape hatch responsive against a long delivery, which was the ceiling's actual motivation. 'Never worse' is now true per pair, and the four claim-sites that overstated it are corrected. 2. UNREPORTED DEGRADED INTERRUPT BODY. A ceiling-expired interrupt writes its own body unserialized into a still-pacing predecessor, yet the row was claimed delivered up front and the response said delivered:true with no qualification. Claim-first is kept (un-claiming risks double delivery) but the truth is now surfaced: /api/send returns degraded:true + degradedReason, threaded through the SDK client and warned about by afx send. An indicator nobody surfaces is half a fix. Tests: op-op never degrades; a third operator does not bypass a QUEUED one; a body-bearing interrupt crossing the ceiling behind a delivery reports degraded. Note the pre-existing ceiling test needed its holder changed from an operator to a delivery -- that fixture change IS the behaviour change, not a workaround. 4882 passed / 0 failed. Refs #1492. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ion, claim sweep The two non-blocking correctness notes from the claude lane, plus the residual doc claims. Both blocking findings were already fixed in ece06a5. 1. THE BYPASS COUNTER NOW COUNTS BYTES, NOT INTENT. It was bumped on ceiling expiry unconditionally, but the delayed `^C` re-checks isStillLive()/writable INSIDE the lock and can return having written nothing. That no-op was still counted, forcing a concurrent delivery into a spurious `preempted` re-delivery -- a duplicate charged for a race that never happened. SubmitOptions.wroteBytes is consulted straight after the write callback, with NO await in between, so the ordering guarantee the old placement provided (a delivery cannot observe our bytes without also observing the bump) is unchanged. 2. unserializedWrites NOW SELF-EVICTS. It was retained for the life of the Tower -- one entry per session that ever degraded, the leak class #1472 fixed. It cannot self-delete on drain the way chains and pendingOperators do, because it must OUTLIVE the submission whose watcher is about to compare against it: a reset landing between a watcher's two reads reads as "nobody raced me", the exact false `delivered` this issue exists to eliminate. So eviction is interlocked with an explicit watchBypasses() window -- refused while a watch is open, attempted from BOTH the chain's drain cleanup and the last watch's release, so whichever runs second evicts and no ordering leaks. Needs no session-teardown hook and therefore no terminal/ -> agent-farm/ layer crossing. Also: the residual "never worse than the status quo" claim-sites now state the guarantee per pair (tower-routes' logCeilingExpired doc + the degraded-path inline comment; arch.md and the session-submit boundary comment were done in ece06a5). Stale {@link writeMessagePaced} repointed. DEGRADED_SUBMIT_REASON moved out from between logCeilingExpired's JSDoc and its function. Two nits refused with reasons rather than silently skipped (ceiling-timer cancellation needs abort semantics on the injected SubmitClock; the timing assertions are load-bearing) -- see the rebuttal file. Review doc carries the full REQUEST_CHANGES disposition, the two files outside the stated PR scope, and the codev/evidence/ placement the maintainer may veto. Tests: a degraded write that writes NOTHING is not counted; the counter is evicted once the session goes idle; eviction cannot land inside a watcher window and mask a race. Build clean (codev + sdk). 4885 passed / 0 failed / 48 skipped, 246 files. Refs #1492. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Architect fix-verification — review iteration 1 dispositions (pre-gate)Verified against the branch at 1. Kind-aware ceiling (converged codex/claude blocking finding) — verified fixed. 2. Degraded interrupt no longer reports unqualified success (blocking) — verified fixed. 3. Bytes-only bypass counting — verified. The bump moved inside the chain: 4. Counter eviction — verified. 5. Claim-site sweep — re-ran the grep myself. Every surviving 6. Pinning tests — all present, and I re-ran both files: 129/129 green ( Refusals — accepted as reasonable. Ceiling-timer cancellation would force abort semantics onto the injected Escalated to the human at the gate (not resolved here): the hot-tier Two out-of-scope files ( Contributor-workflow reminder: this PR parks for the maintainer — no self-merge. |
main advanced 25 commits during the #1365 review round (47477ca..9129ab8): AIR #1489 (`afx reset` → `afx refresh`), secfix-1 (Tower auth hardening), PIR #1495 (Stream Deck architect action). Merged rather than rebased so the verified #1365 history stays intact. One conflict, in `codev/resources/lessons-learned.md` § Architecture — an append-only collision, not a competing edit: this branch appended three #1365 lessons and main appended one from secfix-1 at the same point. Resolved KEEP-BOTH; all four entries are present and unmodified. Every code file auto-merged. The only changes the merge made to this PR's files come from main, not from #1365: the `afx reset` → `afx refresh` rename in two comments (session-submit.ts, mailbox-wiring.ts), and secfix-1's auth work in tower-routes.ts, tower-client.ts (`codev-web-key` → `codev-tower-key`) and tower-routes.test.ts. Every #1365 hunk survives byte-identical — message-write.ts, mailbox-delivery.ts, commands/send.ts and spec-1365-serializer-convergence.test.ts are untouched by the merge. PR #1492 stays OPEN and parked for the maintainer; #1365 stays open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Post-merge figures: build clean (codev + sdk), full suite 4934 passed / 0 failed / 48 skipped, 248 files (was 4885 / 246 pre-merge; the delta is main's own tests). Merge ebbc495, keep-both resolution of the lessons-learned append collision, and the verification that every #1365 hunk survived byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PIR Review: Serializer convergence — one lock at the terminal write edge
Fixes #1365
Summary
The gated mailbox delivery path and the
--interrupt/--escapepaths held disjointlocks (per-agent vs per-terminal), so they could interleave on one terminal. The failure that
mattered was not a garbled composer but a false
delivered: a^Clanding inside adelivery's own text→Enter window cleared the composer, the delivery's Enter submitted nothing,
every byte still reached the PTY so the write reported success, and the row was marked
delivered for a message the agent never saw. This PR routes the delivery's write edge through
the same per-terminal submission lock, taken as a leaf inside the per-agent serializer
(order: per-agent → per-terminal, no cycle), and lands the resulting model as one documented
boundary instead of three separately-reasoned decisions.
The issue asked for an evaluation before a remedy. That evaluation is in
codev/plans/1365-serializer-convergence-route-m.mdPart 1; it ratified convergence, andthis is its implementation.
Files Changed
Implementation:
packages/codev/src/agent-farm/servers/session-submit.ts—trySubmitToSession,isSubmissionInFlight,OPERATOR_SUBMIT_WAIT_CEILING_MS+SubmitOptions,SubmissionKindpendingOperators,unserializedWriteCount/watchBypasses, and the rewritten boundarycomment
packages/codev/src/agent-farm/servers/mailbox-delivery.ts(+144 / −20) —DeliverySession.id,WriteAbort/WriteResult, the in-lock precheck, the outcome mappingpackages/codev/src/agent-farm/servers/message-write.ts—submitMessagePaced(replaces
writeMessagePaced),PacedSubmitResultpackages/codev/src/agent-farm/servers/tower-routes.ts— wait ceiling at the threeoperator call sites,
logCeilingExpired,degradedon the send response, updated scope commentspackages/codev/src/agent-farm/servers/mailbox-wiring.ts— binds the new write edgepackages/codev/src/agent-farm/commands/send.ts— warns the sender on a degraded writepackages/sdk/src/tower-client.ts—degraded/degradedReasonon the sendMessage resultTwo of those are outside the 21-file scope this PR originally stated (
commands/send.tsand the SDK client), added by the review round below. They are the minimum needed to make a
degraded operator write visible to the sender rather than only to the Tower log, which
required crossing the server→client boundary. Flagged so the diff holds no surprises; the
boundary rule itself is respected (
codev-sdkstill imports onlycodev-types).Tests:
packages/codev/src/agent-farm/__tests__/spec-1365-serializer-convergence.test.ts— newpackages/codev/src/agent-farm/__tests__/spec-1313-paced-write-drop.test.ts— re-pointedtower-routes.test.ts,send-delivery.test.ts,send-mailbox-repro.test.ts,cron-delivery.test.ts,send-architect-identity.test.ts— fakes updatedEvidence + docs:
packages/codev/scripts/spec-1365-e2e-evidence.mts(+420 / −0) — dev-approval evidence scriptcodev/evidence/1365-dev-approval-transcript.txt(+80 / −0) — its transcriptcodev/resources/arch.md(+10 / −2),codev/resources/arch-critical.md,codev/resources/lessons-learned.mdcodev/plans/1365-...md,codev/state/pir-1365_thread.mdCommits
2adbe0b9[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Plan draft: evaluation of the three write paths + convergence designa6cdbe27[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Plan revised (rev 2): all 5 blocking review items + interrupt-latency ceiling30af22b2[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Lock primitives: try-acquire, wait ceiling, degraded-write countere9fd2d42[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Route the mailbox write edge through the per-terminal submission lock194685e1[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Tests: interleaving, in-lock precheck, liveness, ceiling, key hygienedcf22a5f[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Document the converged write-edge model in one place54d57008[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Thread log: implement phaseee3a17df[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] dev-approval evidence: 4 scenarios against an isolated live Tower1483d65c[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Thread log: dev-approval evidencead824bf2[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Review + retrospectiveece06a5e[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Fix codex/claude finding: kind-aware ceiling + report degraded writes0dc75d8e[PIR Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365] Review round 2: byte-accurate bypass count, counter eviction, claim sweepTest Results
pnpm --filter @cluesmith/codev build: ✓ passpnpm --filter @cluesmith/codev-sdk build: ✓ passpnpm --filter @cluesmith/codev test: ✓ pass — 4934 passed / 0 failed / 48 skipped,248 files, after merging
origin/main. 28 new tests inspec-1365-serializer-convergence.test.ts, plus the degraded-interrupt response test intower-routes.test.ts. (Before the merge, on the Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 work alone: 4885 / 0 / 48, 246 files— the delta is main's own tests arriving, not tests changing behaviour here.)
origin/mainafter theprgate was approved (mergeebbc495dc, not a rebase —the reviewed history is preserved).
mainhad advanced 25 commits during the review round(AIR Rename
afx resettoafx refresh(keepresetas a deprecated alias for one release) #1489'safx reset→afx refresh, secfix-1's Tower auth hardening, PIR Stream Deck: Architect Action key — scope the fleet to one architect's builders #1495). Oneconflict, in
codev/resources/lessons-learned.md§ Architecture: an append-only collision(three Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 entries vs one secfix-1 entry at the same point), resolved keep-both, all
four entries present and unmodified. Every code file auto-merged; every Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 hunk survives
byte-identical, and
message-write.ts,mailbox-delivery.ts,commands/send.tsandspec-1365-serializer-convergence.test.tswere untouched by the merge.afx devwas not usable —4100 is shared by design and restarting the live Tower kills every builder session — so the
running-worktree evidence was scripted against an isolated Tower on port 14650 with real
shellper-backed PTYs and real HTTP endpoints (routes → mailbox → render gate → locks → PTY,
nothing stubbed). 66/66 checks. Full transcript:
codev/evidence/1365-dev-approval-transcript.txt; script:packages/codev/scripts/spec-1365-e2e-evidence.mts.--interrupt: 10/10 bodies reached the wirewhole, zero fragmentation, zero duplication.
--delay 5 --interruptmid-turn: scheduled;^Cat due time; body did not landmid-turn; landed exactly once after the prompt cleared;
^Cbefore body.paced write; Tower logged the degradation at WARN; the raced delivery reported
preemptedand held its row (delivered=false held=true reason=busy).--escapeunchanged (ESC + Enter on the wire); a dead terminal is refused, neversilently dropped.
Architecture Updates
COLD —
codev/resources/arch.md§7 item 5 (rewritten). The old text described the twolocks as disjoint with the cross-path race as an accepted boundary; that is now false. The
replacement carries the whole model in one place: which writers take the lock and which stay
deliberately uncovered, the per-agent → per-terminal order and why there is no cycle, the
deliveries-decline / operators-block asymmetry and the reason each side differs, the wait
ceiling and its degradation, the delayed-interrupt sequencing, and — stated honestly — that
serialization is the structural guarantee while the in-lock precheck only narrows the
echo-lag residual (#1473).
HOT —
codev/resources/arch-critical.md: the existing mailbox-first fact already governs"any new message writer", so the lock-order invariant was appended to that fact rather
than added as an eleventh. This keeps the tier at its 10-fact cap with no displacement —
the hot tier gains the one clause a future author actually needs at decision time ("take
submitToSession; order is per-agent → per-terminal"), not a second entry on the samesubject.
Lessons Learned Updates
COLD —
codev/resources/lessons-learned.md→ Architecture, three entries:transport acceptance, not semantic loss — a
^Cthat clears the composer leaves everywrite returning
true. Any success signal derived from "did the transport accept thebytes" needs a second question, answered from a source the transport can't lie about.
becomes a key (lock, cache, registry), assert its presence at the boundary — type-checking
the shape does not check the key. Here a double without an
idkeyed every per-terminallock on
undefined: a silently global lock, no failing assertion anywhere.match each caller's liveness needs. Blocking would have regressed both sides (the
sequential drainer, and the human's escape hatch). The shape that works is asymmetric:
the background writer declines contention and retries on its existing schedule; the
operator waits, but boundedly, degrading to documented prior behaviour rather than a hang.
Considered for HOT and deliberately not promoted: both are real but narrower than the
current ten hot lessons, and promoting either would require displacing an existing one.
Displacement at the cap is the maintainer's call, not a builder's — flagged here rather than
taken unilaterally.
Review Round: two REQUEST_CHANGES, and what happened to each
PIR's consultation is single-pass — there is no second automated round — so the human at
the
prgate is the only remaining reviewer of these dispositions. They are written out infull rather than summarised.
Two independent review sets ran, and they did not agree:
codev/projects/1365-.../)My own codex lane approved; the architect's codex lane found a real bug. I verified every
finding against the code before acting on it, and none was dismissed on the strength of
another lane's APPROVE. Both REQUEST_CHANGES lanes converged independently on the same two
blocking findings.
Blocking 1 — the ceiling could bypass another operator's submission. ACCEPTED, real,
fixed in
ece06a5e.boundedkeyed only off "is anything in flight" without asking whatkind of writer was ahead, so a second
--interruptcould skip a first one carrying a longbody after 2 s. Operator-vs-operator was always fully serialized before #1365
(
submitToSessionhad no ceiling at all — it is Spec 1273's/clearfusion bug), so myceiling made that one pair strictly worse than the status quo. That also falsified this
document's own "never worse" claim. Fix: chain entries carry a
SubmissionKind, apendingOperatorscount tracks operators queued as well as in flight, and the ceiling armsonly when nothing ahead is an operator. Queued has to count — bypassing an operator that has
not started yet is the same violation as bypassing one mid-write. Pinned by "operator vs
operator NEVER degrades" and "a THIRD operator does not bypass a QUEUED one". Note the
pre-existing ceiling test needed its holder changed from an operator to a delivery: that
fixture change is the behaviour change, not a workaround for it.
Blocking 2 — a ceiling-degraded
--interruptstill reported unqualified success.ACCEPTED, real, fixed in
ece06a5e. The row is claimeddeliveredbefore the write, so adegraded interrupt returned
delivered: truewith only a Tower-side WARN — the samelying-success-signal class this whole issue exists to remove, relocated from the delivery path
to the operator path. Claim-first is kept (un-claiming risks a double delivery, reasoned
through at CMAP round 3 of the implement phase); what changed is that the truth is now
surfaced:
/api/sendreturnsdegraded: true+degradedReason, threaded through the SDKclient and warned about by
afx send. An indicator nobody surfaces is half a fix. Pinned by"a body-bearing interrupt that crosses the wait ceiling reports degraded" in
tower-routes.test.ts.Non-blocking, taken anyway (this commit):
delayed
^Cre-checksisStillLive()/writableinside the lock and can return havingwritten nothing; that no-op was still counted, forcing a concurrent delivery into a spurious
preemptedre-delivery. The counter answers "did bytes bypass the lock while I held it?", soonly bytes may bump it:
SubmitOptions.wroteBytesis consulted straight after the writecallback, with no
awaitin between, so the ordering guarantee the old placement provided isunchanged. Pinned by "a degraded write that writes NOTHING is not counted as a bypass".
unserializedWriteswas never pruned — one entry per session that ever degraded, retainedfor the life of the Tower. The leak class VSCode: bound the mailbox escalation-toast
seenSet (dedupe by mailboxId with eviction) #1472 just fixed. It cannot self-delete on drainthe way
chainsandpendingOperatorsdo, because it must outlive the submission whosewatcher is about to compare against it: a reset landing between a watcher's two reads would
read as "nobody raced me" — the exact false
deliveredthis issue exists to eliminate. Soeviction is interlocked with an explicit
watchBypasseswindow: refused while a watch isopen, attempted from both the chain's drain cleanup and the last watch's release, so
whichever runs second is the one that evicts and no ordering leaks. This needs no
session-teardown hook and therefore no
terminal/→agent-farm/layer crossing. Pinned by"the degraded-write counter is evicted once the session goes idle" and "eviction cannot
land inside a watcher window and mask a race".
{@link writeMessagePaced}inmessage-write.ts— repointed atsubmitMessagePaced.DEGRADED_SUBMIT_REASONwas inserted betweenlogCeilingExpired's JSDoc and its function,orphaning the comment — moved above it.
the guarantee per pair (op↔op unchanged and unbounded; op↔delivery serialized under the
ceiling and degraded to the old disjoint-lock behaviour above it; delivery↔delivery
unchanged).
arch.md§7 item 5 and thesession-submit.tsboundary comment were corrected inece06a5e;tower-routes.ts'slogCeilingExpireddoc comment and the degraded-path inlinecomment in this one.
Non-blocking, NOT taken — flagged instead:
Promise.raceleaves a ≤2 s
setTimeoutpending whose resolution is then discarded. Cancelling it meansadding abort semantics to the injected
SubmitClockinterface, which every test doubleimplements. The cost of leaving it is one short-lived timer per contended operator
submission; the cost of fixing it is a broader interface change late in a review round. A
reviewer who disagrees should say so — it is a small change, just not a free one.
waited < 100/tickMs < 250are timing-sensitive under CI load. Real, and deliberate:these are the assertions that make "the drainer does not stall" and "the escape hatch stays
responsive" testable claims rather than prose. Both have ≥2.5× headroom over the behaviour
they exclude. If they flake in CI, raising the bounds preserves the property.
arch-critical.mdwas appended to directly, where the plan said it would beproposed. Disclosed below; it is the human's call, and reverting it is a one-line edit.
Things to Look At During PR Review
so in production today the precheck cannot observe a state change a pre-lock check missed —
no macrotask can interleave between them. I kept it (both plan reviewers asked for it, the
human ratified it) but documented its actual value rather than implying it closes a live
race: it backstops the injected port boundary, and it is what keeps the acquisition policy
a free choice if afx send: add --interrupt-after <seconds> (hold, then force-deliver after a bounded wait) #1481 later wants the delivery to wait behind an interrupt. If a reviewer
would rather not carry code whose value is conditional on a future change, this is the
place to say so.
OPERATOR_SUBMIT_WAIT_CEILING_MS = 2000, human-ratified at the plan gate). It exists because
--interruptpreviously never waited, and apaced write runs
(lines−1)×10+80ms against a body capped only byparseJsonBody's1 MiB — a 48 KB
--fileof short lines is ~8 minutes. It arms only against a deliverywrite: behind another operator the wait stays unbounded, exactly as before Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 (see the
review round above — a ceiling that could skip an operator made that pair strictly worse,
and that was a real blocking finding, not a hypothetical). So past the ceiling the operator
write falls back to precisely the pre-Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 operator-vs-delivery behaviour — two disjoint
locks, no serialization — which is no worse for that pair, only no longer silent. The
guarantee is per pair, and the 2 s value itself remains a judgment call.
preemptedtrades a possible duplicate for never falsely reporting delivery. Adelivery raced by a ceiling-expired write holds its row instead of marking it delivered, so
if the message did land intact the gate may deliver it again later. That is the same call
the existing dropped-write branch already makes, and the opposite of the interrupt path's
claim-first tradeoff — the asymmetry is deliberate (an operator's own message vs an
autonomous background delivery), but it is worth a second opinion.
writeMessagegained a 4th parameter and a typed result acrosssix test files. One override (
send-delivery.test.ts:604) previously returnedundefinedand relied on falsy ⇒ hold; it is now explicit.
writeMessagePacedwas removed, not deprecated — its only live caller was the mailboxwiring. Its drop-semantics test is re-pointed at
submitMessagePacedso the Shellper reconnect error is swallowed: terminal becomes a silent zombie (no input/output, 'Message sent' logged for dropped frames) until next Tower restart #1198silent-loss guard stays on the live write edge rather than on a function nothing calls.
codev/evidence/, a new directory. Thatplacement is a deliberate choice, not an accident: the evidence is part of the PIR record
for this project, the way
codev/specs/,codev/plans/andcodev/reviews/are, and agate approved on evidence that then vanishes leaves the approval unauditable. It is
nonetheless a new top-level convention in the repo, and the maintainer may veto it —
moving or dropping the file changes nothing else in the PR (the generating script,
packages/codev/scripts/spec-1365-e2e-evidence.mts, is re-runnable and is the durableartifact).
Interlock for #1481 (
--interrupt-after): "interrupt, then deliver this body" is nowexpressible as ordered acquisitions of one lock rather than a race between two. Two
residuals it must design against, both documented: the
^C→body gap is gate-mediated anddeliberately not atomic (the delayed interrupt guarantees "the turn was ended", never
"this body is next"), and a no-op
^Cis only logged.How to Test Locally
pir-1365→ Review Diffafx dev pir-1365— but note it will contend for the live Tower's port; theisolated-Tower script below is why the dev-approval evidence took that route instead
pnpm --filter @cluesmith/codev build && node --experimental-strip-types packages/codev/scripts/spec-1365-e2e-evidence.mts(isolated Tower on 14650, ~90 s, exits non-zero on any failed check)
pnpm --filter @cluesmith/codev test spec-1365-serializer-convergenceafx interruptnever fuses and neverleaves an
afx inbox showrow readingdeliveredwhose text is absent or partial; anotheragent's mail keeps flowing while a large body is mid-delivery;
afx interruptstaysresponsive against a busy line;
--escapebehaviour is unchanged.Flaky Tests
None. No tests were skipped or quarantined, and no pre-existing unrelated failures were
touched.