Skip to content

[PIR #1365] Serializer convergence: one lock at the terminal write edge - #1492

Open
mohidmakhdoomi wants to merge 27 commits into
mainfrom
builder/pir-1365
Open

[PIR #1365] Serializer convergence: one lock at the terminal write edge#1492
mohidmakhdoomi wants to merge 27 commits into
mainfrom
builder/pir-1365

Conversation

@mohidmakhdoomi

@mohidmakhdoomi mohidmakhdoomi commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

PIR Review: Serializer convergence — one lock at the terminal write edge

Fixes #1365

Summary

The gated mailbox delivery path and the --interrupt / --escape paths held disjoint
locks (per-agent vs per-terminal), so they could interleave on one terminal. The failure that
mattered was not a garbled composer but a false delivered: a ^C landing inside a
delivery's own text→Enter window cleared the composer, the delivery's Enter submitted nothing,
every byte still reached the PTY so the write reported success, and the row was marked
delivered for a message the agent never saw. This PR routes the delivery's write edge through
the same per-terminal submission lock, taken as a leaf inside the per-agent serializer
(order: per-agent → per-terminal, no cycle), and lands the resulting model as one documented
boundary instead of three separately-reasoned decisions.

The issue asked for an evaluation before a remedy. That evaluation is in
codev/plans/1365-serializer-convergence-route-m.md Part 1; it ratified convergence, and
this is its implementation.

Files Changed

Implementation:

  • packages/codev/src/agent-farm/servers/session-submit.tstrySubmitToSession,
    isSubmissionInFlight, OPERATOR_SUBMIT_WAIT_CEILING_MS + SubmitOptions, SubmissionKind
    • pendingOperators, unserializedWriteCount / watchBypasses, and the rewritten boundary
      comment
  • packages/codev/src/agent-farm/servers/mailbox-delivery.ts (+144 / −20) — DeliverySession.id,
    WriteAbort / WriteResult, the in-lock precheck, the outcome mapping
  • packages/codev/src/agent-farm/servers/message-write.tssubmitMessagePaced
    (replaces writeMessagePaced), PacedSubmitResult
  • packages/codev/src/agent-farm/servers/tower-routes.ts — wait ceiling at the three
    operator call sites, logCeilingExpired, degraded on the send response, updated scope comments
  • packages/codev/src/agent-farm/servers/mailbox-wiring.ts — binds the new write edge
  • packages/codev/src/agent-farm/commands/send.ts — warns the sender on a degraded write
  • packages/sdk/src/tower-client.tsdegraded / degradedReason on the sendMessage result

Two of those are outside the 21-file scope this PR originally stated (commands/send.ts
and the SDK client), added by the review round below. They are the minimum needed to make a
degraded operator write visible to the sender rather than only to the Tower log, which
required crossing the server→client boundary. Flagged so the diff holds no surprises; the
boundary rule itself is respected (codev-sdk still imports only codev-types).

Tests:

  • packages/codev/src/agent-farm/__tests__/spec-1365-serializer-convergence.test.ts — new
  • packages/codev/src/agent-farm/__tests__/spec-1313-paced-write-drop.test.ts — re-pointed
  • tower-routes.test.ts, send-delivery.test.ts, send-mailbox-repro.test.ts,
    cron-delivery.test.ts, send-architect-identity.test.ts — fakes updated

Evidence + docs:

  • packages/codev/scripts/spec-1365-e2e-evidence.mts (+420 / −0) — dev-approval evidence script
  • codev/evidence/1365-dev-approval-transcript.txt (+80 / −0) — its transcript
  • codev/resources/arch.md (+10 / −2), codev/resources/arch-critical.md,
    codev/resources/lessons-learned.md
  • codev/plans/1365-...md, codev/state/pir-1365_thread.md

Commits

Test Results

  • pnpm --filter @cluesmith/codev build: ✓ pass
  • pnpm --filter @cluesmith/codev-sdk build: ✓ pass
  • pnpm --filter @cluesmith/codev test: ✓ pass — 4934 passed / 0 failed / 48 skipped,
    248 files, after merging origin/main. 28 new tests in
    spec-1365-serializer-convergence.test.ts, plus the degraded-interrupt response test in
    tower-routes.test.ts. (Before the merge, on the Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 work alone: 4885 / 0 / 48, 246 files
    — the delta is main's own tests arriving, not tests changing behaviour here.)
  • Merged origin/main after the pr gate was approved (merge ebbc495dc, not a rebase —
    the reviewed history is preserved). main had advanced 25 commits during the review round
    (AIR Rename afx reset to afx refresh (keep reset as a deprecated alias for one release) #1489's afx resetafx refresh, secfix-1's Tower auth hardening, PIR Stream Deck: Architect Action key — scope the fleet to one architect's builders #1495). One
    conflict, in codev/resources/lessons-learned.md § Architecture: an append-only collision
    (three Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 entries vs one secfix-1 entry at the same point), resolved keep-both, all
    four entries present and unmodified. Every code file auto-merged; every Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 hunk survives
    byte-identical, and message-write.ts, mailbox-delivery.ts, commands/send.ts and
    spec-1365-serializer-convergence.test.ts were untouched by the merge.
  • Manual verification (dev-approval gate, human-approved): afx dev was not usable —
    4100 is shared by design and restarting the live Tower kills every builder session — so the
    running-worktree evidence was scripted against an isolated Tower on port 14650 with real
    shellper-backed PTYs and real HTTP endpoints (routes → mailbox → render gate → locks → PTY,
    nothing stubbed). 66/66 checks. Full transcript:
    codev/evidence/1365-dev-approval-transcript.txt; script:
    packages/codev/scripts/spec-1365-e2e-evidence.mts.
    • S1 — 10× long multi-line send raced by --interrupt: 10/10 bodies reached the wire
      whole, zero fragmentation, zero duplication.
    • S2 — --delay 5 --interrupt mid-turn: scheduled; ^C at due time; body did not land
      mid-turn; landed exactly once after the prompt cleared; ^C before body.
    • S3 — interrupt vs a busy line: returned in 2156 ms rather than waiting out a ~4.1 s
      paced write; Tower logged the degradation at WARN; the raced delivery reported
      preempted and held its row (delivered=false held=true reason=busy).
    • S4 — --escape unchanged (ESC + Enter on the wire); a dead terminal is refused, never
      silently dropped.
    • Live Tower on 4100 verified healthy and untouched afterwards; no orphan processes.

Architecture Updates

COLD — codev/resources/arch.md §7 item 5 (rewritten). The old text described the two
locks as disjoint with the cross-path race as an accepted boundary; that is now false. The
replacement carries the whole model in one place: which writers take the lock and which stay
deliberately uncovered, the per-agent → per-terminal order and why there is no cycle, the
deliveries-decline / operators-block asymmetry and the reason each side differs, the wait
ceiling and its degradation, the delayed-interrupt sequencing, and — stated honestly — that
serialization is the structural guarantee while the in-lock precheck only narrows the
echo-lag residual (#1473).

HOT — codev/resources/arch-critical.md: the existing mailbox-first fact already governs
"any new message writer", so the lock-order invariant was appended to that fact rather
than added as an eleventh. This keeps the tier at its 10-fact cap with no displacement
the hot tier gains the one clause a future author actually needs at decision time ("take
submitToSession; order is per-agent → per-terminal"), not a second entry on the same
subject.

Lessons Learned Updates

COLD — codev/resources/lessons-learned.md → Architecture, three entries:

  1. "Every byte reached the PTY" is not "the message landed." A write-success boolean sees
    transport acceptance, not semantic loss — a ^C that clears the composer leaves every
    write returning true. Any success signal derived from "did the transport accept the
    bytes" needs a second question, answered from a source the transport can't lie about.
  2. A structurally-typed port makes an omitted field compile and pass. When a value
    becomes a key (lock, cache, registry), assert its presence at the boundary — type-checking
    the shape does not check the key. Here a double without an id keyed every per-terminal
    lock on undefined: a silently global lock, no failing assertion anywhere.
  3. Converging two locks is not just "take the same lock" — the acquisition policy has to
    match each caller's liveness needs.
    Blocking would have regressed both sides (the
    sequential drainer, and the human's escape hatch). The shape that works is asymmetric:
    the background writer declines contention and retries on its existing schedule; the
    operator waits, but boundedly, degrading to documented prior behaviour rather than a hang.

Considered for HOT and deliberately not promoted: both are real but narrower than the
current ten hot lessons, and promoting either would require displacing an existing one.
Displacement at the cap is the maintainer's call, not a builder's — flagged here rather than
taken unilaterally.

Review Round: two REQUEST_CHANGES, and what happened to each

PIR's consultation is single-pass — there is no second automated round — so the human at
the pr gate is the only remaining reviewer of these dispositions. They are written out in
full rather than summarised.

Two independent review sets ran, and they did not agree:

Reviewer Protocol CMAP (codev/projects/1365-.../) Architect's CMAP on PR #1492
gemini APPROVE APPROVE
codex APPROVE REQUEST_CHANGES
claude REQUEST_CHANGES REQUEST_CHANGES

My own codex lane approved; the architect's codex lane found a real bug. I verified every
finding against the code before acting on it, and none was dismissed on the strength of
another lane's APPROVE
. Both REQUEST_CHANGES lanes converged independently on the same two
blocking findings.

Blocking 1 — the ceiling could bypass another operator's submission. ACCEPTED, real,
fixed in ece06a5e. bounded keyed only off "is anything in flight" without asking what
kind
of writer was ahead, so a second --interrupt could skip a first one carrying a long
body after 2 s. Operator-vs-operator was always fully serialized before #1365
(submitToSession had no ceiling at all — it is Spec 1273's /clear fusion bug), so my
ceiling made that one pair strictly worse than the status quo. That also falsified this
document's own "never worse" claim. Fix: chain entries carry a SubmissionKind, a
pendingOperators count tracks operators queued as well as in flight, and the ceiling arms
only when nothing ahead is an operator. Queued has to count — bypassing an operator that has
not started yet is the same violation as bypassing one mid-write. Pinned by "operator vs
operator NEVER degrades"
and "a THIRD operator does not bypass a QUEUED one". Note the
pre-existing ceiling test needed its holder changed from an operator to a delivery: that
fixture change is the behaviour change
, not a workaround for it.

Blocking 2 — a ceiling-degraded --interrupt still reported unqualified success.
ACCEPTED, real, fixed in ece06a5e. The row is claimed delivered before the write, so a
degraded interrupt returned delivered: true with only a Tower-side WARN — the same
lying-success-signal class this whole issue exists to remove, relocated from the delivery path
to the operator path. Claim-first is kept (un-claiming risks a double delivery, reasoned
through at CMAP round 3 of the implement phase); what changed is that the truth is now
surfaced: /api/send returns degraded: true + degradedReason, threaded through the SDK
client and warned about by afx send. An indicator nobody surfaces is half a fix. Pinned by
"a body-bearing interrupt that crosses the wait ceiling reports degraded" in
tower-routes.test.ts.

Non-blocking, taken anyway (this commit):

  • The bypass counter was bumped on ceiling expiry regardless of whether bytes went out. The
    delayed ^C re-checks isStillLive() / writable inside the lock and can return having
    written nothing; that no-op was still counted, forcing a concurrent delivery into a spurious
    preempted re-delivery. The counter answers "did bytes bypass the lock while I held it?", so
    only bytes may bump it: SubmitOptions.wroteBytes is consulted straight after the write
    callback, with no await in between, so the ordering guarantee the old placement provided is
    unchanged. Pinned by "a degraded write that writes NOTHING is not counted as a bypass".
  • unserializedWrites was never pruned — one entry per session that ever degraded, retained
    for the life of the Tower. The leak class VSCode: bound the mailbox escalation-toast seen Set (dedupe by mailboxId with eviction) #1472 just fixed. It cannot self-delete on drain
    the way chains and pendingOperators do, because it must outlive the submission whose
    watcher is about to compare against it: a reset landing between a watcher's two reads would
    read as "nobody raced me" — the exact false delivered this issue exists to eliminate. So
    eviction is interlocked with an explicit watchBypasses window: refused while a watch is
    open, attempted from both the chain's drain cleanup and the last watch's release, so
    whichever runs second is the one that evicts and no ordering leaks. This needs no
    session-teardown hook
    and therefore no terminal/agent-farm/ layer crossing. Pinned by
    "the degraded-write counter is evicted once the session goes idle" and "eviction cannot
    land inside a watcher window and mask a race"
    .
  • Stale {@link writeMessagePaced} in message-write.ts — repointed at submitMessagePaced.
  • DEGRADED_SUBMIT_REASON was inserted between logCeilingExpired's JSDoc and its function,
    orphaning the comment — moved above it.
  • The residual "never worse than the status quo" claim-sites — swept and rewritten to state
    the guarantee per pair (op↔op unchanged and unbounded; op↔delivery serialized under the
    ceiling and degraded to the old disjoint-lock behaviour above it; delivery↔delivery
    unchanged). arch.md §7 item 5 and the session-submit.ts boundary comment were corrected in
    ece06a5e; tower-routes.ts's logCeilingExpired doc comment and the degraded-path inline
    comment in this one.

Non-blocking, NOT taken — flagged instead:

  • The ceiling timer is not cancelled when the predecessor wins the race. Promise.race
    leaves a ≤2 s setTimeout pending whose resolution is then discarded. Cancelling it means
    adding abort semantics to the injected SubmitClock interface, which every test double
    implements. The cost of leaving it is one short-lived timer per contended operator
    submission; the cost of fixing it is a broader interface change late in a review round. A
    reviewer who disagrees should say so — it is a small change, just not a free one.
  • waited < 100 / tickMs < 250 are timing-sensitive under CI load. Real, and deliberate:
    these are the assertions that make "the drainer does not stall" and "the escape hatch stays
    responsive" testable claims rather than prose. Both have ≥2.5× headroom over the behaviour
    they exclude. If they flake in CI, raising the bounds preserves the property.
  • Hot-tier arch-critical.md was appended to directly, where the plan said it would be
    proposed.
    Disclosed below; it is the human's call, and reverting it is a one-line edit.

Things to Look At During PR Review

  1. The in-lock precheck's honest status. With try-lock semantics the delivery never waits,
    so in production today the precheck cannot observe a state change a pre-lock check missed —
    no macrotask can interleave between them. I kept it (both plan reviewers asked for it, the
    human ratified it) but documented its actual value rather than implying it closes a live
    race: it backstops the injected port boundary, and it is what keeps the acquisition policy
    a free choice if afx send: add --interrupt-after <seconds> (hold, then force-deliver after a bounded wait) #1481 later wants the delivery to wait behind an interrupt. If a reviewer
    would rather not carry code whose value is conditional on a future change, this is the
    place to say so.
  2. The wait ceiling is a judgment call (OPERATOR_SUBMIT_WAIT_CEILING_MS = 2000, human-
    ratified at the plan gate). It exists because --interrupt previously never waited, and a
    paced write runs (lines−1)×10+80 ms against a body capped only by parseJsonBody's
    1 MiB — a 48 KB --file of short lines is ~8 minutes. It arms only against a delivery
    write
    : behind another operator the wait stays unbounded, exactly as before Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 (see the
    review round above — a ceiling that could skip an operator made that pair strictly worse,
    and that was a real blocking finding, not a hypothetical). So past the ceiling the operator
    write falls back to precisely the pre-Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 operator-vs-delivery behaviour — two disjoint
    locks, no serialization — which is no worse for that pair, only no longer silent. The
    guarantee is per pair, and the 2 s value itself remains a judgment call.
  3. preempted trades a possible duplicate for never falsely reporting delivery. A
    delivery raced by a ceiling-expired write holds its row instead of marking it delivered, so
    if the message did land intact the gate may deliver it again later. That is the same call
    the existing dropped-write branch already makes, and the opposite of the interrupt path's
    claim-first tradeoff — the asymmetry is deliberate (an operator's own message vs an
    autonomous background delivery), but it is worth a second opinion.
  4. Port signature churn. writeMessage gained a 4th parameter and a typed result across
    six test files. One override (send-delivery.test.ts:604) previously returned undefined
    and relied on falsy ⇒ hold; it is now explicit.
  5. writeMessagePaced was removed, not deprecated — its only live caller was the mailbox
    wiring. Its drop-semantics test is re-pointed at submitMessagePaced so the Shellper reconnect error is swallowed: terminal becomes a silent zombie (no input/output, 'Message sent' logged for dropped frames) until next Tower restart #1198
    silent-loss guard stays on the live write edge rather than on a function nothing calls.
  6. The dev-approval transcript is committed under codev/evidence/, a new directory. That
    placement is a deliberate choice, not an accident: the evidence is part of the PIR record
    for this project, the way codev/specs/, codev/plans/ and codev/reviews/ are, and a
    gate approved on evidence that then vanishes leaves the approval unauditable. It is
    nonetheless a new top-level convention in the repo, and the maintainer may veto it
    moving or dropping the file changes nothing else in the PR (the generating script,
    packages/codev/scripts/spec-1365-e2e-evidence.mts, is re-runnable and is the durable
    artifact).

Interlock for #1481 (--interrupt-after): "interrupt, then deliver this body" is now
expressible as ordered acquisitions of one lock rather than a race between two. Two
residuals it must design against, both documented: the ^C→body gap is gate-mediated and
deliberately not atomic (the delayed interrupt guarantees "the turn was ended", never
"this body is next"), and a no-op ^C is only logged.

How to Test Locally

  • View diff: VSCode sidebar → right-click builder pir-1365Review Diff
  • Run dev: afx dev pir-1365 — but note it will contend for the live Tower's port; the
    isolated-Tower script below is why the dev-approval evidence took that route instead
  • Re-run the evidence: pnpm --filter @cluesmith/codev build && node --experimental-strip-types packages/codev/scripts/spec-1365-e2e-evidence.mts
    (isolated Tower on 14650, ~90 s, exits non-zero on any failed check)
  • Unit: pnpm --filter @cluesmith/codev test spec-1365-serializer-convergence
  • What to verify: a long multi-line send raced by afx interrupt never fuses and never
    leaves an afx inbox show row reading delivered whose text is absent or partial; another
    agent's mail keeps flowing while a large body is mid-delivery; afx interrupt stays
    responsive against a busy line; --escape behaviour is unchanged.

Flaky Tests

None. No tests were skipped or quarantined, and no pre-existing unrelated failures were
touched.

mohidmakhdoomi and others added 19 commits August 17, 2026 19:25
…ence design

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rupt-latency ceiling

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e counter

Adds the three pieces the convergence needs from the per-terminal lock:

- trySubmitToSession / isSubmissionInFlight: a non-blocking acquisition for the
  gated delivery path, which must never queue. MailboxDrainer.tick walks agents
  sequentially, so one delivery parked on a terminal lock would stall every other
  agent's mail plus that tick's escalation and prune passes.
- OPERATOR_SUBMIT_WAIT_CEILING_MS (2s) + SubmitOptions: operator submissions block,
  but boundedly. A paced write runs (lines-1)*10+80 ms and a body is capped only by
  parseJsonBody's 1 MiB, so an unbounded wait could stall afx interrupt -- the human's
  escape hatch -- for minutes. Past the ceiling it proceeds unserialized, which is
  exactly the pre-#1365 behaviour, so it is never worse than the old status quo.
- unserializedWriteCount: degraded writes are counted per session so a concurrent
  delivery can detect that it was raced, rather than the ceiling opening a second
  silent-loss route.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mission lock

The gated delivery path and the escape/interrupt paths held disjoint locks, so they
could interleave on one terminal. The failure that mattered was not a garbled composer
but a false 'delivered': a ^C landing inside a delivery's own text->Enter window
cleared the composer, the delivery's Enter submitted nothing, every byte still reached
the PTY so the write reported success, and the row was marked delivered for a message
the agent never saw. --escape produced the truncated variant, and is the more likely
trigger for a long body, whose exposed window is longest.

submitMessagePaced replaces writeMessagePaced as the delivery's write edge: the same
paced write, performed as one submission on the session's per-terminal lock, taken as a
LEAF inside the per-agent serializer. Lock order is always per-agent -> per-terminal,
and PtySession.write() emits no submit signal, so there is no cycle in either direction.
The lock wraps the write only, never the gate classify -- --interrupt must not queue
behind a screen classification.

The port now reports written | dropped | contended | preempted | aborted instead of a
bare boolean, so every not-delivered outcome holds the row rather than collapsing into
'no-live-pty'. Its precheck runs inside the lock and re-checks writability, the gate's
ringToken, and the row's own status -- the last of which stops a dismiss landing during
the write edge from putting bytes on the wire.

Operator call sites (interrupt, escape, delayed ^C) pass the wait ceiling and log loudly
when they degrade.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… key hygiene

23 new tests. The corruption cases are each paired with a bypass-the-lock CONTROL that
reproduces the original bug (body lost to a ^C, body truncated by an ESC, both with the
write still reporting success), so they assert the fix rather than merely exercising it.

Also covered: the drainer keeps serving other agents while one terminal is held; a
delivery declines contention immediately instead of waiting; the ceiling degrades and
announces itself; a raced delivery reports preempted and HOLDS its row; deliveries to
different terminals do not serialize; a session with no id throws.

gateSession in tower-routes.test.ts gains a real id -- it is the hazard the review
predicted: an un-annotated fake reaching the live wiring would have keyed every lock on
undefined, collapsing per-terminal serialization into one global lock with nothing
failing. The runtime guard turns that into a loud throw, and 13 tests duly failed until
the fake was fixed.

spec-1313-paced-write-drop is re-pointed from the retired writeMessagePaced onto
submitMessagePaced, so the silent-loss fix stays guarded at the live write edge rather
than at a function nothing calls.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rewrites session-submit.ts's 'Exactly what it covers' boundary comment (in the code
commits) and arch.md section 7 item 5 so the model lives in one place rather than as a
patchwork of separately-reasoned decisions: which writers take the lock and which stay
deliberately uncovered, the per-agent -> per-terminal order, the deliveries-decline /
operators-block asymmetry and why each side differs, the ceiling and its degradation,
and the delayed-interrupt sequencing (^C on the timer, body through the gate, not atomic
by design).

States the guarantee honestly, per the plan review: serialization is the structural
guarantee; the in-lock precheck NARROWS the echo-lag residual but cannot close it, since
ringToken counts output and un-echoed input from an uncovered writer still reads as
unchanged. That residual is #1473. The precheck's structural value is that it keeps the
acquisition policy a free choice -- switching the delivery from declining to waiting
(e.g. #1481 ordering an interrupt ahead of its body) stays safe.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ve Tower

afx dev from this worktree would bind the live Tower's port (4100 is shared by design)
and restarting the live Tower kills every builder session, so the running-worktree
evidence is produced the way send-integration.e2e.test.ts does: this worktree's built
Tower spawned on port 14650, real shellper-backed PTY sessions, real HTTP endpoints.
Nothing stubbed -- routes -> mailbox -> render gate -> locks -> PTY is the wire path.

The oracle is the existing echo-terminal fixture (stty raw -echo; exec cat): the PTY
re-emits every byte written to it, in order, so GET /api/terminals/:id/output is a
faithful ordered transcript of what every writer actually put on the terminal. The
profile resolves through the real wrapped-launch fallback (.builder-start.sh), which is
how a live builder's profile resolves -- without it the gate holds everything no-profile
and there is nothing to assert about.

66/66 checks. Scenario 3 is the sharpest: the interrupt returned in 2156ms rather than
waiting out a ~4.1s paced write, Tower logged the degradation at WARN, and the raced
delivery reported preempted and HELD its row -- the degraded path declining to claim a
delivery it could not youch for, end to end.

Two honest limits, stated in the transcript rather than approximated: the 503
TERMINAL_NOT_WRITABLE branch needs a shellper socket that died while the session still
reports running, which cannot be produced from the public API without staging it (covered
by tower-routes.test.ts:1560, and untouched by this change); and this fixture's agent
exists only as a live terminal, never as a registry-known builder, so a send after its
death correctly 404s rather than exercising the hold-instead-of-404 seam.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Routes the governance updates by tier: the arch.md section 7 rewrite landed with the code;
the hot-tier lock-order invariant is APPENDED to the existing mailbox-first fact rather than
added as an eleventh, so the 10-fact cap holds with no displacement. Three cold lessons.
Both hot lessons candidates were considered and deliberately not promoted -- promoting either
needs a displacement, which is the maintainer's call, not a builder's.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mohidmakhdoomi

Copy link
Copy Markdown
Collaborator Author

Architect integration review — 3-way CMAP (risk tier: High — core write edge, session-submit/mailbox-delivery)

Verdicts: gemini APPROVE · codex REQUEST_CHANGES · claude REQUEST_CHANGES — all HIGH confidence, and the two RC lanes independently converged on the same finding and the same fix.

The blocking finding (verified by the architect against the branch)

The 2s operator wait ceiling narrows Spec 1273's operator-vs-operator guarantee: pre-#1365, two operator submissions to one terminal were serialized unboundedly; with the ceiling, a second operator action can degrade past a long body-bearing first one and land inside its text→Enter window — and since the interrupt path claims markMailboxDelivered before its write (claim-first, correctly retained) and has no preempted detection of its own, the false-delivered failure this PR removes for deliveries is reintroduced for operators. The PR's "never worse than the old status quo" claim is therefore wrong for the operator-operator pair (it remains true for operator-vs-delivery, which was never serialized pre-PR).

Directed fix (both lanes agree): record the lock holder's kind in the chain and arm the ceiling only when a delivery write holds the lock — that bounds exactly the new exposure D3 was aimed at (the escape hatch stalling behind a long delivery) while leaving Spec 1273's operator serialization intact and making the "never worse" claim provably true per-pair. Plus: surface a degraded/raced indicator on the operator response alongside the existing WARN; correct the four claim-sites (session-submit.ts:249-253, :161-171, tower-routes.ts:1623-1625, arch.md §7 item 5); add tests for a body-bearing interrupt crossing the ceiling and for two long operator submissions.

Non-blocking (in-PR if cheap, else flagged)

  • Bump the bypass counter only on bytes actually written (a no-op ^C currently forces a spurious redelivery hold); evict unserializedWrites on session teardown (unbounded map on a long-lived Tower).
  • codev/evidence/ + the committed e2e-evidence script: deliberate precedent (evidence lives with the PIR record, like codev/reviews/) — maintainer may veto.

What all three lanes and the architect agree is right

The convergence design itself: leaf per-terminal lock inside the per-agent serializer (cycle-free, all PTY writers accounted for), delivery fail-fast preserving drainer liveness, in-lock precheck with row-status re-check, the preempted counter sampled around the delivery write, honest narrowing (not closing) of the #1473 echo-lag residual, 23 paired tests each with a bypass-the-lock control, and 66/66 live-wire dev-approval evidence (codev/evidence/1365-dev-approval-transcript.txt) demonstrating the ceiling and the preempted-hold end-to-end on real PTYs.

Awaiting the fix push; the pr gate follows. Parked for maintainer approval + merge; we are not maintainers.

mohidmakhdoomi and others added 4 commits August 17, 2026 22:51
…raded writes

Two real defects found by the PR #1492 consultation (codex and claude converged on the
same core finding independently), both verified against the code before fixing.

1. OPERATOR-VS-OPERATOR REGRESSION. Before #1365 submitToSession had no ceiling, so two
   operator submissions to one terminal were ALWAYS fully serialized (Spec 1273). My
   ceiling let a second operator degrade past a long body-bearing first operator, making
   that one pair strictly WORSE than the status quo -- so the 'never worse' claim was
   false in that corner. Chain entries now carry a SubmissionKind and the ceiling arms
   only when nothing ahead is an operator (queued counts, not just in-flight). Restores
   exact pre-#1365 op-op serialization while keeping the escape hatch responsive against
   a long delivery, which was the ceiling's actual motivation. 'Never worse' is now true
   per pair, and the four claim-sites that overstated it are corrected.

2. UNREPORTED DEGRADED INTERRUPT BODY. A ceiling-expired interrupt writes its own body
   unserialized into a still-pacing predecessor, yet the row was claimed delivered up
   front and the response said delivered:true with no qualification. Claim-first is kept
   (un-claiming risks double delivery) but the truth is now surfaced: /api/send returns
   degraded:true + degradedReason, threaded through the SDK client and warned about by
   afx send. An indicator nobody surfaces is half a fix.

Tests: op-op never degrades; a third operator does not bypass a QUEUED one; a
body-bearing interrupt crossing the ceiling behind a delivery reports degraded. Note the
pre-existing ceiling test needed its holder changed from an operator to a delivery --
that fixture change IS the behaviour change, not a workaround.

4882 passed / 0 failed. Refs #1492.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ion, claim sweep

The two non-blocking correctness notes from the claude lane, plus the residual doc claims.
Both blocking findings were already fixed in ece06a5.

1. THE BYPASS COUNTER NOW COUNTS BYTES, NOT INTENT. It was bumped on ceiling expiry
   unconditionally, but the delayed `^C` re-checks isStillLive()/writable INSIDE the lock and
   can return having written nothing. That no-op was still counted, forcing a concurrent
   delivery into a spurious `preempted` re-delivery -- a duplicate charged for a race that
   never happened. SubmitOptions.wroteBytes is consulted straight after the write callback,
   with NO await in between, so the ordering guarantee the old placement provided (a delivery
   cannot observe our bytes without also observing the bump) is unchanged.

2. unserializedWrites NOW SELF-EVICTS. It was retained for the life of the Tower -- one entry
   per session that ever degraded, the leak class #1472 fixed. It cannot self-delete on drain
   the way chains and pendingOperators do, because it must OUTLIVE the submission whose
   watcher is about to compare against it: a reset landing between a watcher's two reads reads
   as "nobody raced me", the exact false `delivered` this issue exists to eliminate. So
   eviction is interlocked with an explicit watchBypasses() window -- refused while a watch is
   open, attempted from BOTH the chain's drain cleanup and the last watch's release, so
   whichever runs second evicts and no ordering leaks. Needs no session-teardown hook and
   therefore no terminal/ -> agent-farm/ layer crossing.

Also: the residual "never worse than the status quo" claim-sites now state the guarantee per
pair (tower-routes' logCeilingExpired doc + the degraded-path inline comment; arch.md and the
session-submit boundary comment were done in ece06a5). Stale {@link writeMessagePaced}
repointed. DEGRADED_SUBMIT_REASON moved out from between logCeilingExpired's JSDoc and its
function.

Two nits refused with reasons rather than silently skipped (ceiling-timer cancellation needs
abort semantics on the injected SubmitClock; the timing assertions are load-bearing) -- see
the rebuttal file. Review doc carries the full REQUEST_CHANGES disposition, the two files
outside the stated PR scope, and the codev/evidence/ placement the maintainer may veto.

Tests: a degraded write that writes NOTHING is not counted; the counter is evicted once the
session goes idle; eviction cannot land inside a watcher window and mask a race.

Build clean (codev + sdk). 4885 passed / 0 failed / 48 skipped, 246 files. Refs #1492.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mohidmakhdoomi

Copy link
Copy Markdown
Collaborator Author

Architect fix-verification — review iteration 1 dispositions (pre-gate)

Verified against the branch at 618925f1 (commits ece06a5e + 0dc75d8e), not from the builder's summary. All accepted findings are actually fixed:

1. Kind-aware ceiling (converged codex/claude blocking finding) — verified fixed.
session-submit.ts: submissions carry a SubmissionKind (default operator — the never-bypassable kind, the safe default); a pendingOperators per-session count includes queued operators (counted before any await, after the self-check); bounded now requires !behindOperator. Operator-vs-operator keeps its pre-#1365 unbounded serialization; the ceiling can only ever bypass a delivery. trySubmitToSession tags the delivery path delivery-write. The conservative direction (an operator queued behind a delivery and an operator waits unbounded) errs toward more serialization, never less.

2. Degraded interrupt no longer reports unqualified success (blocking) — verified fixed.
/api/send returns degraded: true + degradedReason: 'submit-wait-ceiling-expired' on both the interrupt and escape paths (additive, present only when it happened — older clients unaffected); threaded through packages/sdk/src/tower-client.ts (types + passthrough; SDK still imports only types) and surfaced by afx send as a loud, actionable warning. Claim-first is kept and the tradeoff is now told to the sender instead of hidden.

3. Bytes-only bypass counting — verified. The bump moved inside the chain: if (bypassed && (options.wroteBytes?.() ?? true)) immediately after write() with no await between, so a delivery holding the line cannot observe the bytes without also observing the bump. The delayed-^C path passes wroteBytes: () => fired, and fired is set exactly where the ^C bytes go out (tower-routes.ts:1768).

4. Counter eviction — verified. watchBypasses/bypassWatchers interlock: eviction refused while any watch is open, attempted from both the chain drain and the last watch release; submitMessagePaced holds the watch across its write and releases in a finally. No session-teardown hook, no terminal/agent-farm/ crossing. bypassCountedSessions() gives tests the leak assertion.

5. Claim-site sweep — re-ran the grep myself. Every surviving never worse / status quo / never waited site states the corrected per-pair guarantee (op↔op unchanged/unbounded; op↔delivery ceiling-bounded, degraded to old disjoint-lock behaviour above it; delivery↔delivery unchanged).

6. Pinning tests — all present, and I re-ran both files: 129/129 green (spec-1365-serializer-convergence.test.ts + tower-routes.test.ts, Vitest 4.1.4). Includes operator vs operator NEVER degrades, a THIRD operator does not bypass a QUEUED one, a degraded write that writes NOTHING is not counted as a bypass, eviction cannot land inside a watcher window and mask a race, and the degradedReason route assertion.

Refusals — accepted as reasonable. Ceiling-timer cancellation would force abort semantics onto the injected SubmitClock every test double implements, for one ≤2 s timer on the rare contended-operator path; the timing assertions are load-bearing responsiveness claims with ≥2.5× headroom.

Escalated to the human at the gate (not resolved here): the hot-tier arch-critical.md clause was applied directly rather than proposed. It extends the existing mailbox-first fact (no new bullet; cap holds) and reverting is a one-line edit — ratification is the human's call and is being asked alongside the pr gate.

Two out-of-scope files (commands/send.ts, sdk/tower-client.ts) are the minimum server→client surfacing for finding 2 and are disclosed in the PR body.

Contributor-workflow reminder: this PR parks for the maintainer — no self-merge.

mohidmakhdoomi and others added 4 commits August 17, 2026 23:12
main advanced 25 commits during the #1365 review round (47477ca..9129ab8): AIR #1489
(`afx reset` → `afx refresh`), secfix-1 (Tower auth hardening), PIR #1495 (Stream Deck
architect action). Merged rather than rebased so the verified #1365 history stays intact.

One conflict, in `codev/resources/lessons-learned.md` § Architecture — an append-only
collision, not a competing edit: this branch appended three #1365 lessons and main appended
one from secfix-1 at the same point. Resolved KEEP-BOTH; all four entries are present and
unmodified.

Every code file auto-merged. The only changes the merge made to this PR's files come from
main, not from #1365: the `afx reset` → `afx refresh` rename in two comments
(session-submit.ts, mailbox-wiring.ts), and secfix-1's auth work in tower-routes.ts,
tower-client.ts (`codev-web-key` → `codev-tower-key`) and tower-routes.test.ts. Every #1365
hunk survives byte-identical — message-write.ts, mailbox-delivery.ts, commands/send.ts and
spec-1365-serializer-convergence.test.ts are untouched by the merge.

PR #1492 stays OPEN and parked for the maintainer; #1365 stays open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Post-merge figures: build clean (codev + sdk), full suite 4934 passed / 0 failed / 48 skipped,
248 files (was 4885 / 246 pre-merge; the delta is main's own tests). Merge ebbc495, keep-both
resolution of the lessons-learned append collision, and the verification that every #1365 hunk
survived byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape)

1 participant