FE-1604: Allow Voice interruption by speaking

[kostandinang]

🌟 What is the purpose of this PR?

This PR lets a person interrupt Voice assistant playback by speaking without losing the interrupting utterance. It keeps Your turn as an optional half-duplex fallback and rejects likely prompt-regurgitation or assistant-echo transcripts before they become answers.

This is an independent semantic recut of the interruption-only delta from the stale, conflicting #9550, based on post-deployment main at ef0f444987. It does not merge or carry the old branch's unrelated Brunch stack. The automated proof establishes protocol ordering, admission, retention, UI preference, and package integrity; it does not establish real microphone latency or acoustic echo behavior.

🔗 Related links

• FE-1604 (internal)
• Replaces FE-1604: Allow Voice interruption by speaking #9550
• Independent settlement sibling: FE-1580: Port omitted Voice settlement behavior #9588

🚫 Blocked by

• FE-1580: Port omitted Voice settlement behavior #9588 landing and a final update from main
• Real browser/microphone witness for speaker feedback and false interruption
• Review completion

🔍 What does this change?

• Keeps microphone capture active during canonical playback when Interruption by speaking is enabled.
• Preserves immediate speech_started → response.cancel → output_audio_buffer.clear; interruption never clears the input buffer or waits for transcript validation.
• Carries the same input item through completed transcription and submits it exactly once through the existing Brunch admission path.
• Retains an accepted interruption while the previous Brunch turn settles, including across follow-on canonical speech and later rejected or failed transcripts.
• Rejects completed interruption transcripts with strong ordered overlap against the configured transcription vocabulary or the exact canonical text active when interruption began.
• Uses medium Semantic VAD eagerness while keeping response creation and interruption under Petrinaut control.
• Adds a browser-saved, default-on Interruption by speaking checkbox to the playback menu and shows Your turn only when it is disabled.
• Preserves mute, pause, Stop, end, reconnect, exact question replay, and exact full-response replay.
• Updates the website README, Petrinaut Voice guide, Voice architecture decision record, and Petrinaut patch changeset.

🏗️ Agent notes

Recut provenance

The stale PR head f69ac17034 contained four interruption product commits after its FE-1580 parent. Those commits were applied onto current main; range comparison found one cancellation-settlement hunk already present on main, with the remaining semantic patches unchanged. The current NodeNext website boundary additionally required the new shared transcription import to use its .js extension.

Constraints retained

• Completed provider transcription remains the only Voice-answer authority.
• Cancellation is immediate, output-only, and never gated by classification.
• Only playback-overlapping input is classified; ordinary capture is unchanged.
• Self-echo compares only with canonical text active when speech began, not queued speech or history.
• Rejection diagnostics contain only operational metadata and a reason.
• Short novel answers such as “stop”, “no”, and “wait” remain admissible.
• FE-1580: Port omitted Voice settlement behavior #9588 remains an independent sibling and is not included in this branch.

Fog-line and stop conditions

Browser echo cancellation may still permit a false VAD event to stop playback before the completed transcript is rejected. Stop or reorient if current-main behavior clears the interrupting input, delays cancellation, admits an interruption twice, erases an earlier retained answer, or fails to restore acknowledged half-duplex handoff when the preference is disabled.

The complete live mission contract is in libs/@hashintel/brunch-agent/MISSION.md.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

• Modifies an npm-publishable library and includes a patch changeset.

📜 Does this require a change to the docs?

• Requires docs changes, made in apps/petrinaut-website/README.md, libs/@hashintel/petrinaut/docs/ai-assistant.md, and libs/@hashintel/brunch-agent/docs/adr/0009-openai-voice-ui-turn-shell.md.

🕸️ Does this require a change to the Turbo Graph?

• Does not affect the execution graph.

⚠️ Known issues

• A false VAD event can still stop playback even when the completed transcript is discarded.
• The deterministic classifier is conservative and heuristic; real speaker/microphone behavior still needs a human witness.
• Protocol tests do not benchmark audible interruption latency.
• Website ESLint reports one pre-existing set-state-in-effect warning in the Voice control component; there are zero errors.

🛡 What tests cover this?

Coverage includes immediate cancellation and input preservation; duplicate completion; delayed and pending admission; follow-on canonical speech; retained-answer display; prompt-regurgitation and active-playback self-echo rejection; short novel answers; unchanged ordinary capture; preference/UI behavior; and lifecycle cleanup.

Fresh verification on this recut:

• Focused Voice suites: 13 files, 246 tests passed.
• Full @apps/petrinaut-website suite: 41 files, 417 tests passed.
• Full @hashintel/petrinaut suite: 94 files, 800 tests passed.
• Dependency-aware TypeScript checks, ESLint, and builds passed for both affected workspaces (19/19 Turbo tasks).
• Petrinaut architecture lint passed: 78 layers, 409 edges, 786 files.
• Repository formatting and git diff --check passed.

❓ How to test this?

1. Start Voice and open Voice playback options. Confirm Interruption by speaking is checked by default.
2. While Brunch is speaking, begin answering, including a short answer such as “stop”, “no”, or “wait”. Confirm playback stops immediately and the same utterance appears once, including if follow-on canonical speech is queued before transcription completes.
3. Interrupt while the previous Brunch turn is still settling. Confirm Answer captured. Waiting for Brunch. remains visible and submits once input reopens. Confirm a later rejected echo does not erase that notice.
4. Uncheck Interruption by speaking. Confirm Your turn returns and the microphone stays closed during playback until the acknowledged handoff completes. Reload and confirm the preference is remembered.
5. Exercise mute, pause, Stop, end, reconnect, Repeat question, and Read full response in both modes.

📹 Demo

The original interruption-only capture remains representative of this semantic recut:

fe-1604-voice-interruption-by-speaking.mp4

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Sep 9, 2026 3:46pm UTC
hashdotdesign-tokens	Ready	Preview	Sep 9, 2026 3:46pm UTC
petrinaut	Ready	Preview	Sep 9, 2026 3:46pm UTC
petrinaut-docs	Ready	Preview	Sep 9, 2026 3:46pm UTC

[cursor]

PR Summary

Medium Risk
Changes Realtime turn-taking, transcript admission, and retained-answer timing on the Voice path; misclassification or race bugs could drop or duplicate answers, though existing Brunch submission remains the authority.

Overview
Adds interruption by speaking so users can talk over canonical assistant playback without losing the utterance. When enabled (default), Realtime keeps the mic open, immediately sends response.cancel and output_audio_buffer.clear on speech start (never clearing the input buffer), and still submits the completed transcript once through the existing Brunch bridge.

The bridge retains interrupting answers while Brunch is busy, drains them when the turn reopens, and runs interruption-only checks for prompt regurgitation and self-echo against active canonical playback (with content-free rejected diagnostics). Disabling the browser-saved preference restores half-duplex capture and the acknowledged Your turn handoff; the Petrinaut playback menu exposes a checkbox and hides Your turn while interruption is on.

Server Realtime policy moves semantic VAD to medium eagerness and shares the transcription vocabulary via voiceTranscriptionPrompt. Docs, ADR, mission text, a Petrinaut patch changeset, and broad unit/integration coverage (including history remount continuity) accompany the behavior change.

^{Reviewed by Cursor Bugbot for commit a1940d8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[lunelson]

I found four behavioral blockers and one mission-contract gap:

1. Short exact self-echo is admitted. classify-interruption.ts returns null for every transcript under six tokens before comparing canonical playback. An exact echo such as “Who approves this?” therefore becomes an answer. The contract permits short novel answers, not short exact echoes.

2. Generated audio awaiting playback escapes spoken interruption. #interruptOutputBySpeaking() cancels only active or already-speaking responses. After response.done but before output_audio_buffer.started, speech does not cancel/clear the terminal buffered response. The manual-cancellation suite models this phase, but the spoken-interruption “generated” case starts playback first.

3. Pause or host Stop silently drops a retained answer. VoiceTurnController.pause() / host Stop call cancelPendingSpeech(), which sets RealtimeBrunchBridge.#pendingInterruption = null. The controller can continue showing “Answer captured. Waiting for Brunch” after the answer has become undeliverable.

4. Cancellation before response.created can leave permanent output ownership. The cancelled-created path returns before recording speechRequestId; its terminal event cannot clear #pendingSpeechRequestIds in the bridge. Later ordinary speech can remain classified against phantom pending output. The bridge test fabricates a terminal speechRequestId that the real session does not emit in this path.

Separately, MISSION.next.md requires every later Voice mission to exercise the typed-origin + Voice-origin + durably-stopped second-tab reopen scenario. FE-1604 exercises Voice but does not include that gate. This is also still deferred in FE-1573 Step B, so the ownership/disposition should be explicit rather than silently left there.

The architecture and overall branch scope otherwise align well: provider logic remains website-owned, Petrinaut stays provider-neutral, and admission still uses the existing Brunch path.

[kostandinang]

Addressed and pushed in 268303c8a4, with the ownership follow-up in a16c19ef0f (mission authority changes remain committed separately):

1. Exact normalized playback echoes are now rejected before the six-token fuzzy-overlap threshold. The regression also proves a short novel answer is still admitted unchanged.
2. Spoken interruption clears completed canonical audio buffered between response.done and output_audio_buffer.started; response.cancel is sent only while the response is active. The buffer-clear acknowledgement now carries speechRequestId downstream, so both the bridge and controller release generated-but-unplayed output ownership and restore capture.
3. Output cancellation no longer deletes a finalized retained interruption. Delivery waits for both cancellation handoff and chat readiness. Both Stop/cancellation race orders now assert that the first gate alone cannot drain the retained answer.
4. Response-to-speech correlation is recorded as soon as canonical response.created arrives and retained independently through output cleanup, so a pre-created cancellation's terminal event carries speechRequestId and releases bridge ownership.
5. MISSION.md owns the combined continuity gate. voice-history-continuity.integration.test.tsx exercises typed history, supported Voice client-tool provenance, durable composer Stop, close/reopen through a second mounted Flue observation, stopped rendering, and local End Voice without another durable Stop. Per the owner decision, direct-user Voice source reconstruction is explicitly deferred: Flue 2.0.3 exposes no supported caller-metadata/idempotency-key projection, and the prohibited browser-side/text-encoding alternatives were not introduced.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit a16c19e. Configure here.}

[lunelson]

I still see four concerns:

1. High — the continuity test is a projection fixture, not the claimed throughline. voice-history-continuity.integration.test.tsx injects source: "voice" into a prebuilt observation, makes the mocked requestStop publish a prebuilt aborted snapshot, reuses the same mocked FlueClient promise for both mounts, and substitutes a local fake VoiceMode. It therefore proves projection/rendering against two observations, not durable Voice origin, production Stop, or second-tab reopening. MISSION.next.md explicitly says not to treat the prepared fixture or mechanical witness as a permanent substitute for that skipped check.

2. High — overlapping provider input items can produce two Brunch submissions. openai-realtime-session.ts accepts every valid speech_started item into #acceptedInputItemIds. In realtime-brunch-bridge.ts, if A completes while admission is open it submits immediately; if overlapping B then completes while A is active, B becomes #pendingInterruption and is drained after A settles. The existing overlap test covers two answers while admission is already closed, not this submit-then-retain order. This conflicts with ADR-0009's requirement that overlapping provider events cannot create another Brunch submission.

3. Medium — one global output-buffer clear can release ownership for only one of several buffered responses. Spoken interruption marks every active or terminal buffered canonical response cancelled, then sends one output_audio_buffer.clear. #handleOutputBufferEvent() emits output-interrupted only for the acknowledgement's responseId; its loop over the remaining terminal responses runs only for manual cancelOutput(). If A is playing while completed B is buffered, the clear acknowledgement for A can leave B's bridge/controller speech ownership pending indefinitely.

4. Low — the checkbox-style preference has no checkbox semantics. playback-menu.tsx uses selectedStyle: "checkbox", but the shared menu renders it as role="menuitem" with only data-selected. Assistive technology receives neither role="menuitemcheckbox" nor aria-checked.

[kostandinang]

Addressed in bc9fc13 and a1940d8:

1. Recut MISSION.md and the test wording so voice-history-continuity.integration.test.tsx is explicitly a prepared projection/remount guard. It no longer claims production Voice provenance, production Flue Stop/abort persistence, or a fresh second-tab client. The real Flue-backed continuity witness remains an explicit oracle gap.
2. OpenAIRealtimeSession now admits only the first unresolved provider input item; a regression proves a distinct overlapping item cannot surface a second completed transcript downstream.
3. A global output-buffer clear now releases and emits output-interrupted for every additional terminal buffered response, including each speechRequestId; the A-playing/B-buffered sequence is covered.
4. Checkbox-styled menu actions now render through Ark Menu.CheckboxItem, with the Petrinaut regression asserting menuitemcheckbox and live aria-checked state.