Skip to content

Fix waiting for the data channel send buffer to drain - #1267

Open
hiroshihorie wants to merge 4 commits into
hiroshi/close-engine-on-terminal-pathsfrom
hiroshi/fix-buffer-status-wait
Open

hiroshihorie wants to merge 4 commits into
hiroshi/close-engine-on-terminal-pathsfrom
hiroshi/fix-buffer-status-wait

Conversation

@hiroshihorie

@hiroshihorie hiroshihorie commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Stacked on #1268. Review only the last commit, bb1c5b8. The merge commit before it brings in #1268.

Engine.waitForBufferStatusLow is the backpressure wait for data stream writes. It had three bugs:

  1. Could hang after a full reconnect. It polled only a cached "buffer is low" flag that a full reconnect never reset, so a waiter on a full buffer could keep waiting on the replaced channel. The flag is now reset when the channels are created, and the wait also checks the live channel. Flagged by Devin on Apply backpressure to native data stream sends #1266. Didn't reproduce on macOS (see E2E), but the fix no longer relies on that.
  2. Resolved normally when the engine closed. The caller then went on to send on a closing engine. It now fails with 'Engine disconnected'.
  3. Leaked a listener per blocking wait. The EngineClosingEvent listener was never cancelled. The wait no longer uses a listener at all.

Bugs 2 and 3 are fixed by checking the engine's closed state in the wait loop instead of listening for EngineClosingEvent. That only works because #1268 marks the engine closed synchronously on every terminal path, including room disposal and a reconnect that gives up. Hence the stack.

This matters more with #1166, where every native data stream packet goes through this wait.

Tests

test/core/buffer_status_wait_test.dart has one test per bug. All three fail on #1268 and on main, and pass here. MockDataChannel gains fakeBufferedAmount, identical to #1266's, so the two merge cleanly. analyze, format, import_sorter, version check and flutter test all pass.

E2E (macOS, local livekit-server --dev)

A local server drains the channel faster than the 2 MB low-water mark, so these runs used test-only builds with the mark lowered to 32 KB (not in this PR):

main this PR
Full reconnect while the buffer is past the mark (3 runs) the in-flight write fails cleanly, no hang same, and a send after the reconnect goes out
Engine listeners after 16 MiB of backpressured writes +4 +0

On macOS the old channel's low-buffer event during teardown releases the waiter, so bug 1 doesn't show there. Other platforms are unchecked.

Merge order

#1264, then #1268, then this PR. Once #1268 merges, this PR retargets to main and I'll merge main back in.

waitForBufferStatusLow had three problems:

- It polled only the cached buffer status, which changes on a send or on
  the channel's low-buffer event. When a full reconnect replaced a backed
  up channel, the new one fired neither, so a waiter could hang until
  disconnect. The cache is now reset when the channels are created, and
  the wait also checks the live channel.
- When the engine closed during a wait, the wait resolved normally and
  the caller went on to send. It now fails.
- Every wait that blocked left an EngineClosingEvent listener behind
  until the engine closed. The listener is now cancelled.

MockDataChannel gains fakeBufferedAmount to simulate a channel that has
not drained.
…-paths' into hiroshi/fix-buffer-status-wait

# Conflicts:
#	lib/src/core/engine.dart
Every terminal path now marks the engine closed synchronously, so the wait
can check that state directly. This drops the listener the wait had to
remove again, and a wait started after closure fails the same way.
@hiroshihorie
hiroshihorie changed the base branch from main to hiroshi/close-engine-on-terminal-paths October 29, 2026 20:11
…-paths' into hiroshi/fix-buffer-status-wait

waitForBufferStatusLow() keeps this branch's polling wait on the closed state.
The base now throws UnexpectedConnectionState instead of a string, so the wait
throws that too.
@hiroshihorie
hiroshihorie marked this pull request as ready for review November 3, 2026 04:22

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment thread lib/src/core/engine.dart
),
);
_reliableDCPub?.bufferedAmountLowThreshold = 2 * 1024 * 1024;
_dcBufferStatus[Reliability.reliable] = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Stream chunks lost during full reconnect

When a full reconnect replaces a full channel, _dcBufferStatus releases a waiting stream write before reconnection finishes. sendDataPacket defers that chunk, but full reconnect never resends it; the write succeeds without delivery.

Learn more

A data stream write waits in WritableStream.write whenever the reliable channel is full. During restartConnection, the engine remains reconnecting while the replacement data channels are created. Resetting the cached status here releases the waiting write as soon as that channel exists, before the reconnect succeeds. sendDataPacket then buffers the reliable packet and returns early because _isReconnecting is true. Only the resume response calls _resendReliableMessagesForResume; a full restart does not send those queued packets.

Example: A stream chunk waits on a full channel. A full reconnect creates an empty reliable channel, releasing the write. The write completes while reconnecting, but the recipient never gets the chunk.

Recommended fix: Keep buffer waits pending until the replacement channel is usable and reconnection has completed, or explicitly resend reliable packets deferred during a full reconnect. Test an in-flight stream write across an actual full reconnect and assert receipt of the chunk.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@hiroshihorie
hiroshihorie added this pull request to stack #1276 November 3, 2026 05:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant