Skip to content

runtime-node: end the LLM span for streams read with toReadableStream() or tee() - #61

Open
kunalgoesbyken wants to merge 1 commit into
Autter-dev:mainfrom
kunalgoesbyken:fix/llm-stream-readable-span
Open

kunalgoesbyken wants to merge 1 commit into
Autter-dev:mainfrom
kunalgoesbyken:fix/llm-stream-readable-span

Conversation

@kunalgoesbyken

@kunalgoesbyken kunalgoesbyken commented Nov 17, 2026 •

Copy link
Copy Markdown

Closes #60

Why

A streamed OpenAI or Anthropic call only got its span ended when the stream was iterated. new Response(stream.toReadableStream()), the usual Next.js route handler, and stream.tee() read the stream without going through the proxy, so the span never ended and the call, its tokens and its cost were never exported.

Binding those methods to the proxy is not an option: the SDK Stream keeps private state keyed by the real instance, so they throw when this is a proxy.

What changed

wrapStream no longer proxies the stream. Both SDKs read a public iterator field from [Symbol.asyncIterator], tee() and toReadableStream(), so it wraps that one function on the stream itself and returns the stream as is. Async iterables without an iterator field (Google, Bedrock) are wrapped at [Symbol.asyncIterator], as before.

  • One finisher is shared by every way of reading the stream, so a tee ends the span once, after the last branch is done. Closing one branch leaves it open.
  • Cancel ends the span with the usage seen so far and an error mid-stream ends it as an error. Both go through the same return() and next() paths as plain iteration.
  • If the stream cannot be modified (frozen), it still works and the span ends without usage instead of throwing into the app.
  • The result is now the SDK's own stream object, not a proxy of it.

A stream that is never read or cancelled keeps its span open, same as when it is never iterated.

How it was checked

  • 10 new tests in llm-instrument.test.mjs, using a fake Stream shaped like the openai and Anthropic one (public iterator, native private fields, tee() through the iterator field and through Symbol.asyncIterator). They cover toReadableStream to the end, cancel and error, tee to the end, one branch closed, both closed, plain iteration and break, and a frozen stream. All 10 fail on main.
  • Ran the same cases against the real openai 7.57.0 and @anthropic-ai/sdk 1.9.0 with a local server: toReadableStream, tee, cancel, a dropped connection, new Response(stream).text(), and the Anthropic messages.stream() helper. Each gives one span with the right usage and status.
  • npm run build and npm test -w @autter/runtime-node pass (299 passed, 4 skipped).

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

…() or tee()

wrapStream proxied the stream and only intercepted Symbol.asyncIterator.
Every other method was bound to the raw stream, so toReadableStream() and
tee() read chunks the finisher never saw, and the span of a call served
with new Response(stream.toReadableStream()) never ended or exported.

The openai and @anthropic-ai/sdk streams read a public iterator field from
[Symbol.asyncIterator], tee() and toReadableStream(), and use private
fields that only work with the stream as this, so a proxy receiver is not
an option. wrapStream now wraps that source on the stream itself (or
Symbol.asyncIterator for other async iterables) and returns the stream
unchanged. One finisher covers iteration, both branches of a tee, cancel
and error. A stream that cannot be modified still works and its span ends
without usage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

instrumentLlmClient: an OpenAI stream read through toReadableStream() never ends its span, so the call is never exported

1 participant