Repository navigation
runtime-node: end the LLM span for streams read with toReadableStream() or tee() - #61
Open
kunalgoesbyken wants to merge 1 commit into
Open
kunalgoesbyken wants to merge 1 commit into
kunalgoesbyken wants to merge 1 commit into
Conversation
…() or tee() wrapStream proxied the stream and only intercepted Symbol.asyncIterator. Every other method was bound to the raw stream, so toReadableStream() and tee() read chunks the finisher never saw, and the span of a call served with new Response(stream.toReadableStream()) never ended or exported. The openai and @anthropic-ai/sdk streams read a public iterator field from [Symbol.asyncIterator], tee() and toReadableStream(), and use private fields that only work with the stream as this, so a proxy receiver is not an option. wrapStream now wraps that source on the stream itself (or Symbol.asyncIterator for other async iterables) and returns the stream unchanged. One finisher covers iteration, both branches of a tee, cancel and error. A stream that cannot be modified still works and its span ends without usage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #60
Why
A streamed OpenAI or Anthropic call only got its span ended when the stream was iterated.
new Response(stream.toReadableStream()), the usual Next.js route handler, andstream.tee()read the stream without going through the proxy, so the span never ended and the call, its tokens and its cost were never exported.Binding those methods to the proxy is not an option: the SDK
Streamkeeps private state keyed by the real instance, so they throw whenthisis a proxy.What changed
wrapStreamno longer proxies the stream. Both SDKs read a publiciteratorfield from[Symbol.asyncIterator],tee()andtoReadableStream(), so it wraps that one function on the stream itself and returns the stream as is. Async iterables without aniteratorfield (Google, Bedrock) are wrapped at[Symbol.asyncIterator], as before.return()andnext()paths as plain iteration.A stream that is never read or cancelled keeps its span open, same as when it is never iterated.
How it was checked
llm-instrument.test.mjs, using a fakeStreamshaped like the openai and Anthropic one (publiciterator, native private fields,tee()through theiteratorfield and throughSymbol.asyncIterator). They covertoReadableStreamto the end, cancel and error,teeto the end, one branch closed, both closed, plain iteration and break, and a frozen stream. All 10 fail on main.openai7.57.0 and@anthropic-ai/sdk1.9.0 with a local server:toReadableStream,tee, cancel, a dropped connection,new Response(stream).text(), and the Anthropicmessages.stream()helper. Each gives one span with the right usage and status.npm run buildandnpm test -w @autter/runtime-nodepass (299 passed, 4 skipped).Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.