Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## [Unreleased]

### Streams read with `toReadableStream()` or `tee()` are recorded

- `runtime-node`: `instrumentLlmClient` ended the span of a streamed OpenAI or Anthropic call only when the stream was iterated. `new Response(stream.toReadableStream())` (the usual Next.js route handler) and `stream.tee()` read the stream without iterating it, so the span never ended and the call, its tokens and its cost were never exported. The stream is now instrumented at its own `iterator` source, so every way of reading it, and every branch of a tee, ends the span once, on completion, error or cancel. The result is the SDK's own stream object instead of a proxy of it.

## [1.21.0] - 2026-11-17

Packages: `runtime-node` 1.17.0, and the Python adapter (`adapters/python`, copied, not published). The other packages are unchanged; the `otlp-ingester` README is updated without a release.
Expand Down
97 changes: 56 additions & 41 deletions packages/runtime-node/src/llm-instrument.ts
Original file line number Diff line number Diff line change
Expand Up @@ -153,11 +153,18 @@ function streamFinisher(span: Span, model: string, seen: RawUsage) {
}

/**
* Wrap a streaming result so the span ends when the stream finishes (or
* errors), accumulating usage from the chunks that carry it: OpenAI's final
* chunk (with `stream_options: { include_usage: true }`), OpenAI Responses'
* Make a streaming result end the span when the stream finishes (or errors),
* accumulating usage from the chunks that carry it: OpenAI's final chunk
* (with `stream_options: { include_usage: true }`), OpenAI Responses'
* `response.completed`, Anthropic's message_start/message_delta events,
* Google's per-chunk usageMetadata, Bedrock's ConverseStream `metadata`.
*
* The stream itself is instrumented, not proxied. The openai and
* @anthropic-ai/sdk streams read their own `iterator` field from
* `[Symbol.asyncIterator]`, `tee()` and `toReadableStream()`, and use private
* fields that only work with the stream as `this`, so wrapping that one
* source covers every way of reading it and every branch of a tee. Other
* async iterables are wrapped at `[Symbol.asyncIterator]`.
*/
function wrapStream<T extends object>(
stream: T,
Expand All @@ -167,46 +174,54 @@ function wrapStream<T extends object>(
): T {
const seen: RawUsage = {};
const finish = streamFinisher(span, model, seen);
return new Proxy(stream, {
get(target, prop, receiver) {
if (prop === Symbol.asyncIterator) {
return () => {
const inner = (
target as unknown as AsyncIterable<unknown>
)[Symbol.asyncIterator]();
return {
async next(): Promise<IteratorResult<unknown>> {
try {
const step = await inner.next();
if (!step.done) {
mergeRawUsage(seen, extractRawUsage(step.value, provider));
} else {
finish();
}
return step;
} catch (err) {
finish({ error: err });
throw err;
}
},
async return(value?: unknown): Promise<IteratorResult<unknown>> {
finish();
return inner.return
? inner.return(value)
: { done: true, value: undefined };
},
async throw(err?: unknown): Promise<IteratorResult<unknown>> {
finish({ error: err });
if (inner.throw) return inner.throw(err);
throw err;
},
};
};
const entry =
typeof (stream as { iterator?: unknown }).iterator === "function"
? "iterator"
: Symbol.asyncIterator;
const open = (stream as Record<PropertyKey, () => AsyncIterator<unknown>>)[entry];
try {
Object.defineProperty(stream, entry, {
value: () => trackIterator(open.call(stream), finish, seen, provider),
configurable: true,
writable: true,
});
} catch {
finish();
}
return stream;
}

function trackIterator(
inner: AsyncIterator<unknown>,
finish: ReturnType<typeof streamFinisher>,
seen: RawUsage,
provider: string,
): AsyncIterator<unknown> {
return {
async next(): Promise<IteratorResult<unknown>> {
try {
const step = await inner.next();
if (!step.done) {
mergeRawUsage(seen, extractRawUsage(step.value, provider));
} else {
finish();
}
return step;
} catch (err) {
finish({ error: err });
throw err;
}
const value = Reflect.get(target, prop, receiver);
return typeof value === "function" ? value.bind(target) : value;
},
});
async return(value?: unknown): Promise<IteratorResult<unknown>> {
finish();
return inner.return ? inner.return(value) : { done: true, value: undefined };
},
async throw(err?: unknown): Promise<IteratorResult<unknown>> {
finish({ error: err });
if (inner.throw) return inner.throw(err);
throw err;
},
};
}

interface EventStreamHelper {
Expand Down
Loading