Repository navigation
runtime-node: keep the SDK's promise from instrumented LLM calls - #63
Open
kunalgoesbyken wants to merge 1 commit into
Open
kunalgoesbyken wants to merge 1 commit into
kunalgoesbyken wants to merge 1 commit into
Conversation
traceCall returned result.then(...), a plain Promise, in place of the APIPromise the openai and @anthropic-ai/sdk clients return. That dropped withResponse() and asResponse(), so code that reads rate-limit headers or request ids threw a TypeError once the client was instrumented. The APIPromise parses the body only through parse(), which then, catch, finally and withResponse() all call. traceCall now returns the original promise and observes its parse() instead, once, so the span still ends with usage however the call is read, a stream still reaches the caller wrapped, and asResponse() leaves the body unread. Other promises (Google, Bedrock) are chained as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #62
Why
instrumentLlmClientreturnedresult.then(...), a plainPromise, in place of theAPIPromisethe openai and Anthropic clients return..withResponse()and.asResponse()were gone, so an app that reads rate-limit headers or request ids got aTypeErroras soon as its client was instrumented.What changed
traceCallnow returns the original promise. The SDK'sAPIPromiseparses the body only throughparse(), andthen,catch,finallyandwithResponse()all go through it, so the span is observed there, once (the observed promise is kept, like the SDK's own).await,.then()or.withResponse(), and several of them on one call still end it once..withResponse()..asResponse()hands over the response with its body unread. Observing the promise with a side.thenwould have parsed the body first, which is whyparse()is hooked instead. A call read only through.asResponse()is not recorded, because its body belongs to the caller (the changelog says so).parse()(Google, Bedrock) are chained as before.How it was checked
llm-instrument.test.mjs, using a fakeAPIPromiseshaped like the SDKs' (Promise subclass, parse throughparse(), native private state). They cover the promise keeping its identity and_request_id,withResponse()data and usage,asResponse()with the body unread, awaiting pluswithResponse()on one call, a stream throughwithResponse(), and a failed call with no unhandled rejection. All 6 fail on main.openai7.57.0 and@anthropic-ai/sdk1.9.0 with a local server, including a 429: one span with the right usage or error status each time, and no unhandled rejection.npm run buildandnpm test -w @autter/runtime-nodepass (295 passed, 4 skipped).Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.