Skip to content

runtime-node: keep the SDK's promise from instrumented LLM calls - #63

Open
kunalgoesbyken wants to merge 1 commit into
Autter-dev:mainfrom
kunalgoesbyken:fix/llm-keep-api-promise
Open

kunalgoesbyken wants to merge 1 commit into
Autter-dev:mainfrom
kunalgoesbyken:fix/llm-keep-api-promise

Conversation

@kunalgoesbyken

@kunalgoesbyken kunalgoesbyken commented Nov 17, 2026 •

Copy link
Copy Markdown

Closes #62

Why

instrumentLlmClient returned result.then(...), a plain Promise, in place of the APIPromise the openai and Anthropic clients return. .withResponse() and .asResponse() were gone, so an app that reads rate-limit headers or request ids got a TypeError as soon as its client was instrumented.

What changed

traceCall now returns the original promise. The SDK's APIPromise parses the body only through parse(), and then, catch, finally and withResponse() all go through it, so the span is observed there, once (the observed promise is kept, like the SDK's own).

  • The span ends with usage however the call is read: await, .then() or .withResponse(), and several of them on one call still end it once.
  • A stream still reaches the caller wrapped, including through .withResponse().
  • A failed call ends the span as an error once and rejects exactly as the SDK does. Nothing is attached on the side, so there is no extra rejection to leak.
  • .asResponse() hands over the response with its body unread. Observing the promise with a side .then would have parsed the body first, which is why parse() is hooked instead. A call read only through .asResponse() is not recorded, because its body belongs to the caller (the changelog says so).
  • Promises without parse() (Google, Bedrock) are chained as before.

How it was checked

  • 6 new tests in llm-instrument.test.mjs, using a fake APIPromise shaped like the SDKs' (Promise subclass, parse through parse(), native private state). They cover the promise keeping its identity and _request_id, withResponse() data and usage, asResponse() with the body unread, awaiting plus withResponse() on one call, a stream through withResponse(), and a failed call with no unhandled rejection. All 6 fail on main.
  • Ran the same cases against the real openai 7.57.0 and @anthropic-ai/sdk 1.9.0 with a local server, including a 429: one span with the right usage or error status each time, and no unhandled rejection.
  • npm run build and npm test -w @autter/runtime-node pass (295 passed, 4 skipped).

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

traceCall returned result.then(...), a plain Promise, in place of the
APIPromise the openai and @anthropic-ai/sdk clients return. That dropped
withResponse() and asResponse(), so code that reads rate-limit headers or
request ids threw a TypeError once the client was instrumented.

The APIPromise parses the body only through parse(), which then, catch,
finally and withResponse() all call. traceCall now returns the original
promise and observes its parse() instead, once, so the span still ends
with usage however the call is read, a stream still reaches the caller
wrapped, and asResponse() leaves the body unread. Other promises (Google,
Bedrock) are chained as before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

instrumentLlmClient breaks .withResponse() and .asResponse() on OpenAI and Anthropic clients

1 participant