Custom "anthropic" provider does not enforce provider.max_prompt_tokens — sessions grow past the model context window until a hard 400

[kevnwang]

Summary

When a custom provider of type: "anthropic" is configured with max_prompt_tokens, the SDK does not appear to enforce that budget. A long-running session keeps accumulating transcript until the request exceeds the model's context window and the provider rejects it with HTTP 400 -- after which the session is permanently unusable.

Configuration

The provider is created on every session create/resume with an explicit prompt budget:

{
  "type": "anthropic",
  "base_url": "...",
  "model_id": "claude-sonnet-5",
  "max_output_tokens": 32768,
  "max_prompt_tokens": 967232
}

max_prompt_tokens is derived as context_window (1,000,000) - max_output_tokens (32,768) = 967,232.

Expected

The SDK compacts (or otherwise bounds the prompt) before crossing max_prompt_tokens = 967232.

Actual

The transcript grew unbounded to 1,001,142 tokens -- 33,910 past the configured budget, and past the model's 1,000,000 hard limit:

400 invalid_request_error
"prompt is too long: 1001142 tokens > 1000000 maximum"

The session had run ~18 successful turns over ~3 hours, with the serialized request growing steadily (~1.98 MB -> ~2.04 MB) before crossing the limit. Tool count was constant throughout, so the growth is accumulated conversation history rather than tool schemas.

Two additional observations

1. infinite_sessions thresholds also appear inert on this path. background_compaction_threshold / buffer_exhaustion_threshold are sent on every turn but appear to have no effect for the anthropic provider (they do take effect on the Copilot backend path). So neither the threshold-based compaction nor the max_prompt_tokens budget bounded the transcript.

2. The session actively degrades after the first failure. Once over the limit, continued turns keep appending to the transcript -- request size grew from ~2.044 MB to ~2.065 MB across ~30 consecutive failed turns. There is no back-off, trim, or compaction triggered by the 400, so the session can never self-recover; every subsequent turn fails immediately (~1.5s vs. the 38s first failure).

Impact

Every turn in an affected session fails permanently. The only recovery is to start a new session, and nothing in the surfaced error indicates that to the user. Because the failure is a deterministic 400, retry suppression correctly kicks in -- but that just means the session is durably wedged.

Environment

• SDK 1.0.7 / Copilot CLI 1.0.71
• Custom anthropic provider over an OpenAI-incompatible relay endpoint
• Model claude-sonnet-5 (1,000,000-token context window)

Ask

Should provider.max_prompt_tokens be enforced on the anthropic provider path (and/or should infinite_sessions compaction apply there)? If enforcement is intentionally backend-only today, it would help to document that clearly, since the field is accepted without warning and silently has no effect.

[github-actions]

Investigation findings

I looked at this issue across the SDK codebase. Here's what I found:

What I investigated

• nodejs/src/types.ts — the ProviderConfig interface and maxPromptTokens field
• nodejs/src/generated/rpc.ts — the wire-level ProviderConfig schema with maxPromptTokens
• docs/auth/byok.md — the ProviderConfig reference table
• E2E and unit tests covering maxPromptTokens on various provider types
• SDK client code (nodejs/src/client.ts) — how provider configs are forwarded to the CLI runtime

What I found

Confirmed bug. The ProviderConfig interface in the SDK exposes maxPromptTokens with a JSDoc comment that explicitly says:

"The runtime triggers conversation compaction before sending a request when the prompt (system message, history, tool definitions, user message) would exceed this limit."

However, your report — along with the note that background_compaction_threshold / buffer_exhaustion_threshold are also inert on the anthropic provider path — strongly indicates that this compaction enforcement is not applied on the Anthropic provider path. The field is accepted without error, forwarded over the RPC wire, but silently ignored by the CLI runtime for type: "anthropic" sessions. The session then accumulates context unboundedly until the hard provider 400 hits and permanently wedges the session.

There is also a secondary documentation gap: the ProviderConfig reference table in docs/auth/byok.md does not list maxPromptTokens, maxOutputTokens, modelId, or wireModel at all — so users have no way to discover that maxPromptTokens exists or whether it applies to their provider type. If the docs explicitly noted "compaction enforcement via maxPromptTokens is currently only supported on the Copilot backend path, not on type: "anthropic" or other direct-provider paths," the misconfiguration would be obvious.

Root cause

The compaction logic in the CLI runtime appears to only trigger for Copilot backend (CAPI) sessions. Custom anthropic provider sessions bypass this path, so neither maxPromptTokens enforcement nor infiniteSessions threshold-based compaction applies to them. The field is accepted and forwarded, giving the false impression that the budget will be enforced.

Classification

This is a bug — the field is documented (in code) as enforcing compaction, but it silently has no effect on the Anthropic provider path. The session overflow and permanent wedging are the concrete customer impact.

Workaround (until fixed)

Until this is resolved at the runtime level, you can:

1. Monitor token count via session.rpc.metadata.getContextInfo() and call session.rpc.history.compact() manually before approaching the limit.
2. Start a new session when you detect a persistent HTTP 400 with prompt is too long in the error.
3. Track the session.compaction_start / session.compaction_complete events — their absence confirms compaction never fires on this path.

Generated by Bug Handler for #2355 · sonnet46 95.7 AIC · ⌖ 5.31 AIC · ⊞ 5.9K · ◷