Repository navigation
Models: a reasoning setting; resend an effort the upstream refuses - #126
Merged
Merged
Conversation
A model can now set its reasoning (`models.reasoning_effort`): `off`
takes reasoning out of requests that have it on, and a level (low,
medium, high, xhigh, max) changes the effort of requests whose
reasoning is on. Requests without reasoning, such as title generation,
go out unchanged. The setting applies to the caller's request in its
own format, before routing, so a converted hop carries it the way the
upstream's format writes it, and the response cache keys on the
changed body.
An upstream that refuses the effort with a 400 gets the request once
more on the same route, with the level it named or without one. The
resend does not count against the retry budget. When it is answered,
the refused level is remembered in Redis per provider and upstream
model for 7 days (`effort_memory:{provider:<id>}`, one hash field per
model), and later requests send what worked straight away. A
provider's memory is cleared when its URL or credentials change, a
route relearns its protocol, or it is deleted.
A hop that sent a different effort than the caller asked is recorded
on its gateway_logs row as `"effort": {asked, sent, reason}`.
The admin console gets the setting as a segmented control in the
model editor and shows it in the model detail sheet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Align the gateway with core's reasoning effort fixes:
- Refusals are read against the effort the hop writes
(`written_effort`), whether or not reasoning is on, so Claude's
"effort 'xhigh' is not supported when thinking is disabled ... Use
effort 'high' or below" is resent with high. A request carrying only
a budget is never treated as an effort refusal and never asks the
memory.
- A `none` the upstream does not take (`none_rejection`) is resent
without the field, and remembered.
- The memory is keyed on the provider, the format sent and the model
(`effort_memory:{provider:<id>}`, field `<dialect>:<model>`). A
remembered fix refused itself is forgotten, or replaced when the
resend that follows is answered; one answered at the level first
refused is dropped. No more chained lookups.
- The setting is applied per hop, for the model named upstream
(Gemini budgets are capped per model), and keyed into the response
cache like the output cap. `off` writes `none` for OpenAI's formats.
- The log row's `effort.asked` is what the hop would have sent before
the change: the caller's level for the setting, the replaced level
for a refusal or the memory. `none` is a valid slug; `sent: null`
means removed.
The model editor's reasoning control now takes its whole row, so its
seven options fit the dialog.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pins the core crates to the merge of ThinkWatch-Core#374 until the next core release, when the pin moves back to a tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some upstreams accept only some reasoning-effort levels, so a client that sends
mediumorxhighcan get a 400. Enterprise follows ThinkWatch-Core#374:A reasoning setting per model.
models.reasoning_effort:off,low,medium,high,xhighormax; empty leaves requests unchanged. A level changes the effort of requests that already have reasoning on;offturns reasoning off (nonefor OpenAI Chat and Responses). The model editor gets a segmented control for it, and the model detail shows the value. The setting is applied per upstream hop with the model name sent upstream, and it is part of the response-cache key, as the output cap is.Resending a refused effort. When an upstream answers 400 because of the effort, the request is sent again once on the same route with a level it accepts (or without the effort; a refused
noneis resent without the field). After a successful resend the fix is remembered in Redis for 7 days per provider, wire format, model and level (effort_memory:{provider:<id>}), and is dropped when the provider's base URL or headers change, the provider is removed, or a route relearns its protocol. The resend does not use the retry budget. Budget-only requests are never resent.Each change is recorded in the gateway log detail as
effort(asked,sent,reason:setting/rejected/remembered).The core crates are pinned to the merge of ThinkWatch-Core#374 until the next core release.
🤖 Generated with Claude Code