Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ Two rules govern this file, and both are load-bearing:
1. **These are weak priors, never task-quality labels.** They inform the `speed` and `reliability` terms only. A model is not "better" because it is fast.
2. **Historical numbers are never presented as a current provider SLA.** Hosts are expected to inject fresher observations (see below); the committed snapshot exists so the engine is safe and useful when a catalog is temporarily unavailable.

The repository ships 68 live performance profiles (three samples per model), 9 auditable historical seeds, 82 function-calling probe results, and a built-in snapshot of the 92 text models in the public catalog (2026-11-08 sync, plus routed ids the listing omits). Eleven newly listed models passed a real structured tool-call probe and are admitted as availability fallbacks for tool-required turns. DeepSeek V4 Flash Vision Experimental and Qwen 3.8 Flash did not return a valid tool call and remain fail-closed. GPT-6.1 Sol and DeepSeek V4.1 Flash also passed a structured tool-call probe in the subsequent capability refresh. Mistral Large 4 and Gemini 4 Argon have no probe evidence yet. A model with no probe evidence can serve plain chat but never a tool-required turn.
The repository ships 68 live performance profiles (three samples per model), 9 auditable historical seeds, 83 function-calling probe results, and a built-in snapshot of the 93 text models in the public catalog (2026-11-08 sync, plus routed ids the listing omits). Eleven newly listed models passed a real structured tool-call probe and are admitted as availability fallbacks for tool-required turns. DeepSeek V4 Flash Vision Experimental and Qwen 3.8 Flash did not return a valid tool call and remain fail-closed. GPT-6.1 Sol and DeepSeek V4.1 Flash also passed a structured tool-call probe in the subsequent capability refresh. Gemini 4 Carbon, added 2026-11-11 ahead of its gateway listing, returned its own next tool call through a two-round tool history on both the streaming and non-streaming gateway paths, so it is admitted as a tool fallback too. Mistral Large 4 and Gemini 4 Argon have no probe evidence yet. A model with no probe evidence can serve plain chat but never a tool-required turn.

Passing the function-call probe does **not** promote a model to primary. New tool-capable models are appended after the configured and trajectory-benchmarked candidate chains with neutral task affinity; they can recover an unavailable route immediately (cheapest first outside Premium), while primary promotion still requires an end-to-end agent evaluation.

Expand Down Expand Up @@ -345,7 +345,7 @@ The engine never makes a network call. Everything current is **injected** by the
const decision = route(prompt, systemPrompt, maxOutputTokens, {
config: DEFAULT_ROUTING_CONFIG,
modelPricing, // required — current prices from your catalog
modelCapabilities, // optional — overrides the built-in 92-model snapshot
modelCapabilities, // optional — overrides the built-in 93-model snapshot
modelPerformance, // optional — fresh speed/reliability, e.g. the Observatory feed
routingProfile: "auto",
hasTools, toolCount, toolNames, requiresTools,
Expand Down
7 changes: 7 additions & 0 deletions dist/index.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion dist/index.js.map

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions scripts/probe-tools.json
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,11 @@
"supportsTools": true,
"probedAt": "2026-10-03T04:27:11Z"
},
"google/gemini-4-carbon": {
"supportsTools": true,
"probedAt": "2026-11-11",
"source": "BlockRun base gateway before listing (BlockRunAI/blockrun#1035), not probe-model-profiles.mjs: a two-round tool history through both the stream and non-stream paths returned Carbon's own next tool call (finish_reason tool_calls), no catalog fallback. No latency profile yet; a re-probe overwrites this entry."
},
"minimax/minimax-m2.7": {
"supportsTools": true,
"probedAt": "2026-08-29T16:51:33Z"
Expand Down
60 changes: 40 additions & 20 deletions src/decisions.snapshot.json

Large diffs are not rendered by default.

8 changes: 8 additions & 0 deletions src/model-capabilities.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ describe("generated model capability snapshot", () => {
"anthropic/claude-opus-5.5",
"anthropic/claude-sonnet-5.5",
"google/gemini-3.8-flash",
"google/gemini-4-carbon",
"openai/gpt-5.1",
"openai/gpt-6-astra",
"openai/gpt-6-luna",
Expand All @@ -25,4 +26,11 @@ describe("generated model capability snapshot", () => {
])("keeps %s out of tool-required routes after an invalid tool response", (model) => {
expect(DEFAULT_MODEL_CAPABILITIES[model]?.supportsTools).toBe(false);
});

it.each([
"google/gemini-4-argon",
"mistral/mistral-large-4",
])("keeps unprobed %s out of tool-required routes", (model) => {
expect(DEFAULT_MODEL_CAPABILITIES[model]?.supportsTools).toBe(false);
});
});
6 changes: 6 additions & 0 deletions src/model-capabilities.ts
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,12 @@ export const DEFAULT_MODEL_CAPABILITIES: Readonly<Record<string, ModelCapabiliti
supportsTools: false,
supportsVision: true,
},
"google/gemini-4-carbon": {
contextWindow: 1_048_576,
maxOutputTokens: 65_536,
supportsTools: true,
supportsVision: true,
},
"minimax/minimax-m2.7": {
contextWindow: 204_800,
maxOutputTokens: 16_384,
Expand Down
1 change: 1 addition & 0 deletions src/portfolio.ts
Original file line number Diff line number Diff line change
Expand Up @@ -930,6 +930,7 @@ const VERIFIED_TOOL_FALLBACKS = [
"anthropic/claude-sonnet-5.5",
"deepseek/deepseek-v4.1-flash",
"google/gemini-3.8-flash",
"google/gemini-4-carbon",
"openai/gpt-5.1",
"openai/gpt-6-astra",
"openai/gpt-6-luna",
Expand Down
Loading