Skip to content

Full audit: src/ layout, table-driven affinity, 14 routing/hardening fixes, catalog resync - #12

Merged
VickyXAI merged 6 commits into
mainfrom
improve/full-audit
Nov 8, 2026
Merged

VickyXAI merged 6 commits into
mainfrom
improve/full-audit

Conversation

@VickyXAI

@VickyXAI VickyXAI commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Structure / DX

  • Sources and tests move to src/; tests are now typechecked (noUnusedLocals on).
  • affinity()'s duplicated if/else ladders become named affinity tables picked by affinityBand(). Default portfolio weights now point at DEFAULT_ROUTING_CONFIG instead of keeping a copy. Affinity and cost are computed once per candidate. Cost/baseline/savings logic and the fail-open filters each have one shared helper. None of this moves any of the 88 snapshot decisions.

Correctness fixes (regressions in src/hardening.test.ts, 23 tests)

  • Chinese debug/code-edit/extraction/reasoning keywords sat inside an ASCII-only \b and never matched, so 请修复这个报错 routed as chat.
  • A bare language name on a tool turn ("java coffee prices online") forced code_agent, with Codex as primary.
  • Tool-name verbs matched as substrings ("add" inside "address"). Six-letter capitalised words ("CANCEL", "FLIGHT") counted as reservation ids. Japanese prompts got the Mandarin extraction band. "e.g." counted as a location connector.
  • The rules multi-step signal fired on any sentence ending in a number. Full-width ? was not counted.
  • An unpriced candidate scored a neutral 0.5 on cost and outranked priced models. NaN prices and NaN performance profiles leaked NaN into costs and into the sort order.
  • A promotion with an unparseable date stayed active forever. sampleForScan stopped bounding the scan at limit 1 or NaN.
  • calculateModelCost reported ~100% savings for unpriced models. toRoutingCatalog threw away the bundled vision evidence and the nemotron fail-closed override.

Catalog freshness

  • I probed all 54 routed ids directly today: every one answered 402 (paid) or 200 (free).
  • sync-model-capabilities used to drop any id missing from the listing, which would have removed moonshot/kimi-k3. That model is still routed and still served. The script now keeps routed ids that are absent from the listing.
  • Resynced to the 2026-11-08 catalog: 4 models added, 2 unrouted models removed. The catalog's new "vision" tag on deepseek-chat, deepseek-reasoner and deepseek-v4.1-flash fails closed via overrides until an image probe confirms it.

Docs: the README no longer claims catalog-absence filtering or "never fail open". Shadow evaluation is documented as host-side. Counts and paths are refreshed.

Decision snapshot

8 of 88 entries move, all from the multi-step fix: prompt 12 ("…(d) 29. Answer…") and prompt 13 ("…= 0. Show your work") no longer pick up a false multi-step signal. Over its four profiles, prompt 12 moves MEDIUM→SIMPLE.

Follow-ups

  • Python port (blockrun-llm) has to mirror the behavior fixes before its parity tests are repinned to this commit.
  • Not changed, flagged for calibration: the code_edit/debug/reasoning/vision affinity bands only reach models already in the tier chain, because evidenceCandidates returns [] for those tasks.
  • The listing now has moonshot/kimi-k3.1-pro. K3 is still served, so no chain was changed without calibration evidence.

Test plan

  • npm run check — 189 tests across 11 files, typecheck, build
  • dist/ rebuilt and committed

🤖 Generated with Claude Code

1bcMax and others added 5 commits November 8, 2026 08:05
Tests were excluded from tsc; they now typecheck with noUnusedLocals.
dist/ changes only in the module-path comments tsup emits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
affinity() repeated the web-research, retail-policy and terminal bands
across task types; they are now named tables selected by affinityBand().
The portfolio weights reference DEFAULT_ROUTING_CONFIG instead of a copy,
affinity and cost are computed once per candidate, the reasoning keyword
scan runs once, and the fail-open filters share one helper.

One intended behavior change rides on the shared savings helper:
calculateModelCost now reports savings 0 for a model with no pricing,
matching selectModel. It previously costed an unpriced model at the
minimum payment and reported ~100% savings.

The 88-request decision snapshot is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Classification
- Chinese debug/code-edit/extraction/reasoning keywords sat inside an
  ASCII-only \b and never matched; 请修复这个报错 routed as chat.
- A bare language name on a tool turn ("java coffee prices online")
  forced code_agent with Codex primary; it now needs a code artifact or a
  coding action.
- Tool-name operation verbs matched by substring ("add" in "address"),
  inflating parallel-tool detection; now whole words.
- Six-letter shouted words ("CANCEL", "FLIGHT") counted as airline
  reservation ids and raised agentRisk; ids now need a letter and a digit.
- Japanese prompts got the Mandarin extraction band; kana now excludes it.
- "e.g." counted as a Spanish/Italian multi-location connector.
- Rules multi-step fired on any sentence ending in a number; it now needs a
  line-leading list item. Full-width question marks now count.

Ranking and cost
- An unpriced candidate scored a neutral 0.5 on cost and outranked priced
  models; it now scores as the most expensive (it is still never filtered).
- NaN/negative prices are treated as unknown instead of propagating NaN
  into costEstimate and savings.
- A host-injected performance profile with a non-numeric field made the
  score NaN and froze that candidate in the sort; it is now ignored.

Hardening
- A promotion with an unparseable date was active forever; it is skipped.
- sampleForScan returned the whole prompt at limit 1 (slice(-0)) and at a
  NaN limit, defeating the scan bound.
- toRoutingCatalog took vision only from catalog categories, discarding
  the bundled Haiku 4.5 / Sonnet 4.6 vision evidence and the nemotron
  nano-omni fail-closed override; verified evidence now wins.

Snapshot: 8 of 88 entries move, all from the multi-step fix (prompts 12
and 13 end a sentence with a number).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bsent from the listing

The sync script dropped any id missing from GET /api/v1/models. Today
that would have removed moonshot/kimi-k3, which three chains and four
affinity bands still name and which the gateway still serves (direct call
returns 402). It now retains the previous entry for routed ids absent from
the listing, annotated, and removes one only on an explicit
available:false row. It also refuses to overwrite the snapshot when the
response has no data array.

Catalog changes: +kimi-k3.1-pro, +nvidia/minimax-m3.1-flash,
+nvidia/nemotron-3-super-120b, +nvidia/nemotron-3.5-bolt (all unprobed,
tools fail closed); -openai/gpt-4.1-nano, -nvidia/llama-3.2-11b-vision
(unrouted).

The catalog newly tags deepseek-chat, deepseek-reasoner and
deepseek-v4.1-flash as vision. No image probe backs that, so overrides
keep them text-only (fail closed) until one passes.

All 54 ids named by config.ts and portfolio.ts were probed directly
today: every one answered 402 (paid) or 200 (free).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Stage 2 claimed candidates absent from the catalog are removed; they are
  not (the gateway serves hidden ids). Document that, and the two
  deliberate fail-open edges, instead of claiming hard filters never fail
  open.
- unavailableModels: document the all-rungs-dead exception.
- Shadow evaluation is host-side; route() never reads config.shadow.
- Refresh counts (tests, profiles, probe results, snapshot size) and the
  src/ paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test files import node:fs and read process.env. Locally an
extraneous @types/node hid this; CI's clean npm ci did not have it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@VickyXAI
VickyXAI merged commit 273c2af into main Nov 8, 2026
2 checks passed
VickyXAI pushed a commit to BlockRunAI/blockrun-llm that referenced this pull request Nov 8, 2026
Ports the router-core audit (BlockRunAI/router-core#12) onto the 8773a99
port. All 88 cross-language snapshot decisions match again.

- portfolio: Chinese task keywords outside the ASCII \b; coding-action
  gate for code_agent; whole-word tool verbs; letter+digit reservation ids;
  kana excluded from zh; strict multi-location connector; unpriced
  candidates score cost 0; malformed performance profiles ignored.
- rules: line-leading numbered lists only; full-width question marks.
- selector: usable_pricing; no savings for an unpriced model in
  calculate_model_cost.
- tokens: bounded scan at limit 1 and for non-numeric limits.
- model_capabilities: 2026-11-08 catalog (kimi-k3 retained, DeepSeek
  vision fails closed).

29 new tests port upstream hardening.test.ts; 15 fail on the previous port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VickyXAI added a commit to BlockRunAI/blockrun-llm that referenced this pull request Nov 8, 2026
Ports the router-core audit (BlockRunAI/router-core#12) onto the 8773a99
port. All 88 cross-language snapshot decisions match again.

- portfolio: Chinese task keywords outside the ASCII \b; coding-action
  gate for code_agent; whole-word tool verbs; letter+digit reservation ids;
  kana excluded from zh; strict multi-location connector; unpriced
  candidates score cost 0; malformed performance profiles ignored.
- rules: line-leading numbered lists only; full-width question marks.
- selector: usable_pricing; no savings for an unpriced model in
  calculate_model_cost.
- tokens: bounded scan at limit 1 and for non-numeric limits.
- model_capabilities: 2026-11-08 catalog (kimi-k3 retained, DeepSeek
  vision fails closed).

29 new tests port upstream hardening.test.ts; 15 fail on the previous port.

Co-authored-by: 1bcMax <viewitter@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
VickyXAI added a commit to BlockRunAI/blockrun-llm-ts that referenced this pull request Nov 8, 2026
Picks up the 2026-11 router audit (BlockRunAI/router-core#12): Chinese
task keywords, code-agent gating, look-alike tool verbs and reservation
ids, unpriced/NaN cost handling, toRoutingCatalog vision evidence, and the
2026-11-08 capability snapshot. Decisions match the Python SDK port
(blockrun-llm#102) again.

Co-authored-by: 1bcMax <viewitter@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
VickyXAI added a commit to BlockRunAI/ClawRouter that referenced this pull request Nov 8, 2026
Picks up the 2026-11 router audit (BlockRunAI/router-core#12): Chinese
task classification, code-agent gating, look-alike tool verbs and
reservation ids, unpriced/NaN cost handling, toRoutingCatalog vision
evidence and the 2026-11-08 capability snapshot. Decisions match
@blockrun/llm 3.26.4 and blockrun-llm 1.21.6.

Co-authored-by: 1bcMax <viewitter@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
VickyXAI added a commit to BlockRunAI/Franklin that referenced this pull request Nov 8, 2026
* chore: bump @blockrun/router-core to 273c2af, release 3.49.25

Picks up the 2026-11 router audit (BlockRunAI/router-core#12). Auto
routing decisions match @blockrun/llm 3.26.4 and blockrun-llm 1.21.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): declare the same router-core pin as the runtime

prepare-runtime requires Desktop to declare every embedded runtime
dependency at the runtime's exact version; the pin bump missed
apps/desktop/package.json, which failed the macOS and Windows builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: 1bcMax <viewitter@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
VickyXAI added a commit to BlockRunAI/Franklin-Trading that referenced this pull request Nov 8, 2026
Picks up the 2026-11 router audit (BlockRunAI/router-core#12). Auto
routing decisions match @blockrun/llm 3.26.4 and blockrun-llm 1.21.6.

Co-authored-by: 1bcMax <viewitter@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant