[wmlba]

Resubmission of #236

This PR replaces #236, whose evaluation was stopped twice with exit code 143 near the end (9,205 and 9,160 of 9,209 entries).

• Cause: the LiveCodeBench evaluator runs each code answer in a forked process with a wall-clock timeout but no memory limit. Two answers in Add Compass CoreRouter submission (compass-corerouter) #236's optional optimality rows (Qwen3-235B's answers for LiveCodeBench_437 and LiveCodeBench_491) allocate memory without bound. Run alone, each reached a 96 GiB cgroup cap within about two minutes. These rows are evaluated last, so the 16 GB runner's memory-pressure killer stopped the job.
• Change: this file contains only the 8,400 scored rows, byte-identical to Add Compass CoreRouter submission (compass-corerouter) #236's. The 809 optimality rows are omitted, as in several other submissions. Routes, answers, token usage, cost accounting and the robustness file are unchanged.
• Local check: the official evaluator completed on this file with exit 0 under a 30 GiB memory cap. Peak memory was 23.5 GiB, from transient spikes during the LiveCodeBench phase.
• Disclosure: to diagnose the failure, we ran the official evaluator locally after Add Compass CoreRouter submission (compass-corerouter) #236 was submitted. Only memory, signals and exit codes were recorded; the scored copies, logs and metrics were deleted unread. Nothing from it changed the router.
• Suggestion for maintainers: a per-test memory limit in llm_evaluation/livecodebench_util.py would keep evaluation within the runner's memory for every submission. We can share the reproduction details.

Files

• router_inference/config/compass-corerouter.json
• router_inference/predictions/compass-corerouter.json — 8,400 routed queries (scored rows only; optimality entries omitted, see above), with generated_result populated
• router_inference/predictions/compass-corerouter-robustness.json — 420 routing decisions, with generated_result populated, because each decision depends on Qwen's answer to the perturbed prompt

Prediction entries were generated with RouterArena's own generate_predictions / save_predictions, and answers were attached with llm_inference/run.py, at commit 04c99fdbe4a3ec1f44475694f46dd25065c8fd8a. Both files pass check_config_prediction_files.py (full --check-generated-result and robustness), and tools/audit_token_accounting.py reports 0 failed inferences and 0 hidden reasoning tokens.

Router

• Model pool: qwen/qwen3-235b-a22b-2507 and deepseek/deepseek-v4-pro, with prices from the repository's model_cost.json.
• Method: an answer-aware cascade.
  1. Qwen3-235B answers every query.
  2. A fine-tuned Cloudflare/clef-flash answer critic reads the query and Qwen's answer and predicts the expected grader score in [0, 1]. The critic is a native JointSchemaHead plus a rank-256 LoRA, at base revision 8b2e5fd17c09fd49bc2880b805d5323ec7ff4ff3.
  3. If the score is below 0.9, the same query goes to DeepSeek V4 Pro and its answer is returned. Otherwise Qwen's answer is returned.
• Why a cascade: in our five-fold cross-validation on non-RouterArena data, prompt-only routing between these two models was statistically indistinguishable from always using one model. Reading the cheap model's answer is what separates easy queries from hard ones. The critic's out-of-fold AUROC for detecting a wrong answer was 0.885–0.904.
• Critic training data: 19,759 graded answers from Qwen3-235B, DeepSeek V4 Pro, DeepSeek V4 Flash and GPT-OSS-120B to the same 4,955 non-RouterArena prompts used by our earlier router. Training used the fold-0 training split, with the cross-validation recipe unchanged.
• Threshold: 0.9, the modal per-fold choice in cross-validation. It was frozen and hash-recorded before any RouterArena answer was scored.
• Router inference: native BF16 weights with FP32 recurrent state on AMD MI350X; no quantization. Scoring took under 3 minutes per 1,200 queries per GPU, including model load. Critic compute is router compute and is not billed, as for other routers.

Cost accounting (please review)

• Kept rows: token_usage is Qwen's provider-reported usage.
• Escalated rows: both calls were really paid for. token_usage is DeepSeek V4 Pro's reported usage plus the Qwen first call's dollar cost, converted to Pro input-price-equivalent tokens and rounded up. RouterArena's evaluator therefore charges the exact two-call cost, never less.
  • In total, 486,678 equivalent tokens were added, worth $0.2107 across 4,569 escalated rows.
• Disclosure: every regular row carries generated_result.cascade_accounting with the raw usage of each call, so reviewers can recompute the cost either way.
• Alternative: if maintainers prefer another convention, for example summing raw Qwen tokens into the Pro usage as cruq-router does, we can regenerate the file.

Evaluation-only compliance

• No RouterArena queries, answers, label files or scores were used to train, fit, tune or select any router component, the critic or the threshold.
• The critic, threshold, scoring code and accounting rule were frozen in a hash-recorded record before any RouterArena answer was scored.
  • A pre-declared sanity gate on held-out non-RouterArena data was passed: AUROC 0.901, and cascade accuracy at least as high as the cross-validation critic's.
  • One amendment, written before any full-split score existed, fixed a JSONL line-splitting bug. It changed parsing only.
• Training prompts come from public publisher training splits of public benchmarks, plus 987 MMLU-Pro test/validation items screened to exclude RouterArena items. A decontamination check of all 4,955 training prompts against the 8,400 RouterArena prompts found 0 exact and 0 near-duplicates.
• No local grading against RouterArena answers was performed before submission.

Solver generation

• qwen/qwen3-235b-a22b-2507: self-hosted original BF16 Qwen/Qwen3-235B-A22B-Instruct-2507 on vLLM with TP4 on AMD MI350X; no quantization.
  • Each request carried only the model and one user message, as in the repository's OpenRouter path, so the released sampling defaults applied: temperature 0.7, top-p 0.8, top-k 20. Maximum context was 16,384 tokens.
  • Robustness answers were generated in a separate cache. RouterArena's cache is keyed by global index, and all 420 robustness indices coincide with full-split indices whose prompts differ.
• deepseek/deepseek-v4-pro: generated through the Core42 Compass gateway with temperature 0.7 and max_tokens=8192. This model ID has no model_to_provider entry at this commit.

Final statistics

Item	Value
Full split	8,400 scored entries; 8,400 / 8,400 generations succeeded
Final answers (regular)	Qwen kept 3,831 · escalated to DeepSeek V4 Pro 4,569 (54.4%)
Robustness split	Qwen kept 162 · escalated 258
Finish reasons	Qwen: 8,398 stop, 2 length at the 16,384-token limit (full) and 420 stop (robustness). DeepSeek V4 Pro: all stop
Mean tokens	Kept Qwen 198 in / 83 out. Escalated: Qwen first call 307 in / 243 out, then Pro 295 in / 101 out
Generation cost at model_cost.json prices, both calls billed	$0.1532 per 1,000 queries (router compute excluded)
Escalation pattern	Near 100% on categories absent from the critic's training data or scored by continuous metrics (WMT19, LiveCodeBench, ChessInstruct, FinQA, AIME, NarrativeQA); 3–45% on most multiple-choice categories
Local accuracy	Not computed: no grading against RouterArena labels before submission

SHA-256: predictions 10e406c6972d29aaa8ffc578733ab6bdae45a49036973a2a5e047c454db6297e, robustness 8bf7ceb04b69c1a8455763003962456ccf5b6c30b0d4e17fe27006f744d8acf3, config 5515036fc2872bead0b34992a72cf7a32829fde65c34b773bbf8d52801638056.

[wmlba]

/evaluate

[wmlba]

The evaluation stopped again with exit code 143, this time during the LiveCodeBench phase (1,220/8,400 entries). The cause is a missing memory limit in the LiveCodeBench evaluator (#238); a fix is in #239. With that fix, this file's evaluation completes inside a 16 GiB cgroup with no swap (exit 0, 7.9 GiB peak). Once the fix is merged, we will merge main into this branch and comment /evaluate again. The router's files are unchanged.

[wmlba]

Superseded by #240 (CoreRouter v2). v1's unofficial verification with the #239 memory fix scored 73.40 (wmlba/routerarena-verification#1); v2 is described, pre-registered and predicted in #240.