[wmlba]

Files

• router_inference/config/compass-corerouter.json
• router_inference/predictions/compass-corerouter.json — 8,400 routed queries plus 809 official optimality entries, with generated_result populated
• router_inference/predictions/compass-corerouter-robustness.json — 420 routing decisions, with generated_result populated, because each decision depends on Qwen's answer to the perturbed prompt

Prediction entries were generated with RouterArena's own generate_predictions / save_predictions, and answers were attached with llm_inference/run.py, at commit 04c99fdbe4a3ec1f44475694f46dd25065c8fd8a. Both files pass check_config_prediction_files.py (full --check-generated-result and robustness), and tools/audit_token_accounting.py reports 0 failed inferences and 0 hidden reasoning tokens.

Router

• Model pool: qwen/qwen3-235b-a22b-2507 and deepseek/deepseek-v4-pro, with prices from the repository's model_cost.json.
• Method: an answer-aware cascade.
  1. Qwen3-235B answers every query.
  2. A fine-tuned Cloudflare/clef-flash answer critic reads the query and Qwen's answer and predicts the expected grader score in [0, 1]. The critic is a native JointSchemaHead plus a rank-256 LoRA, at base revision 8b2e5fd17c09fd49bc2880b805d5323ec7ff4ff3.
  3. If the score is below 0.9, the same query goes to DeepSeek V4 Pro and its answer is returned. Otherwise Qwen's answer is returned.
• Why a cascade: in our five-fold cross-validation on non-RouterArena data, prompt-only routing between these two models was statistically indistinguishable from always using one model. Reading the cheap model's answer is what separates easy queries from hard ones. The critic's out-of-fold AUROC for detecting a wrong answer was 0.885–0.904.
• Critic training data: 19,759 graded answers from Qwen3-235B, DeepSeek V4 Pro, DeepSeek V4 Flash and GPT-OSS-120B to the same 4,955 non-RouterArena prompts used by our earlier router. Training used the fold-0 training split, with the cross-validation recipe unchanged.
• Threshold: 0.9, the modal per-fold choice in cross-validation. It was frozen and hash-recorded before any RouterArena answer was scored.
• Router inference: native BF16 weights with FP32 recurrent state on AMD MI350X; no quantization. Scoring took under 3 minutes per 1,200 queries per GPU, including model load. Critic compute is router compute and is not billed, as for other routers.

Cost accounting (please review)

• Kept rows: token_usage is Qwen's provider-reported usage.
• Escalated rows: both calls were really paid for. token_usage is DeepSeek V4 Pro's reported usage plus the Qwen first call's dollar cost, converted to Pro input-price-equivalent tokens and rounded up. RouterArena's evaluator therefore charges the exact two-call cost, never less.
  • In total, 486,678 equivalent tokens were added, worth $0.2107 across 4,569 escalated rows.
• Disclosure: every regular row carries generated_result.cascade_accounting with the raw usage of each call, so reviewers can recompute the cost either way.
• Alternative: if maintainers prefer another convention, for example summing raw Qwen tokens into the Pro usage as cruq-router does, we can regenerate the file.

Evaluation-only compliance

• No RouterArena queries, answers, label files or scores were used to train, fit, tune or select any router component, the critic or the threshold.
• The critic, threshold, scoring code and accounting rule were frozen in a hash-recorded record before any RouterArena answer was scored.
  • A pre-declared sanity gate on held-out non-RouterArena data was passed: AUROC 0.901, and cascade accuracy at least as high as the cross-validation critic's.
  • One amendment, written before any full-split score existed, fixed a JSONL line-splitting bug. It changed parsing only.
• Training prompts come from public publisher training splits of public benchmarks, plus 987 MMLU-Pro test/validation items screened to exclude RouterArena items. A decontamination check of all 4,955 training prompts against the 8,400 RouterArena prompts found 0 exact and 0 near-duplicates.
• No local grading against RouterArena answers was performed before submission.

Solver generation

• qwen/qwen3-235b-a22b-2507: self-hosted original BF16 Qwen/Qwen3-235B-A22B-Instruct-2507 on vLLM with TP4 on AMD MI350X; no quantization.
  • Each request carried only the model and one user message, as in the repository's OpenRouter path, so the released sampling defaults applied: temperature 0.7, top-p 0.8, top-k 20. Maximum context was 16,384 tokens.
  • Robustness answers were generated in a separate cache. RouterArena's cache is keyed by global index, and all 420 robustness indices coincide with full-split indices whose prompts differ.
• deepseek/deepseek-v4-pro: generated through the Core42 Compass gateway with temperature 0.7 and max_tokens=8192. This model ID has no model_to_provider entry at this commit.

Final statistics

Item	Value
Full split	8,400 regular + 809 optimality entries; 9,209 / 9,209 generations succeeded
Final answers (regular)	Qwen kept 3,831 · escalated to DeepSeek V4 Pro 4,569 (54.4%)
Robustness split	Qwen kept 162 · escalated 258
Finish reasons	Qwen: 8,398 stop, 2 length at the 16,384-token limit (full) and 420 stop (robustness). DeepSeek V4 Pro: all stop
Mean tokens	Kept Qwen 198 in / 83 out. Escalated: Qwen first call 307 in / 243 out, then Pro 295 in / 101 out
Generation cost at model_cost.json prices, both calls billed	$0.1532 per 1,000 queries (router compute excluded)
Escalation pattern	Near 100% on categories absent from the critic's training data or scored by continuous metrics (WMT19, LiveCodeBench, ChessInstruct, FinQA, AIME, NarrativeQA); 3–45% on most multiple-choice categories
Local accuracy	Not computed: no grading against RouterArena labels before submission

SHA-256: predictions 8f069176699fab96ae2b62eeb10bdc26cc05c6d50b5b6f17ce012232feb9ff46, robustness 8bf7ceb04b69c1a8455763003962456ccf5b6c30b0d4e17fe27006f744d8acf3, config 5515036fc2872bead0b34992a72cf7a32829fde65c34b773bbf8d52801638056.

[wmlba]

/evaluate

[wmlba]

/evaluate

[wmlba]

Superseded by #237. Evaluation here was stopped twice with exit code 143: two code answers in the optional optimality rows (Qwen3-235B on LiveCodeBench_437 and LiveCodeBench_491) allocate memory without bound in the LiveCodeBench evaluator, which has no per-test memory limit, so the 16 GB runner's memory-pressure killer ends the job. #237 contains the same 8,400 scored rows without the optimality rows; details are in its description.