[wmlba]

Compass CoreRouter v2 (compass-corerouter)

This replaces #237, the v1 resubmission, which is closed as superseded. v1 routed from Qwen3-235B to DeepSeek V4 Pro with
an answer critic.

Router

Input. The router uses the prompt text only. A task classifier predicts the task type from the prompt:

• TF-IDF word and character n-grams, logistic regression;
• trained on our own practice prompts;
• 96% accurate on our held-out practice test.

Pool. All models run at native precision. Gemini is called through Google's API with RouterArena's own
_call_google request semantics.

Model	Served
qwen/qwen3-235b-a22b-2507	self-hosted, BF16
google/gemma-4-31b-it	self-hosted, BF16
gemini-3-flash-preview	Google API
deepseek/deepseek-v4-flash	self-hosted, original release precision
qwen/qwen3-next-80b-a3b-instruct	self-hosted, BF16
deepseek/deepseek-v4.1-flash-1017-reasoning-low	self-hosted, original release precision; reasoning effort "low" (25), as OpenRouter's reasoning: low

deepseek/deepseek-v4.1-flash-1017-reasoning-low is registered at OpenRouter's listed price for
deepseek/deepseek-v4.1-flash-1017 ($0.15/M input, $0.60/M output). The -reasoning-<effort> suffix sends OpenRouter's
reasoning.effort (same handling as #220).

Policy.

• Per-task model picks and the quality–cost trade-off (λ = 0.02) are fitted by 5-fold cross-validation on our own
  practice data: graded answers of the candidate models on public datasets' train/dev splits, scored with RouterArena's
  metric code.
• Where the pick is Gemini-3-Flash, Qwen3-235B and Gemma-4-31B answer first. Gemini is called when:
  • their final answers disagree; or
  • they agree, but a self-hosted answer critic scores the shared answer below 0.5. The critic is a Clef model
    fine-tuned on our own practice data; it reads only the prompt and our models' answers. The threshold was chosen by
    cross-validation on practice data.
• For the MMLU-Pro task type, DeepSeek V4.1 Flash (reasoning low) is called instead of Gemini in that step; AIME and
  MATH go to it directly. These choices come from the same cross-validation on our practice data.
• Calls per row: one call on 1,743 rows, two calls on 4,468 rows, and three calls on 2,189 rows.

Cost accounting.

• Every call is charged on the returned row. The other calls' dollar cost is added as returned-model input-price tokens,
  rounded up.
• Per-call usage and cost are kept in generated_result.cascade_accounting.
• Charged cost is $0.2309 per 1K queries. check_config_prediction_files.py passes for both splits, and
  tools/audit_token_accounting.py --strict is clean.

Optimality rows. The optional optimality rows are omitted, as in #237 (see #238 / #239). The 8,400 scored rows and
the 420 robustness rows are complete.

[wmlba]

/evaluate

[github-actions]

Router Evaluation Results

Router: compass-corerouter
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7643
Accuracy	77.71%
Total Cost	$1.473410
Avg Cost per Query	$0.000175
Avg Cost per 1K Queries	$0.1754
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.8381

---

Evaluation completed by RouterArena automated workflow

[wmlba]

/evaluate

[github-actions]

Router Evaluation Results

Router: compass-corerouter
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7645
Accuracy	77.73%
Total Cost	$1.477005
Avg Cost per Query	$0.000176
Avg Cost per 1K Queries	$0.1758
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7381

---

Evaluation completed by RouterArena automated workflow

[wmlba]

/evaluate

[github-actions]

Router Evaluation Results

Router: compass-corerouter
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7688
Accuracy	78.43%
Total Cost	$1.722366
Avg Cost per Query	$0.000205
Avg Cost per 1K Queries	$0.2050
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7524

---

Evaluation completed by RouterArena automated workflow

[wmlba]

/evaluate

[github-actions]

Router Evaluation Results

Router: compass-corerouter
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7687
Accuracy	78.59%
Total Cost	$1.939322
Avg Cost per Query	$0.000231
Avg Cost per 1K Queries	$0.2309
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.6405

---

Evaluation completed by RouterArena automated workflow