[VBremer]

Router: nuroutev2-artemis

• Affiliation: NuRoute (https://nuroute.ai)
• What it is: NuRoute's Artemis Task router. For each query a small fine-tuned encoder estimates the query's task types; a capability matrix gives each pool model's expected quality and cost for them; the model with the best quality-for-cost trade-off answers (the default Balanced preset). Deciding makes no extra LLM call.
• How the predictions were made: every query went to NuRoute's production API with model: "auto", as a single user message with no other settings. Models that answered the full split: gpt-5.6-luna (5,341), DeepSeek-V4-Flash-0731 (2,899), Kimi-K2.6 (116), NVIDIA-Nemotron-3-Nano-30B-A3B (42), ministral-8b-latest (2). 160 queries (1.9%) were decided by NuRoute's fallback router because the router was busy, as they would be in production.
• Not trained or tuned on RouterArena data.
• Costs: the token counts the providers reported, at NuRoute's list prices (added to model_cost/model_cost.json).
• Failed generations: on 18 queries the chosen model used its whole output budget (32,768 tokens) without returning an answer. They are marked success: false, so they score as wrong.

Files

• router_inference/config/nuroutev2-artemis.json
• router_inference/predictions/nuroutev2-artemis.json (full split, generated_result populated)
• router_inference/predictions/nuroutev2-artemis-robustness.json
• model_cost/model_cost.json and universal_model_names.py: the five answering models

Local checks with RouterArena's own code

• router_inference/check_config_prediction_files.py nuroutev2-artemis full --check-generated-result: all checks pass.
• llm_evaluation/run.py nuroutev2-artemis full: arena score 0.7279, accuracy 0.7538, $0.6005 per 1K queries, 18 abnormal entries. robustness: 0.8000.
• pre-commit run on the changed files passes, except the SPDX-header hook, which needs flock (missing on macOS); the edited Python file keeps its header.

🤖 Generated with Claude Code

[VBremer]

/evaluate

[github-actions]

Router Evaluation Results

Router: nuroutev2-artemis
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7287
Accuracy	75.47%
Total Cost	$5.044231
Avg Cost per Query	$0.000601
Avg Cost per 1K Queries	$0.6005
Number of Queries	8400
Abnormal Entries	18
Robustness Score	0.8000

⚠️ 18 of 8400 queries (0.2%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow

[yl231]

/evaluate

[github-actions]

Router Evaluation Results

Router: nuroutev2-artemis
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7287
Accuracy	75.47%
Total Cost	$5.044231
Avg Cost per Query	$0.000601
Avg Cost per 1K Queries	$0.6005
Number of Queries	8400
Abnormal Entries	18
Robustness Score	0.8000

⚠️ 18 of 8400 queries (0.2%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow

[yl231]

Thanks, @VBremer. We re-ran /evaluate with the LiveCodeBench fixes merged: Arena 0.7287, accuracy 75.47%. After the Divyam and BizRouter submissions merged, this branch conflicted with their price-table entries. We can't push to your fork, so we're landing it through #247, which is your commits plus one merge commit keeping both sides; no price differs. This PR will show as merged once #247 lands. Prices for all routers will be revisited together when we rebuild the price table.

One question: #208 (NuRoute, by @vibre06) is still open. Does this v2 entry replace it, so that we can close #208?

[yl231]

A correction to my note above: the /evaluate re-run graded with this branch's own copy of the evaluator, so it did not include the LiveCodeBench fixes after all. The full re-grade in #253 uses the current evaluator for every router. There, NuRoute Artemis scores 73.47 (accuracy 76.19%).

[VBremer]

@yl231 Thanks for reaching out. This is fine. We can keep v2 and discard the other older version of our router.