Skip to content

feat(catalogue): add GSMA's Open Telco benchmark on the standard - #226

Merged
aitianshi merged 6 commits into
mainfrom
feat/integrate-gsma-bench
Nov 19, 2026
Merged

aitianshi merged 6 commits into
mainfrom
feat/integrate-gsma-bench

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #225

What

GSMA's Open Telco becomes a bundled benchmark, open-telco, on the Nullius standard. It has eight telecom tests (TeleQnA, TeleTables, TeleMath, TeleLogs, 3GPP TSG, ORANBench, srsRANBench, 6G-Bench), each as a Lite dataset (GSMA's sample, the one its leaderboard ranks: 1,850 questions in all) and a Full one (20,588). The driver runs GSMA's own Inspect AI tasks, gsma-labs/evals v1.1.0 (708f1c5), unchanged, one question per unit. The Original calls the run's model through the relay. An agent that proves agent.task/v1 (Codex, Goose, JiuwenSwarm, OpenCode, the reference agent) gets GSMA's exact prompt. Its last message goes back to the task as the model's reply, and the task's own solver and scorer judge it in Inspect's eval loop. The data are GSMA/ot-lite and GSMA/ot-full at pinned revisions, each file checked by its sha256. data-source.json may now name a repository per dataset, so the page names each dataset's source. The options are temperature, subject, difficulty, task family and answer match (TeleLogs' soft or hard). All 16 datasets are proven: nop 0, and oracle 100% except on 3GPP TSG, 94/100 and 1871/2000, because its answer pattern cannot read RAN_AH1. The standard takes one bound per control for a whole benchmark, so the oracle's is at least 93%.

The page says three upstream defects and leaves them as published:

  • 3GPP TSG cannot read RAN_AH1 (gsma-labs/evals#81).
  • TeleMath's 0.01 absolute tolerance passes 0 on 10 Lite and 47 Full questions whose answer is not 0. This one is not reported upstream.
  • The multiple-choice prompt asks for 'ANSWER: $LETTER', and a reply that keeps the $ is scored wrong (gsma-labs/evals#73).

Two Nullius limitations: TeleMath's system prompt reaches an agent inside its one prompt, since ACP has no system message, and TeleLogs' maj@k is not computed.

Proofs short of the full mark. A control proven within a bound short of the full mark had the same green tick as a full pass, and its hover said it must score everything. Now:

  • Its tick takes the warning colour.
  • Its hover says what it reached and the bound it met ("93.55%, at least 93% needed").
  • The control's hover says what it must score ("must score at least 93%").

prove.status decides this once, for both the builder's Dataset step and the benchmark page. Besides Open Telco's two 3GPP TSG oracles, it changes LongMemEval's ten ticks (oracle at least 92%, nop at most 7%) and OSWorld2's nop on All tasks (at most 1%).

Touches

catalogue (benchmarks/open-telco/: benchmark.json, standard.json, data-source.json, upstream-settings.json, driver.py; catalogue/sources.py) · the standard (prove.py, recipe.py: what a proof reached and its bound) · templates and static (catalogue/benchmark.html, builder/command-builder.js, style.css, catalogue/marks.svg) · tests (test_open_telco_driver.py new, test_catalogue_sources.py, test_standard_ui.py, test_catalog.py, ui_tests/test_catalogue_routes.py) · docs (generated: docs/benchmarks/supported/, docs/agents/supported/)

Before and after

A scratch profile, light theme, 1280 px.

Benchmarks page: Open Telco is listed, with GSMA Labs' mark.

Before (main) After
Benchmarks page before Benchmarks page after

LongMemEval's proofs: every control is within its bound but short of the full mark, so its ticks turn amber.

Before (main) After
LongMemEval proofs before LongMemEval proofs after

New: Open Telco's page, its header and setup.

Open Telco's page header

New: Its proofs. The two 3GPP TSG oracles, 94/100 and 1871/2000, are amber.

Open Telco's proofs

New: The builder's Dataset step, with the same ticks on its 16 cards.

Open Telco's Dataset step

New: Good to know and the limitations.

Open Telco's Good to know and limitations

Verified

  • Suites, at the branch head (911ab51, after merging main): 3,015 backend tests and 656 route tests pass. Two tests failed once while the machine was loaded, a run-credentials route test and a kit isolation test. Both passed 3 times out of 3 alone and on full reruns, and this branch touches neither.
  • main gained feat(standard): reach the desktop through acting calls alone once the task is given #224 after that, so I merged it again (f845efd). The merge had no conflicts, and regenerating the docs changed nothing. The affected tests pass locally (74 backend, 32 route), and CI's backend and route suites pass on f845efd.
  • The generated docs are current (catalogue_docs.py), and cli check on the definition is ok.
  • Proofs: cli prove on all 16 datasets, 22,438 questions. Results: nop 0, oracle 100% except 3GPP TSG at 94/100 and 1871/2000, units stable, no leak. The proofs ran before three driver edits that change no behaviour: the Original no longer caches a model client per question, the main loop moved under __main__, and a file is now closed. The hand runs and the unit tests ran on the final driver.
  • Real runs with Muse Spark 1.3 Contributor through OpenRouter:
    • The Original, 2 questions of each Lite test: 11/16.
    • The reference agent on the container backend: 3/4. The Original on the container backend: 1/1.
    • Codex 2.2.2 on the container backend: 6/6 (TeleQnA, TeleMath and 3GPP TSG, 2 Lite questions each), 0.0195 USD.
    • JiuwenSwarm 0.2.8, same questions: 5/6, 0.0113 USD. Its miss answered 0 where the answer is 0.033.
  • Rate limits: with a stand-in provider that refused every call with 429, an Original question ended as RateLimitError, not as a timeout.
  • TeleMath's tolerance counts and the $LETTER scoring were checked against the pinned scorer and data.
  • Browser checks, light and dark, at 390 and 1280 px:
    • Pages: Open Telco's card, page and builder steps.
    • The amber ticks on Open Telco, LongMemEval and OSWorld2, and their hovers.
    • No sideways scroll and no page errors.
  • Not verified:
    • a whole dataset run by the Original with a real model;
    • agents other than Codex, JiuwenSwarm and the reference agent on Open Telco;
    • a TeleLogs run with answer match set to hard.

Risk

  • prove.py is in the standard, but only status() changes: each row gains what the control must score and how far short of the full mark its proof is. recipe.py passes both on, and command-builder.js and benchmark.html only draw them. Nothing else changes: how a control is proven, its bounds, its verdict and its records. A mistake there would mislabel a tick on the benchmark pages and the Dataset step (amber for a full pass, or green for a short one). It would never change a proof, a run or a score. Tests cover both cases and the default bounds.
  • Open Telco's oracle bound, at least 93%, holds for all 16 datasets, because the standard takes one bound per control for a whole benchmark. On the 14 datasets whose oracle reaches 100%, a regression that left it between 93% and 100% would still prove; its amber tick would show it. Bounds per dataset would close this. That is a change to the standard and is not made here.
  • Open Telco is a new entry. It installs GSMA's project from its lock file at v1.1.0 (with Inspect AI) and openai==2.26.0 in the benchmark's own environment. It fetches GSMA's data at pinned revisions, checked by sha256. A fault in its driver affects only Open Telco runs. If GSMA moves or removes a revision, a run fails at setup; it does not score differently.
  • sources.py: a dataset may name its own repository. Entries that do not are unchanged, and tests cover both.
  • Undo: revert the squash commit. Nothing is migrated, and no other benchmark's stored runs or proof records change.

GSMA's eight telecom tests (TeleQnA, TeleTables, TeleMath, TeleLogs,
3GPP TSG, ORANBench, srsRANBench, 6G-Bench), each as a Lite and a Full
dataset, run by GSMA's own Inspect AI tasks (gsma-labs/evals v1.1.0),
one question per unit. The Original calls the run's model through the
relay; an agent's last message is read and scored by the task's own
solver and scorer, in Inspect's eval loop. The data are the GSMA/ot-lite
and GSMA/ot-full repositories at pinned revisions, each file checked by
its sha256.

All 16 datasets are proven: nop 0, oracle 100% (3GPP TSG 94% and 93.6%:
its pattern cannot read RAN_AH1), units stable, no leak.

data-source.json may now name a repository per dataset, so a benchmark
page names GSMA/ot-lite or GSMA/ot-full as each dataset's source.
The multiple-choice prompt (Inspect's own template, on all five
multiple-choice tests) asks for 'ANSWER: $LETTER'; a reply that keeps
the dollar sign is read as no answer and scored wrong, as
gsma-labs/evals#73 reports and the pinned scorer confirms.

3GPP TSG's RAN_AH1 line cites gsma-labs/evals#81, which reports it.

TeleMath's line now says how far the 0.01 tolerance reaches: it is
looser than the scorer's 1% for every answer between -1 and 1 (44 Lite,
245 Full). 0 passes on 10 Lite and 47 Full questions, not 12 and 51:
2 Lite and 4 Full answers are 0, where 0 is right.
…ench

# Conflicts:
#	app/static/catalogue/marks.svg
#	app/tests/test_catalog.py
#	docs/agents/supported/README.md
#	docs/agents/supported/codex.md
#	docs/agents/supported/goose.md
#	docs/agents/supported/jiuwenswarm.md
#	docs/agents/supported/mini-swe-agent.md
#	docs/agents/supported/opencode.md
#	docs/agents/supported/terminus-2.md
…Telco

A control proven within a bound short of the full mark showed the same
green tick as a full pass, and its hover said it must score everything:
Open Telco's oracle on 3GPP TSG (94/100 and 1871/2000, at least 93%
needed), LongMemEval's oracle and nop, OSWorld2's nop. Its tick now
takes the warning colour, its hover says what it reached and the bound
it met ("93.55%, at least 93% needed"), and the control's hover says
what it must score. prove.status decides both, once, for the builder
and the benchmark page.

Open Telco's page:
- The rate-limit line was wrong and is gone. A stand-in provider that
  refused every call with 429 showed an Original question ending as
  RateLimitError, as Nullius labels a time run out under refused calls,
  not as a timeout.
- Answer match's two values say what TeleLogs' scorer compares without
  LaTeX markup.
- The temperature option carries the example main now requires of a
  pattern.
@github-actions

This comment has been minimized.

@aitianshi

Copy link
Copy Markdown
Collaborator Author

/judge

@github-actions

This comment has been minimized.

@aitianshi

Copy link
Copy Markdown
Collaborator Author

/judge

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown

🟡 PR judge · 7 of 8 yes · #226 ↔ #225

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/standard/prove.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/standard/prove.py
Question Verdict
Changes do what the issue asks · 7 of 7 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ⚠️ partly
Before and after shown · 8 images · the diff touches what is visible ✅ yes
Acceptance criteria, 7 of 7 met
# Criterion Verdict
1 The Benchmarks page lists Open Telco with its own mark and 16 datasets: the eight tests, each Lite and Full. ✅ met (cut)
2 The driver runs gsma-labs/evals v1.1.0 unchanged. The data are GSMA/ot-lite and GSMA/ot-full at pinned revisions,… ✅ met (cut)
3 The Original sends every model call through the relay. Agents that prove agent.task/v1 are offered, and their answers… ✅ met (cut)
4 All 16 datasets are proven: nop 0, oracle 100% except 3GPP TSG (94/100 and 1871/2000), units stable, no leak. ✅ met (cut)
5 Upstream limitations: 3GPP TSG's scorer cannot read RAN_AH1, TeleMath's 0.01 tolerance, the multiple-choice prompt's `$… ✅ met (cut)
6 A control proven short of the full mark (Open Telco's 3GPP TSG oracle, LongMemEval's controls, OSWorld2's nop on All ta… ✅ met (cut)
7 The generated docs list Open Telco, and both suites pass. ✅ met (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit f745f42 into main Nov 19, 2026
2 checks passed
@aitianshi
aitianshi deleted the feat/integrate-gsma-bench branch November 19, 2026 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Catalogue: run GSMA's Open Telco telecom tests on the standard

1 participant