Repository navigation
feat(catalogue): add GSMA's Open Telco benchmark on the standard - #226
Merged
Merged
Conversation
GSMA's eight telecom tests (TeleQnA, TeleTables, TeleMath, TeleLogs, 3GPP TSG, ORANBench, srsRANBench, 6G-Bench), each as a Lite and a Full dataset, run by GSMA's own Inspect AI tasks (gsma-labs/evals v1.1.0), one question per unit. The Original calls the run's model through the relay; an agent's last message is read and scored by the task's own solver and scorer, in Inspect's eval loop. The data are the GSMA/ot-lite and GSMA/ot-full repositories at pinned revisions, each file checked by its sha256. All 16 datasets are proven: nop 0, oracle 100% (3GPP TSG 94% and 93.6%: its pattern cannot read RAN_AH1), units stable, no leak. data-source.json may now name a repository per dataset, so a benchmark page names GSMA/ot-lite or GSMA/ot-full as each dataset's source.
The multiple-choice prompt (Inspect's own template, on all five multiple-choice tests) asks for 'ANSWER: $LETTER'; a reply that keeps the dollar sign is read as no answer and scored wrong, as gsma-labs/evals#73 reports and the pinned scorer confirms. 3GPP TSG's RAN_AH1 line cites gsma-labs/evals#81, which reports it. TeleMath's line now says how far the 0.01 tolerance reaches: it is looser than the scorer's 1% for every answer between -1 and 1 (44 Lite, 245 Full). 0 passes on 10 Lite and 47 Full questions, not 12 and 51: 2 Lite and 4 Full answers are 0, where 0 is right.
…ench # Conflicts: # app/static/catalogue/marks.svg # app/tests/test_catalog.py # docs/agents/supported/README.md # docs/agents/supported/codex.md # docs/agents/supported/goose.md # docs/agents/supported/jiuwenswarm.md # docs/agents/supported/mini-swe-agent.md # docs/agents/supported/opencode.md # docs/agents/supported/terminus-2.md
…Telco
A control proven within a bound short of the full mark showed the same
green tick as a full pass, and its hover said it must score everything:
Open Telco's oracle on 3GPP TSG (94/100 and 1871/2000, at least 93%
needed), LongMemEval's oracle and nop, OSWorld2's nop. Its tick now
takes the warning colour, its hover says what it reached and the bound
it met ("93.55%, at least 93% needed"), and the control's hover says
what it must score. prove.status decides both, once, for the builder
and the benchmark page.
Open Telco's page:
- The rate-limit line was wrong and is gone. A stand-in provider that
refused every call with 429 showed an Original question ending as
RateLimitError, as Nullius labels a time run out under refused calls,
not as a timeout.
- Answer match's two values say what TeleLogs' scorer compares without
LaTeX markup.
- The temperature option carries the example main now requires of a
pattern.
This comment has been minimized.
This comment has been minimized.
Collaborator
Author
|
/judge |
This comment has been minimized.
This comment has been minimized.
Collaborator
Author
|
/judge |
This comment has been minimized.
This comment has been minimized.
🟡 PR judge · 7 of 8 yes · #226 ↔ #225
Acceptance criteria, 7 of 7 met
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #225
What
GSMA's Open Telco becomes a bundled benchmark,
open-telco, on the Nullius standard. It has eight telecom tests (TeleQnA, TeleTables, TeleMath, TeleLogs, 3GPP TSG, ORANBench, srsRANBench, 6G-Bench), each as a Lite dataset (GSMA's sample, the one its leaderboard ranks: 1,850 questions in all) and a Full one (20,588). The driver runs GSMA's own Inspect AI tasks,gsma-labs/evalsv1.1.0 (708f1c5), unchanged, one question per unit. The Original calls the run's model through the relay. An agent that provesagent.task/v1(Codex, Goose, JiuwenSwarm, OpenCode, the reference agent) gets GSMA's exact prompt. Its last message goes back to the task as the model's reply, and the task's own solver and scorer judge it in Inspect's eval loop. The data areGSMA/ot-liteandGSMA/ot-fullat pinned revisions, each file checked by its sha256.data-source.jsonmay now name a repository per dataset, so the page names each dataset's source. The options are temperature, subject, difficulty, task family and answer match (TeleLogs' soft or hard). All 16 datasets are proven: nop 0, and oracle 100% except on 3GPP TSG, 94/100 and 1871/2000, because its answer pattern cannot read RAN_AH1. The standard takes one bound per control for a whole benchmark, so the oracle's is at least 93%.The page says three upstream defects and leaves them as published:
gsma-labs/evals#81).'ANSWER: $LETTER', and a reply that keeps the$is scored wrong (gsma-labs/evals#73).Two Nullius limitations: TeleMath's system prompt reaches an agent inside its one prompt, since ACP has no system message, and TeleLogs' maj@k is not computed.
Proofs short of the full mark. A control proven within a bound short of the full mark had the same green tick as a full pass, and its hover said it must score everything. Now:
prove.statusdecides this once, for both the builder's Dataset step and the benchmark page. Besides Open Telco's two 3GPP TSG oracles, it changes LongMemEval's ten ticks (oracle at least 92%, nop at most 7%) and OSWorld2's nop on All tasks (at most 1%).Touches
catalogue (
benchmarks/open-telco/:benchmark.json,standard.json,data-source.json,upstream-settings.json,driver.py;catalogue/sources.py) · the standard (prove.py,recipe.py: what a proof reached and its bound) · templates and static (catalogue/benchmark.html,builder/command-builder.js,style.css,catalogue/marks.svg) · tests (test_open_telco_driver.pynew,test_catalogue_sources.py,test_standard_ui.py,test_catalog.py,ui_tests/test_catalogue_routes.py) · docs (generated:docs/benchmarks/supported/,docs/agents/supported/)Before and after
A scratch profile, light theme, 1280 px.
Benchmarks page: Open Telco is listed, with GSMA Labs' mark.
main)LongMemEval's proofs: every control is within its bound but short of the full mark, so its ticks turn amber.
main)New: Open Telco's page, its header and setup.
New: Its proofs. The two 3GPP TSG oracles, 94/100 and 1871/2000, are amber.
New: The builder's Dataset step, with the same ticks on its 16 cards.
New: Good to know and the limitations.
Verified
main): 3,015 backend tests and 656 route tests pass. Two tests failed once while the machine was loaded, a run-credentials route test and a kit isolation test. Both passed 3 times out of 3 alone and on full reruns, and this branch touches neither.maingained feat(standard): reach the desktop through acting calls alone once the task is given #224 after that, so I merged it again (f845efd). The merge had no conflicts, and regenerating the docs changed nothing. The affected tests pass locally (74 backend, 32 route), and CI's backend and route suites pass on f845efd.catalogue_docs.py), andcli checkon the definition is ok.cli proveon all 16 datasets, 22,438 questions. Results: nop 0, oracle 100% except 3GPP TSG at 94/100 and 1871/2000, units stable, no leak. The proofs ran before three driver edits that change no behaviour: the Original no longer caches a model client per question, the main loop moved under__main__, and a file is now closed. The hand runs and the unit tests ran on the final driver.$LETTERscoring were checked against the pinned scorer and data.Risk
prove.pyis in the standard, but onlystatus()changes: each row gains what the control must score and how far short of the full mark its proof is.recipe.pypasses both on, andcommand-builder.jsandbenchmark.htmlonly draw them. Nothing else changes: how a control is proven, its bounds, its verdict and its records. A mistake there would mislabel a tick on the benchmark pages and the Dataset step (amber for a full pass, or green for a short one). It would never change a proof, a run or a score. Tests cover both cases and the default bounds.openai==2.26.0in the benchmark's own environment. It fetches GSMA's data at pinned revisions, checked by sha256. A fault in its driver affects only Open Telco runs. If GSMA moves or removes a revision, a run fails at setup; it does not score differently.sources.py: a dataset may name its own repository. Entries that do not are unchanged, and tests cover both.