Repository navigation
feat(standard): reach the desktop through acting calls alone once the task is given - #224
Merged
aitianshi merged 5 commits intoNov 19, 2026
Merged
Conversation
… once the task is given The computer port serves one set of rules. Once the agent is given its task, everything it learns of the desktop comes in an acting call's answer: screenshot returns the observation the last step fixed and takes no new picture, and a step names the files of the agent's folder it wants back, read once its actions and pauses have run. The python and read tools are gone, with python_seconds. Before the task, in a window from the session's start, upload writes inside the agent's folder and prepare runs actions there and reads files back; neither is a step, and an acting call is refused until the window closes. A benchmark's driver that sets driver.task_given is sent task_given before the agent's first prompt or command, and the task waits for its answer: the driver closes the window there (Bridge.close_window, or the lifeline's line task-given). A failure ends the unit before the task is sent. The bridge takes the agent's folder, made empty at the loop's first predict, the window and the file of the prepare record. A desktop that keeps the loop's own observation gives it by observation(); otherwise the bridge takes a picture once the step's last pause has passed. OSWorld1's runner and the kit's driver build the bridge the new way, with no window yet. The port's page describes the window, the observation and the known limits.
…a ninth check, late The kit's driver sets driver.task_given: the unit's server opens the window before the task, closed by task_given or, for the reference, as it starts, and every prepare call is recorded in the trial's verifier/computer-prepare.jsonl. Input sent by prepare fails a check, as input outside an acting call does. The ninth check, late, is a skill: one labelled button appears 3 s after the task is given, away from the screen's centre at a place drawn anew for each unit, and the agent clicks it. It passes when the click is the click check's and a picture was taken at a step's end after the button appeared and before the action holding the press began. A script that clicks the centre without looking is a control that must fail it, on the fake desktop and in the docker tier. The fixture's click screen takes the point to draw around as at. Its kit_version is 2, by the rule on a kit's version: the kit tests something new. The records of its controls, taken on the eight checks, are removed; none was taken on the kit as it is now.
…d the loop's own observation OSWorld1 sets driver.task_given. Its driver's task_given tells the unit's runner on its lifeline; the runner closes the port's window before the task and says so in the socket's folder, and the driver answers then, within 30 s. The runner records each prepare call, and score copies that record to the trial's verifier/osworld/ beside the call log. The runner hands run_single_example the DesktopEnv through a view that keeps the screenshot of each env.step's observation: a step's observation is the one the loop hands its agent for the next decision, with no capture added. The agent's folder in the VM is /tmp/agent. A step's files are read with no retry: a missing file is null at once. The benchmark's page says what a step's observation is.
…of nine checks Codex 2.2.2 passed every check of the computer kit at kit_version 2 in each of three runs: the four basic checks and the five skills, late included. The record replaces the one taken on the eight checks. The generated pages do not change: they already gave Codex 2.2.2 the mark for computer use and for OSWorld1.
The kit's own proof, as prove wrote it on a machine with Docker: the nine units' ids, the same twice and unique; no reference found in the unit's prompt (9 of 9); the null control passing 0 of 9 checks and the reference 9 of 9. It takes the place of the record made on the eight checks. No generated page shows it.
aitianshi
approved these changes
Nov 19, 2026
aitianshi
added a commit
that referenced
this pull request
Nov 19, 2026
Closes #225 ## What GSMA's Open Telco becomes a bundled benchmark, `open-telco`, on the Nullius standard. It has eight telecom tests (TeleQnA, TeleTables, TeleMath, TeleLogs, 3GPP TSG, ORANBench, srsRANBench, 6G-Bench), each as a Lite dataset (GSMA's sample, the one its leaderboard ranks: 1,850 questions in all) and a Full one (20,588). The driver runs GSMA's own Inspect AI tasks, `gsma-labs/evals` v1.1.0 (708f1c5), unchanged, one question per unit. The Original calls the run's model through the relay. An agent that proves `agent.task/v1` (Codex, Goose, JiuwenSwarm, OpenCode, the reference agent) gets GSMA's exact prompt. Its last message goes back to the task as the model's reply, and the task's own solver and scorer judge it in Inspect's eval loop. The data are `GSMA/ot-lite` and `GSMA/ot-full` at pinned revisions, each file checked by its sha256. `data-source.json` may now name a repository per dataset, so the page names each dataset's source. The options are temperature, subject, difficulty, task family and answer match (TeleLogs' soft or hard). All 16 datasets are proven: nop 0, and oracle 100% except on 3GPP TSG, 94/100 and 1871/2000, because its answer pattern cannot read RAN_AH1. The standard takes one bound per control for a whole benchmark, so the oracle's is at least 93%. The page says three upstream defects and leaves them as published: - 3GPP TSG cannot read RAN_AH1 (`gsma-labs/evals#81`). - TeleMath's 0.01 absolute tolerance passes 0 on 10 Lite and 47 Full questions whose answer is not 0. This one is not reported upstream. - The multiple-choice prompt asks for `'ANSWER: $LETTER'`, and a reply that keeps the `$` is scored wrong (`gsma-labs/evals#73`). Two Nullius limitations: TeleMath's system prompt reaches an agent inside its one prompt, since ACP has no system message, and TeleLogs' maj@k is not computed. **Proofs short of the full mark.** A control proven within a bound short of the full mark had the same green tick as a full pass, and its hover said it must score everything. Now: - Its tick takes the warning colour. - Its hover says what it reached and the bound it met ("93.55%, at least 93% needed"). - The control's hover says what it must score ("must score at least 93%"). `prove.status` decides this once, for both the builder's Dataset step and the benchmark page. Besides Open Telco's two 3GPP TSG oracles, it changes LongMemEval's ten ticks (oracle at least 92%, nop at most 7%) and OSWorld2's nop on All tasks (at most 1%). ## Touches catalogue (`benchmarks/open-telco/`: `benchmark.json`, `standard.json`, `data-source.json`, `upstream-settings.json`, `driver.py`; `catalogue/sources.py`) · the standard (`prove.py`, `recipe.py`: what a proof reached and its bound) · templates and static (`catalogue/benchmark.html`, `builder/command-builder.js`, `style.css`, `catalogue/marks.svg`) · tests (`test_open_telco_driver.py` new, `test_catalogue_sources.py`, `test_standard_ui.py`, `test_catalog.py`, `ui_tests/test_catalogue_routes.py`) · docs (generated: `docs/benchmarks/supported/`, `docs/agents/supported/`) ## Before and after A scratch profile, light theme, 1280 px. **Benchmarks page:** Open Telco is listed, with GSMA Labs' mark. | Before (`main`) | After | | --- | --- | | <img width="420" alt="Benchmarks page before" src="https://github.com/user-attachments/assets/811859bf-6195-44c3-a151-71206a3e3293" /> | <img width="420" alt="Benchmarks page after" src="https://github.com/user-attachments/assets/45fcad79-e640-4405-b178-205d5286b5f5" /> | **LongMemEval's proofs:** every control is within its bound but short of the full mark, so its ticks turn amber. | Before (`main`) | After | | --- | --- | | <img width="420" alt="LongMemEval proofs before" src="https://github.com/user-attachments/assets/8a96ffd0-be4e-48a0-ace7-ecccb6cd5523" /> | <img width="420" alt="LongMemEval proofs after" src="https://github.com/user-attachments/assets/4993f06b-49dc-4235-9968-d68153a73290" /> | **New:** Open Telco's page, its header and setup. <img width="720" alt="Open Telco's page header" src="https://github.com/user-attachments/assets/2d9bfc19-07bc-4ff6-a0a0-31031c57eb31" /> **New:** Its proofs. The two 3GPP TSG oracles, 94/100 and 1871/2000, are amber. <img width="520" alt="Open Telco's proofs" src="https://github.com/user-attachments/assets/c7f5d203-85b7-46cc-9f76-3824218b0d5a" /> **New:** The builder's Dataset step, with the same ticks on its 16 cards. <img width="640" alt="Open Telco's Dataset step" src="https://github.com/user-attachments/assets/1da5a5a0-cfb3-447a-a690-1a829efb9a23" /> **New:** Good to know and the limitations. <img width="520" alt="Open Telco's Good to know and limitations" src="https://github.com/user-attachments/assets/488b97e2-00c8-46d9-9a33-604469d05f74" /> ## Verified - Suites, at the branch head (911ab51, after merging `main`): 3,015 backend tests and 656 route tests pass. Two tests failed once while the machine was loaded, a run-credentials route test and a kit isolation test. Both passed 3 times out of 3 alone and on full reruns, and this branch touches neither. - `main` gained #224 after that, so I merged it again (f845efd). The merge had no conflicts, and regenerating the docs changed nothing. The affected tests pass locally (74 backend, 32 route), and CI's backend and route suites pass on f845efd. - The generated docs are current (`catalogue_docs.py`), and `cli check` on the definition is ok. - Proofs: `cli prove` on all 16 datasets, 22,438 questions. Results: nop 0, oracle 100% except 3GPP TSG at 94/100 and 1871/2000, units stable, no leak. The proofs ran before three driver edits that change no behaviour: the Original no longer caches a model client per question, the main loop moved under `__main__`, and a file is now closed. The hand runs and the unit tests ran on the final driver. - Real runs with Muse Spark 1.3 Contributor through OpenRouter: - The Original, 2 questions of each Lite test: 11/16. - The reference agent on the container backend: 3/4. The Original on the container backend: 1/1. - Codex 2.2.2 on the container backend: 6/6 (TeleQnA, TeleMath and 3GPP TSG, 2 Lite questions each), 0.0195 USD. - JiuwenSwarm 0.2.8, same questions: 5/6, 0.0113 USD. Its miss answered 0 where the answer is 0.033. - Rate limits: with a stand-in provider that refused every call with 429, an Original question ended as RateLimitError, not as a timeout. - TeleMath's tolerance counts and the `$LETTER` scoring were checked against the pinned scorer and data. - Browser checks, light and dark, at 390 and 1280 px: - Pages: Open Telco's card, page and builder steps. - The amber ticks on Open Telco, LongMemEval and OSWorld2, and their hovers. - No sideways scroll and no page errors. - Not verified: - a whole dataset run by the Original with a real model; - agents other than Codex, JiuwenSwarm and the reference agent on Open Telco; - a TeleLogs run with answer match set to hard. ## Risk - `prove.py` is in the standard, but only `status()` changes: each row gains what the control must score and how far short of the full mark its proof is. `recipe.py` passes both on, and `command-builder.js` and `benchmark.html` only draw them. Nothing else changes: how a control is proven, its bounds, its verdict and its records. A mistake there would mislabel a tick on the benchmark pages and the Dataset step (amber for a full pass, or green for a short one). It would never change a proof, a run or a score. Tests cover both cases and the default bounds. - Open Telco's oracle bound, at least 93%, holds for all 16 datasets, because the standard takes one bound per control for a whole benchmark. On the 14 datasets whose oracle reaches 100%, a regression that left it between 93% and 100% would still prove; its amber tick would show it. Bounds per dataset would close this. That is a change to the standard and is not made here. - Open Telco is a new entry. It installs GSMA's project from its lock file at v1.1.0 (with Inspect AI) and `openai==2.26.0` in the benchmark's own environment. It fetches GSMA's data at pinned revisions, checked by sha256. A fault in its driver affects only Open Telco runs. If GSMA moves or removes a revision, a run fails at setup; it does not score differently. - `sources.py`: a dataset may name its own repository. Entries that do not are unchanged, and tests cover both. - Undo: revert the squash commit. Nothing is migrated, and no other benchmark's stored runs or proof records change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #223
What
The computer port keeps its name,
agent.computer/v1, and gets stricter rules. With them a benchmark's count of steps holds for any agent. Once the agent has its task, only acting calls reach the desktop:pythonandreadare removed,screenshotreturns the last step's observation, andwaitlooks again at the cost of a step. Before the task, a window lets the agentuploadfiles andpreparethe desktop at no step; an agent's own driver starts there. Astepreturns the files of the agent's folder that it names. The kit gains a ninth check,late, and its version goes to 2. OSWorld's agent side serves the window and learns that the task was handed over through a new driver message,task_given.Touches
backend (the standard: the port's server, the hand-over of the task in a run) · catalogue (the kit, OSWorld's runner and driver, the proof in the
codexentry) · docs and site (the port's page, the generated pages).Verified
mainin the sandbox used here.proveon the kit: the null control 0 of 9, the reference 9 of 9, a control that clicks without looking failslate.codex2.2.2 withgpt-6-lunapasses the kit's nine checks in 27 runs of 27, with no code of its own; its proof is in its catalogue entry.Risk
A proof taken on the first kit still reads as "older" and still pairs, and an agent written for the first rules would then find
pythonmissing. No such proof was released, and the catalogue's own was taken again. Undone by a revert: no stored data changes and nothing migrates.