Skip to content

feat(standard): reach the desktop through acting calls alone once the task is given - #224

Merged
aitianshi merged 5 commits into
ThinkFlowLab:mainfrom
pierreleca:feat/computer-port-strict
Nov 19, 2026
Merged

aitianshi merged 5 commits into
ThinkFlowLab:mainfrom
pierreleca:feat/computer-port-strict

Conversation

@pierreleca

Copy link
Copy Markdown
Contributor

Closes #223

What

The computer port keeps its name, agent.computer/v1, and gets stricter rules. With them a benchmark's count of steps holds for any agent. Once the agent has its task, only acting calls reach the desktop: python and read are removed, screenshot returns the last step's observation, and wait looks again at the cost of a step. Before the task, a window lets the agent upload files and prepare the desktop at no step; an agent's own driver starts there. A step returns the files of the agent's folder that it names. The kit gains a ninth check, late, and its version goes to 2. OSWorld's agent side serves the window and learns that the task was handed over through a new driver message, task_given.

Touches

backend (the standard: the port's server, the hand-over of the task in a run) · catalogue (the kit, OSWorld's runner and driver, the proof in the codex entry) · docs and site (the port's page, the generated pages).

Verified

  • The suites, each module in its own process: 2,982 backend and 649 route tests. The three modules that fail also fail on main in the sandbox used here.
  • prove on the kit: the null control 0 of 9, the reference 9 of 9, a control that clicks without looking fails late.
  • codex 2.2.2 with gpt-6-luna passes the kit's nine checks in 27 runs of 27, with no code of its own; its proof is in its catalogue entry.
  • An agent outside the catalogue that brings its own driver ran 18 OSWorld units with a real model: in each the port's steps equal the steps of the upstream's own trace, and after the task only acting calls reached the desktop.
  • Not verified: OSWorld2.

Risk

A proof taken on the first kit still reads as "older" and still pairs, and an agent written for the first rules would then find python missing. No such proof was released, and the catalogue's own was taken again. Undone by a revert: no stored data changes and nothing migrates.

… once the task is given

The computer port serves one set of rules. Once the agent is given its
task, everything it learns of the desktop comes in an acting call's
answer: screenshot returns the observation the last step fixed and takes
no new picture, and a step names the files of the agent's folder it
wants back, read once its actions and pauses have run. The python and
read tools are gone, with python_seconds.

Before the task, in a window from the session's start, upload writes
inside the agent's folder and prepare runs actions there and reads files
back; neither is a step, and an acting call is refused until the window
closes. A benchmark's driver that sets driver.task_given is sent
task_given before the agent's first prompt or command, and the task
waits for its answer: the driver closes the window there
(Bridge.close_window, or the lifeline's line task-given). A failure ends
the unit before the task is sent.

The bridge takes the agent's folder, made empty at the loop's first
predict, the window and the file of the prepare record. A desktop that
keeps the loop's own observation gives it by observation(); otherwise
the bridge takes a picture once the step's last pause has passed.
OSWorld1's runner and the kit's driver build the bridge the new way,
with no window yet. The port's page describes the window, the
observation and the known limits.
…a ninth check, late

The kit's driver sets driver.task_given: the unit's server opens the
window before the task, closed by task_given or, for the reference, as
it starts, and every prepare call is recorded in the trial's
verifier/computer-prepare.jsonl. Input sent by prepare fails a check, as
input outside an acting call does.

The ninth check, late, is a skill: one labelled button appears 3 s after
the task is given, away from the screen's centre at a place drawn anew
for each unit, and the agent clicks it. It passes when the click is the
click check's and a picture was taken at a step's end after the button
appeared and before the action holding the press began. A script that
clicks the centre without looking is a control that must fail it, on the
fake desktop and in the docker tier. The fixture's click screen takes
the point to draw around as at.

Its kit_version is 2, by the rule on a kit's version: the kit tests
something new. The records of its controls, taken on the eight checks,
are removed; none was taken on the kit as it is now.
…d the loop's own observation

OSWorld1 sets driver.task_given. Its driver's task_given tells the unit's
runner on its lifeline; the runner closes the port's window before the
task and says so in the socket's folder, and the driver answers then,
within 30 s. The runner records each prepare call, and score copies that
record to the trial's verifier/osworld/ beside the call log.

The runner hands run_single_example the DesktopEnv through a view that
keeps the screenshot of each env.step's observation: a step's
observation is the one the loop hands its agent for the next decision,
with no capture added. The agent's folder in the VM is /tmp/agent. A
step's files are read with no retry: a missing file is null at once.
The benchmark's page says what a step's observation is.
…of nine checks

Codex 2.2.2 passed every check of the computer kit at kit_version 2 in
each of three runs: the four basic checks and the five skills, late
included. The record replaces the one taken on the eight checks. The
generated pages do not change: they already gave Codex 2.2.2 the mark
for computer use and for OSWorld1.
The kit's own proof, as prove wrote it on a machine with Docker: the
nine units' ids, the same twice and unique; no reference found in the
unit's prompt (9 of 9); the null control passing 0 of 9 checks and the
reference 9 of 9. It takes the place of the record made on the eight
checks. No generated page shows it.
@aitianshi
aitianshi merged commit 109aa27 into ThinkFlowLab:main Nov 19, 2026
2 checks passed
aitianshi added a commit that referenced this pull request Nov 19, 2026
Closes #225

## What

GSMA's Open Telco becomes a bundled benchmark, `open-telco`, on the
Nullius standard. It has eight telecom tests (TeleQnA, TeleTables,
TeleMath, TeleLogs, 3GPP TSG, ORANBench, srsRANBench, 6G-Bench), each as
a Lite dataset (GSMA's sample, the one its leaderboard ranks: 1,850
questions in all) and a Full one (20,588). The driver runs GSMA's own
Inspect AI tasks, `gsma-labs/evals` v1.1.0 (708f1c5), unchanged, one
question per unit. The Original calls the run's model through the relay.
An agent that proves `agent.task/v1` (Codex, Goose, JiuwenSwarm,
OpenCode, the reference agent) gets GSMA's exact prompt. Its last
message goes back to the task as the model's reply, and the task's own
solver and scorer judge it in Inspect's eval loop. The data are
`GSMA/ot-lite` and `GSMA/ot-full` at pinned revisions, each file checked
by its sha256. `data-source.json` may now name a repository per dataset,
so the page names each dataset's source. The options are temperature,
subject, difficulty, task family and answer match (TeleLogs' soft or
hard). All 16 datasets are proven: nop 0, and oracle 100% except on 3GPP
TSG, 94/100 and 1871/2000, because its answer pattern cannot read
RAN_AH1. The standard takes one bound per control for a whole benchmark,
so the oracle's is at least 93%.

The page says three upstream defects and leaves them as published:
- 3GPP TSG cannot read RAN_AH1 (`gsma-labs/evals#81`).
- TeleMath's 0.01 absolute tolerance passes 0 on 10 Lite and 47 Full
questions whose answer is not 0. This one is not reported upstream.
- The multiple-choice prompt asks for `'ANSWER: $LETTER'`, and a reply
that keeps the `$` is scored wrong (`gsma-labs/evals#73`).

Two Nullius limitations: TeleMath's system prompt reaches an agent
inside its one prompt, since ACP has no system message, and TeleLogs'
maj@k is not computed.

**Proofs short of the full mark.** A control proven within a bound short
of the full mark had the same green tick as a full pass, and its hover
said it must score everything. Now:
- Its tick takes the warning colour.
- Its hover says what it reached and the bound it met ("93.55%, at least
93% needed").
- The control's hover says what it must score ("must score at least
93%").

`prove.status` decides this once, for both the builder's Dataset step
and the benchmark page. Besides Open Telco's two 3GPP TSG oracles, it
changes LongMemEval's ten ticks (oracle at least 92%, nop at most 7%)
and OSWorld2's nop on All tasks (at most 1%).

## Touches

catalogue (`benchmarks/open-telco/`: `benchmark.json`, `standard.json`,
`data-source.json`, `upstream-settings.json`, `driver.py`;
`catalogue/sources.py`) · the standard (`prove.py`, `recipe.py`: what a
proof reached and its bound) · templates and static
(`catalogue/benchmark.html`, `builder/command-builder.js`, `style.css`,
`catalogue/marks.svg`) · tests (`test_open_telco_driver.py` new,
`test_catalogue_sources.py`, `test_standard_ui.py`, `test_catalog.py`,
`ui_tests/test_catalogue_routes.py`) · docs (generated:
`docs/benchmarks/supported/`, `docs/agents/supported/`)

## Before and after

A scratch profile, light theme, 1280 px.

**Benchmarks page:** Open Telco is listed, with GSMA Labs' mark.

| Before (`main`) | After |
| --- | --- |
| <img width="420" alt="Benchmarks page before"
src="https://github.com/user-attachments/assets/811859bf-6195-44c3-a151-71206a3e3293"
/> | <img width="420" alt="Benchmarks page after"
src="https://github.com/user-attachments/assets/45fcad79-e640-4405-b178-205d5286b5f5"
/> |

**LongMemEval's proofs:** every control is within its bound but short of
the full mark, so its ticks turn amber.

| Before (`main`) | After |
| --- | --- |
| <img width="420" alt="LongMemEval proofs before"
src="https://github.com/user-attachments/assets/8a96ffd0-be4e-48a0-ace7-ecccb6cd5523"
/> | <img width="420" alt="LongMemEval proofs after"
src="https://github.com/user-attachments/assets/4993f06b-49dc-4235-9968-d68153a73290"
/> |

**New:** Open Telco's page, its header and setup.

<img width="720" alt="Open Telco's page header"
src="https://github.com/user-attachments/assets/2d9bfc19-07bc-4ff6-a0a0-31031c57eb31"
/>

**New:** Its proofs. The two 3GPP TSG oracles, 94/100 and 1871/2000, are
amber.

<img width="520" alt="Open Telco's proofs"
src="https://github.com/user-attachments/assets/c7f5d203-85b7-46cc-9f76-3824218b0d5a"
/>

**New:** The builder's Dataset step, with the same ticks on its 16
cards.

<img width="640" alt="Open Telco's Dataset step"
src="https://github.com/user-attachments/assets/1da5a5a0-cfb3-447a-a690-1a829efb9a23"
/>

**New:** Good to know and the limitations.

<img width="520" alt="Open Telco's Good to know and limitations"
src="https://github.com/user-attachments/assets/488b97e2-00c8-46d9-9a33-604469d05f74"
/>

## Verified

- Suites, at the branch head (911ab51, after merging `main`): 3,015
backend tests and 656 route tests pass. Two tests failed once while the
machine was loaded, a run-credentials route test and a kit isolation
test. Both passed 3 times out of 3 alone and on full reruns, and this
branch touches neither.
- `main` gained #224 after that, so I merged it again (f845efd). The
merge had no conflicts, and regenerating the docs changed nothing. The
affected tests pass locally (74 backend, 32 route), and CI's backend and
route suites pass on f845efd.
- The generated docs are current (`catalogue_docs.py`), and `cli check`
on the definition is ok.
- Proofs: `cli prove` on all 16 datasets, 22,438 questions. Results: nop
0, oracle 100% except 3GPP TSG at 94/100 and 1871/2000, units stable, no
leak. The proofs ran before three driver edits that change no behaviour:
the Original no longer caches a model client per question, the main loop
moved under `__main__`, and a file is now closed. The hand runs and the
unit tests ran on the final driver.
- Real runs with Muse Spark 1.3 Contributor through OpenRouter:
  - The Original, 2 questions of each Lite test: 11/16.
- The reference agent on the container backend: 3/4. The Original on the
container backend: 1/1.
- Codex 2.2.2 on the container backend: 6/6 (TeleQnA, TeleMath and 3GPP
TSG, 2 Lite questions each), 0.0195 USD.
- JiuwenSwarm 0.2.8, same questions: 5/6, 0.0113 USD. Its miss answered
0 where the answer is 0.033.
- Rate limits: with a stand-in provider that refused every call with
429, an Original question ended as RateLimitError, not as a timeout.
- TeleMath's tolerance counts and the `$LETTER` scoring were checked
against the pinned scorer and data.
- Browser checks, light and dark, at 390 and 1280 px:
  - Pages: Open Telco's card, page and builder steps.
- The amber ticks on Open Telco, LongMemEval and OSWorld2, and their
hovers.
  - No sideways scroll and no page errors.
- Not verified:
  - a whole dataset run by the Original with a real model;
- agents other than Codex, JiuwenSwarm and the reference agent on Open
Telco;
  - a TeleLogs run with answer match set to hard.

## Risk

- `prove.py` is in the standard, but only `status()` changes: each row
gains what the control must score and how far short of the full mark its
proof is. `recipe.py` passes both on, and `command-builder.js` and
`benchmark.html` only draw them. Nothing else changes: how a control is
proven, its bounds, its verdict and its records. A mistake there would
mislabel a tick on the benchmark pages and the Dataset step (amber for a
full pass, or green for a short one). It would never change a proof, a
run or a score. Tests cover both cases and the default bounds.
- Open Telco's oracle bound, at least 93%, holds for all 16 datasets,
because the standard takes one bound per control for a whole benchmark.
On the 14 datasets whose oracle reaches 100%, a regression that left it
between 93% and 100% would still prove; its amber tick would show it.
Bounds per dataset would close this. That is a change to the standard
and is not made here.
- Open Telco is a new entry. It installs GSMA's project from its lock
file at v1.1.0 (with Inspect AI) and `openai==2.26.0` in the benchmark's
own environment. It fetches GSMA's data at pinned revisions, checked by
sha256. A fault in its driver affects only Open Telco runs. If GSMA
moves or removes a revision, a run fails at setup; it does not score
differently.
- `sources.py`: a dataset may name its own repository. Entries that do
not are unchanged, and tests cover both.
- Undo: revert the squash commit. Nothing is migrated, and no other
benchmark's stored runs or proof records change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: on agent.computer/v1 an agent can reach the desktop outside its steps

2 participants