Skip to content

feat(standard): run a desktop benchmark with an agent of the Agents page, through a computer port and its kit - #188

Merged
aitianshi merged 96 commits into
ThinkFlowLab:mainfrom
pierreleca:feat/agent-computer
Nov 19, 2026
Merged

aitianshi merged 96 commits into
ThinkFlowLab:mainfrom
pierreleca:feat/agent-computer

Conversation

@pierreleca

@pierreleca pierreleca commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #187

What

An agent of the Agents page could not run OSWorld: the entry asked for agent.computer/v1, a port with no page, no server and no kit. This adds the three.

  • The port: the benchmark's runner serves the desktop to the agent as an MCP server computer of fifteen tools. Five read (info, screenshot, read, python, upload), eight act (click, move, drag, scroll, type, key, wait, step) and two end the task (done, fail). One acting call is one step, counted by the benchmark's own loop. A step may also end the task with end (done or fail): a decision that acts and then stops is one step. The port's records (calls, desktop events and targets, OSWorld's result and runner log) are listed on a trial's Artifacts tab.
  • The kit: a small desktop in a container and eight checks: four basic (see, click, type, limit) and four skills (drag, scroll, recover, batch), with a null control and a reference.
  • OSWorld's agent side: an agent that holds the port runs OSWorld's tasks through the upstream's own loop and evaluator, unmodified, with the options Steps and Pause. Steps is the entry's one max_steps, for OSWorld's own scripts and for an agent through the port, where default is 100.
  • The core: what a driver needs for this (driver.participant, driver.unit_end, limits.setup_seconds, an option for agents alone), and conform going on past a kit that cannot run on the machine.

Read first: the port's page, then computer_port.py (the server and its bridge to a loop), the kit's checks.py and driver.py, then OSWorld's runner.py and the new part of its driver.py. main is merged in, with the fixes the merge needed.

Two limits: OSWorld's agent side ran on the local backend alone, and the port bounds an agent's steps, not its model calls. A run that would put the agent in a VM of its own is refused with the port.

Touches

backend and the standard · templates and static · docs and site · catalogue

Before and after

The builder for OSWorld, with an agent that holds the port chosen: Pause on the Agent step, Steps with the run's limits. Each picture opens at full size.

Before: the Agent step After: the Agent step After: the Limits step
The Agent step on main The Agent step with Pause The Limits step with Steps per task

Verified

  • The kit's own proof (prove), on a machine with Docker: the null control passes 0 of 8 checks and the reference 8 of 8. The kit's tests that need Docker pass there (100 tests).
  • A real agent on the kit: codex 2.2.2 with gpt-6-luna passes the four basic checks and shows the four skills, each in three runs of three (conform --repeat 3, the container backend).
  • OSWorld on the upstream's real VM, through the port, with a scripted agent: every tool answered as the page says, and the steps as the port counts them equal the upstream's own trace. A turn with no done, a fail, a used-up budget and a stop each left no VM, container or process. Two units at a time kept their desktops apart.
  • Unchanged: OSWorld with its own agents, and the null control, give the same records as on main.
  • Tests: the backend and route suites, each module in its own process, pass but for one module that fails the same way on main in the sandbox used here. After the merge of main, prove and the suites were run again; the other lines above date from the first push.
  • Not verified: the container backend for OSWorld's agent side, and OSWorld's tasks run by a model-driven agent of the Agents page.

Risk

A run with no agent of the Agents page takes the paths it took: the new hooks act only for a driver that asks for them (the comparison above). The kit needs Docker; on a machine without it conform reports that kit as refused and goes on. The port's page is a contract: agents built against it would break if it changed. Undo with a revert.

Pass participant kinds and agent setup limits to drivers that request them. Notify opted-in drivers once each agent unit ends, including failed and stopped units.

Apply agent-only benchmark options in runs, recipes and the builder. Keep leak checks on the control setup contract and document the hooks.
Report benchmark-specific refusals per port and continue checking the remaining candidates. Save the proofs collected by the command while preserving existing records for refused kits.

Keep explicit port refusals and command-wide failures fatal. Distinguish benchmark prerequisites and installs from agent installs, and validate provider credentials before checking kits.
…loop

A desktop benchmark's runner serves one MCP server, computer, whose tools
are the desktop's basic operations; the agent is a client alone. The
benchmark's loop runs as it is and is handed a Bridge: each predict gives
it the actions of the agent's next acting call, and the reading calls are
served before it, one desktop operation at a time, in the order they came.
An acting call is answered once its actions and their pause have run.

- computer_port: the Desktop protocol (python, upload, read, capture,
  password), the Bridge (the step budget, the refusal of every call past
  it, the turn's end), the Server over a Unix socket (every connection of
  the session, one call log line per tool call: tool, times, failure),
  files sent once for a run by their sha256, and the runner's ready file,
  lifeline and one stop.
- computer_pipe: the session's stdio pipe to the socket. A read of the
  input is counted under the lock the exit decision takes; the input's
  end is seen by its writer's hang-up while a full queue holds the reader;
  the pipe's end has one deadline from the input's end.
- Tests on a fake desktop with a scripted MCP client, a stand-in loop that
  counts steps as the upstream's does, and the upstream's own
  run_single_example where a checkout of it is at hand.
A page for agent.computer/v1: its fifteen tools, what a step is, the step
budget, the turn's end, uploads by content, the call log, and what a
desktop benchmark's runner gives the shared code. The standard's row of
the port links to it, and the docs folder map names ports/.
…dicts

The computer port's conformance kit needs a small real desktop on which an
agent shows that it sees the screen, points and types, decided by a program.
This adds the desktop and the fixture the agent acts on; the kit's driver,
runner and standard.json come with the port's server.

- desktop.Dockerfile: an environment of the kit on the base image, tagged
  nullius/env-agent-computer-v1-desktop:<hash> as container.env_image tags
  any environment. Xvfb at 1920 x 1080 on :0, Openbox, a session bus for
  uid 1000, Python 3 with Tk, pyautogui and Pillow pinned by version and
  hash (desktop-requirements.txt). Two users: user (uid 1000, no sudo) and
  fixture.
- session.py: the container's command; the display, its cookie, the bus and
  the window manager, then a ready file.
- desktop.py: start a desktop (no network, no new privileges), the five
  operations a benchmark's side gives the port's server (python, upload,
  read, capture, password), one action run as user with its start and end
  read on the desktop's clock by a root process, the fixture's start and log,
  and removal.
- fixture.py: a Tk program that draws codes and labels as bitmaps, with
  Tk's send removed, and logs each press, release, scroll and key with its
  time and what it hit, in a folder only fixture reads.
- checks.py: the values each check draws, and the verdicts of see, click and
  type from the fixture's log, the actions' times and the last message;
  click and type fail on any input outside an action's time.
- pyproject.toml: the kits' Dockerfiles and requirement files go in the
  package.

Tests: the verdicts on hand-written logs, the action's times taken from the
desktop's reply, the image's declaration; the docker tier starts real
desktops and judges a click and a word sent by actions right, and the same
click sent through python wrong.
A run stopped after a unit passed its stop check, and before its setup request was written, still sent that unit's end to the driver. A unit now counts as set up once its setup line is written.

A request cut short by a stop leaves its late reply in the driver's queue, where a forced end would read it as its own. Such a driver is no longer sent an end.

Document that a driver closed after a request got no reply, or one left with an unanswered request, is not sent an end. Test the forced wait after a stop, and replace a test whose mock passed for the wrong reason.
The setup time limit was turned into a float, so the live feed recorded the env_setup phase's timeout as 600.0 for every run, where it had recorded 600. The limit is now kept as given.

Test the env_setup phase's timeout in the live feed for an agent and a control, with and without limits.setup_seconds.
An option whose setups named agent beside another scope passed the check, though the builder never showed it in a setup's tab. One on a capability benchmark passed too, where no agent ever runs it.

The check now accepts agent only on its own, and only on a benchmark that agents run.
Conform without a port named went on past a kit refused for what its benchmark needs of the machine, and then exited 0 as if every port had been proven. When every kit was refused it still saved the definition and exited 0.

A refused kit is now returned as refused beside the proofs gained, which are written, and the command exits 1. When every kit is refused, nothing is written and the check is refused, with exit status 2.

The key is checked before the first kit only when some port has a kit: a command that names only ports without one says so and needs no key.

Document the refusals of conform and its exit statuses.
…ny client

- Each connection's answers are written by a thread of its own, at most
  OUT_BYTES waiting: a client that stops reading holds up neither the
  loop nor another connection, and a close never waits for a write.
  A connection accepted as the server closes is closed, and a closed one
  leaves the server's list.
- One table gives what tools/list says and what a call may hold. A call
  is checked in full, an upload's base64 decoded, before it is queued;
  the JSON-RPC envelope is checked before any tool, a frame's length
  with its newline, and a number past any integer is refused and
  recorded. An answer over the bound is recorded as failed.
- The turn's end stops the actions of a step not yet run and refuses
  that call; a done or fail taken before it still reaches the loop.
- Uploads: an append goes through a name of the server's own, removed
  on every path; a file taken by its sha256 starts the file, and append
  is refused with it; the chunks being kept are the unit's own, dropped
  when the file starts again and at the unit's end, up to KEEP_BYTES; a
  kept file is written through an exclusive temporary file and read only
  when it is no link.
- The last resort gives its own callback LAST_SECONDS before it exits.
- The budget test holds the first action behind an event in place of a
  250 ms wait.
Once the server's side ends, the pipe reads its input on to its end and
sends it, by its one deadline; it exits 0 only when it saw that end and
sent everything, and 1 otherwise. The queue test waits for the first
send and measures bytes, assuming no read's size, and runs again with
reads of 4 KiB.
…nd bounds

The page says that a call is checked against its listed schema before it
waits, what the turn's end does to a step not yet run and to a done or
fail taken before it, that a session may connect again, the bounds on
a connection's waiting answers and on a kept file, what an error is, and
that done and fail answer before the loop runs them.
… cookie and no key repeat

The desktop's X server ran as root with access control off, and a key held
from one action to the next repeated between them, which the verdicts count
as input outside every action.

- The server runs as `display`, an account of its own, with access control
  on: root, user and fixture connect with the cookie the session writes.
- The core key repeat is turned off, and the server's repeat delay is the
  longest XKB allows, which also covers the input-test device.
- The session ends, and the container with it, when the display, the session
  bus or the window manager ends, not the display alone.
- The image adds the accessibility bus (at-spi2-core), which the session bus
  starts on demand for programs that look for it.
…il loudly

A pass must come from input sent inside actions, and a correct agent must not
be failed by the desktop's own timing.

- An action answers only once the verdicts' slack has passed on the desktop,
  so a request sent after it cannot fall inside its time.
- The action's times and the fixture's events are read on one monotonic
  clock: a step of the wall clock no longer fails a correct action.
- click fails on a press of any mouse button on another button; Enter in the
  field is input whose time is checked, as keys are.
- The fixture says ready only when its window is mapped at the screen's size
  and holds the focus within its deadline.
- read gives a regular file alone, at most 12 MiB, and refuses the rest as
  the agent's error.
- A start that fails or is interrupted removes the container it made, its log
  read or not; remove checks the engine's answer and that the container is
  gone; a fixture log that cannot be read, or holds a line that is no event,
  is an error.
- The docker tier's desktop tests run wherever an engine answers and build
  the image first; the removal test's probe is bounded and must succeed.
…hers

When conform passed over a refused kit and wrote the other kits' proofs, it dropped every record of a known port that was not in the offers. A port proven earlier whose offer had been left out lost its proof although its kit never ran.

The record of a port whose kit was refused in the same command now stays as it was.
… ends

The driver.unit_end row said that a driver closed after a request got no reply has its process group ended with it. A child that redirects its streams can outlive that. The row keeps saying that such a driver is not sent an end.
…red-request test

The test stopped the run from a timer started before setup was sent; on a slow machine the stop could come before the line was written and fail the test for the wrong reason. The stop now comes from the request's on_sent call, and the test waits for the driver to read all it was sent instead of sleeping.
For a unit an agent runs (driver.participant), setup starts runner.py
behind the start gate. The runner builds OSWorld's DesktopEnv as the null
control does, serves the computer port's MCP server over a Desktop of
five methods on the upstream's controllers, and calls the upstream's own
run_single_example, unmodified, with the port's bridge as the loop's
agent. Setup answers once the loop asks for the agent's first step, with
the prompt and the session's server computer: the port's pipe, whose path
the driver takes from the package it imports, joined to the runner's
socket. Score sends the turn's end on the runner's lifeline, waits for
the runner within its bound and reads the upstream's result.txt; end
(driver.unit_end) reaps the runner.

- The bounds follow from the unit's limits (limits.setup_seconds,
  score's and end's); the runner's clean-up, now the bridge, the server
  and the desktop's bounded close, fits inside the driver's reap. A reap
  the driver's end cuts is handed to a reaper of its own.
- The options Steps (100) and Pause (3.0 s) apply to an agent's run
  alone; driver.command names runner.py for a host's copy of the entry.
- A unit with no agent runs as before.
…erywhere

- The last resort's word goes to stderr from a thread it waits for 0.1 s
  at most: an exit never waits on a pipe nobody reads.
- Every outgoing frame is bounded, the answers of any method and the
  errors standing in for them, with no id when even that is over.
- NaN, Infinity and -Infinity are no JSON. Each tool's schema lists
  every argument it takes (additionalProperties false), and any other
  argument is refused.
- The listed patterns use explicit classes, which Python and JSON
  Schema's regular expressions read alike; a final $ is the text's end.
- upload's data is base64 by its pattern, and its listed bound is the
  check's: at the longest length the padding says the chunk is not over.
- A failed append removes its part as far as the desktop answers, and a
  kept file's temporary file and descriptor go on every path.
- The tests close the wires they make, and the queue test's helper ends
  the pipe before it closes an input a feeder may still write.
The computer port had a server and the kit a desktop; this joins them so
that an agent can prove agent.computer/v1 like any other port.

- The kit agent-computer-v1: eight checks, four basic (see, click, type,
  limit) and four skills (drag, scroll, recover, batch), each decided by
  the fixture's log, the actions' times and the server's call log. Every
  check fails on input that came outside an acting call.
- driver.py: each unit's desktop and fixture, the port's server over it in
  the driver's process, and a loop of the kit's own that runs each step's
  actions and waits the pause. score ends the turn, judges and removes the
  desktop and the socket. A unit that never reaches score is let go by the
  next setup or the driver's end; reaper.py, started in a session of its
  own, removes what a killed driver left.
- The reference makes the right calls through the unit's own server; the
  null control passes nothing. The kit's proof is recorded in its
  standard.json.
- ports.py names the kit; the port's page gets its section, the ports
  table its row, and the generated agent pages the port's column.
- fake_computer_agent.py: a scripted ACP agent that looks and clicks once,
  for the tests and a scripted conformance.
A runner's snapshot for a remote host takes a benchmark's server command
only from files of its own folder, and refused the kit's path to the port's
pipe two folders up. pipe.py in the kit's folder runs that pipe.

The kit's proof is recorded again on this tree, and the recipe test now
expects the computer kit among the kits an ACP agent takes.
… own last 429, not a side call's or a context window's stop

The rule reads the agent's own last call, a guard's line in the stop file and a score stopped by other than the context window; a cap's refusal is still the cap's.
…h --served-by replaces, as ThinkFlowLab#124 has the user choose

A model the definition pins (setups.original.pin, score.judge.pin) goes to the
user's providers when --served-by gives it others: the run says so, records the
definition's under standard.pins_replaced, and the Original's run does not
compare with the paper. The recipe's compile gives each field's default
(served_pins) and sends a field's list, its own or the default, with every job
that calls its model; the builder's Served by starts with it, and Review says
when a field's differ. --served-by without MODEL= names the model an Original
sends, and prove --job refuses a control whose pins were replaced.
…r place, the recipe keeping the value; OSWorld1's agents by their upstream's names
…e the work runs on Where, a single value as fixed, and what the datasets need on Dataset; OSWorld1's VM provider on Where
…eclares, with their route errors; Served by is the user's alone, as ThinkFlowLab#124 has it
… upstream's own runners, mock websites, GitLab and proxy, proven on test_v2, and how to prepare a machine for it

OSWorld2 runs each of the 108 tasks of test_v2 in a VM of its own through the upstream's docker provider, with the mock websites its tasks need served on the machine, and GitLab and a proxy started for each unit that needs one. Its Original runs the upstream's agents with their models pinned to their providers on OpenRouter, and its null control is proven on test_v2 at 0 of 108. The tasks are gated, so a failing endpoint is kept as its status alone. Its machine page says what a machine needs, how to check it and what a run costs.
…g programs, and fake the sweep's clock for the sweep alone

Two tests made a Run of OSWorld2, which asks for ImageMagick, ffmpeg and
tesseract on the machine: they now say the programs are there, as the
test of their refusal does. The sweep's tests replaced time.sleep for
every module, so a slow docker's wait in subprocess advanced the fake
clock and the test's own steps: the fake now replaces the sweep's time
module alone.
…he rest to its machine setup and a page on reading its runs
… keep OSWorld2's option and setup lines short and named

An option with a pattern may carry an example its pattern allows besides
its default, checked when given, so an entry from before still loads;
every bundled pattern carries one. Pages, the builder, a refused value
and the generated pages show the example; the pattern only validates. A
default named default reads Provider default for any option.

A machine setup may carry a label, shown in place of its key on the
benchmark page, in the builder, in Review and in the container note.
OSWorld2's option and setup lines say what a value does and when to set
it, without internal names; the machine guide keeps the rest.
…es and datasets, its refusals by the option's name, and its results page to what a trial says

The definition's summaries, dataset texts, option values and the null control's
source say what a user chooses, without script names or code literals; the one
VM provider is fixed. The mock websites' refusals name Website host suffix, and
the driver's say stopped and the judge's calls. Read an OSWorld2 run keeps what
differs from the paper and a table of what a trial says; the machine guide keeps
one rule for write limits.
…root token of its own

Tasks 026 and 041 no longer need a GitLab set up by hand. Their first unit
on a machine clones Task-Web/gitlab at its pin beside the upstream's clone,
configures it for the mock websites' suffix, pulls its images and makes
its snapshot, under the GitLab lock and its warden; later units start
from that snapshot. The snapshot keeps a stamp of the stack, the host and
the token, and is made anew when any of them changes. Nullius makes the
root token there, a file for this user alone beside the stack, and gives
it to the processes of those two tasks only. The run goes where its
driver runs, here or on a host.

The GitLab host option and the GITLAB_PRIVATE_TOKEN licence are gone, so
the builder asks for no token: GitLab's host is gitlab. before the
websites option, whose Caddy serves it. Step 9 of the machine guide goes.
…esh, and ask nothing about GitLab

The GitLab write share option goes: a GitLab unit's own GitLab takes the
VMs' write limits, or 16 MB/s and 160 IOPS where they are none, and the
stack is always configured with that default. The snapshot's stamp
covers the compose override and gitlab.rb too, and carries the day it
was made: the upstream's token step makes the root token for a year,
so a snapshot 300 days old is made anew. A stamp that cannot be read
fails the unit instead of removing a good snapshot. Preparation waits
within the unit's time. A folder that is no clone is refused, never
removed; the clone goes through a folder of its own, removed if it
fails, as is the token's temporary file. The token step's output is
masked before it reaches an error.
… and keep a snapshot Docker could not read

GitLab is made under the limits of the unit that makes it, as it later
runs under them. A snapshot whose stamp Docker could not read fails the
unit instead of being made anew; only a missing stamp counts as none.
By hand, prepare refuses a folder that is not the stack Nullius cloned,
and removes nothing. Website host suffix says that GitLab is served with the mock websites.
…er could not read

A snapshot is none only when Docker says there is no such volume, or
when it holds no stamp; any other failure to look for it or read it
fails the unit and keeps it. With GitLab's image gone, after a prune or
a new pin, the snapshot counts as none, so the images are pulled and
GitLab is made anew. The token step runs under timeout, so a driver
killed during it leaves no shell retrying for longer than an hour. Task
041 counts among those that need the websites option.
…y where Docker says so

The volume, GitLab's image and the snapshot's stamp count as absent
only where Docker or cat says there is no such volume, image or file,
in any case; every other failure fails the unit and keeps the snapshot.
The token step's timeout runs in the foreground and ends before
Python's, so the shell under it never outlives it.
…ls, as its other refusals do, and say that its GitLab tasks run one at a time
…s write-limit refusals, not their environment
…th each agent's own budget, show its VM provider on Where and ask its website host suffix on Dataset, for the datasets that need it
aitianshi added a commit that referenced this pull request Nov 19, 2026
Closes #192

Depends on #188. **Merge #188 first, then #193.**

## What

Conformance writes several kit jobs in one execution step. The shared
running state and exit code made already completed kits remain running,
then labelled all of them failed when a later kit failed. For standard
jobs sharing a step, execution now reads a complete, error-free result
as that job's own finish, with its own timestamp. This also works after
restoring an execution from disk. The overall execution keeps its
failure.

Incomplete, errored, unreadable or partially written results retain
runner status. Harbor and single-job steps keep their existing exit
semantics. Scores and execution commands are unchanged.

## Touches

Backend (`execution`) and backend and route regression tests.

## Before and after

Synthetic three-kit fixture, with two clean completions and one failing
kit:

| Moment | Before | After |
| --- | --- | --- |
| Last kit running | 3 running | 2 finished, 1 running |
| Last kit failed | 3 failed | 2 finished, 1 failed |

Before/after screenshots were captured and reviewed locally at 1280 and
390 pixels in light and dark themes. Image attachments are not uploaded
to GitHub; this PR records the visible labels above.

## Verified

- 86 backend tests passed: `test_execution_job_state`, `test_execution`,
`test_execution_stop`, `test_execution_stop_restart`.
- 31 route tests passed: `test_runs_by_launch`, `test_routes`.
- Focused Chromium check used the regression fixture through Flask's
test client, comparing the original and changed `job_state`
implementations. Both running and failed states checked at 1280 and 390
pixels, light and dark; no page JavaScript errors. At mobile width the
existing layout hides the State column; the summary shows the corrected
counts.
- Regression coverage includes restored executions,
incomplete/error/cancellation counts, malformed files, and preservation
of Harbor and single-job state behavior.
- `git diff --check` passed.
- No paid conformance run or Docker benchmark was launched; validation
uses synthetic results written by the standard's own result writer.

## Risk

The state reader now reads one small result file for a standard job
sharing a step. A wrong completion decision could hide a failed process
behind a finished job row. It requires a matching job ID, a finish
timestamp, a positive planned count fully completed, and zero errored,
cancelled, running or pending trials; otherwise it falls back to runner
state. No record format or migration changes. Revert this commit to
restore the previous display.
@aitianshi
aitianshi merged commit 55c9f73 into ThinkFlowLab:main Nov 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: an agent of the Agents page cannot run a computer-use benchmark

3 participants