Repository navigation
feat(standard): run a desktop benchmark with an agent of the Agents page, through a computer port and its kit - #188
Merged
Conversation
Pass participant kinds and agent setup limits to drivers that request them. Notify opted-in drivers once each agent unit ends, including failed and stopped units. Apply agent-only benchmark options in runs, recipes and the builder. Keep leak checks on the control setup contract and document the hooks.
Report benchmark-specific refusals per port and continue checking the remaining candidates. Save the proofs collected by the command while preserving existing records for refused kits. Keep explicit port refusals and command-wide failures fatal. Distinguish benchmark prerequisites and installs from agent installs, and validate provider credentials before checking kits.
…loop A desktop benchmark's runner serves one MCP server, computer, whose tools are the desktop's basic operations; the agent is a client alone. The benchmark's loop runs as it is and is handed a Bridge: each predict gives it the actions of the agent's next acting call, and the reading calls are served before it, one desktop operation at a time, in the order they came. An acting call is answered once its actions and their pause have run. - computer_port: the Desktop protocol (python, upload, read, capture, password), the Bridge (the step budget, the refusal of every call past it, the turn's end), the Server over a Unix socket (every connection of the session, one call log line per tool call: tool, times, failure), files sent once for a run by their sha256, and the runner's ready file, lifeline and one stop. - computer_pipe: the session's stdio pipe to the socket. A read of the input is counted under the lock the exit decision takes; the input's end is seen by its writer's hang-up while a full queue holds the reader; the pipe's end has one deadline from the input's end. - Tests on a fake desktop with a scripted MCP client, a stand-in loop that counts steps as the upstream's does, and the upstream's own run_single_example where a checkout of it is at hand.
A page for agent.computer/v1: its fifteen tools, what a step is, the step budget, the turn's end, uploads by content, the call log, and what a desktop benchmark's runner gives the shared code. The standard's row of the port links to it, and the docs folder map names ports/.
…dicts The computer port's conformance kit needs a small real desktop on which an agent shows that it sees the screen, points and types, decided by a program. This adds the desktop and the fixture the agent acts on; the kit's driver, runner and standard.json come with the port's server. - desktop.Dockerfile: an environment of the kit on the base image, tagged nullius/env-agent-computer-v1-desktop:<hash> as container.env_image tags any environment. Xvfb at 1920 x 1080 on :0, Openbox, a session bus for uid 1000, Python 3 with Tk, pyautogui and Pillow pinned by version and hash (desktop-requirements.txt). Two users: user (uid 1000, no sudo) and fixture. - session.py: the container's command; the display, its cookie, the bus and the window manager, then a ready file. - desktop.py: start a desktop (no network, no new privileges), the five operations a benchmark's side gives the port's server (python, upload, read, capture, password), one action run as user with its start and end read on the desktop's clock by a root process, the fixture's start and log, and removal. - fixture.py: a Tk program that draws codes and labels as bitmaps, with Tk's send removed, and logs each press, release, scroll and key with its time and what it hit, in a folder only fixture reads. - checks.py: the values each check draws, and the verdicts of see, click and type from the fixture's log, the actions' times and the last message; click and type fail on any input outside an action's time. - pyproject.toml: the kits' Dockerfiles and requirement files go in the package. Tests: the verdicts on hand-written logs, the action's times taken from the desktop's reply, the image's declaration; the docker tier starts real desktops and judges a click and a word sent by actions right, and the same click sent through python wrong.
A run stopped after a unit passed its stop check, and before its setup request was written, still sent that unit's end to the driver. A unit now counts as set up once its setup line is written. A request cut short by a stop leaves its late reply in the driver's queue, where a forced end would read it as its own. Such a driver is no longer sent an end. Document that a driver closed after a request got no reply, or one left with an unanswered request, is not sent an end. Test the forced wait after a stop, and replace a test whose mock passed for the wrong reason.
The setup time limit was turned into a float, so the live feed recorded the env_setup phase's timeout as 600.0 for every run, where it had recorded 600. The limit is now kept as given. Test the env_setup phase's timeout in the live feed for an agent and a control, with and without limits.setup_seconds.
An option whose setups named agent beside another scope passed the check, though the builder never showed it in a setup's tab. One on a capability benchmark passed too, where no agent ever runs it. The check now accepts agent only on its own, and only on a benchmark that agents run.
Conform without a port named went on past a kit refused for what its benchmark needs of the machine, and then exited 0 as if every port had been proven. When every kit was refused it still saved the definition and exited 0. A refused kit is now returned as refused beside the proofs gained, which are written, and the command exits 1. When every kit is refused, nothing is written and the check is refused, with exit status 2. The key is checked before the first kit only when some port has a kit: a command that names only ports without one says so and needs no key. Document the refusals of conform and its exit statuses.
…ny client - Each connection's answers are written by a thread of its own, at most OUT_BYTES waiting: a client that stops reading holds up neither the loop nor another connection, and a close never waits for a write. A connection accepted as the server closes is closed, and a closed one leaves the server's list. - One table gives what tools/list says and what a call may hold. A call is checked in full, an upload's base64 decoded, before it is queued; the JSON-RPC envelope is checked before any tool, a frame's length with its newline, and a number past any integer is refused and recorded. An answer over the bound is recorded as failed. - The turn's end stops the actions of a step not yet run and refuses that call; a done or fail taken before it still reaches the loop. - Uploads: an append goes through a name of the server's own, removed on every path; a file taken by its sha256 starts the file, and append is refused with it; the chunks being kept are the unit's own, dropped when the file starts again and at the unit's end, up to KEEP_BYTES; a kept file is written through an exclusive temporary file and read only when it is no link. - The last resort gives its own callback LAST_SECONDS before it exits. - The budget test holds the first action behind an event in place of a 250 ms wait.
Once the server's side ends, the pipe reads its input on to its end and sends it, by its one deadline; it exits 0 only when it saw that end and sent everything, and 1 otherwise. The queue test waits for the first send and measures bytes, assuming no read's size, and runs again with reads of 4 KiB.
…nd bounds The page says that a call is checked against its listed schema before it waits, what the turn's end does to a step not yet run and to a done or fail taken before it, that a session may connect again, the bounds on a connection's waiting answers and on a kept file, what an error is, and that done and fail answer before the loop runs them.
… cookie and no key repeat The desktop's X server ran as root with access control off, and a key held from one action to the next repeated between them, which the verdicts count as input outside every action. - The server runs as `display`, an account of its own, with access control on: root, user and fixture connect with the cookie the session writes. - The core key repeat is turned off, and the server's repeat delay is the longest XKB allows, which also covers the input-test device. - The session ends, and the container with it, when the display, the session bus or the window manager ends, not the display alone. - The image adds the accessibility bus (at-spi2-core), which the session bus starts on demand for programs that look for it.
…il loudly A pass must come from input sent inside actions, and a correct agent must not be failed by the desktop's own timing. - An action answers only once the verdicts' slack has passed on the desktop, so a request sent after it cannot fall inside its time. - The action's times and the fixture's events are read on one monotonic clock: a step of the wall clock no longer fails a correct action. - click fails on a press of any mouse button on another button; Enter in the field is input whose time is checked, as keys are. - The fixture says ready only when its window is mapped at the screen's size and holds the focus within its deadline. - read gives a regular file alone, at most 12 MiB, and refuses the rest as the agent's error. - A start that fails or is interrupted removes the container it made, its log read or not; remove checks the engine's answer and that the container is gone; a fixture log that cannot be read, or holds a line that is no event, is an error. - The docker tier's desktop tests run wherever an engine answers and build the image first; the removal test's probe is bounded and must succeed.
…hers When conform passed over a refused kit and wrote the other kits' proofs, it dropped every record of a known port that was not in the offers. A port proven earlier whose offer had been left out lost its proof although its kit never ran. The record of a port whose kit was refused in the same command now stays as it was.
… ends The driver.unit_end row said that a driver closed after a request got no reply has its process group ended with it. A child that redirects its streams can outlive that. The row keeps saying that such a driver is not sent an end.
…red-request test The test stopped the run from a timer started before setup was sent; on a slow machine the stop could come before the line was written and fail the test for the wrong reason. The stop now comes from the request's on_sent call, and the test waits for the driver to read all it was sent instead of sleeping.
For a unit an agent runs (driver.participant), setup starts runner.py behind the start gate. The runner builds OSWorld's DesktopEnv as the null control does, serves the computer port's MCP server over a Desktop of five methods on the upstream's controllers, and calls the upstream's own run_single_example, unmodified, with the port's bridge as the loop's agent. Setup answers once the loop asks for the agent's first step, with the prompt and the session's server computer: the port's pipe, whose path the driver takes from the package it imports, joined to the runner's socket. Score sends the turn's end on the runner's lifeline, waits for the runner within its bound and reads the upstream's result.txt; end (driver.unit_end) reaps the runner. - The bounds follow from the unit's limits (limits.setup_seconds, score's and end's); the runner's clean-up, now the bridge, the server and the desktop's bounded close, fits inside the driver's reap. A reap the driver's end cuts is handed to a reaper of its own. - The options Steps (100) and Pause (3.0 s) apply to an agent's run alone; driver.command names runner.py for a host's copy of the entry. - A unit with no agent runs as before.
…erywhere - The last resort's word goes to stderr from a thread it waits for 0.1 s at most: an exit never waits on a pipe nobody reads. - Every outgoing frame is bounded, the answers of any method and the errors standing in for them, with no id when even that is over. - NaN, Infinity and -Infinity are no JSON. Each tool's schema lists every argument it takes (additionalProperties false), and any other argument is refused. - The listed patterns use explicit classes, which Python and JSON Schema's regular expressions read alike; a final $ is the text's end. - upload's data is base64 by its pattern, and its listed bound is the check's: at the longest length the padding says the chunk is not over. - A failed append removes its part as far as the desktop answers, and a kept file's temporary file and descriptor go on every path. - The tests close the wires they make, and the queue test's helper ends the pipe before it closes an input a feeder may still write.
The computer port had a server and the kit a desktop; this joins them so that an agent can prove agent.computer/v1 like any other port. - The kit agent-computer-v1: eight checks, four basic (see, click, type, limit) and four skills (drag, scroll, recover, batch), each decided by the fixture's log, the actions' times and the server's call log. Every check fails on input that came outside an acting call. - driver.py: each unit's desktop and fixture, the port's server over it in the driver's process, and a loop of the kit's own that runs each step's actions and waits the pause. score ends the turn, judges and removes the desktop and the socket. A unit that never reaches score is let go by the next setup or the driver's end; reaper.py, started in a session of its own, removes what a killed driver left. - The reference makes the right calls through the unit's own server; the null control passes nothing. The kit's proof is recorded in its standard.json. - ports.py names the kit; the port's page gets its section, the ports table its row, and the generated agent pages the port's column. - fake_computer_agent.py: a scripted ACP agent that looks and clicks once, for the tests and a scripted conformance.
…ake desktop's limit plainly
A runner's snapshot for a remote host takes a benchmark's server command only from files of its own folder, and refused the kit's path to the port's pipe two folders up. pipe.py in the kit's folder runs that pipe. The kit's proof is recorded again on this tree, and the recipe test now expects the computer kit among the kits an ACP agent takes.
… own last 429, not a side call's or a context window's stop The rule reads the agent's own last call, a guard's line in the stop file and a score stopped by other than the context window; a cap's refusal is still the cap's.
…h --served-by replaces, as ThinkFlowLab#124 has the user choose A model the definition pins (setups.original.pin, score.judge.pin) goes to the user's providers when --served-by gives it others: the run says so, records the definition's under standard.pins_replaced, and the Original's run does not compare with the paper. The recipe's compile gives each field's default (served_pins) and sends a field's list, its own or the default, with every job that calls its model; the builder's Served by starts with it, and Review says when a field's differ. --served-by without MODEL= names the model an Original sends, and prove --job refuses a control whose pins were replaced.
…tcome and prove text and their messages
…r place, the recipe keeping the value; OSWorld1's agents by their upstream's names
…e the work runs on Where, a single value as fixed, and what the datasets need on Dataset; OSWorld1's VM provider on Where
…eclares, with their route errors; Served by is the user's alone, as ThinkFlowLab#124 has it
… upstream's own runners, mock websites, GitLab and proxy, proven on test_v2, and how to prepare a machine for it OSWorld2 runs each of the 108 tasks of test_v2 in a VM of its own through the upstream's docker provider, with the mock websites its tasks need served on the machine, and GitLab and a proxy started for each unit that needs one. Its Original runs the upstream's agents with their models pinned to their providers on OpenRouter, and its null control is proven on test_v2 at 0 of 108. The tasks are gated, so a failing endpoint is kept as its status alone. Its machine page says what a machine needs, how to check it and what a run costs.
…reach OSWorld2's VMs and services
…g programs, and fake the sweep's clock for the sweep alone Two tests made a Run of OSWorld2, which asks for ImageMagick, ffmpeg and tesseract on the machine: they now say the programs are there, as the test of their refusal does. The sweep's tests replaced time.sleep for every module, so a slow docker's wait in subprocess advanced the fake clock and the test's own steps: the fake now replaces the sweep's time module alone.
…he rest to its machine setup and a page on reading its runs
… keep OSWorld2's option and setup lines short and named An option with a pattern may carry an example its pattern allows besides its default, checked when given, so an entry from before still loads; every bundled pattern carries one. Pages, the builder, a refused value and the generated pages show the example; the pattern only validates. A default named default reads Provider default for any option. A machine setup may carry a label, shown in place of its key on the benchmark page, in the builder, in Review and in the container note. OSWorld2's option and setup lines say what a value does and when to set it, without internal names; the machine guide keeps the rest.
…es and datasets, its refusals by the option's name, and its results page to what a trial says The definition's summaries, dataset texts, option values and the null control's source say what a user chooses, without script names or code literals; the one VM provider is fixed. The mock websites' refusals name Website host suffix, and the driver's say stopped and the judge's calls. Read an OSWorld2 run keeps what differs from the paper and a table of what a trial says; the machine guide keeps one rule for write limits.
…root token of its own Tasks 026 and 041 no longer need a GitLab set up by hand. Their first unit on a machine clones Task-Web/gitlab at its pin beside the upstream's clone, configures it for the mock websites' suffix, pulls its images and makes its snapshot, under the GitLab lock and its warden; later units start from that snapshot. The snapshot keeps a stamp of the stack, the host and the token, and is made anew when any of them changes. Nullius makes the root token there, a file for this user alone beside the stack, and gives it to the processes of those two tasks only. The run goes where its driver runs, here or on a host. The GitLab host option and the GITLAB_PRIVATE_TOKEN licence are gone, so the builder asks for no token: GitLab's host is gitlab. before the websites option, whose Caddy serves it. Step 9 of the machine guide goes.
…esh, and ask nothing about GitLab The GitLab write share option goes: a GitLab unit's own GitLab takes the VMs' write limits, or 16 MB/s and 160 IOPS where they are none, and the stack is always configured with that default. The snapshot's stamp covers the compose override and gitlab.rb too, and carries the day it was made: the upstream's token step makes the root token for a year, so a snapshot 300 days old is made anew. A stamp that cannot be read fails the unit instead of removing a good snapshot. Preparation waits within the unit's time. A folder that is no clone is refused, never removed; the clone goes through a folder of its own, removed if it fails, as is the token's temporary file. The token step's output is masked before it reaches an error.
… and keep a snapshot Docker could not read GitLab is made under the limits of the unit that makes it, as it later runs under them. A snapshot whose stamp Docker could not read fails the unit instead of being made anew; only a missing stamp counts as none. By hand, prepare refuses a folder that is not the stack Nullius cloned, and removes nothing. Website host suffix says that GitLab is served with the mock websites.
…er could not read A snapshot is none only when Docker says there is no such volume, or when it holds no stamp; any other failure to look for it or read it fails the unit and keeps it. With GitLab's image gone, after a prune or a new pin, the snapshot counts as none, so the images are pulled and GitLab is made anew. The token step runs under timeout, so a driver killed during it leaves no shell retrying for longer than an hour. Task 041 counts among those that need the websites option.
…y where Docker says so The volume, GitLab's image and the snapshot's stamp count as absent only where Docker or cat says there is no such volume, image or file, in any case; every other failure fails the unit and keeps the snapshot. The token step's timeout runs in the foreground and ends before Python's, so the shell under it never outlives it.
…ls, as its other refusals do, and say that its GitLab tasks run one at a time
…s write-limit refusals, not their environment
… its model, now that no provider is pinned
…th each agent's own budget, show its VM provider on Where and ask its website host suffix on Dataset, for the datasets that need it
aitianshi
added a commit
that referenced
this pull request
Nov 19, 2026
Closes #192 Depends on #188. **Merge #188 first, then #193.** ## What Conformance writes several kit jobs in one execution step. The shared running state and exit code made already completed kits remain running, then labelled all of them failed when a later kit failed. For standard jobs sharing a step, execution now reads a complete, error-free result as that job's own finish, with its own timestamp. This also works after restoring an execution from disk. The overall execution keeps its failure. Incomplete, errored, unreadable or partially written results retain runner status. Harbor and single-job steps keep their existing exit semantics. Scores and execution commands are unchanged. ## Touches Backend (`execution`) and backend and route regression tests. ## Before and after Synthetic three-kit fixture, with two clean completions and one failing kit: | Moment | Before | After | | --- | --- | --- | | Last kit running | 3 running | 2 finished, 1 running | | Last kit failed | 3 failed | 2 finished, 1 failed | Before/after screenshots were captured and reviewed locally at 1280 and 390 pixels in light and dark themes. Image attachments are not uploaded to GitHub; this PR records the visible labels above. ## Verified - 86 backend tests passed: `test_execution_job_state`, `test_execution`, `test_execution_stop`, `test_execution_stop_restart`. - 31 route tests passed: `test_runs_by_launch`, `test_routes`. - Focused Chromium check used the regression fixture through Flask's test client, comparing the original and changed `job_state` implementations. Both running and failed states checked at 1280 and 390 pixels, light and dark; no page JavaScript errors. At mobile width the existing layout hides the State column; the summary shows the corrected counts. - Regression coverage includes restored executions, incomplete/error/cancellation counts, malformed files, and preservation of Harbor and single-job state behavior. - `git diff --check` passed. - No paid conformance run or Docker benchmark was launched; validation uses synthetic results written by the standard's own result writer. ## Risk The state reader now reads one small result file for a standard job sharing a step. A wrong completion decision could hide a failed process behind a finished job row. It requires a matching job ID, a finish timestamp, a positive planned count fully completed, and zero errored, cancelled, running or pending trials; otherwise it falls back to runner state. No record format or migration changes. Revert this commit to restore the previous display.
6 tasks
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #187
What
An agent of the Agents page could not run OSWorld: the entry asked for
agent.computer/v1, a port with no page, no server and no kit. This adds the three.computerof fifteen tools. Five read (info,screenshot,read,python,upload), eight act (click,move,drag,scroll,type,key,wait,step) and two end the task (done,fail). One acting call is one step, counted by the benchmark's own loop. Astepmay also end the task withend(doneorfail): a decision that acts and then stops is one step. The port's records (calls, desktop events and targets, OSWorld's result and runner log) are listed on a trial's Artifacts tab.see,click,type,limit) and four skills (drag,scroll,recover,batch), with a null control and a reference.max_steps, for OSWorld's own scripts and for an agent through the port, wheredefaultis 100.driver.participant,driver.unit_end,limits.setup_seconds, an option for agents alone), andconformgoing on past a kit that cannot run on the machine.Read first: the port's page, then
computer_port.py(the server and its bridge to a loop), the kit'schecks.pyanddriver.py, then OSWorld'srunner.pyand the new part of itsdriver.py.mainis merged in, with the fixes the merge needed.Two limits: OSWorld's agent side ran on the local backend alone, and the port bounds an agent's steps, not its model calls. A run that would put the agent in a VM of its own is refused with the port.
Touches
backend and the standard · templates and static · docs and site · catalogue
Before and after
The builder for OSWorld, with an agent that holds the port chosen: Pause on the Agent step, Steps with the run's limits. Each picture opens at full size.
Verified
prove), on a machine with Docker: the null control passes 0 of 8 checks and the reference 8 of 8. The kit's tests that need Docker pass there (100 tests).codex2.2.2 withgpt-6-lunapasses the four basic checks and shows the four skills, each in three runs of three (conform --repeat 3, the container backend).done, afail, a used-up budget and a stop each left no VM, container or process. Two units at a time kept their desktops apart.main.mainin the sandbox used here. After the merge ofmain,proveand the suites were run again; the other lines above date from the first push.Risk
A run with no agent of the Agents page takes the paths it took: the new hooks act only for a driver that asks for them (the comparison above). The kit needs Docker; on a machine without it
conformreports that kit as refused and goes on. The port's page is a contract: agents built against it would break if it changed. Undo with a revert.