Repository navigation
feat(builder): choose hosts or sandboxes and run VM tasks on Boat - #228
Merged
Merged
Conversation
Machines are Hosts (this machine and Settings hosts, one shard each) or Sandboxes (a Boat VM per unit, as many at once as the account's limits say), never both. Sandboxes offer the sizes the account's limits list; a size without more memory than a unit's own VM is refused, read from a Where option's new needs.<value>.vm (OSWorld1 and OSWorld2: 4 GB). How it runs is fixed in a sandbox and for a benchmark whose units boot their own VM (no container: it cannot start them), with a single VM provider on the same line. A setup or a check on Boat no longer falls back silently to local processes: Review and run.Run refuse it, as they refuse a computer-port agent. Boat's limits are read once a minute for both sizes and slots.
The boat backend moves only an agent's process into a VM, so OSWorld, whose driver boots the task's own VM on its machine, could not use it. On Boat, a benchmark whose units boot their own VM now runs each unit whole in a Boat VM of its own: this runner's snapshot is sent there and runs one unit as a shard on local processes, installing what it needs itself; each shard's job comes back as its unit ends, the VM is deleted, and the shards are merged here into one job recorded as a Boat run. Shards gain a position partition (--shard-by position) so shard I of N is exactly one unit. For now only what calls no model goes this way, since the model key would otherwise travel to each VM: OSWorld1's null control. Proven on two test_small tasks in Default VMs: both scored 0.0, no VM left.
7 tasks
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
🟡 PR judge · 8 of 8 yes · #228 ↔ #227
Acceptance criteria, 7 of 7 met
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #227
What
The Where step now chooses Hosts or Sandboxes first, then shows the relevant machines and isolation. It follows the approved layout: a separate Boat card, account capacity and size cards, compact equal-sized cards across benchmarks, and the short “Too small” warning. Ordinary Boat runs retain agent-only isolation. Benchmarks that boot their own VMs run each task whole in its own Boat VM, with a separate relay container for model calls.
Machine requirements now use compact named rows, scoped Check and Install actions, and a collapsed manual command with Copy. The redundant prerequisite subtitle and shard count are removed.
Touches
Before and after
OSWorld2 on Hosts, both captured at a 1280-pixel viewport. Before is the branch's base commit. After shows the current Where renderer with missing scoring tools and the manual command expanded; the app shell is omitted from this focused capture.
Boat: light, dark. Screenshots live on the separate review-evidence branch, outside this diff.
Current requirement layout at 390 pixels, with Copy beside the expanded command:
These focused captures use the actual Where renderer and stylesheet with stubbed missing-tool state and clipboard access. “Copied” shows the successful Copy interaction.
Verified
Compact requirement update: all 664 route tests passed, plus 11 focused Where/catalogue documentation checks. Headless checks at 1280/390 pixels in light/dark passed for disclosure, copied command text and overflow.
After merging current main:
TMPDIR=/var/tmp python3 -X cpu_count=4 app/tests/parallel.pyran 3,073 tests across 156 modules. One failure remains:tests.test_osworld2_api_logs.Reaper.test_an_ended_run_is_packedexpected three restored logs but got zero. The 18-test module also failed in an isolated rerun. These test and implementation files are unchanged by this PR; the cause has not been established. An earlier serial recheck passed, so do not treat the latest backend suite as green.TMPDIR=/var/tmp uv run --no-project --with-requirements app/requirements.txt python app/tests/parallel.py app/ui_tests: after merging main, all 664 tests across 55 modules passed. The seven focused Where tests also passed after the computer-port guard adjustment.Real local Docker relay with a fake model endpoint: host-side agent calls reached the relay container, records were saved, and the saved job contained no real model key. Seven focused tests passed; the additional credential-scope test passed separately.
Sixteen headless browser captures of HumanEval and OSWorld2 on Hosts/Sandboxes, light/dark, 1280/390 pixels: no JavaScript errors, horizontal overflow or clipped card text. VM-size card dimensions match across the two benchmarks at each viewport.
Earlier real Boat validation on this branch: OSWorld1 null control, two test_small tasks in Default VMs, both scored 0.0 on the repeat run; no VMs remained. The first attempt had one upstream desktop boot timeout, which did not recur on the repeat.
Not verified: a model-backed OSWorld task or OSWorld2 task on a real Boat VM after adding relay containers. No paid model call was made for that change.
Risk
The remaining backend log-reaper failure above needs investigation before merge.
Major: this changes where tasks execute and how credentials reach whole-task Boat VMs. Those VMs receive the model key in runner memory and declared benchmark credentials; the relay container receives the real model key and task processes receive a placeholder. Ordinary agent-only Boat execution retains its existing key boundary.
Failures in task setup, SSH transfer, cancellation, relay startup or merge could leave incomplete results or require VM cleanup. Whole-task Boat runs do not accept
--unitsor--shard; choose a dataset and limit. Splitting a total spending cap can stop one task while another has unused budget. OSWorld's nested desktop startup can still time out upstream.Revert these commits to restore the previous Where layout and execution paths. No persistent storage migration is required.