Skip to content

feat(builder): choose hosts or sandboxes and run VM tasks on Boat - #228

Merged
aitianshi merged 7 commits into
mainfrom
feat/where-step-machines
Nov 19, 2026
Merged

aitianshi merged 7 commits into
mainfrom
feat/where-step-machines

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #227

What

The Where step now chooses Hosts or Sandboxes first, then shows the relevant machines and isolation. It follows the approved layout: a separate Boat card, account capacity and size cards, compact equal-sized cards across benchmarks, and the short “Too small” warning. Ordinary Boat runs retain agent-only isolation. Benchmarks that boot their own VMs run each task whole in its own Boat VM, with a separate relay container for model calls.

Machine requirements now use compact named rows, scoped Check and Install actions, and a collapsed manual command with Copy. The redundant prerequisite subtitle and shard count are removed.

Touches

  • Builder UI and routes: exclusive machine choices, API-derived Boat limits and sizes, and fixed VM isolation/provider descriptions.
  • Standard, catalogue and execution planning: declared guest memory, size checks, explicit refusal of unsupported placements, one-unit positional shards, remote task setup, result retrieval, cleanup and merge.
  • Credential and model handling: selected provider and declared benchmark credentials cross SSH stdin; task processes use placeholder model keys. Model options and limits are forwarded; a total spending cap is split equally between tasks, without transferring unused shares.
  • Unit/route coverage and builder, standard and CLI documentation.

Before and after

OSWorld2 on Hosts, both captured at a 1280-pixel viewport. Before is the branch's base commit. After shows the current Where renderer with missing scoring tools and the manual command expanded; the app shell is omitted from this focused capture.

Before After
Before After

Boat: light, dark. Screenshots live on the separate review-evidence branch, outside this diff.

Current requirement layout at 390 pixels, with Copy beside the expanded command:

Light Dark
Current light requirement layout Current dark requirement layout

These focused captures use the actual Where renderer and stylesheet with stubbed missing-tool state and clipboard access. “Copied” shows the successful Copy interaction.

Verified

  • Compact requirement update: all 664 route tests passed, plus 11 focused Where/catalogue documentation checks. Headless checks at 1280/390 pixels in light/dark passed for disclosure, copied command text and overflow.

  • After merging current main: TMPDIR=/var/tmp python3 -X cpu_count=4 app/tests/parallel.py ran 3,073 tests across 156 modules. One failure remains: tests.test_osworld2_api_logs.Reaper.test_an_ended_run_is_packed expected three restored logs but got zero. The 18-test module also failed in an isolated rerun. These test and implementation files are unchanged by this PR; the cause has not been established. An earlier serial recheck passed, so do not treat the latest backend suite as green.

  • TMPDIR=/var/tmp uv run --no-project --with-requirements app/requirements.txt python app/tests/parallel.py app/ui_tests: after merging main, all 664 tests across 55 modules passed. The seven focused Where tests also passed after the computer-port guard adjustment.

  • Real local Docker relay with a fake model endpoint: host-side agent calls reached the relay container, records were saved, and the saved job contained no real model key. Seven focused tests passed; the additional credential-scope test passed separately.

  • Sixteen headless browser captures of HumanEval and OSWorld2 on Hosts/Sandboxes, light/dark, 1280/390 pixels: no JavaScript errors, horizontal overflow or clipped card text. VM-size card dimensions match across the two benchmarks at each viewport.

  • Earlier real Boat validation on this branch: OSWorld1 null control, two test_small tasks in Default VMs, both scored 0.0 on the repeat run; no VMs remained. The first attempt had one upstream desktop boot timeout, which did not recur on the repeat.

  • Not verified: a model-backed OSWorld task or OSWorld2 task on a real Boat VM after adding relay containers. No paid model call was made for that change.

Risk

The remaining backend log-reaper failure above needs investigation before merge.

Major: this changes where tasks execute and how credentials reach whole-task Boat VMs. Those VMs receive the model key in runner memory and declared benchmark credentials; the relay container receives the real model key and task processes receive a placeholder. Ordinary agent-only Boat execution retains its existing key boundary.

Failures in task setup, SSH transfer, cancellation, relay startup or merge could leave incomplete results or require VM cleanup. Whole-task Boat runs do not accept --units or --shard; choose a dataset and limit. Splitting a total spending cap can stop one task while another has unused budget. OSWorld's nested desktop startup can still time out upstream.

Revert these commits to restore the previous Where layout and execution paths. No persistent storage migration is required.

Machines are Hosts (this machine and Settings hosts, one shard each) or
Sandboxes (a Boat VM per unit, as many at once as the account's limits
say), never both. Sandboxes offer the sizes the account's limits list;
a size without more memory than a unit's own VM is refused, read from
a Where option's new needs.<value>.vm (OSWorld1 and OSWorld2: 4 GB).

How it runs is fixed in a sandbox and for a benchmark whose units boot
their own VM (no container: it cannot start them), with a single VM
provider on the same line.

A setup or a check on Boat no longer falls back silently to local
processes: Review and run.Run refuse it, as they refuse a computer-port
agent. Boat's limits are read once a minute for both sizes and slots.
The boat backend moves only an agent's process into a VM, so OSWorld,
whose driver boots the task's own VM on its machine, could not use it.
On Boat, a benchmark whose units boot their own VM now runs each unit
whole in a Boat VM of its own: this runner's snapshot is sent there and
runs one unit as a shard on local processes, installing what it needs
itself; each shard's job comes back as its unit ends, the VM is deleted,
and the shards are merged here into one job recorded as a Boat run.

Shards gain a position partition (--shard-by position) so shard I of N
is exactly one unit. For now only what calls no model goes this way,
since the model key would otherwise travel to each VM: OSWorld1's null
control. Proven on two test_small tasks in Default VMs: both scored 0.0,
no VM left.
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown

🟡 PR judge · 8 of 8 yes · #228 ↔ #227

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/standard/boat.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/standard/boat.py
Question Verdict
Changes do what the issue asks · 7 of 7 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown · 6 images · the diff touches what is visible ✅ yes
Acceptance criteria, 7 of 7 met
# Criterion Verdict
1 Hosts and Sandboxes are mutually exclusive; host selections still run one shard per host. ✅ met (cut)
2 Sandboxes show a separate fixed Boat card, API-derived concurrent capacity and machine sizes, with no invented fallback… ✅ met (cut)
3 Size eligibility uses declared guest memory; the short “Too small” label and compact cards have matching dimensions acr… ✅ met (cut)
4 Ordinary benchmarks offer Local processes or Container on Hosts and fixed agent-only VM isolation on Boat; VM benchmark… ✅ met (cut)
5 Each whole-task Boat VM runs one positional shard, returns its results, is deleted, and contributes to the merged job; … ✅ met (cut)
6 Whole-task model calls use a separate relay container inside the Boat VM; task processes receive a placeholder key. Mod… ✅ met (cut)
7 Unit and route coverage checks placement, sizing, credentials and limits; focused browser checks cover light/dark theme… ✅ met (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit 269e6db into main Nov 19, 2026
2 checks passed
@aitianshi
aitianshi deleted the feat/where-step-machines branch November 19, 2026 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redesign Where around hosts or Boat sandboxes and run VM tasks wholly on Boat

1 participant