Skip to content

Redesign Where around hosts or Boat sandboxes and run VM tasks wholly on Boat #227

Description

@aitianshi

Priority

extreme: this reaches the standard runner, machine placement and credential delivery. Native Priority: High; P2 is planned feature work.

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)

Today

The Where step mixes the machine choice with the execution backend. VM benchmarks can be offered a container that cannot start their desktop VMs. The agent-only Boat backend cannot run a whole desktop task, and setup/check execution can fall back to the launching machine. The interface does not express the account's VM capacity and benchmark size requirements together.

Wanted

Use the approved layout: a Hosts/Sandboxes switch, host cards or a fixed Boat card, compact VM-size cards, then Isolation. Hosts and Boat must never mix in a recipe. Read Boat sizes and concurrent capacity from the API, disable sizes that cannot fit the benchmark's declared guest memory, and show “Too small.”

Ordinary Boat runs keep the agent alone in its VM and the driver, answers and model key on the originating machine. VM benchmarks run each task whole in a Boat VM, with Docker providing the nested desktop and a separate relay container handling model calls. Forward only the chosen provider credentials and declared benchmark credentials over SSH stdin.

Screenshots

Before After
Before After

Acceptance

  • Hosts and Sandboxes are mutually exclusive; host selections still run one shard per host.
  • Sandboxes show a separate fixed Boat card, API-derived concurrent capacity and machine sizes, with no invented fallback count in the UI.
  • Size eligibility uses declared guest memory; the short “Too small” label and compact cards have matching dimensions across HumanEval and OSWorld2 at the same viewport.
  • Ordinary benchmarks offer Local processes or Container on Hosts and fixed agent-only VM isolation on Boat; VM benchmarks show fixed desktop VM isolation and their provider.
  • Each whole-task Boat VM runs one positional shard, returns its results, is deleted, and contributes to the merged job; unsupported agent-only placements are refused rather than silently run locally.
  • Whole-task model calls use a separate relay container inside the Boat VM; task processes receive a placeholder key. Model settings, declared credentials and limits are forwarded, with total spending caps divided between tasks.
  • Unit and route coverage checks placement, sizing, credentials and limits; focused browser checks cover light/dark themes and desktop/phone layouts.

Dependencies and related

Builds on merged #150 (agent-only Boat execution), #134 (OSWorld2), #188 (computer port) and #213 (Where icons). Duplicate searches found no open implementation of this redesign.

Implementation: #228. After integrating main, 664 route tests passed; the 16 browser captures cover the approved layout. The backend suite ran 3,073 tests with one remaining OSWorld2 log-reaper failure, also reproduced in an isolated rerun. Its cause is not established. Full validation and real-Boat coverage limits are recorded in the PR.

Requirement layout refinement

Compact named requirements with Check and Install scoped to each requirement; manual commands stay collapsed with Copy. Removed the prerequisite subtitle and shard count. All 664 route tests and 11 focused checks passed; light/dark desktop/mobile disclosure and Copy checks passed. Screenshots: light | dark. Implemented in #228.

Activity

  1. self-assigned this
    on Nov 19, 2026
  2. added
    enhancementNew feature or request
    P2Planned: limited in scope, has a way around it, or a missing feature
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Planned: limited in scope, has a way around it, or a missing featureenhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions