Run builder turns your choices into a recipe and runs it only when you press Run from Review.
- Choose a benchmark. It sets the runner: Nullius when the benchmark runs there, Harbor otherwise. A benchmark that runs on both lets you switch.
- Choose a dataset.
- Choose who runs it and the model: only the agents and capabilities that fit, and the providers set in Settings.
- Set the limits; keep the task limit small the first time.
- Choose the machine in Where.
- In Review, meet every prerequisite, then press Run.
!!! warning "A run can call paid model APIs"
Review names what the run will do. The controls nop and oracle run no agent; a judge or embedding model can still be called.
Then read its results.
| Stage | Decides |
|---|---|
| Benchmark | A catalogue entry of either runner, and a version when it has several (key@version) |
| Runner | Nullius, the standard, or Harbor with its pin (Which Harbor). A choice only when the benchmark runs on both; Nullius is preselected |
| Dataset | One or more; each dataset with each agent is a job of its own. Each shows its proofs. A benchmark option its datasets need (step: dataset) is asked here, for the datasets it applies to. A missing or expired credential is named once, linked to Settings → Data providers (Hugging Face) or Custom keys |
| Agent (Memory on a memory benchmark) | Reference agents first, or No agent · Original for a direct-model benchmark. Memory benchmarks offer Original memory alongside compatible memory capabilities. Controls follow separately; each choice has its own settings tab |
| Model | Like the paper keeps what the paper fixed: the model picker shows only documented paper models when the benchmark lists them (the filter applies to the selected agent or capability too), and the user simulator and judge are their official models. A benchmark that lists no paper models but has an official simulator or judge leaves the model to you. Like the leaderboard, on a benchmark that declares its leaderboard's rules, leaves the model to you and fixes what the leaderboard does: its user simulator and judge, the options it sets and at least its attempts per unit. Any of your choice opens the provider's full list and lets the simulator and judge be changed. Nothing fixed by the paper: the paper choice is disabled. Served by, a row after each model field's when the provider is OpenRouter, waits for the provider list and stays hidden when it has exactly one entry and the field has no pin; an existing pin always shows, as does an empty or failed list. Two endpoints of one provider count as two entries. It chooses the serving providers that may answer that model, tried in their order and no other (Run a benchmark). Its picker is the model's: typing filters the model's providers, and each pick, or a slug typed and Enter, joins the list under it, each removable. Empty: any of them. Review shows them in its Model row |
| Limits | Task limit (empty is every task), a task-name glob, concurrency, attempts per unit, retries per attempt, timeouts in minutes. On the Nullius runner, a spending cap for the run and one per unit, turns per conversation for a benchmark that holds conversations, a cap on model calls per unit, time limits per phase in minutes, forced on every participant (empty: the benchmark's or agent's own value in text colour, Nullius's default in grey), then the benchmark's own bounds in a last section of its name (τ³-bench's steps and tool errors per conversation, OSWorld's steps per task); a list of units is set only on the command line |
| Where | Hosts: this machine or Settings hosts. Sandboxes: Boat VMs, within the account’s sizes and capacity. Choose isolation where available; VM benchmarks keep their own VM provider |
| Review | One row per decision and the prerequisites. Run |
Original setups run on local processes or in Docker; Docker changes isolation only. A unit whose credential is unset is not run; an unset credential the driver needs to list the units stops the run.
Checked on each machine when Review opens, on Check again, and as the run starts (app/nullius/prerequisites.py).
| Need | Asked of | Needed by |
|---|---|---|
uv |
command -v |
The standard; a pinned Harbor. On a host, installed by the run |
python3, git |
command -v on the host |
The standard on a host |
| Docker or Podman, its engine answering | The engine's version here, docker ps or podman ps on a host |
The standard's container backend; a control in a unit's own image |
| Docker | docker version here, docker ps on a host |
Harbor |
| A benchmark option's machine setup | Its programs | The chosen value; the install commands are shown |
harbor on PATH |
command -v |
A Harbor recipe without a pin |
For the standard, the Docker row says when the engine has userns-remap, which the standard's containers run outside of.
If the recipe's provider failed its last test in Settings, Review shows a notice. Run still starts.
A provider is set when Settings holds its key (OPENROUTER_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY), or the base URL of a custom provider. Each custom provider is a choice of its own, listed by its name. The Model step offers a provider only where it serves what the recipe's agents speak (protocols): Anthropic for an agent of the standard that speaks messages, never for a Harbor recipe.
| Set in Settings | Model stage | Run |
|---|---|---|
| One | That provider, nothing to choose | Open |
| Several | A choice among them | Open |
| None | Every provider that fits | Disabled until a provider is configured |
| A recipe naming one that is not set | Kept, marked not set in Settings | Disabled until another is chosen |
A recipe of controls alone needs no key unless the benchmark uses a judge or embedding model.
| Provider | Key | Endpoint |
|---|---|---|
| OpenRouter | OPENROUTER_API_KEY |
https://openrouter.ai/api/v1, fixed |
| OpenAI | OPENAI_API_KEY |
https://api.openai.com/v1, fixed |
| Anthropic | ANTHROPIC_API_KEY |
https://api.anthropic.com/v1, fixed |
| Custom | The provider's key, optional (placeholder local without it) |
The provider's base URL, required |
A recipe on a custom provider records "provider": "custom" and
the endpoint's id, "endpoint": "GPU_BOX". The id does not change when the
endpoint is renamed. A recipe without an endpoint uses the one at
MODEL_API_URL. A recipe on an endpoint that was removed keeps it, marked
not set in Settings, and Run stays disabled until another is chosen.
The Model step searches the selected provider's model catalogue: OpenRouter,
OpenAI, Anthropic, or the /models endpoint of the selected custom provider.
Keys stay on the Nullius host. Choose a result to use its exact model ID, or
enter an ID manually if the catalogue is unavailable. An Original setup with
a declared model list keeps those published choices.
| What | Where | When |
|---|---|---|
| Draft | The browser, one per browser | Saved on every change; Start over forgets it |
| Saved recipes | recipes/<id>.json under the user data directory, one file each |
Any browser lists and loads them. Save and Delete work only from a browser on the application host. Saving a name that exists updates that recipe, and saving a loaded recipe under a new name renames it. Import and export as JSON. A list kept in the browser by an earlier version moves here when the builder opens on the host. Recipes the server refuses stay in the browser under nullius.recipes.v1.rejected |
| A run's recipe | runs/<name>/run.json |
Job page, New run like this: the same recipe, at Review |
| A job's record | Its config.json and lock.json |
New run like this without a kept recipe: rebuilt, what it cannot carry named |
The Recipes page (Manage, Recipes) lists the saved recipes, a column per step. Each row opens its recipe in the builder, shows it step by step with its JSON (Info), duplicates it under a free name, exports its JSON or deletes it. Import adds a recipe file, or a list, and refuses the whole file when one recipe is bad. A row whose recipe names something this Nullius does not have, such as a benchmark, an agent or a provider without a key, shows Cannot run here. Its tooltip gives the builder's reason for each step. Import, Duplicate and Delete work only from a browser on the application host.
| Rule | Detail |
|---|---|
| Order | nop, oracle, then the agents under evaluation |
| Each Run | A new job (recipe name + timestamp); never resumes |
| Key and endpoint | Passed as environment (over stdin to a host), the key to a docker relay as a file; logs redacted. Other secrets withheld |
| A host | Gets a content-addressed snapshot of the running backend and effective catalogue into ~/.nullius/app/<content-id>, including source edits and user entries but no configuration stores; a Harbor recipe gets its pin's wheels in ~/.nullius/harbor/<pin>/wheels, and uv if missing |
| Proofs | Local runs save to the user catalogue. A host uses a writable copy in its run's catalogue/, isolated from the host's own entries; that copy and its proofs mirror back alongside jobs/, without changing the active user catalogue |
| Docker or Podman | The engine must answer ps (docker ps, or podman ps for the standard's container backend), or the run stops before it starts |
| Stop | Ends the current part (exit 130), records stopped; the next part never starts |
| Delete launch | On the launch in Runs, By launch, once every run of it has ended; asks first. The job page links to it. Deletes each run's <RUNS_DIR>/<name>/ (a shard per machine included), every job of the launch, their judgements in every workspace, and each run's copy on its host when ticked. A run still going is refused: stop it first. A launch with jobs from elsewhere has no Delete |
| Restart of Nullius | execution.restore reattaches from runs/<name>/run.json; the job goes back into the workspace it started in, while that workspace exists, without opening it. The workspace open before the restart opens again |
| Cannot run from the page | Generated tasks and custom agents are not supported by Run |
| Refused | Reads refused on the runs page, with why; the job was never written |
| Files | <RUNS_DIR>/<name>/: script, log, run.json, jobs/; default runs/ under the user data directory (nullius paths) |
Harbor's tasks, run by Harbor's agents through the Harbor the recipe pins. What changes:
| Stage | On Harbor |
|---|---|
| Dataset | Hub, registry, a local folder or generated tasks; one dataset per recipe |
| Agent | The pinned Harbor's agents; custom agents as module:Class; the controls. Common settings apply to every agent; an agent's --ak options sit under its tab; Extra parameters override the agent, harness, environment or verifier |
| Model | <provider>/<model> (a custom endpoint as openai/), with LLM_API_KEY, OPENAI_API_KEY and the provider's *_BASE_URL. The selected provider is kept; model IDs retain their vendor prefix |
| Where | One machine, with Docker. An agent's own vendor credentials come from Custom keys |
| Podman | A Harbor recipe runs Harbor's docker environment. On Podman alone, run Harbor 0.23.0 or later itself with its podman environment, harbor run -e podman (with podman-compose, or docker-compose and Podman's socket), and open its jobs here |
Chosen on the Runner stage, kept in the recipe's harbor field.
| Choice | The recipe keeps | Offered |
|---|---|---|
| Release | {kind: "release", version, extras} |
PyPI's five newest releases, the newest preselected |
| Repository | {kind: "repo", repo, revisionKind, revision, commit, extras} |
Any git URL, private forks included; then a branch, tag or commit |
| Rule | Detail |
|---|---|
| The pin | The commit: a branch or tag is resolved when picked. Review names it (harbor 0.24.0 · PyPI); lock.json records which Harbor ran |
| Built here | A repository's commit, once, into harbor-pins/<pin>/ beside the runs (HARBOR_PINS); a release is not built |
| Run | Through uvx, outside this application's interpreter (app/nullius/harbor/source.py) |
| Live board | app/nullius/harbor/feed.py registers Harbor's trial hooks and writes events.jsonl; a Harbor that writes the feed itself is left alone |
| No pin | Runs the harbor on PATH; Run asks for a pin first |
| New run like this | A job whose lock.json names a PyPI release opens pinned to it |
-
From a score to the trial behind it.
-
Providers, keys and hosts.
-
Ports, proofs, setups and controls in full.
Settings shared by Original and selected participants appear once under Every agent or Every memory. For LongMemEval, Reading applies to every memory; Separate notes is available only for Original with chain-of-note reading.