Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
## [Unreleased]
### Added
- Added Kernel as a managed remote browser runtime with live view and downloaded replay recordings. Thanks to @[rgarcia](https://github.com/rgarcia).
- Added a `--browser-runtime kernel` mode to the Harbor adapter that runs each task against one Kernel cloud browser, exposing only a credential-free CDP bridge to the agent, and finalizes the replay and deletes the browser during verification.

## [0.9.2] - 2026-08-18
### Added
Expand Down
23 changes: 21 additions & 2 deletions docs/harbor.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ uv run clawbench-harbor-adapt \

## 2. Wire up the judge

Harbor's verifier calls the same judge ClawBench uses. Export the four variables once, then forward them into each run with `--ve`:
Harbor's verifier applies both ClawBench V2 judge rubrics to every intercepted request. `reward` and `reward_lenient` use the public leaderboard's no-explicit-contradiction rubric; `reward_strict` requires the payload to demonstrate complete fulfillment. The two judge calls run concurrently against the same request and use the model configured below. Export the four variables once, then forward them into each run with `--ve`:

```bash
export CLAWBENCH_JUDGE_BASE_URL="https://your-judge-provider.example/v1"
Expand Down Expand Up @@ -108,6 +108,25 @@ uvx --from harbor==0.15.0 harbor run \
--jobs-dir ./harbor-jobs/hermes-deepseek-flash
```

## Kernel browser runtime (control arm)

By default each Harbor trial runs Chromium inside its own container. Pass `--browser-runtime kernel` to the adapter to run the same tasks against one Kernel cloud browser per task instead:

```bash
uv run clawbench-harbor-adapt \
--output-dir ./harbor-datasets/clawbench-v2-kernel \
--browser-runtime kernel \
--browser-runtime-options '{"stealth": true}' \
--task-ids v2-1134-chapter-finder-redcross \
--overwrite
```

During task setup, the environment creates exactly one Kernel browser and replay, starts the ClawBench runtime server against it, and exposes only the local credential-free CDP bridge (`http://127.0.0.1:7878`) to the agent — the Kernel API key is never visible to the benchmark agent. Session identity and cleanup metadata land in `/my-info/kernel_browser.json`. During verification the provider replay is finalized, `recording.mp4` is downloaded into `/data`, and the browser is deleted (idempotently, including failure paths via a setup trap).

Generated tasks register a pinned Playwright MCP package (`@playwright/mcp@0.0.79`) pointed at the CDP bridge, so Harbor's stock Claude Code and Codex agents drive the Kernel browser with native Playwright MCP tool calls — structurally identical to ClawBench's native Claude/Codex harnesses.

Export `KERNEL_API_KEY` (and optionally `KERNEL_BASE_URL` for non-production gateways) before `harbor run`; no extra flags are needed.

## Making it fast

A full V2 sweep is 129 containerized browser sessions, each capped by the task's `time_limit`. Serial, that is a very long night. What actually moves the needle, in order:
Expand Down Expand Up @@ -140,7 +159,7 @@ uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2-smoke \

Each converted task directory carries its own `environment/` (Chromium, the ClawBench recorder/interceptor, noVNC, runtime helper scripts), a `run/` step with `instruction.md`, the original `task.json`, the `eval-schema.json`, and a verifier under `tests/`. It deliberately contains **no ClawBench-native harness** — Harbor installs and runs whatever agent you pass to `-a` inside the task container.

Scoring is the same two-stage rule as the native runner: the interceptor must catch a request matching the task schema, and the judge must agree the payload fulfills the instruction.
Scoring uses the same two-stage rule as the native runner: the interceptor must catch a request matching the task schema, then the verifier emits both the public lenient reward and the conservative strict reward. Harbor uses the lenient result as the primary `reward` metric and retains both verdicts and reasons in `clawbench-result.json`.

## Troubleshooting

Expand Down
Loading
Loading