Repository navigation
Conversation
The proxy now also rejects WebMCP calls from code in the Playwright-only arm, whose tool description tells the agent to list WebMCP tools first.
Sort the per-task table by task so the arms sit next to each other.
Adding custom tools is still sent once, since a lost response may hide a registration that happened.
yummybomb
marked this pull request as ready for review
November 9, 2026 21:40
The proxy passed the server's initialize instructions and tool descriptions through unchanged. Both tell the agent to prefer WebMCP, which that arm rejects, and the instructions name tools no approach arm provides.
A logged call with a non-string code or non-object arguments crashed kernel-mcp-verify. Harbor reads reward.json whatever the test script's exit status, so the reward the grader wrote stood without the toolset check. The Playwright-only arm now rejects any code that refers to webmcp, since matching the method names missed optional chaining, bracket access and bound methods. With an allowlist the proxy refuses lines that are not one JSON-RPC message, and a server request is no longer logged as a call's result.
Registration overwrites its namespace, so sending it again replaces whatever an earlier attempt registered. A failed setup is never retried by Harbor, and only the treatment arm registers tools. The cart check reads the page right after the agent finishes, when its last action may still be loading. The pinned runner tries such a read again; do the same here.
Bot blocks, site and Kernel errors, unreadable pages and trials that errored before grading were recorded as rewards but not shown, so they read as wrong answers. Browser calls from the proxy's log replaced the trajectory's tool-call count for every benchmark, which changed ClawBench's median calls. They are now a separate metric.
Also say what the cart page check does not cover.
Harbor records a failed agent run on its step and still runs the verifier, so the trial has rewards and was scored as an ordinary zero. A rejected credential, a rate limit or a crashed CLI now makes the trial errored and the run incomplete. Timeouts, context and output limits and refusals stay scored, as does work the agent finished before the failure.
github-latest-release is a task no tool at the pinned revision covers, so treatment should gain nothing on it. xwebmcp's runner stops an agent at a turn cap and a spend cap. Harbor's agents share neither, so the call proxy stops browser calls at the task's turn cap for every runner, and the task carries the spend limit for the report to apply. A WebMCP invocation names its tool by an opaque ref. The proxy now remembers what each list returned and logs which tool was invoked, whether it was injected or the site's own, and whether the invocation succeeded.
Arms ran as separate Harbor jobs, so the two trials of a task could run minutes or hours apart and in no particular order. A benchmark can now pair two arms: for every task and attempt, one trial of each runs back to back, in an order balanced within the task and fixed by the run ID. Each trial is a one-trial Harbor job, gathered into a job directory per arm that reports read like any other. The run manifest records the pair order and each task's limits. --unpaired keeps the old behavior.
The headline is now the pass rate on tasks whose answer is checked against the page or a pinned value, with a Wilson interval. Shape/range passes are reported apart as format-only. For paired arms the report compares each task's pairs: passes in both arms or one only with an exact McNemar test, the speedup over both-pass pairs with a sign test, and whether the two answers agree where the task defines that. The placebo task is marked and left out of the pooled row. Tool use is split into injected and native tools, tried and completed. Trials that used up the call cap or overspent are listed under what stopped them, and errors name their exception. The call proxy sees only MCP calls, so the report reads each trajectory and flags trials with a tool call that reached the task's site, the Kernel API or the task's files outside the arm's browser tools.
Codex 0.163 calls its tools from a script, so the trajectory records one exec call whose code names the tool. Every such call looked like a call outside the browser tools, which flagged any trial that opened a URL. A script that only calls browser tools is now a browser call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
public-webto the existing Harbor benchmark harness, as a standalone change againstmainwith no benchmark PR stack.Nine signed-out tasks across six sites run under three arms:
public-control: Playwright, persistent browser REPL, computer actions and native WebMCP, without injected tools.public-treatment: identical model, browser settings and instructions, with the site's eligible xwebmcp tools registered before the agent starts.public-playwright: only the Playwright browser MCP tool, with instructions prohibiting WebMCP, REPL, computer actions, raw CDP and direct HTTP.Control and treatment differ only in the injected tools, so they run as matched pairs: for every task and attempt, one trial of each runs back to back, in an order balanced within the task and recorded in the run manifest. The Playwright-only arm changes more than the tools and stays an independent Harbor job;
--unpairedruns all three that way. One task,github-latest-release, is a placebo no injected tool covers.Task definitions, tool batches and grading code come from one pinned xwebmcp main revision; task-specific development branches are not followed. Kernel MCP is also pinned. The benchmark runs here, not through xwebmcp's development-eval runner.
Model switching
--runnerselects a pinned preset fromrunners/(codex,claude, orpi), and--modeloverrides the model for every selected browser arm. The same selectors are exposed in the workflow. The default remains Codex; no browser arm needs to be duplicated for another model.For model-only comparisons, keep the Pi runner fixed and switch between IDs such as
anthropic/claude-sonnet-5andopenai/gpt-5.4. The native Claude Code preset also supports Anthropic, but comparing it against native Codex changes the agent as well as the model. Presets hold agent version and options; changing presets replaces the entire option set, preventing incompatible options from carrying over.Resolved runner/model/version/options are written into the run manifest. Reusing an experiment ID is rejected atomically instead of overwriting another model's results.
Harness and reporting
--attempts, preserving existing ClawBench defaults.webmcp. A rejected call never reaches the browser, so it is counted and does not fail the trial. The proxy also withholds the server's own usage instructions, which rank WebMCP first and name tools the arms lack, and in the Playwright-only arm cuts the WebMCP guidance from the tool's descriptions.pairs). Each paired trial is a one-trial Harbor job, gathered into a Harbor-shaped job directory per arm, so the report and the Braintrust publisher read it unchanged. The report compares pairs per task: passes in both arms or one only (exact McNemar), speedup over both-pass pairs (sign test), and answer agreement where the task defines it.tool_refto the tool a list returned and records how the invocation ended.Scope and limitations
This is an initial screening catalog, not a held-out evaluation. Answer-rule passes are distinct from page-checked passes and do not independently verify every returned fact. Only control and treatment are paired; the Playwright-only arm runs independently in the same window and is not compared with a test. At 5 attempts a single task's sign test reaches p < 0.1 only when all 5 pairs agree, so per-task results are a screen. The spend limit applies only to trials whose cost Harbor can price, and is applied by the report, not while the agent runs. Tool filtering is not an arbitrary-code sandbox, so trajectories still need review for bypasses before publishing results.
Testing
uv run ruff check .anduv run ruff format --check .uv run pytest -q: 48 passednode --test suites/public-web/*.test.mjs: 17 passed locally and on CI's Node 22bun testinreport/: 29 passed on local Bun and CI's Bun 1.3.3; TypeScript and Prettier checks passed. Fixed a filesystem-order-dependent report test.github-merged-prand loaded its pinned gradergithub-merged-pr: setup registered the pinned tools, the proxy listed only the arm's tools and rejectedlist_custom, a call without the session ID andwebmcp.listTools()from Playwright code, an injected tool and plain Playwright both returned the pinned answer, and the grader run from/testsgave reward 1 with the rejected calls counted. The cart task's page check read the live page and failed a false success claim. Through the pinned server in those images, the Playwright-only arm received no server instructions and no mention of WebMCP in its tool, the treatment arm kept its tool descriptions as sent, and a ClawBench-style run without an allowlist was unchanged.listandinvokeresponses for the pinned GitHub tool batch (a completed call, a call whose tool returned an error, and a staletool_ref) through the call proxy and the verifier; each was attributed correctly.harbor run --path <task directory>resolves to a single task, which the pair runner relies on.bench run public-web --tasks github-merged-pr,github-latest-release --attempts 2 --concurrency 2on this branch with Codex 0.163.0,gpt-6-lunaat low reasoning effort: 12 trials, all passed and finalized,harborexited 0 for every arm. The four control/treatment pairs ran back to back in the planned order, treatment completedget_pull_requestin bothgithub-merged-prpairs and invoked no tool on the placebo, and the report rendered the matched-pairs table. Two attempts per task say nothing about speed.claude-haiku-5-5got no model output: the Anthropic key was rejected. Harbor recordedAgentAuthenticationErroron each step, counted no errored trial and exited 0; the report listed all 12 trials as errored and marked the run incomplete.