Add retail-cart-providers: retail-cart on competing providers' browsers

.github/workflows/_run.yml

@@ -65,6 +65,14 @@ jobs:
       CLAWBENCH_JUDGE_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
       CLAWBENCH_JUDGE_MODEL: deepseek/deepseek-v4-pro
       CLAWBENCH_JUDGE_API_TYPE: openai-completions
+      # Other browser providers, for remote-browser arms.
+      BROWSERBASE_API_KEY: ${{ secrets.BROWSERBASE_API_KEY }}
+      BROWSERBASE_PROJECT_ID: ${{ secrets.BROWSERBASE_PROJECT_ID }}
+      STEEL_API_KEY: ${{ secrets.STEEL_API_KEY }}
+      STEEL_PROJECT_ID: ${{ secrets.STEEL_PROJECT_ID }}
+      BROWSER_USE_API_KEY: ${{ secrets.BROWSER_USE_API_KEY }}
+      HYPERBROWSER_API_KEY: ${{ secrets.HYPERBROWSER_API_KEY }}
+      ANCHOR_API_KEY: ${{ secrets.ANCHOR_API_KEY }}
       BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
       BRAINTRUST_PROJECT: ${{ vars.BRAINTRUST_PROJECT }}
     steps:

.github/workflows/ci.yml

@@ -26,6 +26,8 @@ jobs:
         run: uv run bench prepare kernel-mcp-clawbench --arms codex-kernel-mcp,claude-kernel-mcp --tasks v2-1134-chapter-finder-redcross --out "$RUNNER_TEMP/prepared"
       - name: Prepare the static retail-cart arm
         run: uv run bench prepare retail-cart --arms codex-repl --out "$RUNNER_TEMP/prepared-retail-cart"
+      - name: Prepare the retail-cart provider arms
+        run: uv run bench prepare retail-cart-providers --out "$RUNNER_TEMP/prepared-retail-cart-providers"
 
   report:
     runs-on: ubuntu-latest

.github/workflows/retail-cart-providers.yml

@@ -0,0 +1,45 @@
+name: Retail cart flows on each provider's browser
+
+on:
+  workflow_dispatch:
+    inputs:
+      arms:
+        description: Comma-separated arm names (empty runs every arm in benchmarks/retail-cart-providers.toml)
+        required: false
+        default: ""
+        type: string
+      tasks:
+        description: Comma-separated task IDs, or all
+        required: false
+        default: all
+        type: string
+      attempts:
+        description: Trials per task and arm
+        required: false
+        default: "10"
+        type: string
+      concurrency:
+        description: Concurrent trials per arm. Every arm's task containers run on the one runner.
+        required: false
+        default: "4"
+        type: string
+      playwright_mcp_version:
+        description: "@playwright/mcp version to install (empty installs the toolset's pinned version)"
+        required: false
+        default: ""
+        type: string
+
+permissions:
+  contents: read
+
+jobs:
+  benchmark:
+    uses: ./.github/workflows/_run.yml
+    secrets: inherit
+    with:
+      benchmark: retail-cart-providers
+      arms: ${{ inputs.arms }}
+      tasks: ${{ inputs.tasks }}
+      attempts: ${{ inputs.attempts }}
+      concurrency: ${{ inputs.concurrency }}
+      toolset_ref: ${{ inputs.playwright_mcp_version }}

README.md

@@ -91,6 +91,7 @@ A run of two arms on two tasks (Red Cross and Indeed) produced this summary:
 | `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` | Codex and Claude Code arms; the `site-config` configurator's model |
 | `PURELY_MAIL_API_KEY`, `PURELY_MAIL_DOMAIN` | ClawBench creates a disposable [PurelyMail](https://purelymail.com) mailbox on your domain for every task, so agents can sign up and receive confirmation emails. |
 | `CLAWBENCH_JUDGE_BASE_URL`, `CLAWBENCH_JUDGE_API_KEY`, `CLAWBENCH_JUDGE_MODEL`, `CLAWBENCH_JUDGE_API_TYPE` | ClawBench's LLM judge, which checks the intercepted request against the task. Any OpenAI-compatible endpoint works with `CLAWBENCH_JUDGE_API_TYPE=openai-completions`; we use `deepseek/deepseek-v4-pro` through OpenRouter. See [ClawBench](https://github.com/TIGER-AI-Lab/ClawBench) for details. |
+| `BROWSERBASE_API_KEY`, `BROWSERBASE_PROJECT_ID`, `BROWSER_USE_API_KEY`, `STEEL_API_KEY` (plus `STEEL_PROJECT_ID` for an organization-scoped key), `HYPERBROWSER_API_KEY`, `ANCHOR_API_KEY` | Arms on those providers' browsers (`environment = "remote-browser"` or `"kernel-vm-remote-browser"`); checked before a run that includes them |
 | `BRAINTRUST_API_KEY`, `BRAINTRUST_PROJECT` | Optional: publishing results |
 
 ## Running in GitHub Actions
@@ -121,7 +122,7 @@ ClawBench's primary score is the lenient reward. `kernel_mcp_valid` records whet
 
 ### `retail-cart`
 
-Guest add-to-cart tasks on 16 retail and industrial-supply sites with heavy bot protection. Each task starts on the site's home page and asks the agent to put one item in the site's cart as a guest, by whatever route works, and show the cart.
+Guest add-to-cart tasks on 28 retail and industrial-supply sites, in three tiers: 2 `easy` controls with little bot protection, 13 `standard` protected sites, and 13 `hard` sites that block most automated browsers. Run one tier with `--suite-arg tiers=hard`. Each task starts on the site's home page and asks the agent to put one item in the site's cart as a guest, by whatever route works, and show the cart.
 
 | Arm | Agent | Browser |
 |---|---|---|
@@ -139,6 +140,28 @@ Run several attempts per task to get a pass rate per site:
 uv run bench run retail-cart --attempts 10
 ```
 
+### `retail-cart-providers`
+
+The `retail-cart` tasks on each browser provider's browser. Within an agent, every arm uses the same tools ([Playwright MCP](https://github.com/microsoft/playwright-mcp) attached over CDP) and the same task container, so arms differ only in the browser. The `codex-pw-*` arms (Codex, `gpt-6-luna`) run the container on the runner's Docker (`environment = "remote-browser"`). The `claude-pw-*` arms (Claude Code, `claude-sonnet-5-5`) run it on a Kernel VM used only as compute (`environment = "kernel-vm-remote-browser"`), which needs no local Docker; the Kernel arms still get a separate browser session, reached over the network like every other provider's. Each provider gets its own anti-bot settings:
+
+| Provider | Arms | Browser |
+|---|---|---|
+| Kernel | `codex-pw-kernel`, `claude-pw-kernel` | `stealth: true` (managed ISP proxy and CAPTCHA solver) |
+| Kernel, macOS | `claude-pw-kernel-macos` | as above, plus the macOS fingerprint (`fingerprint: {os: macos}`, closed beta) |
+| Browserbase | `codex-pw-browserbase`, `claude-pw-browserbase` | CAPTCHA solving and its residential proxies. Advanced Stealth is left out: Browserbase rejects it outside the Enterprise plan |
+| Browser Use | `codex-pw-browser-use`, `claude-pw-browser-use` | its US proxy |
+| Steel | `codex-pw-steel`, `claude-pw-steel` | residential proxy and CAPTCHA solving; 15-minute sessions, the Launch plan's maximum |
+| Hyperbrowser | `codex-pw-hyperbrowser`, `claude-pw-hyperbrowser` | stealth, proxy and CAPTCHA solving |
+| Anchor | `codex-pw-anchor` | its US proxy, extra stealth and CAPTCHA solving |
+
+The start page, the agent and the verifier each connect to the browser separately, so sessions are created to outlive a CDP disconnect (Browserbase's `keepAlive`, Hyperbrowser's `keepAlive=true` on the CDP URL, Anchor's idle timeout). A Steel key scoped to the whole organization also needs `STEEL_PROJECT_ID`. Scoring is the same as `retail-cart`.
+
+```bash
+uv run bench run retail-cart-providers --attempts 10 --concurrency 4
+```
+
+All arms' task containers run on one machine, so keep `--concurrency` low on a GitHub runner.
+
 ## Extending
 
 ### Add an arm
@@ -162,6 +185,8 @@ steering = true
 
 `toolset_args` are passed to the toolset; `kernel-mcp-local` takes `toolsets`, the Kernel MCP toolsets to enable: `playwright` (the default), `repl`, `computer`, or several. `agent_kwargs` are passed to the Harbor agent.
 
+An arm with `environment = "remote-browser"` runs the task container on local Docker instead of in the Kernel browser's VM, and its `config` names a `provider` from `harness/providers.py` and that provider's session create `params`. The environment writes the browser's CDP URL to `/run/browser/cdp-url` in the container; the `playwright-mcp` toolset connects to it.
+
 `config` takes any [`POST /browsers`](https://kernel.sh/docs/api-reference/browsers/create-a-browser-session) parameter, such as `stealth`, `proxy`, `region`, `extensions`, or `viewport`. Browsers are always headful.
 
 ### Add a configurator

arms/claude-pw-browser-use.toml

@@ -0,0 +1,16 @@
+# Claude Code driving a browser-use browser through Playwright MCP over CDP, the
+# same agent, tools and task container as every claude-pw-* arm. The container
+# runs on a Kernel VM used only as compute; the browser is a separate session.
+# Browser: Browser Use's cloud browser with its managed US proxy.
+agent = "claude-code"
+model = "claude-sonnet-5-5"
+agent_version = "2.1.238"
+toolset = "playwright-mcp"
+environment = "kernel-vm-remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "browser-use"
+params = { proxyCountryCode = "us", timeout = 30 }

arms/claude-pw-browserbase.toml

@@ -0,0 +1,17 @@
+# Claude Code driving a browserbase browser through Playwright MCP over CDP, the
+# same agent, tools and task container as every claude-pw-* arm. The container
+# runs on a Kernel VM used only as compute; the browser is a separate session.
+# Browser: Browserbase's managed proxies and CAPTCHA solving. Advanced stealth is left
+# out: Browserbase rejects it outside the Enterprise plan ("Verified mode").
+agent = "claude-code"
+model = "claude-sonnet-5-5"
+agent_version = "2.1.238"
+toolset = "playwright-mcp"
+environment = "kernel-vm-remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "browserbase"
+params = { keepAlive = true, timeout = 1800, region = "us-east-1", proxies = true, browserSettings = { solveCaptchas = true } }

arms/claude-pw-hyperbrowser.toml

@@ -0,0 +1,16 @@
+# Claude Code driving a hyperbrowser browser through Playwright MCP over CDP, the
+# same agent, tools and task container as every claude-pw-* arm. The container
+# runs on a Kernel VM used only as compute; the browser is a separate session.
+# Browser: Hyperbrowser's stealth, managed US proxy and CAPTCHA solving.
+agent = "claude-code"
+model = "claude-sonnet-5-5"
+agent_version = "2.1.238"
+toolset = "playwright-mcp"
+environment = "kernel-vm-remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "hyperbrowser"
+params = { useStealth = true, useProxy = true, proxyCountry = "US", solveCaptchas = true, timeoutMinutes = 30 }

arms/claude-pw-kernel-macos.toml

@@ -0,0 +1,17 @@
+# Claude Code driving a kernel browser through Playwright MCP over CDP, the
+# same agent, tools and task container as every claude-pw-* arm. The container
+# runs on a Kernel VM used only as compute; the browser is a separate session.
+# Browser: Kernel's anti-detection mode plus its macOS fingerprint (closed beta): macOS
+# user agent and platform, Apple GPU renderer.
+agent = "claude-code"
+model = "claude-sonnet-5-5"
+agent_version = "2.1.238"
+toolset = "playwright-mcp"
+environment = "kernel-vm-remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "kernel"
+params = { stealth = true, timeout_seconds = 1800, fingerprint = { os = "macos" } }

arms/claude-pw-kernel.toml

@@ -0,0 +1,16 @@
+# Claude Code driving a kernel browser through Playwright MCP over CDP, the
+# same agent, tools and task container as every claude-pw-* arm. The container
+# runs on a Kernel VM used only as compute; the browser is a separate session.
+# Browser: Kernel's anti-detection mode (`stealth`): managed ISP proxy and CAPTCHA solver.
+agent = "claude-code"
+model = "claude-sonnet-5-5"
+agent_version = "2.1.238"
+toolset = "playwright-mcp"
+environment = "kernel-vm-remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "kernel"
+params = { stealth = true, timeout_seconds = 1800 }

arms/claude-pw-steel.toml

@@ -0,0 +1,17 @@
+# Claude Code driving a steel browser through Playwright MCP over CDP, the
+# same agent, tools and task container as every claude-pw-* arm. The container
+# runs on a Kernel VM used only as compute; the browser is a separate session.
+# Browser: Steel's managed residential proxy and CAPTCHA solving. 15-minute sessions: the
+# Launch plan's maximum (other arms get 30), so a long trial can lose its browser.
+agent = "claude-code"
+model = "claude-sonnet-5-5"
+agent_version = "2.1.238"
+toolset = "playwright-mcp"
+environment = "kernel-vm-remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "steel"
+params = { useProxy = true, solveCaptcha = true, timeout = 900000 }

arms/claude-repl.toml

@@ -0,0 +1,16 @@
+# Same as codex-repl, but with Claude Code driving the browser only through
+# Kernel MCP's browser REPL, with a fixed stealth browser.
+agent = "claude-code"
+model = "claude-sonnet-5-5"
+agent_version = "2.1.238"
+toolset = "kernel-mcp-local"
+
+# Claude Code's own web tools would let the agent answer without the browser.
+[agent_kwargs]
+disable_web_search = true
+
+[toolset_args]
+toolsets = ["repl"]
+
+[browser]
+config = { stealth = true }

arms/codex-pw-anchor.toml

@@ -0,0 +1,24 @@
+# Codex driving an Anchor browser through Playwright MCP over CDP, with
+# Anchor's anti-bot settings: its US proxy, extra stealth and CAPTCHA solving
+# (both need the proxy).
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "anchor"
+
+[browser.config.params.session]
+proxy = { active = true, type = "anchor_proxy", country_code = "us" }
+# Minutes. The idle timeout covers the gaps between the start page, agent and
+# verifier connections.
+timeout = { max_duration = 45, idle_timeout = 30 }
+
+[browser.config.params.browser]
+extra_stealth = { active = true }
+captcha_solver = { active = true }

arms/codex-pw-browser-use.toml

@@ -0,0 +1,14 @@
+# Codex driving a Browser Use cloud browser through Playwright MCP over CDP, with
+# Browser Use's managed US proxy.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "browser-use"
+params = { proxyCountryCode = "us", timeout = 45 }

arms/codex-pw-browserbase.toml

@@ -0,0 +1,22 @@
+# Codex driving a Browserbase browser through Playwright MCP over CDP, with
+# Browserbase's anti-bot settings: CAPTCHA solving and its residential proxies.
+# Advanced Stealth is left out: Browserbase rejects it outside the Enterprise
+# plan ("Verified mode is only available on the Enterprise plan"). keepAlive keeps the session across the
+# start page, agent and verifier connections.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "browserbase"
+
+[browser.config.params]
+keepAlive = true
+timeout = 2700
+proxies = true
+browserSettings = { solveCaptchas = true }

arms/codex-pw-hyperbrowser.toml

@@ -0,0 +1,14 @@
+# Codex driving a Hyperbrowser browser through Playwright MCP over CDP, with
+# Hyperbrowser's anti-bot settings: stealth, proxy and CAPTCHA solving.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "hyperbrowser"
+params = { useStealth = true, useProxy = true, solveCaptchas = true, timeoutMinutes = 45 }

arms/codex-pw-kernel.toml

@@ -0,0 +1,14 @@
+# Codex driving a Kernel stealth browser through Playwright MCP over CDP, the
+# same agent, tools and container as the other providers' codex-pw-* arms.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "kernel"
+params = { stealth = true, timeout_seconds = 1800 }

arms/codex-pw-steel.toml

@@ -0,0 +1,15 @@
+# Codex driving a Steel browser through Playwright MCP over CDP, with Steel's
+# anti-bot settings: residential proxy and CAPTCHA solving.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "steel"
+# timeout is in milliseconds: 15 minutes, the Launch plan's maximum.
+params = { useProxy = true, solveCaptcha = true, timeout = 900000 }

benchmarks/retail-cart-providers.toml

@@ -0,0 +1,17 @@
+# The retail-cart tasks on each provider's browser, with the same agent,
+# Playwright MCP tools and task container, so arms differ only in the browser.
+suite = "retail-cart"
+arms = [
+  "codex-pw-kernel",
+  "codex-pw-browserbase",
+  "codex-pw-steel",
+  "codex-pw-hyperbrowser",
+  "codex-pw-anchor",
+  "codex-pw-browser-use",
+  "claude-pw-kernel",
+  "claude-pw-kernel-macos",
+  "claude-pw-browserbase",
+  "claude-pw-browser-use",
+  "claude-pw-hyperbrowser",
+  "claude-pw-steel",
+]

benchmarks/retail-cart.toml

@@ -1,5 +1,6 @@
 # Guest add-to-cart flows on bot-protected retail and industrial-supply sites,
 # driven through the browser REPL, with a stealth browser and with the config
-# registry's per-site settings, and with a fixed per-site table.
+# registry's per-site settings, and with a fixed per-site table, plus the
+# stealth arm with Claude Code in place of Codex.
 suite = "retail-cart"
-arms = ["codex-repl", "codex-repl-site-config", "codex-repl-pinned"]
+arms = ["codex-repl", "codex-repl-site-config", "codex-repl-pinned", "claude-repl"]

configurators/pinned/configure

@@ -25,7 +25,12 @@ def main() -> int:
     site = tomllib.loads(task["config"]).get("metadata", {}).get("site")
     entry = TABLE["sites"].get(site)
     if entry is None:
-        raise SystemExit(f"No pinned configuration for site {site!r}")
+        # Sites added after the table was measured run on the arm's static config.
+        json.dump(
+            {"browser": {}, "steering": None, "provenance": {"site": site, "pinned": False}},
+            sys.stdout,
+        )
+        return 0
     browser = dict(entry["browser"])
     browser["proxy_create"] = {"type": entry["proxy"], "config": {"country": "US"}}
     persona = entry.get("persona")