Browser-fetch lane backed by Cloudflare Browser Run (Kitesurf) — replace the deferred real-Chrome lane

[belikh]

Problem

Blocked fetches (bot walls, auth walls, JS-only pages) currently dead-end into the escalation queue (escalation list --status queued) and wait for a human. The original design reserved a hyperresearch-browser-fetcher lane driving the user's real Chrome; in the opencode port that agent was never built, so the queue only accumulates. Meanwhile every run hits walls a plain HTTP fetcher can't pass: SPA docs that extract as junk, soft UA/bot walls, IP-throttled endpoints.

Proposal

Make Cloudflare Browser Run the supported browser-fetch provider, defaulting to the Kitesurf engine (agent-first, stateless, runs in CF isolates — nothing executes locally):

1. New fetch provider

[web.browser_lane]
enabled = true
engine = "kitesurf"              # kitesurf | chromium (same API)
account_id = "{env:CF_ACCOUNT_ID}"
api_token_env = "CLOUDFLARE_BROWSER_RUN_TOKEN"   # token read from env, never stored

Two integration depths:

• Quick Actions lane (v1): REST one-shots — HTML/content extraction, screenshot, PDF. Maps 1:1 onto $HPR fetch and the existing --save-assets pipeline (screenshot lands as note asset exactly like today).
• CDP session lane (v2, optional): chrome-remote-interface/puppeteer-core against the wss://…browser-run/devtools/browser?browser=kitesurf endpoint for multi-step JS-heavy pages.

2. Escalation-drain semantics

On fetch failure with wall signatures (anti-bot 403, JUNK_CONTENT on script-heavy pages, IP-throttled 429), auto-retry once through the browser lane before queueing. Notes record provenance in existing frontmatter: fetch_provider: browser-run-kitesurf. Login walls, CAPTCHAs and 2FA skip the lane entirely and stay needs_human — consistent with the standing policy that challenges are always the human's.

3. Cost/run discipline

Per-run cap on browser-lane fetches (same spirit as locus source_budget), plus per-account limits honored with backoff. Engine choice recorded per note so downstream steps know extraction came from a rendered page.

Why Browser Run/Kitesurf fits this project specifically

• No local browser: page-isolated execution happens in Cloudflare's sandbox — strictly safer than automating a real Chrome profile, and removes the entire "drive the user's actual browser" hazard that got the original lane deferred
• Crawls what crawlers can't: Kitesurf exists precisely for agents needing rendered pages; Chromium flavor remains available per-request for stubborn sites (drop the browser=kitesurf param)
• Stateless by design: matches the fetcher's bursty fan-out economics; sessions are disposable
• Known non-goals respected: challenge handshakes and long authenticated sessions are explicitly unsupported upstream — which enforces our human-in-the-loop policy rather than fighting it

Acceptance criteria

• $HPR fetch --provider browser-run works standalone (extraction + screenshot asset)
• Auto-fallback: simulated builtin failure → browser lane retry → note written with correct provenance frontmatter
• Seeded 403/JUNK_CONTENT escalation drains end-to-end without human intervention
• CAPTCHA/login-wall escalations still surface as needs_human (regression test)
• Per-run lane budget enforced; token sourced from env only (never argv/config-file plaintext)
• Docs: [web.browser_lane] config reference + beta-limit caveats

Non-goals

Solving challenges, storing site credentials, automating local browsers. Related but independent: #1 (Postgres backend).

[belikh]

v1 lane built and landed (da6d681)

The Quick Actions depth (v1) of this issue is implemented, tested, and on main:

Provider (browser-run) — src/hyperresearch/web/browser_run_provider.py

• fetch() → POST /browser-rendering/content?browser=kitesurf (Kitesurf default, chromium per-config), rendered HTML → readable text, networkidle2 for SPA pages
• screenshot=True rides PNG bytes on WebResult.screenshot → the existing --save-assets pipeline
• fetch_markdown / fetch_screenshot extras
• Auth ENV-only: CLOUDFLARE_BROWSER_RUN_TOKEN + CF_ACCOUNT_ID (never argv/config plaintext — matches jupiterOS's sops→launcher-export pattern). Missing creds fail at call time (BrowserRunAuthError ⊂ ProviderAuthError) so the provider chain falls through
• BrowserRunApiError: 429/5xx chain-falls; 4xx + CF-envelope errors surface with the Browser Rendering – Edit guidance
• SSRF-guarded on both the API host and the fetched target

Escalation-drain semantics — hpr fetch

• OA rescue first (a legal copy outranks a re-fetch), then one Kitesurf retry for non-interactive bot walls before queueing
• Clean retry → saved with fetch_provider: browser-run provenance (serving-provider restamp across frontmatter / sources row / JSON)
• Login walls and interactive challenges NEVER take the lane — standing policy; captcha-family blocks (widget + act vocabulary over title AND body) escalate with reason captcha
• Lane failures degrade to the pre-P6 enqueue; lane-off default is byte-identical old behaviour

Gates: 22 provider + 7 lane + 68 chain tests green, ruff clean, mypy --strict clean.

⚠️ Live smoke BLOCKED — one human step

The jupiterOS sops slot (cloudflare_browser_run_token) holds the blind-generated placeholder from WP7-lite prep — Cloudflare's token-verify returns Invalid API Token. The lane's 401 error path is verified live.

To unblock: mint a real custom API token at dash.cloudflare.com/profile/api-tokens with Browser Rendering – Edit permission, then rotate the sops secret. Zero code changes needed after that.

Remaining for full issue close (v2, out of scope for this drop)

• CDP session lane (wss://…/devtools/browser?browser=kitesurf) for multi-step flows
• Per-run lane budget cap
• [web.browser_lane] docs page