[GoDiao]

Summary

The agent can now operate the built-in browser of its own session, and the user can send text and elements from a page to the chat.

This is phases 3 and 4 of the RFC (agent control on local pages, then on external sites with per-site grants), plus picking from the page, which is not in the RFC. The agent drives the view through webContents.debugger and a sidecar MCP server. There is no remote debugging port, no control of the user's own Chrome, and no new dependency.

Closes #189

Why

The browser tab from #207 is something only the user can use. The agent could not reproduce a UI bug, read the console error or check its own fix in the page the user is looking at. The design is in docs/agent-browser-control-tech-design.md; it records the decisions, the threat model and the limits.

What changed

Agent control

• Removes the Claude in Chrome integration (it depended on a stub package and could not work), so there is one browser path.
• The Electron main process opens a loopback WebSocket to the sidecar. The token travels over the sidecar's stdin, never the environment, argv or a file.
• Fourteen tools: navigate, snapshot, scroll, click, type, press_key, screenshot, console, network, back, wait_for, styles, evaluate, emulate. Pages are read as a compact snapshot of text and numbered elements with delta updates. Each view has its own call queue and a CDP method allowlist, and a session can only reach its own views.
• Page text in tool results is wrapped in a fenced untrusted block with a per-call nonce: snapshot, console, network, dialog text, element names and addresses.
• External origins ask before the agent clicks, types, presses keys, runs a script or navigates (deny, allow once, trust for the session). Trusted sites are managed under Settings → Permissions. A local page that embeds another site asks once per embedded site, and that answer is never remembered. An activity bar shows when the agent is operating and has a Stop button.
• Also: address-bar search, an empty new-tab panel, and the context card fix on desktop with the side panel open.

Picking from the page

• A new chat reference kind browser, with a fenced prompt format that is parsed back when a session is restored.
• Right-click selected text for "Add to chat" or "Copy". A selection that touches a password or card field, or lies in a subframe, is not offered.
• "Pick an element" in the toolbar: a hover outline, an orange mark that stays until the message is sent, one element screenshot, and a payload aligned with Claude Code (tag, classes, role, name, text, selector, link, key styles, HTML, neighbours, React component info). Password and card fields, and anything containing one, carry only where the element is.
• The agent's click, type and key presses are refused with a retryable user_picking error while the user is picking. A picked element is registered in the sidecar, so the agent can click the same id.

Security notes

• Everything from the page is data, not instructions. The fence is defence in depth; the real gate is the grant.
• Known and documented, not fixed here: the sidecar treats a loopback Origin as local-trusted (classifyH5Request), and the agent can now reach a loopback page itself, since local pages need no grant. The session MCP token still reaches the CLI through its environment; it is removed from process.env once read, so child processes do not inherit it, but another process of the same user may still read the CLI's initial environment.
• In plan mode the browser tools are allowed like in other modes. Local pages and trusted sites do not prompt there either; external sites still go through the grants.

Testing

• bun run lint and bun run test -- --run in desktop/: type check passes, 175 files and 2181 tests pass (4 skipped).
• bun test: the changed server test files pass in random order (seeds 1, 7 and 23, 501 tests each). The full run on my machine was not green: some tests timed out under load and e2e/business-flow.test.ts hit EADDRINUSE because another app on the machine holds ports in the range it picks from. Those files pass when run alone or are not touched here.
• Headless Electron 45 scripts, not committed: the host and sidecar against a real <webview> for picking, marks, screenshots and the hover outline (48 checks); the frame origins in inspect, the redirect chain kept in the network buffer, and a crashed page recovering after a reload (12 checks).
• Tried by hand in the dev app on macOS: the right-click menu, element picking with the hover outline and marks, and the once-only grant card.
• Not tested: Windows, the H5 page (desktop only), same-origin and cross-origin iframes for picking, and marks on a long page that keeps scrolling.

Not covered

• A hover label (tag.class) while picking. Only the mark after the pick has one.
• The snapshot comparison benchmark (listed in the design doc as a follow-up) and Windows verification.
• The classifyH5Request weakness above.

CHANGELOG.md has the entries, and the built-in browser line in README.md and README_zh.md now mentions agent control and picking.