Skip to content

Agentic Mode: talk to your crew from the iPhone; agents run xBot and the Mac - #49

Open
MasterYoav wants to merge 49 commits into
masterfrom
agentic
Open

MasterYoav wants to merge 49 commits into
masterfrom
agentic

Conversation

@MasterYoav

Copy link
Copy Markdown
Owner

Agentic Mode

The iPhone gets a second mode. Settings › Mode: Developer / Agentic. Developer is today's app. Agentic is a Telegram-like chat with your crew: HeadMaster pinned on top, one thread per agent. The agents control xBot and the Mac through xBot's own MCP server.

Spec: docs/superpowers/specs/2026-10-13-agentic-mode-design.md.

iPhone

  • Chats: the crew list with still pixel faces, previews, unread marks, "typing…" and search.
  • Threads:
    • accent bubbles with Markdown and code;
    • one activity line that updates as the agent works, folding to "Did N things ›" when it's done;
    • cards for screenshots (full-screen viewer), approvals (Allow / Don't Allow), hand-offs, finished work, notes and lessons.
  • Composer: grows with the text; on-device dictation; photos (≤1600 px JPEG); long-press for Copy, Retry, or Edit and resend.
  • Performance:
    • the last 200 messages per thread;
    • pictures decoded at display size, with a 12 MB cache;
    • no timers or world animation.

Mac

  • Threads: one chat per agent (chats.thread, schema 7), in full access, in the Inbox. Photos are saved and attached by path. A lost session continues in a new thread that starts with recent messages.
  • XBot --mcp: a headless MCP server (JSON-RPC over stdio). It reaches the running app over a 0600 Unix socket, with a per-turn token.
    • 28 tools: xBot (projects, chats, crew, hand-offs, automations, notes, settings, worlds, xbot_ask, xbot_remember, xbot_lessons) and the Mac (mac_screenshot via ScreenCaptureKit, mac_ui via Accessibility, click, type, keys, scroll, open, apps, quit).
  • Attached per CLI:
    • Claude Code: --mcp-config, plus a PreToolUse hook (XBot --hook) that also covers its own shell;
    • Codex: -c mcp_servers.xbot.*;
    • Copilot: --additional-mcp-config;
    • Antigravity: agy mcp add, once.
  • RiskRules + ApprovalCenter: delete, send, push, purchase, quit and System Settings changes ask on the phone (the Mac when no phone is attached). No answer within 10 minutes counts as Don't allow.
  • Experience (Harness2's cue → lesson):
    • agents call xbot_remember;
    • their best lessons (12 at most, 1,500 characters) go into their next turns;
    • low-confidence, unused lessons are dropped after 30 days;
    • an Experience section in the agent editor shows and edits them.
  • Permissions: Accessibility and Screen Recording are in the onboarding Files step, with the floating guide, and asked again when a tool needs them.

Tests

  • Package: 688 tests pass. New suites: RiskRules, XBotTools, ToolHost, Threads, AgenticWire, RemoteHostAgentic, Experience, ApprovalCenter.
  • Live, phone simulator → sandboxed Mac → real Codex:
    • testAgenticMode: HeadMaster replies pong;
    • testAgenticRealTools: a real mac_screenshot card arrives on the phone; a mac_quit approval comes to the phone, is refused, and the agent carries on;
    • testAgenticCards;
    • testAgenticActivity;
    • Developer Mode testNewChatRunsOnTheMac.
  • Live, real Claude Code turn: xbot_crew, mac_apps and TextEdit control; the hook blocks git push when refused.

Gaps

  • Codex, Copilot and Antigravity have no per-run hook, so risky commands in their own shell aren't caught; only xBot's tools are checked, and the agents are told to ask first.
  • Copilot and Antigravity attachment are unit-tested only.
  • Photo picking and dictation weren't driven live.
  • Pending approvals expire rather than being cancelled when a turn is stopped.

…me reminder, existing libraries count as done
…isk Access detection and polling, behind a probe
… › Welcome Tour; Home's Finish setting up row
…dy centred, close-only guide panel); a debug-only fake command to look at the install sheet
One clock per scene (30 fps TimelineView that stops under the motion gate and
never starts under Reduce Motion, which draws the final frame), easing as
functions of time, a stage that scales a 720x420 composition to its room, the
founding crew's looks, a mock xBot window, the world's balloon and name tag.
Welcome (the crew walks into the village at dawn and greets; the mark settles),
Crew (HeadMaster on top, a CTO jumps onto the tier below, Forge and Hawk walk
in; a task travels down and Done comes back up), Worlds (five living
postcards) and Ready (the crew gathered, a warm light, one hop each).
A mock xBot window: a folder drops in as a project, a tab opens, tool rows fold
into one line and the reply streams in; a plan unfolds, a step is dragged up
and the steps tick off; the inspector slides in with changes, a diff line
turning green, a terminal running swift test and a pull request's checks.
Three cards flip in (a note writing itself, connectors switching on, an
automation snapping together under a clock); Desktop, Documents and Downloads
turn from locked to allowed in turn; a Mac draws its pairing code, an iPhone
slides up, scans it and a pulse travels between them.
DEBUG only. XBOT_SCENE_SIZE=WxH sets the scene's size (default 720x420). The
RootView change is the marked #if DEBUG branch at the top of body.
…, unknown parts tolerated), ApprovalCenter, ExperienceStore
…s, decay of stale guesses, forget/restore/edit; kept beside the library (tests never touch the person's) and decayed when it opens
…ou've learnt", and turns with xBot's tools end with the reflection (instructions(for:toolsAttached:))
…full access in the Inbox, photos saved as JPEGs ≤ 1600 px and attached by path, a lost session continues in a new thread
…AgenticDigest; RemoteHost+Agentic (approvals answered in the device's name; DEBUG demo cards)
…ols, PreToolUse hook via --settings), Codex (-c mcp_servers.xbot.*), Copilot (--additional-mcp-config); the token in the CLI's environment
… when learnt and uses; reword in place, forget with Undo (toast in the sheet), Forget everything… with a confirmation; a lessons chip by the name, and the count on the crew card
…y xbot_* and mac_* tool through Workspace and MacControlling, RiskRules then approvals, xbot_ask; approval cards in the chat; tools, full access and guidance on thread turns; LiveMac, MacControlAccess, the PreToolUse hook's decision
…on quit), the permission guide for Accessibility and Screen Recording; debug-only XBOT_DEBUG_AGENTIC_PROMPT end-to-end turn
…nt's question while its turn waits, Settings › Agents › xBot tools (every chat, permissions, Antigravity), Accessibility and Screen Recording rows in the onboarding's files step
… and the hook's honest gap (security), the flow (architecture), per-CLI attachment with sources (providers), modules and tests, CLAUDE.md invariant 5, changelog
…lert in front was invisible to the agent; found in the live check)
…told to ask twice); window buttons named in mac_ui; the debug turn can answer xbot_ask
…rew, threads with bubbles, one activity line, Did N things, screenshot/approval/hand-off/lesson cards, dictation and photos
…Cards (approval, screenshot viewer, hand-off)
…ed), text either side of a tool as paragraphs, the agent's own notices folded into one line, back to the bottom when a turn starts or a card arrives
…e testAgenticRealTools waits for its own reply

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant