Conversation
Wire tool calling end to end: a per-room /ai session spawns a long-lived ACP agent and tells it (via session/new mcpServers) to treat Robrix itself as an MCP server, so its model can call host tools instead of only answering prompts. Agent plumbing (a2app-agent): - mcp: minimal MCP server protocol (stdio framing, initialize/tools/list/ tools/call dispatch, Tool trait) plus McpServerConfig, the shape agents expect for a stdio MCP server - AcpClient advertises the configured servers in session/new; new start_backend_with_mcp for sessions (the generation pipeline's start_backend is unchanged, advertising none) Session host (src/a2app/ai): - server: session-scoped Unix-socket MCP server, torn down with the session - bridge: robrix --mcp-bridge --socket <path> headless relay child agents spawn as the MCP server; main.rs intercepts it before any UI starts - tools: launch_splash_app and send_message as thin Tool impls over the AiHost trait, registered in one place - session: AiSession owns one room's tool server, its real AiHost (SessionHost marshals tool calls from serve threads onto the UI thread and blocks only there), and the agent pointed back at its own socket Runtime wiring (src/a2app/runtime.rs): - /ai slash command starts/stops a room's session and prompts it; replies post back into the room, requests queue while the agent is busy - tool calls execute each event pass: launch_splash_app drives the existing generation pipeline and answers the waiting call with the installed-app summary on completion (running it docked in the room); send_message posts through the room send path - generation cancel/failure resolve pending tool calls instead of hanging Tests: MCP protocol, socket round-trips, bridge relay, full transport smoke test, host rendezvous, and a wire test proving session/new carries the tool server to a spawned agent. Known follow-up: the octos backend registers MCP servers from its own config file rather than session/new, so octos sessions run without host tools until that path injects a per-session config; claude-code-acp and ROBRIX_AGENT_CMD agents get the tools today. Signed-off-by: gterzian <2792687+gterzian@users.noreply.github.com>
The octos ACP handler now connects per-session mcpServers and registers their tools (see the octos commit on acp-embedding-seam); Robrix's side of that channel was sending servers the agent never saw: - acp_client: session/new server objects now carry the schema-required "env": [] field. Without it the agent's VecSkipError deserializer silently dropped the server before its handler saw it (verified on the wire). Param tests + the canned-wire test assert env rides the wire. - lib: the embedded branch forwards the session's tool-server config to EmbeddedOctos::start instead of dropping it (iOS still passes none: the agent cannot exec the relay child there). Stale doc comment fixed. - octos_embedded: build_agent translates the Robrix McpServerConfigs to octos config (command/args, 600s tool-call timeout for the minutes-long app generation) and builds the agent through the factory's build_with_mcp, the same per-session mcpServers path the child serves. Cargo: octos-agent/-core/-cli patched to the sibling ../octos checkout (acp-embedding-seam) so this branch consumes the patched octos before it is pushed to project-robius/octos; drop the patch and repin Cargo.lock once that commit lands upstream.
Writes the octos config that makes /ai run with NO API key: provider 'scenario' (the deterministic offline model) plus a placeholder ANTHROPIC_API_KEY so Robrix's setup gate passes. Preserves any existing config fields; honors OCTOS_CONFIG_DIR and the config-candidate precedence octos/Robrix actually use.
The Claude Code bridge is only reachable from the child-backend paths (not(embedded) arms of start_backend_with_mcp, blocker and runtime), so under an embedded build the helper was compiled but never used, warning every time. Gate the function itself; embedded builds now build warning-free.
Replace the /ai slash-command model with AI Rooms: ordinary Matrix rooms, listed and opened like any other, that are backed by a Robrix agent session. The room is the session's transcript — member messages drive the agent, and its output is stored as rs.robius.robrix.ai_reply state events so it never loops back as input and survives restarts. - Add-room screen gains "start a new AI room": creates a private encrypted room with an ai_room marker in initial_state (no unmarked window); app.rs navigates to it on creation. - runtime: checks the marker when a room is first opened — only after the room's state has synced (a fresh room's state lags sliding sync; own creations are recorded immediately from the create response) — and attaches a per-room session. Member messages after the persisted forwarding cursor (room account data) are forwarded as prompts, priming a (re)started session with a short transcript preamble. - Agent output goes through Robrix's own MCP tools (send_message, launch_splash_app) or a final turn text; a turn that already spoke via send_message drops its redundant trailing text, so one turn = one ai_reply, rendered as a timeline card (ai_room_events.rs). - /ai removed from slash commands and the input bar; event preview for ai_reply added. Fix the MCP tool bridge on macOS: accept() inherits the listener's O_NONBLOCK there, so the blocking serve loop read EAGAIN between requests and dropped the connection (Linux never hit it) — sessions silently lost their host tools. New tests/mcp_rmcp.rs regression harness drives the real rmcp client octos uses; the hand-written client in mcp_transport.rs could not reproduce it. Offline testing via the octos scenario provider (sibling ../octos change): deterministic per-session sequence through the real tools — 1st message pongs via send_message, 2nd launches the minimal "this is a mini-app" sample, then always pong. Sample app avoids glass.* widgets and Fill heights, which render blank in the host's Fit-mounted Splash. See the new "AI Rooms" section in a2app/README.md. Signed-off-by: gterzian <2792687+gterzian@users.noreply.github.com>
Signed-off-by: gterzian <2792687+gterzian@users.noreply.github.com>
Signed-off-by: gterzian <2792687+gterzian@users.noreply.github.com>
…vents An AI room's agent now runs host-managed (octos `hosted` profile): its native tools (shell/bash, files, memory, search) are gone, so every tool call flows through Robrix's MCP bridge and is capability-gated exactly like a mini-app. Everything the agent does while a turn is live is reflected in the room as rendered state events: - `ai_tool_call` — one row per call: posted "Started" when the agent picks the tool, rewritten "Done" (ok/summary) when Robrix executes or refuses it, including permission-prompt refusals. - `ai_activity` — "thinking…" marker when the model starts reasoning, plus rows for turn errors and the session stopping. - `ai_reply` unchanged for the final text, now carrying a `send_message` receipt too. Rendering: new AiEventTimelineCard widget wired into room_screen, room-list previews, and the non-a2app stubs. Also: send_message's tool description now instructs the agent to always use Markdown formatting and matrix.to links in its replies. Signed-off-by: gterzian <2792687+gterzian@users.noreply.github.com>
New `post_room_message` tool posts an ai_reply state-event card into another joined room, so the message renders like the agent's own-room replies and can never loop back as input. Permission is granted per room: the first post into each room prompts the user, an Allow unlocks exactly that room (stored as a durable per-subject room allowlist in the PermissionStore), and a Deny only blocks that room — the group is reset to Ask rather than killing the capability. Worker plumbing (PostToRoom / PostToRoomResult) mirrors the granted-read path; tool outcomes ride the existing tool-call state rows and receipts. Signed-off-by: gterzian <2792687+gterzian@users.noreply.github.com>
When the homeserver rejects an ai_reply state write with 403/M_FORBIDDEN (account power below the room's state_default, usually 50 = Moderator), report it as what it is instead of echoing the raw server error. The full explanation now reaches the model's post_room_message tool result (so its reply says exactly why a post failed), the room's own ai_reply-failure popup, and the worker log. Shared by both posting paths through post_reply's new friendly_state_post_error helper, which detects the error via the same client_api_error_kind == ErrorKind::Forbidden check used elsewhere. Signed-off-by: gterzian <2792687+gterzian@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.