Skip to content

[feat] assistant: Navigate to canvas content from chat links #333

Open

Description

@FayeSpica

Goal and scope

When an Agent references content on the canvas through a link in its reply, render a clickable hyperlink. Clicking it should locate the corresponding object on the canvas so the user can inspect and continue editing it.

Cover all canvas content, including text, images, video, audio, subjects, timelines, and groups. For example, clicking “Finished video” in a reply focuses the corresponding video node.

Example

An Agent creates a video, adds it to the canvas, and includes this link in its reply.

[download video](goodnight-head-replacement-assembled-768p.mp4)
Before After
The reply displays download video [blocked]. The user must find the video manually on the canvas. The reply displays download video as a clickable hyperlink. Clicking it selects and highlights the corresponding video node and moves the viewport to show it.

The filename is resolved to the canvas object through the message's execution context and asset provenance.

User flow

  1. The Agent reads or modifies project objects and includes links to relevant canvas content in its reply.
  2. The chat renders each supported reference as a hyperlink using the Agent's display text.
  3. Clicking the link reveals the target, selects and highlights it, and moves the viewport to frame it at a usable zoom.
  4. The user can inspect or edit the focused object and continue the conversation.

Reference resolution

  • Identify targets by stable project and canvas object IDs. Resolve the object's current position when clicked so moving, renaming, or grouping it does not invalidate the reference.
  • Make the link format available through shared tool receipts, CLI guidance, and runtime instructions. Support OpenCode, OpenJiuwen, Codex, dsh, and Bundled.
  • Support relative Markdown links such as [Finished video](output.mp4) when the originating execution context and asset provenance resolve them to canvas content. Persist the resolved identity with the message.
  • When a referenced asset has several canvas representations, offer a target choice unless the message already identifies a specific canvas object. A filename alone must not silently select an unrelated object.
  • Persist references with conversation history so links continue to work after reloading or reopening the project.

Interaction and errors

  • Frame the target using its current geometry, including parent-group transforms. Keep the chat available after navigation.
  • For an existing hidden object, offer an explicit reveal action through the shared canvas command and respect its current revision and permissions.
  • If content exists as a project asset but has no canvas representation, offer to show it on the canvas through the existing business operation.
  • Report deleted, missing, ambiguous, or inaccessible targets with clear English and Chinese messages.
  • For references to another project, show the destination and require an explicit project-opening action before navigation.
  • Preserve unsubmitted edits during navigation. Respect existing object locks and project access rules.
  • Keep ordinary external links working through the existing link policy.

Implementation impact

  • Extend assistant link rendering with a project-reference resolver and canvas navigation callback.
  • Reuse project queries and existing canvas selection, viewport, highlighting, and reveal operations.
  • Carry stable references and their originating context through persisted messages and runtime adapters.
  • Render supported project references as navigation links through the shared resolver.

Acceptance criteria

  • Agent links locate text, image, video, audio, subject, timeline, and group objects on the canvas.
  • Clicking a link selects and highlights the correct object and frames it at a usable zoom.
  • Links remain valid after moving, renaming, regrouping, reloading, and reopening the project.
  • A relative file link resolves to its corresponding canvas object through recorded execution context and asset provenance.
  • Duplicate filenames and multiple representations produce a clear choice when the target is ambiguous.
  • Hidden objects and assets without a canvas representation provide the appropriate reveal action.
  • Missing, deleted, inaccessible, and cross-project targets follow the defined interaction and error rules.
  • Verify streamed and completed replies across supported runtimes, with English and Chinese UI evidence.

Activity

  1. added
    area:agentAgent runtime, assistant chat and tools
    area:canvasCanvas, nodes and direct manipulation
    on Nov 15, 2026
  2. Xingkai98 commented on Nov 15, 2026

    @Xingkai98
    Contributor

    Hi @FayeSpica, I'd like to take this one — could you assign it to me? Thanks!

  3. Xingkai98 commented on Nov 19, 2026

    @Xingkai98
    Contributor

    Implemented in #389 (head 563530d8).

    What it does: assistant replies resolve Markdown destinations and @title mentions to stable canvas identities, store only stable IDs, and render them as clickable project references. Focused items call the existing workspace focus callback, hidden items reveal, and non-focusable outcomes (materialize, ambiguous, missing, cross-project) show a bilingual hint. Moving, renaming, or regrouping an item never invalidates a saved link.

    Current validation at HEAD 563530d8:

    • pytest tests/test_canvas_links.py -q → 75 passed
    • node --test tests/canvasLinkClick.test.ts tests/canvasLinkMessages.test.ts tests/canvasLinkReferences.test.ts → 13 passed
    • python3 scripts/generate_contracts.py --check → Creation contracts verified
    • npm run check:i18n → 3018 keys, 0 problems
    • python3 scripts/verify_canvas_link_mutations.py → 10/10 baseline exit 0, mutant exit 1

    Review status: an independent review found three medium issues in an earlier head — a missing-receipt target collision, sentence-final punctuation after trailing prose, and Markdown anchor href normalization. All three are fixed in 563530d8; the re-review returned 0 high / 0 medium, and 11/11 mixed-input rendering+click controls passed. The six earlier findings (wrong-title truncation, full-width sentence punctuation, mid-sentence coverage, imperative copy, reference-chain coverage, glued CJK prose) are also addressed.

    Still pending, not claimed: real browser/Streamdown sanitize execution in this environment (the linked node_modules is incomplete), full canvas selection/highlight/viewport framing, and real runtime-session coverage.

    Happy to address review comments or adjust scope.

  4. Xingkai98 commented on Nov 19, 2026

    @Xingkai98
    Contributor

    CI update: the full checks workflow is green on head 5c260e91 (13/13 checks, including browser (assistant), browser (isolated), browser (voice), frontend, python, static, and CodeQL).

    The previous red run was not a browser-environment issue: npm run build failed with TS2322 on the canvas-reference component type, which aborted Install locked dependencies and skipped every later job. That type is fixed in 5c260e91 (accept Streamdown's Record<string, unknown> & ExtraProps index-signature props), and the build/test/browser lanes now pass.

  5. Xingkai98 commented on Nov 19, 2026

    @Xingkai98
    Contributor

    Local acceptance update at HEAD 5c260e91 (no code changes; full dependency install in this worktree):

    • Build/tests: npm ci + npm run build pass; npm test = 830/831 (the one failure is an unrelated browserHost environment assertion about ambient Node PATH).
    • Browser link acceptance: passed in zh-CN and en-US for 7 outcomes — focused link, hidden/reveal, missing, materialize, cross-project, ambiguous, and an @Finished note mention. Focused/hidden clicks select the correct node id and change the React Flow viewport transform; the other outcomes show the exact bilingual hint. 16 screenshots captured.
    • Bundled OpenCode real session: a real turn through the bundled runtime with a real model completed with text OK (streaming, persistence and turn completion verified).
    • Bundled real-CLI suite: 3 passed (plain/thinking hello, watchdog); 2 errors in the mock tool-call/gateway-write case (JSONDecodeError, no asset written) — mock-model path, not the link code.
    • Codex: the direct CLI works, but an app turn fails at "读取 CLI 模型列表失败" with the installed Codex CLI 0.166.0; this is a CLI model-catalog integration issue, not this change.
    • JiuwenSwarm / dsh: not available in this environment (expected CLIs absent); not validated.

    The PR head is unchanged; the earlier CI run is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:agentAgent runtime, assistant chat and toolsarea:canvasCanvas, nodes and direct manipulationhelp wantedExtra attention is neededtype:featNew capability or behaviour

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions