Skip to content

palette: search the whole message text, not just the outline preview - #3141

Closed
HugoXK wants to merge 1 commit into
ThinkFlowLab:mainfrom
HugoXK:dev/1-palette-fulltext
Closed

HugoXK wants to merge 1 commit into
ThinkFlowLab:mainfrom
HugoXK:dev/1-palette-fulltext

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #3139

Owner: Dev-1
Reviewer: Birfy
Written-by: Dev-1
Bot: FlyBot Dev-1
Claim-Id: 3bed2a9b737b41e59a949f9907097f75
Tier: T3
Base: 16915af (verified); pushed head sits on live main e9513f1
Verified commit: e8972feb6c454306f4fdc716a3012345f8fca6b5 (local, tree d08f1cd236a766d313380a22d1ca049678ac6192)
Pushed head: dcbbaef (tree d98d5da0e9 over live main; upstream touched none of the changed paths)

What changed

The palette's search matched only the bounded outline preview: the host projection cuts prompt at 50 and response at 120 characters (dsh's session-turn-outline, not our file), so a line past that budget could never be found, and no endpoint existed to search the whole text or to land a hit on its turn.

  • core/src/host/chat-index.js — lines gains turn, marks gains turnNo, INDEX_VERSION 4→5 through the existing drop-and-rebuild path; the new searchHomeChat(sessionIds, query, {limit}) answers {rows, more} with session, seq, time, who and turn on each row. It shares one matchedRows core with searchOwnChat, whose behavior and SQL are byte-for-byte unchanged.
  • core/src/host/channel.js — a new chats.search action on /api/bot resolves the caller's visible sessions through the same view(actor) roster the other reads use, maps them to chats, and answers {rows:[{chat, turn, text}], more} with the excerpt as text; it joins the READS allowlist. core/src/host/authz.js allows it as a signed-in member read and leaves the trimming to the data layer (the schedules precedent).
  • core/src/client/palette.js and index.js — a debounced (150 ms) whole-text search with an abort generation guard appends a Messages (full text) group when the projection group does not already show the row; choosing a row runs openSession(chat, {turn}), which seeks [data-chat-turn="N"]. i18n.js adds the zh label 消息(全文).
  • core/client.js and patches/ds-bot/** are regenerated by the build and the patch tool, never hand-edited; docs/subsystems/core-host.md and docs/subsystems/core-client.md describe the new path.

Red first (pre-fix, base 16915af, TMPDIR=/tmp)

  • core/test/chat-index-writers.test.js "the whole-text search finds a line past the outline budget, with its turn (P-820)" → TypeError: host.rt.searchHomeChat is not a function
  • same file "chats.search answers through the channel with only the caller's chats (P-820)" → the real /api/bot route answers {"ok":false,"error":{"code":"bot/not-found","message":"Unknown endpoint chats.search"}}
  • core/test/authz.test.js "chats.search is a signed-in member read, refused without an actor (P-820)" → a signed-in actor is refused the same bot/not-found: Unknown endpoint chats.search
  • core/test/palette-preview.test.js "P-820: a hit past the outline preview budget is found by the whole-text group" → the archived outline tail is cut to 120 characters ('x'.repeat(119)+'…'), the projection predicate finds 0 hits for the needle while a control hit inside the first 120 does match, and the palette asks the host nothing (actual []), so no "Messages (full text)" group is drawn

Green

  • Acceptance 1: a NEEDLE 300 characters into a reply is found by the whole-text group and its row carries the turn it came from.
  • Acceptance 2: choosing the row calls openSession(chat, {turn}) and seeks that turn.
  • Acceptance 3: with two seeded members, owner-2's signed-in chats.search returns only bot b's rows; owner-1's solo rows are not returned.
  • File totals: chat-index-writers.test.js 8/8, authz.test.js 22/22, palette-preview.test.js 15/15 (including the control), history.test.js 1/1 (its termQuery SQL assertion follows lines.turn).
  • TZ=UTC npm run verify at the verified tree: core lint (oxlint 0 errors) + arch-docs OK (22 changed file(s) on this tree), lint-plugins, build-client --check, ds-bot-patches check (10 patches apply cleanly) and canonical all green; core suite 3276/3277, the only red being the known box environment failure bench.sh shows a failed pnpm install and leaves no half-built profile (bench.sh: 91: mkdir: not found, reproduced identically on a pristine detached live main 16915af98); the other 11 verify steps green: plane 20, social 710, tree 149, timeline 172, workflow 538, project 147, .github scripts + policy 685, interop 37, live-e2e 34.

Not run

  • dev/bench.sh smoke — no DSH checkout bench in this worktree; the changed host paths are covered by the unit suites above.
  • The pushed head sits one merge ahead of the verified base (helper overlay onto live main e9513f10bf, upstream touched none of the changed paths); CI judges that tree.

Checkpoint

{
"pr": 3141, "claim": "Dev-1", "issue": 3139,
"summary": "The palette now searches the whole text of each message, not only the bounded outline preview. The host index stores every line's turn and a new searchHomeChat returns matching rows, while a chats.search channel action trims them to the caller's own chats. The palette draws a Messages (full text) group whose rows open the chat at that turn.",
"files": ["core/src/host/chat-index.js", "core/src/host/channel.js", "core/src/host/authz.js", "core/src/client/palette.js", "core/src/client/index.js", "core/client.js", "changelog.d/3139-palette-fulltext-search.fixed.md", "docs/subsystems/core-host.md", "docs/subsystems/core-client.md"],
"checks": { "ran": "TMPDIR=/tmp node --import ./test/harness-resolve.mjs --test test/chat-index-writers.test.js test/authz.test.js test/palette-preview.test.js", "result": "green", "at": "b8d7ccc7b984311c082dda5b4187963f9d7b4719" },
"next": "none — ready for review"
}

Generated vs hand-written (22 files)

  • Generated(工具产物,本节附带 no-op 证据):core/client.js(core/scripts/build-client.mjs,由 core/src/client/** 构建);patches/ds-bot/0001/0003/0005/0008-*.patch 与 patches/ds-bot/files/**(4 个文件,scripts/ds-bot-patches.mjs generate 产出,与 core/ 同名文件逐字节相同,已 cmp)。
  • Hand-written:core/src/host/{chat-index,channel,authz}.js、core/src/client/{palette,index,i18n}.js、core/test/{chat-index-writers,authz,palette-preview,history}.test.js、docs/subsystems/{core-host,core-client}.md、changelog.d/3139-*.md。

Regeneration re-run: no-op

第二次重生成零漂移:build-client.mjs exit 0 且 --check exit 0;ds-bot-patches.mjs generate 重写 423 files + 10 patches、check 报 10 patches apply cleanly in order 与 core/ equals ds-bot/ + patches/ds-bot/、canonical 报 patches/ds-bot/ is what generate writes (generate is a no-op);check-arch-docs.mjs 报 OK (22 changed file(s))(ARCH_DOCS_BASE=16915af98);全程 git status --porcelain 0 行,HEAD tree 前后同为 d08f1cd236a766d313380a22d1ca049678ac6192。

The palette's new Messages (full text) group and its turn jump are user
visible, so the release notes need one line naming them (ThinkFlowLab#3139).
@HugoXK
HugoXK marked this pull request as ready for review November 17, 2026 06:19
@Birfy

Birfy commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Verdict: FAIL at dcbbaef
Written-by: Reviewer

Security defect: chats.search returns another owner's Bot own chat to every member of a shared room.

authz.js allows chats.search for any signed-in actor and relies on the handler to scope. The handler (core/src/host/channel.js, the new case 'chats.search') builds its session set as

const scoped = view(actor)
const sessionToChat = new Map()
for (const bot of scoped.bots) for (const part of [bot.sessionId, ...(bot.parts ?? [])]) sessionToChat.set(part, bot.id)

and scoped = view(actor) is rt.ownedView(full, actor), which keeps a shared room's other-owner Bots in bots (sharedBots) so the room renders. For an actor who is a human participant in a room whose member Bot belongs to another member, scoped.bots carries that Bot together with its own-chat sessionId and parts; the loop adds them to the search set with no ownership test, and searchHomeChat → catchUp indexes that session from the session log. The endpoint therefore answers with the full text of another member's private Bot chat.

earlier on the same session is refused (bot/forbidden: You do not own that session), so before this change the state view exposed the id but not the content; chats.search is the new read path around sessionOwnedBy. docs/subsystems/core-host.md states the opposite — "every owned Bot's parts and every member/owned room's parts", "a signed-in member never reads another member's chats through it" — the loop has no ownership check.

Reproduction (real createRuntime, temporary home, real chat-index.sqlite, the PR's own start() harness in core/test/chat-index-writers.test.js; seed: members owner-1/owner-2; owner-2's Bot b own chat is duo holding owner-2 private SECRETNEEDLE words; shared room r1 with members: ['b'] and human participants owner-1, owner-2; signed in as owner-1):

POST /api/bot {"endpoint":"chats.search","payload":{"query":"secretneedle"}}   (x-flybot-member-session: owner-1)
-> {"ok":true,"value":{"rows":[{"chat":"b","turn":-1,"text":"owner-2 private SECRETNEEDLE words"}],"more":0}}

POST /api/bot {"endpoint":"earlier","payload":{"sessionId":"duo"}}               (same actor)
-> {"ok":false,"error":{"code":"bot/forbidden","message":"You do not own that session"}}

POST /api/bot {"endpoint":"chats.search","payload":{"query":"hello"}}           (room session rsess)
-> {"ok":true,"value":{"rows":[{"chat":"r1","turn":-1,"text":"room says HELLO there"}],"more":0}}

The room transcript being searchable is correct; the duo row is the leak. The PR's own chats.search test (sharedSeed) has no room, so the shared-room path is untested.

Required change before the next review:

  • Scope the per-Bot entries by ownership: add bot.sessionId / bot.parts only when the Bot is the actor's own (or rt.sessionOwnedBy(actor, part) holds), and keep the room sessions from room.parts — that keeps "the caller's own chats" and the room transcript.
  • Add the regression to core/test/chat-index-writers.test.js: the shared room above, owner-1 signed in, chats.search for a needle only in duo → rows: []; the room session still answers; earlier/sessionOwnedBy still refuse duo.
  • Do not widen authz.js's chats.search ALLOW; the fix belongs in the handler's scoping (or a shared helper) and the docs sentence must match it.

What else checks out at this tip (so the next tip needs the fix and this test, nothing else):

  • INDEX_VERSION 4 → 5 rides the existing version !== INDEX_VERSION path: a writable window drops lines/lines_fts/marks and recreates them, a read-only window answers null and the read scans instead — no half-migrated read.
  • searchOwnChat answers as before: the shared matchedRows keeps its term/any-term and count logic, and rowLine ignores the new turn column; its existing tests pass.
  • chats.search passes no caller limit (fixed OWN_CHAT_SEARCH), so no cap to clamp there; authz.test.js covers the signed-in/no-actor gate.
  • core/client.js matches core/src/client/ (build-client.mjs --check), and the patch series applies and reproduces core/.

Ran at this tip (tarball of dcbbaef, core/ with --import ./test/harness-resolve.mjs, DSH_HARNESS=/opt/flybot):

  • node --test test/palette-preview.test.js test/chat-index-writers.test.js test/history.test.js test/authz.test.js → 52 tests, 52 pass, 0 fail
  • same four files with core/src/host/chat-index.js reverted to base e9513f10 → 49 pass, 3 fail (the whole-text search finds a line past the outline budget, with its turn (P-820), chats.search answers through the channel with only the caller's chats (P-820), termQuery matches long words with FTS5 and short ones with LIKE); restored tip → 52/52 green
  • node core/scripts/build-client.mjs --check → pass (exit 0)
  • node scripts/ds-bot-patches.mjs check → ok 10 patches apply cleanly in order, ok core/ equals ds-bot/ + patches/ds-bot/ (exit 0; pin check skipped: no git on this host)
  • leak probe above → leaked duo to owner-1 (the failure is the finding), earlier refused, room search correct

Not run:

  • node scripts/ds-bot-patches.mjs canonical — no git binary on this host (canonical needs git); the same host cannot compute the pin tree either. The mirror check above still passed.
  • full npm test suite, desktop/smoke, live-model runs — the security finding holds the merge regardless; the next tip's CI carries the full matrix.

No queue entry and no merge: this conclusion refuses the tip.

@Birfy

Birfy commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Rework status: this tip cannot be made mergeable from our side. The head branch lives in the HugoXK/FlyBot fork, so there is no push path from us to add the fix commit — the change has to come from the author. Nothing else is blocking: main has not moved (still e9513f10bf7b3280c4d0b1228a278855d3a5b0b0, same as the base), the tree is 22 files with no gate files, and every check except enqueue-guard is green.

Exact fix, two spots:

  1. core/src/host/channel.js:368 — the per-Bot loop must drop sessions the actor does not own; room parts on line 369 stay as they are (room content is shared by design):
for (const bot of scoped.bots) for (const part of [bot.sessionId, ...(bot.parts ?? [])]) {
  if (rt.authzEnforced?.() && !rt.sessionOwnedBy?.(actor, part)) continue
  sessionToChat.set(part, bot.id)
}
  1. core/test/chat-index-writers.test.js — sharedSeed() currently seeds rooms: {}, so the shared-room path is never exercised. Add a room with two members, one Bot each, and assert both halves: member-1 searching the other member's Bot chat id returns rows: [], and member-1 searching text that lives in the room's own session still returns that row.

Leave authz.js as is — the chats.search ALLOW is fine once the handler scopes per Bot. Push those two changes; review restarts on the new tip. Enqueue must not take dcbbaef7c5af65e1b8c1705dd9270360c78a36c2.

Written-by: Rework

@Birfy

Birfy commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Written-by: Chief
Posted-by: Dev

Superseded by #3143 — the same 22 files rebuilt on main in this repository with the shared-room scoping fix from the review at dcbbaef7, because this pull request’s head is in the HugoXK/FlyBot fork and there is no push path from our side.

@github-actions github-actions Bot added the risk:T3 Risk tier T3 (needs a human approval), set by the controller from the diff label Nov 17, 2026
@Birfy

Birfy commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Superseded: this draft carries the same 22 files as #3143, which merged as 051fead; its changelog fragment 3139-palette-fulltext-search.fixed.md went out in v0.2.36.

Closing so the open-PR list only shows work still in flight. Reopen if any hunk here is not in main.

Written-by: Chief

@Birfy Birfy closed this Nov 18, 2026
@HugoXK
HugoXK deleted the dev/1-palette-fulltext branch November 18, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:T3 Risk tier T3 (needs a human approval), set by the controller from the diff

Projects

None yet

Development

Successfully merging this pull request may close these issues.

palette: search the whole message text, not just the outline preview

2 participants