Skip to content

Trace CANNBot round knowledge in managed optimization - #523

Open
jiangkuaixue123 wants to merge 3 commits into
mainfrom
codex/cannbot-round-retrieval
Open

jiangkuaixue123 wants to merge 3 commits into
mainfrom
codex/cannbot-round-retrieval

Conversation

@jiangkuaixue123

@jiangkuaixue123 jiangkuaixue123 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Governing Issue and delivery slice

Refs #522

Add opt-in CANNBot round retrieval to managed benchmark sessions and AB/BA campaigns. The user authorized the scoped implementation and a PR.

  • start --cannbot-platform <platform> freezes the source and Ascend platform. knowledge retains a local context and a separate external result under the next round.
  • knowledge-read opens a card page pinned to a hit's revision and digest. Retained search and read records contain metadata only.
  • round --knowledge-ref records explicit item and read-receipt citations. Round decisions shows both sources, reads, the candidate's declared use, and the evaluator's score and verdict. Read receipts and candidate citations are distinct.
  • The existing standalone CANNBot and paired benchmark comparison flows keep their defaults. The new setting applies to benchmarks and AB/BA campaigns. Historical retrieval backfill and accelerator acceptance are outside this slice.

Checks and dependencies

  • Passed the managed CLI regression from joint query through a pinned read, candidate submission, and Dashboard snapshot. Also checked uncited reads, empty and unavailable external responses, and failed candidate scores.
  • Passed AB/BA projection controls with and without the opt-in flag.
  • Passed a headless Chrome smoke on the generated Dashboard using the same synthetic managed session evidence. Checked round navigation and the recorded score and citation.
  • A separate service-reuse fixture hit a macOS process-cleanup permission error on both this change and unmodified main.
  • Verified the real CANNBot worker subprocess using an approved synthetic checkout. A pinned page read was traced to the candidate's score without exposing the card body.
  • An additional regression checks that Dashboard ignores extra fields even when an external record's digest matches. The final head passed 55 focused tests, the headless Chrome smoke and ruff check src tests tools. The preceding revision passed 304 tests and 287 subtests in the wider optimization, channel, Dashboard and module-boundary selection. The last revision changes only external Dashboard projection and its regression coverage.
  • Production CANNBot and Ascend NPU execution remain unverified. The integration requires the existing per-project grant.

Review follow-up

Commit 8bbc58cc preserves the three-field local knowledge return used by the Codex trace verifier. The round number remains on the retained context reference for Dashboard. All four previously failing citation regressions now pass locally. Return format review

Completed-round attribution now checks citations and the submitted context snapshot against the digest-verified observation. Every cited delivery must still match its submitted reference. Regressions cover a later receipt added to an earlier candidate, a changed snapshot, and a cited receipt replaced at the same path. Attribution review

Local checks on this revision passed 320 tests across citation, optimization, delivery, comparison and Dashboard suites, plus ruff check src tests tools. Two Playwright-dependent offline HTML cases were skipped locally. GitHub checks run on the new head.

Remaining work and Issue closure

An independent Codex reviewer checked head 226f8d0f against base c573f40a. The reviewer found the Dashboard projection issue above, rechecked the correction and reported no blocking findings on the current head. This was a code review within the implementation session. GitHub maintainer approvals and repository checks are tracked on this PR. The requester owns Issue acceptance and closure.

Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Agent-assisted-by: Codex (implementer)
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Agent-assisted-by: Codex (implementer)
zengchuang-hw
zengchuang-hw previously approved these changes Nov 19, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

首轮检视,c573f40a43ceef23ef08507fca662c95ffb65390 → 226f8d0f5fb114443ed76311879e78f24b4b6180,覆盖全部 14 个变更文件,按 #522 的软件交付范围核对。

实际追踪了 CLI → managed session → Harness → CANNBot 渠道与 worker。未开启时不调用外部渠道;搜索和读取分开,读取要求当前轮 locator 并校验 revision、digest 和 page。正文不写入留存 trace;候选只能引用已送达的本地条目或外部读取回执,搜索命中和未引用的读取均不算采用。Dashboard 读取同一记录,分别展示来源、状态、引用及 evaluator 分数,没有更改候选选择。

批准前必做:无。已静态阅读覆盖查找、读取、引用、不可用及留存最小化的用例;未运行代码、测试、浏览器或生产 grant/Ascend 集成,不据此宣称硬件验收或性能收益。

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the managed retrieval and Dashboard attribution paths on this head. Two findings below.

Comment thread src/inferrsi/harness/optimization.py Outdated
reference = {"path": path.relative_to(root).as_posix(), "sha256": fingerprint,
"kind": "delivered_context"}
if number is not None:
reference["round"] = number

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds round to the returned reference even with CANNBot off. capture_delivery.managed_envelopes() still reconstructs the exact three-field {path, sha256, kind} response, so the trace verifier can no longer match these genuine local searches. Both Ready jobs fail four citation regressions. Please keep round on the retained state reference only, or version the envelope and verify both old and new traces.

"status": facts, "evaluation": judged},
usage=record["observations"]["usage"],
measured_score=_number(assessment.get("score")),
knowledge_uses=[citations[ref] for ref in record.get("knowledge_refs") or []

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This resolves mutable state.json citations against the current session-wide context map. _load() verifies the observation file's hash but doesn't compare its recorded citations and context snapshot with state. A completed round can therefore display a later receipt as cited while its verified observation stays unchanged. Please resolve completed-round citations from the verified observation and submitted context snapshot, or check that state matches both, with a regression for later receipts.

@Dong1017 Dong1017 added the kind:task Bounded delivery task with a delegated slice label Nov 19, 2026
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Agent-assisted-by: Codex (implementer)

Copy link
Copy Markdown
Collaborator Author

@zengchuang-hw @david6666666 麻烦重新评审当前提交 8bbc58cc。这版恢复了纯本地知识查询的返回格式,并把已完成候选的知识引用绑定到评测时留存的上下文快照。原来失败的 4 项引用测试已在本地通过。 返回格式 · 引用归属

当前提交的 Dashboard 浏览器检查已通过,Python 3.10 和 3.11 的 full-check 测试仍有失败。 CI 结果

如果复核后认可当前改动,请在 GitHub 提交 Approve,正文第一行单独写 LGTM。

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

增量检视 226f8d0f5fb114443ed76311879e78f24b4b6180 → 8bbc58ccbb05fee2b0ad2f44209a7badea9732b2,承接上轮检视。本轮只有一个提交,覆盖控制器及回归测试。读取已完成候选时,现在先核对原始 observation 摘要,再核对原始引用和当时的 context 快照;已引用的 receipt 不能被后续轮次悄悄换掉。留存时仍保存 round,本地调用恢复原先的精简响应字段。已核对新检查与真实 settle 写入一致,也确认两边都没有 knowledge_refs 的旧记录不会因此失败。

批准前必做:无。合入协调:独立于本轮检视,#526 若先合,旧 apps.cannbot 兼容层不导出 SCORE_SCALE。后合入的一方应把 round_knowledge.py 及相关导入迁到 apps.integrations.cannbot,核对安装包 worker 和 managed attribution smoke;未做组合验证。 本轮未执行代码、测试或实机验证。

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining Dashboard metadata validation concern.

# The digest checks file integrity, not whether the record's
# fields are safe to disclose. Expose only known view metadata.
search = {
**{key: record.get(key) for key in (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The metadata field allowlist does not validate the values. public_value() preserves nested objects and lists, so a digest-valid retained record can still carry card content or diagnostics inside an allowed field. The existing regression only covers extra top-level fields. Please validate the expected scalar types and formats for search and read metadata before projection, and cover malformed values.

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by an agent on behalf of @Dong1017 (incremental review only; points already raised by others are not repeated).

DO NOT PASS — reviewed head 8bbc58ccbb05fee2b0ad2f44209a7badea9732b2. Two blocking points: a new one, and @david6666666's latest comment (r4631124370), which I reproduced and can extend with the exact lines, the visible consequence, and a neighbouring read path he did not mention. @david6666666's two earlier threads are addressed at this head: capture_context again returns the exact three-field {path, sha256, kind} envelope while round stays on the retained state entry, and _load() now rejects a completed round whose citations or submitted context snapshot disagree with its verified observation. Their regressions pass here.

Blocking

  1. Both full-check jobs are red on this head, because the published example page was not regenerated after the view changed.

    test_published_profile_example_matches_fixture (tests/test_profile_story_checks.py:21) fails on 3.10 and 3.11 (run 44545533153, jobs 135634949750 and 135634949698) and is the only failure in each (1 failed, 5512 passed). docs/examples/dashboard-profile-review.html still embeds the pre-change round_decision_view.html; the regenerated page differs by the new managedRetrieval() and knowledgeUses() helpers plus the render() edits. So the feature is documented and tested but the published sample no longer shows the committed view.

    Reproduced and fixed locally at this head:

    $ python tests/fixtures/dashboard_profile_review.py /tmp/regen
    $ diff /tmp/regen/dashboard.html docs/examples/dashboard-profile-review.html  → 57 changed lines, all in round_decision_view
    $ pytest tests/test_profile_story_checks.py -k published_profile_example   → 1 passed (after copying the regenerated page)
    

    Please regenerate the example in this PR so it ships with the view it illustrates.

  2. @david6666666's r4631124370 holds, and the untyped pass-through is broader than public_value() alone.

    managed_knowledge() copies the search record field by field with record.get(key) (src/inferrsi/apps/views/retrieval_projection.py:345-354), and public_value() only recurses and redacts strings — it never checks that count is an integer, state a string, or elapsed_ms a number. Replacing count with {"text": "<card body>"} on a digest-valid retained record puts that object straight into the Dashboard payload; the string is not redacted because it is not a URL, address or credential. The same shape applies to page on a read receipt (:357-358) and to revision, card_sha256 and page in the citation projection (:359-363).

    What the view then does with it, which is why I would treat this as blocking rather than cosmetic: round_decision_view.html:52-54 prints result.count through raw() (a non-scalar renders as [object Object]) and compares result.count > result.items.length, which is silently false against an object. The "some hits lack retainable identifiers" note therefore disappears rather than failing loudly, so a malformed record degrades into a plausible-looking round.

    One related point he did not mention: managed_knowledge() reads each context with json.loads((root / path).read_text("utf-8")) and never calls the module's own _read(), so it skips the inside() containment check, the symlink refusal and the 4 MiB cap that every other record read in this file goes through (:73-87). I confirmed {"path": "../outside.json", ...} resolves and is parsed, while _read() correctly returns unsafe_path. _store_knowledge() only ever writes knowledge/<kind>-<uuid>.json, so the normal path cannot reach this; but this PR explicitly claims the property "even an imported record with a matching digest must not widen the view" (tests/test_optimization_session.py, final block of test_cannbot_query_and_read_trace_to_candidate), and these two gaps are on the same accepted-record path that guarantee is meant to cover. Suggested fix: type-check the allowlisted scalars before projection (as r4631124370 asks) and route the record read through _read(), then extend the existing regression to cover a malformed count/page and a context path outside the root.

Non-blocking

  • I checked the context-snapshot binding for a round I can make fail: a missing or changed capture raises Delivered knowledge context changed in _load() before any projection runs, later receipts appended to an earlier candidate and a receipt replaced at the same path both raise Round knowledge attribution differs… / Cited knowledge context differs…. For older sessions I checked the base rather than relying on the incremental PASS: evaluate_round already wrote both knowledge_refs and contexts into the pending record at a3e33851 (optimization.py:387,389), so the new comparison is comparing a field to itself for pre-change rounds and does not add a new failure surface. Nothing to add beyond that.
  • The local return shape, the --knowledge off rejection, the current-round restriction on knowledge-read, and metadata-only retention all match the task: Trace CANNBot knowledge through optimization rounds and the Dashboard #522 acceptance wording. The combined query being redacted before the local search runs is documented in docs/optimization-benchmarks.md, so I do not read it as a defect.

Scope checked: the 226f8d0f..8bbc58cc delta (optimization.py, tests/test_optimization_session.py) and the full a3e33851..8bbc58cc diff across the 14 source/doc/test files; #523 body and both bot/human comment threads; #522 goal and acceptance; the two earlier @david6666666 threads. Ran tests/test_optimization_session.py, tests/test_project_optimization.py and tests/test_round_decisions_browser.py at this head (all pass), test_profile_story_checks.py::test_published_profile_example_matches_fixture at this head and at base a3e33851 (fails only at this head), and the two projection probes above in an isolated worktree on Python 3.12. Not run: the full suite, real CANNBot grant, Ascend hardware, or the Playwright-dependent browser cases (playwright is not installed here; CI reports these skipped too). PR-head CI state read from run 44545533153.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants