Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions docs/external-channels.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,10 +198,12 @@ An external result is **not locally verified**. Cite it as
`cannbot-knowledge @ <revision>` and say so. It does not participate in a `PASS`
verdict.

When a receipt is added (not in this release), it will hold the locator, the
card's `card_sha256` and the index revision — never card text, never
`frontmatter`, never a `local_path`. A digest proves which card was cited without
redistributing it.
[Managed round retrieval](optimization-benchmarks.md#include-cannbot-in-the-round)
records CANNBot search metadata and pinned read receipts. A receipt identifies
the search, locator, index revision, card SHA-256 and requested page. The
authorized read returns text directly to the caller. The campaign retains
metadata only. Candidate attribution requires an explicit citation when
submitting a round.

Whether CANN OSL v2.0 permits even a local read-only copy is an open question for
the organisation; see #476 Q1. This page documents the mechanism, not a legal
Expand All @@ -213,6 +215,7 @@ conclusion.
| --- | --- |
| `cannbot enable / disable / describe / search / read` | implemented |
| `find` integration with gating, budgets, per-channel states | implemented |
| Managed benchmark and AB/BA round query, read and Dashboard attribution | implemented |
| MindIE as a second channel | not yet; `apps/mindie/` has the reader, not the adapter |
| `cannbot serve` (stdio MCP proxying `discover` / `search` / `read_card`) | not yet |
| `campaign import-cannbot` (receipts into evidence) | not yet |
Expand Down
1 change: 1 addition & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ Effect classes used below:
| Inspect the package catalog | CLI `knowledge search` / `knowledge show`; catalog API | Read-only | Filtered inventory and existing catalog scripts. Uses the installed package's tier/repo/backend/scenario filters and IDs |
| Prepare model-aware engine research | CLI `knowledge research`; Python `research_knowledge` | Read-only | Independent model/framework/task/modality/mechanism queries over the same package catalog; source-grounded known profiles, deduplicated matches and explicit target annotations; no engine/hardware qualification |
| Discover/read packaged or custom skills | CLI `skills list` / `show` / `resource` / `reference` | Read-only | Metadata-first listing; `show` loads one named instruction body; `resource`/`reference` resolve paths for your file reader without loading or executing them |
| Retrieve and cite external round knowledge | CLI `benchmarks session knowledge / knowledge-read`, AB/BA `campaign knowledge / knowledge-read` | Local evidence-writing | Requires an explicit `--cannbot-platform` at start and an approved project grant. Retains search and read metadata. Cite a delivered receipt with `round --knowledge-ref`. See [round retrieval](optimization-benchmarks.md#include-cannbot-in-the-round) |
| Collect selected knowledge | CLI `collect`; Python `KnowledgeReader.collect` | Read-only | Batch selection and budgeted assembly by query or exact IDs, with revisions and omissions. The only accepted explicit `--relevance-method` value is `find-lexical-v1` |
| Read exact material/resources | CLI `read` / `resource`; Python `KnowledgeReader.read` / `KnowledgeReader.resource` | Read-only | Returns the requested revision or an explicit unavailable/stale error; never substitutes newer bytes |
| Export legacy cited material | CLI `context` | Read-only | Cited package export for manual use with a coding agent; not a conversation store and not a full Collect equivalent |
Expand Down
35 changes: 35 additions & 0 deletions docs/optimization-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,41 @@ selection method and the external Agent configuration are caller-controlled
variables. A retained delivery establishes what was supplied. Model consumption
and access to material outside this session require their own observation.

### Include CANNBot in the round

Run from the project with an [approved CANNBot grant](external-channels.md).
Add `--cannbot-platform <platform>` at `benchmarks session start` or AB/BA
`campaign start` to freeze the external source and an explicit Ascend
platform. Omit the option for a local-only session. The paired
`benchmarks compare` conditions keep their existing behavior.

`knowledge` queries both sources for the next round. It returns the local
context and a separate `external` result, with its own ranking, channel state
and `reference.path`. Use task keywords in the query and keep private details out.
InferRSI applies its standard evidence redaction to the combined query before
searching and retaining it. Use a locator from this round's external result:

```bash
inferrsi benchmarks session knowledge-read --session <session-dir> \
--search-ref <external.reference.path> --locator <card-locator> --page 1
```

The read verifies the revision and digest recorded by the search. Card text
returns to the authorized CLI caller. The session retains only the source,
locator, revision, digest, page and search reference. Read additional pages
with `--page`. Use the equivalent `campaign knowledge-read` command for AB/BA.

At candidate submission, repeat `--knowledge-ref` for each delivery you used.
Pass the read's `reference.path` for an external card page, or
`<local-reference.path>#<item-ref>` for a local item with delivered resources.
A local capture path alone cites the whole delivery. You can cite an earlier
delivery from the same session. Search result paths are ineligible for citation.

Open [Dashboard](dashboard.md) → **Round decisions** to see the queries and
channel states before submitting a candidate. Completed attempts show the
agent's explicit citations alongside the candidate ID, verdict and measured
score. A search hit or read receipt alone does not record a citation.

## Submit a round and use its feedback

```bash
Expand Down
58 changes: 58 additions & 0 deletions src/inferrsi/apps/cannbot/round_knowledge.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
"""Connect managed round knowledge to the approved read-only CANNBot worker."""

import math
import re

from ...harness.records.evidence import relative_location
from ...knowledge.channels import ChannelRegistry, search_channels
from . import SCORE_SCALE, CannbotChannel, project

SCORE_SCALES = (SCORE_SCALE, "knowledge-search.v5")


class CannbotRoundKnowledge:
def search(self, query, *, settings, limit, timeout):
channel = CannbotChannel()
results = search_channels(
ChannelRegistry([channel]), query, backend=settings["backend"],
platform=settings["platform"], limit=limit, timeout=min(8.0, timeout),
)
result, = results
revision = result.status.revision
if not isinstance(revision, str) or not re.fullmatch(r"[a-f0-9]{40}", revision):
revision = None
# Select metadata explicitly. The channel view also contains prose, open
# commands and worker diagnostics, none of which belongs in a receipt.
items = [
{"locator": hit.locator, "card_sha256": hit.card_sha256,
"score": hit.score if math.isfinite(hit.score) else None}
for hit in result.items
if relative_location(hit.locator) == hit.locator
and re.fullmatch(r"[a-f0-9]{64}", hit.card_sha256)
]
return {
"state": result.status.state, "revision": revision, "items": items,
"count": len(result.items), "elapsed_ms": result.elapsed_ms,
"score_scale": result.score_scale if result.score_scale in SCORE_SCALES else None,
}

def read(self, locator, *, revision, card_sha256, page, timeout):
try:
result = project.read(
locator, expected_revision=revision, expected_sha256=card_sha256, page=page,
timeout=min(15.0, timeout),
)
except project.ChannelFailure as exc:
reason = f"CANNBot read failed ({exc.state}); check the grant or search again"
raise ValueError(reason) from None
if (
result.get("state") != "ok" or result.get("locator") != locator
or result.get("card_sha256") != card_sha256 or result.get("revision") != revision
or result.get("page") != page or not isinstance(result.get("pages"), int)
):
raise ValueError("CANNBot read did not match the recorded search")
if page > result["pages"]:
raise ValueError("CANNBot page is outside the card")
if not isinstance(result.get("text"), str) or not result["text"].strip():
raise ValueError("CANNBot returned an empty card page")
return {key: result[key] for key in ("text", "page", "pages")}
35 changes: 21 additions & 14 deletions src/inferrsi/apps/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,25 @@ def _print(value: object) -> None:
print(json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False))


def _optimization_session(args, *, task_source=None):
from ...harness.optimization_session import session_command

settings = None
provider = None
if getattr(args, "cannbot_platform", None):
settings = {"channel": "cannbot-knowledge", "backend": "ascend",
"platform": args.cannbot_platform}
if args.session_action in ("knowledge", "knowledge-read"):
from ..cannbot.round_knowledge import CannbotRoundKnowledge

provider = CannbotRoundKnowledge()
return session_command(
args, task_source=task_source, retain_context=capture_context,
knowledge_reader=KnowledgeReader() if args.session_action == "knowledge" else None,
external_knowledge=settings, external_provider=provider,
)


def _fixed_attempt_command(args) -> int:
from ...harness.errors import PublicError
from ..api import read_attempt_view
Expand Down Expand Up @@ -361,20 +380,14 @@ def main(argv: list[str] | None = None) -> int:
_print(compare_command(args))
return 0
if args.benchmark_action == "session":
from ...harness.optimization_session import session_command
from ..delivery import after_final

task_source = None
if args.session_action == "start":
from ...benchmark.registry import task_resources

task_source = Path(str(task_resources(args.task)))
result = session_command(
args, task_source=task_source,
retain_context=capture_context,
knowledge_reader=(KnowledgeReader()
if args.session_action == "knowledge" else None),
)
result = _optimization_session(args, task_source=task_source)
_print(after_final(args, result))
return 2 if result.get("state") == "blocked" else 0
if args.benchmark_action == "run":
Expand Down Expand Up @@ -629,15 +642,9 @@ def main(argv: list[str] | None = None) -> int:
_print({"context_directory": str(path), "reviewed_by_caller": False})
return 0
if getattr(args, "session_action", None) is not None:
from ...harness.optimization_session import session_command
from ..delivery import after_final

result = session_command(
args,
retain_context=capture_context,
knowledge_reader=(KnowledgeReader()
if args.session_action == "knowledge" else None),
)
result = _optimization_session(args)
_print(after_final(args, result))
return 2 if result.get("state") == "blocked" else 0
from ...harness.records.campaign_records import (
Expand Down
12 changes: 10 additions & 2 deletions src/inferrsi/apps/cli/arguments.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,9 +99,13 @@ def _optimization_actions(stages, *, project=False):
start.add_argument("--condition", default="current-agent" if project else None,
required=not project)
start.add_argument("--knowledge", choices=("on", "off"), default="on")
from ..cannbot.project import PLATFORMS

start.add_argument("--cannbot-platform", choices=PLATFORMS,
help="Opt into CANNBot retrieval on the declared Ascend platform")
start.add_argument("--evaluation-prefix", default="", help="Quoted allocator argv prefix")
start.add_argument("--python", default=sys.executable)
actions = ("status", "round", "final", "recover", "knowledge")
actions = ("status", "round", "final", "recover", "knowledge", "knowledge-read")
if project:
actions += ("baseline", "refresh-references", "resume")
for name in actions:
Expand All @@ -113,7 +117,7 @@ def _optimization_actions(stages, *, project=False):
stage.add_argument("--hypothesis", required=True)
stage.add_argument("--strategy", help="Agent-declared optimization strategy")
stage.add_argument("--knowledge-ref", action="append",
help="Delivered context path cited by this round; repeatable")
help="Delivered path, path#item-ref, or CANNBot receipt; repeatable")
if name == "final":
stage.add_argument("--reason", help="Why the caller stopped an up-to-budget search")
if name in ("round", "final"):
Expand All @@ -139,6 +143,10 @@ def _optimization_actions(stages, *, project=False):
type=Path,
help="Investigation receipt proving interrupted resources were cleaned",
)
if name == "knowledge-read":
stage.add_argument("--search-ref", required=True, help="Recorded CANNBot search path")
stage.add_argument("--locator", required=True)
stage.add_argument("--page", type=int, default=1)
if name == "knowledge":
stage.add_argument("--query", required=True)
stage.add_argument(
Expand Down
14 changes: 14 additions & 0 deletions src/inferrsi/apps/views/dashboard_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -694,8 +694,10 @@ def _managed_dashboard(root: Path, campaign: dict) -> dict:
from inferrsi.harness.optimization import read_records

from .optimization_story import build_story
from .retrieval_projection import managed_knowledge

plan, state, summary = read_records(root)
retrievals, citations = managed_knowledge(root, state["contexts"])
rows, rounds, decisions = [], [], {}
for index, record in enumerate(state["rounds"], 1):
candidate_id = record["candidate_id"]
Expand Down Expand Up @@ -730,12 +732,23 @@ def _managed_dashboard(root: Path, campaign: dict) -> dict:
artifacts={**row["artifacts"], "execution": facts, "assessment": judged,
"status": facts, "evaluation": judged},
usage=record["observations"]["usage"],
measured_score=_number(assessment.get("score")),
knowledge_uses=[citations[ref] for ref in record.get("knowledge_refs") or []

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This resolves mutable state.json citations against the current session-wide context map. _load() verifies the observation file's hash but doesn't compare its recorded citations and context snapshot with state. A completed round can therefore display a later receipt as cited while its verified observation stays unchanged. Please resolve completed-round citations from the verified observation and submitted context snapshot, or check that state matches both, with a regression for later receipts.

if ref in citations],
)
row.pop("_comparison_signature")
rows.append(row)
rounds.append({"round": index, "candidates": [candidate_id],
"decision": {"candidate_id": candidate_id, "outcome": decision}})
decisions[(index, candidate_id)] = decision
pending = state["pending"]
pending_candidate = None
if pending and pending["kind"] == "round":
pending_candidate = {
"round": len(rows) + 1, "candidate_id": pending["candidate_id"],
"knowledge_uses": [citations[ref] for ref in pending.get("knowledge_refs") or []
if ref in citations],
}
displayed_summary = _managed_summary(summary)
displayed_summary["final_reference"] = (
file_ref(root, root / state["final"]["path"] / "observation.json", "final_assessment")
Expand All @@ -744,6 +757,7 @@ def _managed_dashboard(root: Path, campaign: dict) -> dict:
"campaign_id": sanitized_text(campaign["campaign_id"]), "synthetic": False,
"rounds": rounds, "rounds_state": "present", "corrupt_round_lines": [],
"candidates": rows, "optimization_session": displayed_summary,
"managed_retrievals": retrievals, "pending_candidate": pending_candidate,
"integration": {"state": "connected" if rows else "not_connected",
"stage_declaration": "missing", "external_records": 0,
"collection": {"state": "", "observed_at": ""}},
Expand Down
55 changes: 55 additions & 0 deletions src/inferrsi/apps/views/retrieval_projection.py
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,61 @@ def read_searches(root: Path, campaign_id, rows, retrievals):
return results


def managed_knowledge(root, contexts):
"""Project managed deliveries and explicit citation targets without card bodies."""
queries, external, citations = {}, {}, {}
for reference in contexts:
path, kind = reference["path"], reference["kind"]
record = json.loads((root / path).read_text("utf-8"))
number = reference.get("round")
if kind == "delivered_context":
delivered = {item["ref"] for item in record["contexts"] if item["resources"]}
items = [
{**{key: item.get(key) for key in ("ref", "title", "revision")},
"delivered": item["ref"] in delivered}
for item in record["items"]
]
if number is not None:
request = record["request"]
queries[path] = {
"round": number, "path": path,
"query": request.get("relevance") or request.get("query"),
"items": items,
}
citation = {"source": "local", "retrieval_round": number}
citations[path] = {**citation, "ref": path}
for item in items:
if item["delivered"]:
citations[path + "#" + item["ref"]] = {
**citation, "ref": item["ref"], "revision": item["revision"],
}
elif kind == "external_search":
# The digest checks file integrity, not whether the record's
# fields are safe to disclose. Expose only known view metadata.
search = {
**{key: record.get(key) for key in (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The metadata field allowlist does not validate the values. public_value() preserves nested objects and lists, so a digest-valid retained record can still carry card content or diagnostics inside an allowed field. The existing regression only covers extra top-level fields. Please validate the expected scalar types and formats for search and read metadata before projection, and cover malformed values.

"channel", "backend", "platform", "state", "revision",
"score_scale", "count", "elapsed_ms",
)},
"path": path, "reads": [],
"items": [{key: hit.get(key) for key in ("locator", "score", "card_sha256")}
for hit in record["items"]],
}
external[path] = search
queries[record["context_ref"]]["external"] = search
elif kind == "external_read":
receipt = {key: record[key] for key in ("locator", "page")}
external[record["search_ref"]]["reads"].append(receipt)
citations[path] = {
"source": record["channel"], "retrieval_round": number,
"ref": record["locator"], "revision": record["revision"],
"card_sha256": record["card_sha256"], "page": record["page"],
}
return public_value(list(queries.values())), {
path: public_value(citation) for path, citation in citations.items()
}


def project_knowledge(root, data):
"""One projection for both layouts, offline export and live refresh."""
story = data.get("story") or {}
Expand Down
Loading
Loading