From 5477a672961f6de3a611b0ae18ac79b19dd71050 Mon Sep 17 00:00:00 2001 From: David Bowman Date: Fri, 30 Oct 2026 18:21:54 +0000 Subject: [PATCH 01/25] plan: register standalone MCP local-inference efficiency workstream --- .../001-opt-in-compact-tool-advertisement.md | 72 +++++++++++++++++ .../002-brief-evidence-and-output-budgets.md | 81 +++++++++++++++++++ ...-studio-consumer-qualification-and-docs.md | 59 ++++++++++++++ plans/registry.md | 6 ++ .../standalone-mcp-local-inference-roadmap.md | 80 ++++++++++++++++++ 5 files changed, 298 insertions(+) create mode 100644 plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md create mode 100644 plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md create mode 100644 plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md create mode 100644 plans/subsystems/standalone-mcp-local-inference-roadmap.md diff --git a/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md b/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md new file mode 100644 index 0000000..58baf07 --- /dev/null +++ b/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md @@ -0,0 +1,72 @@ +# M001 — Opt-in Compact Ten-Tool MCP Advertisement + +Status: ready +Primary class: capability +Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (2026-10-30) +Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` +ADRs: `plans/adrs/ADR-0001-ten-tool-surface-and-adapter-ownership.md`; `plans/adrs/ADR-0006-keyless-deterministic-retrieval-and-optional-extraction.md` +Hard dependencies: none. Previously closed tool-surface M001-M007 are the retained baseline. + +## Objective + +Create a **process-configured, opt-in** compact MCP tool-definition profile for direct local-model hosts that advertise all ten stable canonical tool names but far fewer ordinary arguments and metadata bytes. Full/default `tools/list` and tool-call behavior must be unaffected. This milestone concerns advertised contract *size*, not search quality or response content. + +## Current evidence and ownership + +- `src/mcp/server.rs` assembles ten decorated tools once at construction and returns all from `list_tools()`. `contract_fingerprint()` hashes the current advertised name, descriptions, annotation and schemas. The handler router is separate from the advertised list. +- `src/mcp/output_schema.rs` attaches permissive per-tool output schemas; `src/mcp/tool_contract.rs` already has concise canonical selection descriptions, aliases and discovery metadata. +- `src/mcp/tools/{web_search,repo_search,research_search,security_search,web_fetch,repo_fetch,batch_fetch,repo_map,evidence_bundle,provider_status}.rs` deserialize the canonical runtime argument structs; older advanced fields are sometimes `#[schemars(skip)]` while remaining accepted. +- `src/mcp/mod.rs::build_server` is shared by client-owned stdio and loopback HTTP; `src/commands/mcp.rs` is transport startup. `src/core/config.rs` / `src/config.rs` own config parsing and validation. +- `docs/retrieval-quality-context-efficiency.md` records 72,939 full `tools/list` serialized bytes, 14,008 bytes for a three-tool subset, and ~18.2k tokens by a 4-bytes/token proxy. Do not treat this as a measured LM Studio prompt cost. + +## Invariants / non-goals + +Retain the ten canonical names and their exact tool-call dispatch/permission semantics, default full tool definitions, default diagnostic response behavior, existing protocol revisions, existing validation, hidden legacy argument acceptance, all provider/repo/PDF/browser capabilities, CodeGG wrapping, Python SDK and wire compatibility. No mega-tool, 11th discovery tool, new dynamic-client protocol, inferred model detection, output truncation or per-client identity-based behavior. Compact metadata is not a permission wall. + +## Intended contract + +Add a validated process-level configuration (illustrative spelling; pick one canonical name consistently in code/docs): + +```toml +[mcp] +schema_profile = "full" # full (default) | compact +``` + +`full` must serialize unchanged except for strictly necessary additive configuration behavior. `compact` advertises exactly the same ten names and minimum valid inputs: primarily `query`/`url` or current structured locator, task-specific discriminator (`goal`/`include`/`sources` only where necessary), useful budget (`max_results`/`max_chars`), and a response-detail override. Preserve required keys, JSON type, default, enum and nested-required semantics. Suppress nonessential schema descriptions, advanced provider knobs, nested schemas that are only diagnostics, and potentially `outputSchema` only after verifying compatibility. Do not produce a schema that falsely forbids accepted advanced arguments: use a documented compatible `additionalProperties` policy, or provide truthful constrained schema if the runtime cannot accept extra properties. + +All inputs (including unadvertised advanced fields) must continue through the same existing Serde deserializer and validation path. Advanced capability access is through `full` presentation or explicit arguments from clients that know the contract. All ten tools must remain visible and selectable in compact, not moved into an uncallable deferred category. + +## Ordered work packages + +1. Re-run current locked baseline: enumerate `tools/list` with a real stdio MCP initialization and native loopback HTTP where feasible. Record exact bytes per tool split into name/description/input/output/annotations, instructions bytes, selected-client model-tokenizer counts, and repository SHA. Distinguish total wire bytes from actual LM Studio prompt inclusion. +2. Define the `[mcp]` config owner and validation with default `full`; wire the same immutable `McpSchemaProfile` into the shared `EggsearchServer` construction for both transports. Do not change server settings based on an incoming untrusted MCP request or client `User-Agent`. +3. Build compact metadata from the existing registered tool contracts. Prefer a pure deterministic schema projector applied only to the advertised copy. Preserve `ToolRouter` and actual argument types; avoid duplicating the handlers. Carefully handle required parameters and JSON Schema refs/defs; schema validity is tested using the exact supported protocol representation. +4. Establish profile-aware `tool_fingerprint` based on actual serialized definitions, instructions and schema profile as appropriate. Verify deterministic order, cross-process stability, profile separation, and no compatibility drift in full. +5. Add per-profile contract fixtures, per-tool byte budgets and a frozen accepted-input matrix covering advanced aliases, provider selection, repo/security/research constraints, PDF/browser configuration and exact errors. Run full-profile vs compact-profile identical dispatch inputs; assert same normalized outcomes on mock backends. +6. Update MCP/config/operator docs and code-derived tool-name inventory tests. Avoid hard-coding the local-client implementation into a generic protocol feature. + +## Required verification + +```bash +cargo fmt --check +cargo test --locked --features mock --test mcp_tool_contract +cargo test --locked --features mock --test mcp_schema_slimming +cargo test --locked --features mock --test tool_surface_evaluation -- --nocapture +cargo test --locked --features mock --test mcp_tools +cargo test --locked --all-features --test static_guards +make check +make docs-check +make bench-check +``` + +Add a focused `standalone_mcp_profiles` suite (or extend existing `mcp_tool_contract` and `mcp_2026_protocol` suites) exercising real MCP initialized `tools/list` and mock calls; update test inventory and architecture/testing ownership. Run both stdio and loopback HTTP negotiation in offline integration tests when supported. + +## Acceptance / stop + +- Full profile remains bit-for-bit equivalent for advertised definitions, stable names, per-call behavior and default response mode (excluding unavoidable version/build metadata). No existing consumer config requires migration. +- Compact profile advertises all 10 names, each input schema validates, all existing advanced requests still work, and protocol adapters do not lose access to any capability. +- Recorded pre/post serialized total target **<= 36,000 bytes (~50% of baseline)** for compact across ten tools, with stretch goal <= 24,000; do not silently downgrade semantics to meet the target. If 36 KB cannot be reached without invalid schema/host interoperability, document evidence, stop threshold claim, and propose a narrow follow-up rather than force a breaking shape. +- At least the repository's 43-case deterministic tool-selection/discovery suite remains above its existing accuracy gates; separately record compact-profile results and any tools becoming ambiguous. +- Both MCP transports, mock domain workflows, and CodeGG compatibility cases remain green. + +Stop on a missing required field, JSON Schema incompatibility, material tool-selection regression, authorization bypass, or change to full-mode behavior. Record exact candidate SHA, serialized bytes, compatibility matrix, CI commands/outcomes and residual risk in `plans/closure/standalone-mcp-local-inference/001-status.md`. M002 becomes ready only on accepted M001 closure. diff --git a/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md b/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md new file mode 100644 index 0000000..d576c20 --- /dev/null +++ b/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md @@ -0,0 +1,81 @@ +# M002 — Brief Evidence Projection and Deterministic Output Budgets + +Status: blocked (hard dependency: M001 closure) +Primary class: capability +Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (reconfirm after M001) +Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` +ADRs: ADR-0001, ADR-0006. Related closed predecessor: tool-surface consolidation M004 `plans/closure/mcp-tool-surface-consolidation/004-status.md`. + +## Objective + +Provide a genuinely short, deterministic MCP model-visible output mode for direct local-inference hosts. Preserve useful source evidence while suppressing redundant telemetry and repeated metadata; make it the **opt-in compact schema profile's** omitted-parameter default without changing the current `full`/`diagnostic` default or the existing `compact`/`standard` semantics. + +## Current evidence + +`src/mcp/projection.rs` exposes `ResponseDetail::{Compact,Standard,Diagnostic}` with `Diagnostic` default. Compact trims excerpts (normally one per card) and some top-level telemetry, but often retains most nested `SourceCard` fields: provider arrays, score, quality, metadata, trust and duplicate source fields. `src/core/source_card.rs` describes the canonical source-card identity, snippet/excerpt provenance, trust markers, result quality and domain-specific metadata. `src/mcp/tools/web_search.rs` returns both string and structured warnings plus routing, summaries and suggested next actions. `src/mcp/tools/web_fetch.rs` may return `text` plus a full structured `document` and `focus`. `src/mcp/tools/common.rs::map_tool_result` sends a structured payload with a text fallback: smaller returned payload helps clients that consume either channel. + +The context-efficiency corpus reports only 0.7% compact savings on short result cards; a representative fetch is 46.9% smaller. Compact isn't sufficient as a low-token default for LM Studio. + +## Ownership, non-goals, invariants + +The `core` and adapter own canonical evidence, identity and ranking. The new layer only projects MCP responses after successful canonical retrieval; do not change upstream requests, number of internally retrieved candidates, RRF scores, provider selection, security advisory applicability, local workspace trust, caching, PDF/browser extraction, or compatibility views. No LLM summarization, embeddings, heuristic new ranking, hidden evidence fabrication, stateful opaque result handles, automatic additional requests or rewriting source provenance. + +Do not assume a direct MCP host retains diagnostic structured data. When `brief` is requested, both native structured content and the compatibility text rendering may be brief; explicit `diagnostic` must retrieve the existing complete bounded response. Document this distinction rather than claiming full evidence is secretly stored. Avoid a second result-specific persistent cache. + +## Response contract + +Introduce `response_detail = "brief"` as an additive value for search/fetch tools that already accept detail modes; preserve the existing `compact`, `standard` and `diagnostic` values and their outputs. The config chosen by M001 controls the omitted-argument default: `full` => diagnostic; `compact` => brief. Explicit per-call detail ALWAYS wins. Diagnostic-only `provider_status` and identity-preserving `build_evidence_bundle` remain semantically complete; do not call an identity-trimmed evidence bundle "canonical". + +Brief must be a structured, deterministic result envelope using the existing tool names. Suggested fields: +- Search: query/goal; status (`complete`/`partial`/`failed`/`unknown`) backed by actual retrieval signals; ranked source cards with stable IDs, titles, canonical URLs/locators, source kind/group, one short snippet or relevant excerpt (not both by default), trust and necessary provenance; compact warning codes and result-count/omission counters; at most the few useful next-action hints. +- Repository: path/owner/ref/local workspace provenance, checkout state/dirty state where applicable, symbol or line-span locator and evidence-role context. Preserve source/issue/release distinctions. +- Security: advisory ID, affected/unaffected/unknown applicability, package/version/source, severity where present, relevant exploitation/KEV status, confidence, conflicts and limitations. Never infer safety from missing providers. +- Fetch: known URL/redirect identity, content type, bounded `text` or deterministic focused chunks, retained span/chunk/page coordinates for citations and PDFs, truncation flag, trust markers. Avoid returning both full text and overlapping focus-chunk text. +- Batch: per-item identity, success/failure and one-item content budget; aggregate omissions/budget exhaust. Never suppress a failed member. +- Research: multi-source groups and evidence gaps/counterpoints as available. Keep enough coverage to distinguish support from independent evidence. + +Use existing `structured_warnings` severity/code grouping to deduplicate repeated global trust/framing warnings in agent display; do **not** silently strip critical injection or capability-enforcement warnings. Preserve untrusted framing on all retained excerpts/text. Keep full debug/error detail available via explicit diagnostic requests and operator tools. + +## Budget semantics + +Introduce an optional, bounded display budget at the MCP presentation boundary, e.g. server config `[mcp].brief_max_chars` with explicit per-call override only if it can be added consistently to all supported search/fetch types. Proposed initial **character** ceiling 8,000 for typical direct calls, subject to fixture evidence; character/UTF-8 bytes are not model tokens and must be labelled correctly. + +Prioritize reserved space for status and critical warnings, then complete top-ranked cards and selected snippets, then optional next actions/telemetry. Emit `returned_count`, `omitted_count` and `output_truncated` where applicable; do not prefix-clip JSON or claim a page is complete when its content was clipped. If a single critical warning exceeds budget, use a bounded stable code with accessible diagnostic detail, not silence. Honor explicit fetch `max_chars` and server hard caps independently of presentation budget. + +Implement through one central projection boundary (`src/mcp/projection.rs` or its owned submodules), with per-domain typed/field-aware reductions; keep conversions deterministic and bounded. Avoid repeated full-JSON cloning and serializer roundtrips on low-memory SBC devices when a typed projection is practical. + +## Ordered work packages + +1. Freeze realistic fixture payloads for web/repo/research/security/fetch/PDF/batch, partial provider failures, all-providers-failed, local/remote trust, prompt injection, stale/unknown applicability and missing credential. Measure full/compact/brief bytes and the number of complete source cards under budgets; no invented model-token claim. +2. Extend detail parsing/default resolution in `projection.rs` and all response-bearing argument structs; bind omitted-detail policy to the M001 profile, not a global breaking default. Keep the ten output schemas truthful to all supported result modes (some are already permissive). +3. Implement explicit field allowlists and reserved diagnostic/status slots, with stable source IDs and URL/locator cross-tool handoff. Prefer concise safety/status codes over duplicated prose, retaining threat indicators. +4. Implement display-budget enforcement over complete records, UTF-8 boundaries and bounded memory. Avoid dropping semantically essential security/advisory/version data, conflict indicators, source provenance or partial/failed signals to satisfy a numeric size target. +5. Test MCP `CallToolResult` structured and text compatibility paths against both local profile and full diagnostic profile. Test direct call override `response_detail="diagnostic"` with unchanged output, and plain `tools/call` behavior in stdio and HTTP. +6. Update docs, examples, response fixtures and context-efficiency benchmark output. No new automatically running LLM, no network provider dependency in CI. + +## Verification + +```bash +cargo fmt --check +cargo test --locked --all-features --test mcp_projection +cargo test --locked --all-features --test mcp_2026_protocol +cargo test --locked --features mock --test dispatch_fault_injection +cargo test --locked --features mock --test batch_fetch_retrieval +cargo test --locked --all-features --test evidence_contract +cargo test --locked --all-features --test tool_surface_evaluation -- --nocapture +make check +make docs-check +make bench-check +``` + +Extend existing projection, fetch and domain-workflow suites with status-first budget fixtures; record tokenizer-independent byte/character costs and a named model tokenizer where available in optional live measurements. + +## Acceptance / stop + +- Default full-profile results remain byte/structure compatible in frozen fixtures; legacy detail values preserve their prior meaning. Explicit diagnostic requests still expose prior diagnostic information. +- Every brief response (including 0 results, all providers fail, partial failure, injected snippet, security applicability unknown, local dirty checkout, PDF truncation) retains identity/status/trust, is parseable and declares omissions. +- Brief saves **at least 30% serialized bytes** on representative nontrivial web/repo/research/fetch fixtures relative to diagnostic, with per-domain results published. Do not disguise negative or zero savings in sparse fixtures; the 15-task corpus should report separately rather than enforcing an artificial threshold. +- No lower-level retrieval quality/capability change and no additional upstream requests introduced by projection. Bounded CPU/memory/UTF-8 behavior on large batch and PDF fixtures. +- Test results and documentation explain that returned `structuredContent` is also projected in brief mode, so callers may explicitly request diagnostic if they need full data. + +Stop if any projection masks a material failure, vulnerability applicability, untrusted content marker, provider partial state or content truncation; or if budget fitting requires invalid JSON. Exact-SHA closure: `plans/closure/standalone-mcp-local-inference/002-status.md`. M003 becomes ready only on accepted M002 closure. diff --git a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md new file mode 100644 index 0000000..c6884f7 --- /dev/null +++ b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md @@ -0,0 +1,59 @@ +# M003 — LM Studio Direct-MCP Qualification and Consumer Documentation + +Status: blocked (hard dependencies: M001 and M002 closure) +Primary class: capability +Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (rebase baseline to exact M002 candidate at start) +Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` +ADRs: ADR-0001 and ADR-0006. No CodeGG implementation work. + +## Objective + +Prove the new opt-in compact schema and brief result modes actually improve local-model use in LM Studio while preserving the ability to call all ten Eggsearch tools, understand retrieval uncertainty and fetch needed evidence. Deliver a reproducible installed-binary LM Studio configuration and tested troubleshooting workflow, rather than relying on simulated schema sizes. + +## Research/consumer constraints + +LM Studio documents local/remote MCP servers in `mcp.json` and warns about context-heavy servers: https://lmstudio.ai/docs/app/mcp . As of the researched documentation, LM Studio API MCP integrations can select configured servers and restrict visible tools through `allowed_tools`, which can reduce prompt costs: https://lmstudio.ai/docs/developer/core/mcp . Verify feature/version behavior during implementation; do not assume `allowed_tools` in the API implies interactive chat dynamically fetches tool schemas, or that all clients use both `structuredContent` and JSON text. + +Eggsearch default binary transport is client-owned `eggsearch mcp stdio`; optional HTTP is loopback-only via `eggsearch mcp serve`. Do not make local inference dependent on starting a separate cloud service, building on SBCs, or a proprietary UI plugin. Use the actual CLI syntax and executable-path quoting on each OS before publishing examples. + +## Ordered work packages + +1. Create `docs/integrations/lm-studio.md` (or a well-named existing operator doc section): copyable `mcp.json` JSON for an **installed binary** with the opt-in compact profile (pass `--config` to the actual CLI or use supported env/config controls). Provide standard full-profile fallback, Windows PowerShell executable path, macOS and Linux paths, stdio stdout-is-protocol / stderr-is-logs expectations, and one loopback-HTTP alternative if client support is confirmed. Never hardcode private paths or credentials; no remote/public bind. +2. Record an exact SHA, binary version, LM Studio version/build, local model and context length; enumerate MCP `tools/list` from the actual child process and record schema bytes, tokenizer counts, advertised names and prompt-use attribution. Compare `full + diagnostic`, `compact + brief` and optional client-side `allowed_tools` to avoid confusing profile savings with fewer enabled tools. +3. Build deterministic offline fixtures that replay realistic tool calls through the real MCP stdio server (loopback HTTP as additional protocol qualifier): web search, page/PDF fetch, repo search + specific fetch, research grouping, advisory/version applicability, grouped batch failures, and an evidence bundle. Mock upstream calls in CI; no credential or external network requirement. Confirm status-first brief and explicit diagnostic fallback. +4. Run optional interactive LM Studio qualification with at least one smaller tool-capable local model and one larger reference local model (model sizes are examples, not hard requirements). Use >= 8k context if the application permits, record exact tokenizer and actual observed prompt token usage. Compare tool-choice success, argument validity, repair/retry count, target evidence found, truthful failure interpretation, calls to answer, total latency and prompt-processing time on the same prompts. Include tasks requiring `security_search` and `research_search` to demonstrate capability preservation. Do not rely only on 43-case offline keyword ranking results. +5. Characterize how LM Studio actually handles `outputSchema`, `structuredContent` versus text fallback, `tools/list` refreshing and tool visibility across new chat/session after config changes. Record discrepancies and a verified rollback path. Compare at least one other generic MCP inspector/client if available; keep any vendor-specific advice in docs rather than adding vendor detection to Eggsearch. +6. Implement only minimal fixes necessary for discovered compatibility defects in the already-owned profile or projection modules; otherwise open new corrective milestones and keep the current M003 unclosed. Update `docs/integrations.md`/`docs/installation.md`/`README.md`, `docs/test-inventory.md`, `architecture/mcp.md` as appropriate. Explain optional client `allowed_tools` filtering as an intentional narrowing of per-chat discovery, NOT a server capability loss. + +## Compatibility, security and resources + +Full mode must continue working in ordinary MCP clients and CodeGG; no provider or transport regression, secret/proxy bypass, config-driven SSRF relaxation, startup service change, or new live agent backend in Eggsearch. Test prompt-injection markers and retrieval-failure truthfulness through the exact text visible to LM Studio. No raw diagnostic JSON dumps in brief mode unless explicitly requested. Do not claim local-model benchmark wins without actual measured outcomes. + +Test a fresh installation from precompiled release assets where available; no compilation on low-resource ARM boards. Offline smoke can run on normal hosted CI. A live LM Studio instance is an operational acceptance dependency for closing the consumer claim; if unavailable, document an unexecuted gate and leave M003 blocked/closing accordingly, without misrepresenting offline results. + +## Verification + +```bash +cargo fmt --check +cargo test --locked --all-features --test mcp_2026_protocol +cargo test --locked --all-features --test mcp_tool_contract +cargo test --locked --all-features --test tool_surface_evaluation -- --nocapture +cargo test --locked --features mock --test mcp_tools +cargo test --locked --features mock --test web_search_integration +cargo test --locked --features mock --test web_fetch_integration +make check +make docs-check +make bench-check +``` + +Check exact suite names against `architecture/testing.md` before running; add a dedicated installed-binary/LM Studio compatibility test runner if the existing MCP protocol tests cannot exercise the targeted path. Avoid a permanent CI dependency on LM Studio or a model download. + +## Acceptance / stop + +- A validated `mcp.json` configuration from a published or candidate installed Eggsearch executable connects and the host sees all ten tool names in compact presentation; both generic web and specialist tasks run correctly. +- Capture **actual model-visible** schema/result token counts for full vs compact, plus bytes, tool-call failures, completed-task evidence and latency; do not substitute byte/4 estimates for tokenizer or hosted context metrics. Report negative outcomes honestly. +- No capability, safety, inference-free or CodeGG regression; full profile and explicit diagnostic work. User documentation has one clear lightweight default, opt-in full fallback and detailed troubleshooting. +- If measured compact savings come at a substantial cost in tool correctness, repair, safety or specialist reachability, stop and register a corrective pass with concrete fixtures; do not silently publish a lower-quality default. +- Live LM Studio smoke performed and recorded, or M003 remains explicitly unclosed. + +Closure evidence in `plans/closure/standalone-mcp-local-inference/003-status.md`: implementation SHA, installed binary and client versions, configs and nonsecret fixtures, per-task comparisons, test commands/outcomes, residual findings, docs links, and recommended release disposition. Do not claim a public release was made by finishing this plan. diff --git a/plans/registry.md b/plans/registry.md index 835968e..b9627fa 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,6 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | +| Standalone MCP local-inference efficiency | ready (M001 ready; M002-M003 blocked) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001 compact ten-tool schema advertisement handoff | New opt-in direct-client presentation; full ten-tool/default diagnostic contract unchanged; no CodeGG implementation dependency. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -51,6 +52,9 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Subsystem | Milestone | Status | Implementation plan | Dependencies / handoff note | |---|---|---|---|---| +| Standalone MCP local-inference | M001 opt-in compact tool advertisement | ready | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | No hard dependency; implement first against main `a666517c` after verifying current tool-list baseline. Keep default full ten-tool contract. | +| Standalone MCP local-inference | M002 brief evidence and deterministic budgets | blocked | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Hard: accepted M001 closure. Default profile remains diagnostic; brief is opt-in via compact profile or explicit call. | +| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Hard: accepted M001/M002 closure; live LM Studio test is an operational closure gate, never assumed from synthetic bytes. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | @@ -121,6 +125,8 @@ Corrective closure sequence: `plans/implementation/dependency-evidence-hardening **Python native library gate:** ADR-0005 authorizes a narrow direct Rust SDK facade and Python PyO3 native package without broad Rust module stability. M001–M008 are closed at `067468a` with fully-green qualifying run `38641009246` (5/5 builds, 20/20 smokes, offline sdist rebuild, green audit, digests in `008-status.md`). Earlier runs are historical/deficiency/supporting evidence only. First PyPI publication is separately closed under `python-pypi-distribution` M003 at 0.4.4; Eggpack binary assets remain unchanged. +**Standalone direct-MCP local-inference gate:** New roadmap `plans/subsystems/standalone-mcp-local-inference-roadmap.md` is ready for M001; M002 waits on M001 closure and M003 waits on M001/M002 closure. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. No current LM Studio smoke or token savings are claimed. + **Tool-surface gate:** M001-M007 closed. Do not run evaluation against pre-consolidation bytes and present it as consolidation evidence. diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md new file mode 100644 index 0000000..fa55ad4 --- /dev/null +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -0,0 +1,80 @@ +# Standalone MCP Local-Inference Efficiency Roadmap + +Status: ready (planning handoff; M001 ready, M002-M003 blocked by hard dependencies) +Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration + +Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. +Controlling ADRs: `plans/adrs/ADR-0001-ten-tool-surface-and-adapter-ownership.md` and `plans/adrs/ADR-0006-keyless-deterministic-retrieval-and-optional-extraction.md`. +Closed predecessor: `plans/subsystems/tool-surface-consolidation-roadmap.md` (M001-M007); do not reopen its closure or claim CodeGG work here. + +## Purpose and ownership + +Make eggsearch practical as a direct MCP server in LM Studio and similarly simple local-inference hosts that eagerly pass advertised tool definitions and responses to a model. Eggsearch owns the MCP server's advertised schema, protocol responses, deterministic projection and operator configuration; the client owns the model, tokenizer, context window and tool invocation. CodeGG's native wrappers, progressive disclosure and diagnostic storage are intentionally unchanged. + +Primary classes: M001 capability (compact advertised schema profile); M002 capability (brief bounded output); M003 capability (tested direct-client onboarding/qualification). Infrastructure includes measurement and config; polish follows qualified behavior. + +## Baseline and findings + +Reviewed main `a666517c39adf28f32fad04198c74cb70d935c5b` on 2026-10-30. `src/mcp/server.rs` registers ten stable tools and `list_tools` returns all decorated definitions regardless of client; `src/mcp/output_schema.rs` adds output schemas. `src/mcp/projection.rs` supports compact/standard/diagnostic, with diagnostic as omitted-parameter default; `src/mcp/tools/common.rs` uses native structured results plus a text compatibility fallback. `src/core/config.rs` defaults to 10 search results, and configured fetch defaults can reach 12,000 characters. Closed consolidation work already shortened descriptions and some input schemas but is oriented around clients that own progressive disclosure. + +Recorded offline baseline: `docs/retrieval-quality-context-efficiency.md` reports 72,939 serialized bytes for all ten definitions (~18,235 byte/4 estimated tokens) and 14,008 bytes for the three-tool core palette. Its short-card compact response corpus saved only 0.7%, while a 2,000-character web fetch fixture saved 46.9%. These are historical baselines, NOT current token counts measured inside LM Studio or proof that the client includes all definitions verbatim. + +LM Studio documentation supports local/remote MCP in `mcp.json` and warns about MCP context overhead. Its API can use `allowed_tools` to restrict a configured MCP server; the API restriction is not proof that interactive app chats dynamically hydrate Eggsearch tools. Sources: https://lmstudio.ai/docs/app/mcp and https://lmstudio.ai/docs/developer/core/mcp . Measure the actual client path in M003. + +## Non-negotiable invariants + +- Retain exactly ten canonical registered MCP tools and all current backend capabilities, provider behavior, argument compatibility, response semantics in the default profile, stable IDs, CLI/Python interfaces and both MCP transports. No mega-tool, new tool name, second retrieval engine, or LLM/embedding inference in eggsearch. +- `full` remains the install/default presentation, returning the same ten names and current advertised schemas and default `diagnostic` responses unless a caller opts in. Profile selection is **presentation**, not an authorization, SSRF, local-filesystem, tool execution, or provider-routing control. +- The recommended `compact` profile still advertises all ten canonical names. Each remains normally callable by name; advanced/legacy parameters remain accepted by runtime validation even if not listed in compact schemas. Operators can switch to `full` for complete schema help. +- Do not claim that pruning `tools/list` is capability-preserving: reduced-name palettes may be documented as LM Studio/client `allowed_tools` choices, not silently imposed by eggsearch. +- External material remains untrusted; retain trust markers, source locators, stable identity, provider failure vs evidence absence, partial coverage, security applicability, conflicts, and truncation signals. Never invent synthesis. +- No cache of model responses, persisted result blobs, opaque result-handle store, automatic re-fetch/crawl, or network authorization change. Explicit retrieval budgets remain independent of display budgets. +- Do not promise immediate model-context savings from JSON bytes alone. Record actual advertised bytes, estimated and model-tokenized costs, and LM Studio host-injection behavior separately. + +## Target architecture + +A single `ServerState` / router owns the full tools; an opt-in server presentation profile selects which descriptions, JSON input properties, and optional output-schema metadata appear in `tools/list`, with deterministic fingerprinting of the **advertised** profile. Do not fork handlers or silently reinterpret omitted fields. An independent optional `brief` response projection offers terse, bounded evidence, preserving the existing `compact`/`standard`/`diagnostic` response contracts and making the default brief only when explicitly chosen by the local-inference operator profile (per-call overrides always win). Both stdio and loopback HTTP use the same configuration. + +Avoid an unnecessary additional server-side dynamic discovery tool: direct MCP hosts may not support schema hydration after initialization. All ten tool names remain exposed in the compact profile so specialist research, repository, security, evidence and diagnostics are reachable without a separate host. + +## Dependencies and sequence + +```text +M001 compact ten-tool advertised schema profile (ready) + | + +-- hard --> M002 brief/budgeted agent responses (blocked on M001) + | + +-- hard --> M003 LM Studio/direct-client compatibility and quality (blocked on M001+M002) +``` + +No CodeGG dependency; it is a regression consumer, not the implementation target. M003's offline qualification can run in CI; a real LM Studio interactive smoke is an operational closure requirement, not a reason to block M001/M002 development. + +| Milestone | Primary class | Source plan | Exit | +|---|---|---|---| +| M001 | capability | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Opt-in compact `tools/list` for all ten existing names; measurable schema reduction; parity across MCP transports; default unchanged | +| M002 | capability | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Explicit brief projection and bounded results with critical failure, identity, trust and security semantics intact; unchanged legacy defaults | +| M003 | capability | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Verified LM Studio install/smoke, measured local-model context cost and completion quality, onboarding guides, and stop/rollback evidence | + +## Cross-cutting controls + +Compatibility/protocol: verify current `rmcp` schema serialization and protocol negotiation before implementing profile-specific tool definitions. MCP `tools/list` caching or `listChanged` support MUST NOT be assumed by LM Studio. Profiles are process/config scoped, never inferred from user agent or model size. Fingerprints MUST differ when serialized contracts differ and remain stable across equivalent process launches. + +Safety: every tool uses existing policy and validation gates. Removing an advertised property MUST NOT remove runtime validation. Diagnostics must not disappear from the unmodified default. A brief response must signal incomplete retrieval; when a client ignores structured content, text fallback must still be useful. + +Resources: local inference budget is an agent-context budget; do not reduce remote search recall, upstream concurrency or browser/PDF capabilities as a side effect. Keep memory bounded, deterministic order and fast serialization; avoid doubling large JSON payloads in memory merely to render brief. + +Observability: keep per-tool profile-size fixtures, model-visible response samples, omission counts, source-card retention, constraint-enforcement caveats and error/repair traces. Test actual LM Studio tool-selection success, not merely canned keyword matching. + +Documentation: M001/M002 update `architecture/mcp.md`, `architecture/config.md`, `docs/config.md`, `docs/tool-matrix.md` and `docs/test-inventory.md` as applicable. M003 updates `README.md` / `docs/integrations.md` / `docs/installation.md` with LM Studio guidance. Never rewrite historical closure records. + +## Known risks and decision boundaries + +- Some MCP clients may reject incomplete or overly permissive input JSON schemas. If tests show this, retain required fields or fall back to the current schema instead of weakening validation. +- Omitting `outputSchema` may improve bytes but could reduce typed tool-result handling. Test real clients; keep `full` unchanged and omit only if measured safe. +- The brief view could collapse retrieval failures into apparent absence or hide applicability. Security/failure fixtures are hard gates, not optional benchmarks. +- Models may perform worse with less explicit tool parameter instruction. Do not land profile changes solely on size savings; M003 must measure call and repair quality. +- A host may include both MCP `structuredContent` and fallback `content` in context. Characterize duplication rather than assuming one is used. + +## Definition of done + +Every milestone gets an exact-SHA closure record in `plans/closure/standalone-mcp-local-inference/NNN-status.md`; update this roadmap and `plans/registry.md` in each closure commit. No full rollout, release, claims of LM Studio savings, or marking M003 closed on synthetic tests alone. From bf3a35ff1a3463db73eb6aa9105bef225cf0e3dd Mon Sep 17 00:00:00 2001 From: eggsearch Date: Fri, 30 Oct 2026 20:11:21 +0000 Subject: [PATCH 02/25] feat: add compact MCP profiles and brief outputs --- architecture/config.md | 20 +- architecture/mcp.md | 21 +- architecture/testing.md | 6 +- docs/config.md | 23 ++ docs/installation.md | 5 + docs/integrations.md | 2 + docs/lm-studio.md | 94 +++++ docs/retrieval-quality-context-efficiency.md | 23 +- docs/test-inventory.md | 10 +- .../001-opt-in-compact-tool-advertisement.md | 4 +- .../002-brief-evidence-and-output-budgets.md | 2 +- ...-studio-consumer-qualification-and-docs.md | 2 +- plans/registry.md | 4 +- .../standalone-mcp-local-inference-roadmap.md | 2 +- scripts/lm_studio_stdio_smoke.py | 95 +++++ src/core/config.rs | 41 ++ src/mcp/projection.rs | 370 +++++++++++++++++- src/mcp/server.rs | 234 ++++++++++- src/mcp/tools/batch_fetch.rs | 10 +- src/mcp/tools/evidence_bundle.rs | 2 +- src/mcp/tools/repo_fetch.rs | 41 +- src/mcp/tools/repo_map.rs | 14 +- src/mcp/tools/repo_search.rs | 10 +- src/mcp/tools/research_search.rs | 10 +- src/mcp/tools/security_search.rs | 10 +- src/mcp/tools/web_fetch.rs | 10 +- src/mcp/tools/web_search.rs | 10 +- tests/config_validation.rs | 32 ++ tests/mcp_http.rs | 71 +++- tests/mcp_projection.rs | 258 +++++++++++- tests/mcp_tool_contract.rs | 53 +++ tests/tool_surface_evaluation.rs | 63 +++ tests/tool_surface_live.rs | 4 +- 33 files changed, 1478 insertions(+), 78 deletions(-) create mode 100644 docs/lm-studio.md create mode 100644 scripts/lm_studio_stdio_smoke.py diff --git a/architecture/config.md b/architecture/config.md index d416844..1f1cc01 100644 --- a/architecture/config.md +++ b/architecture/config.md @@ -38,8 +38,8 @@ falls back to `default_config_path()` otherwise, then delegates to ## Section / Field Map -Root type `AppConfig` holds four sections: `search`, `fetch`, -`local`, `egress`. Each section field on `AppConfig` carries +Root type `AppConfig` holds five sections: `search`, `fetch`, +`local`, `egress`, `mcp`. Each section field on `AppConfig` carries `#[serde(default)]`, so an omitted table inherits production defaults. Within a section, only fields with their own `#[serde(default = ...)]` are optional: every `[fetch]`, `[local]`, and `[egress]` field is @@ -166,6 +166,22 @@ environment variable. Raw passwords are never a config field, and binary built without the `egress` feature fails validation instead of being silently ignored. +### `[mcp]` — `McpSection` + +| Field | Type | Default | Purpose | +|-------|------|---------|---------| +| `schema_profile` | `McpSchemaProfile` | `full` | `full` advertises the existing complete schemas; `compact` keeps all ten tool names while shortening descriptions and input schemas for context-heavy direct MCP clients | +| `brief_max_chars` | `usize` | `8000` | Maximum serialized character count for brief MCP responses, bounded to 512–32000 | + +The compact profile is process-wide presentation configuration, shared by +stdio and loopback HTTP. It does not restrict tool execution or change +argument validation. Extra accepted properties remain callable. Compact +advertisement omits `outputSchema` metadata; tool responses continue to use +the same handlers. Omitted `response_detail` remains `diagnostic` under +`full` and becomes `brief` under `compact`; explicit per-call values win. +The default `full` profile keeps the existing advertisement and response +behavior unchanged. + --- ## Provider Resolution diff --git a/architecture/mcp.md b/architecture/mcp.md index 1d4dc7d..874383a 100644 --- a/architecture/mcp.md +++ b/architecture/mcp.md @@ -39,6 +39,15 @@ struct EggsearchServer { Construction (`EggsearchServer::new`) builds the `#[tool_router]` router once, then decorates every entry via `apply_contract_metadata()` and freezes the result. `tool_definitions()` clones the frozen list; `tool_fingerprint()` returns the frozen FNV-1a content hash. Nothing is recomputed per call. +`build_server()` reads `[mcp].schema_profile` once when constructing the +server. `full` is the default and leaves the decorated list unchanged. +`compact` projects only advertised tool copies: all ten names and required +arguments remain, optional input fields and prose are reduced, extra accepted +properties stay allowed, and output schemas are omitted. The request router, +Serde argument types, validation and execution policies are shared. The +profile is presentation configuration, never a permission or dispatch gate; +the profile-specific serialized definitions produce a distinct fingerprint. + `ServerHandler` implementation: - `get_info()` advertises `ServerCapabilities` with tools enabled, implementation `eggsearch` + crate version, and `EGGSEARCH_INSTRUCTIONS`. @@ -46,7 +55,7 @@ Construction (`EggsearchServer::new`) builds the `#[tool_router]` router once, t `EGGSEARCH_INSTRUCTIONS` carries global rules only: external content is untrusted data, search discovers while fetch inspects explicitly selected targets, start with the task-appropriate search primitive (never `provider_status`), specialist tools only when their domain semantics apply, one URL per `web_fetch` with `batch_fetch` for fan-out, follow `next_actions`, `build_evidence_bundle` packages without searching/fetching/summarizing. Tool-specific selection guidance lives in `tools/list` descriptions and schemas, sourced from `tool_contract.rs`. -`build_server()` (`mod.rs`) is the single factory both transports share: `ServerState::build(config)` wrapped in `EggsearchServer::new`. Tool registration, contract metadata, schemas, provider construction, and policy behavior exist in exactly one place. +`build_server()` (`mod.rs`) is the single factory both transports share: `ServerState::build(config)` wrapped in `EggsearchServer::new`, which reads the schema profile from the same immutable config. Tool registration, contract metadata, schemas, provider construction, and policy behavior exist in exactly one place. --- @@ -124,7 +133,10 @@ Registration lives in `server.rs`; behavior lives in `tools/`; semantics live in | `research_search` | Multi-source evidence discovery with grouped bundles, no synthesis | `query`, `depth` (`quick`/`standard`/`deep`), `domain`, `goal`, `include` (`counterpoints`, `primary_sources`, `recent_discussion`, `security_considerations`) | `MetadataSearchAdapter::research_search` | | `build_evidence_bundle` | Deterministic packaging of already-selected evidence for handoff | `goal`, `sources`, `fetches`, `include_unfetched_sources`, `max_sources` (default 50, cap 200), `max_fetched_items` (default 20, cap 100), `max_total_chars` (default 100000, cap 500000), `response_detail` (accepted, identity-preserving) | Pure local: `meta::evidence_bundle::build_evidence_bundle`; no adapter, no fetch, no network | -All search/fetch tools except diagnostic-only `provider_status` accept optional `response_detail` (`compact`/`standard`/`diagnostic`, default `diagnostic`). +All response-bearing search/fetch tools accept optional `response_detail` +(`brief`/`compact`/`standard`/`diagnostic`). Omitted detail is +`diagnostic` in the `full` schema profile and `brief` in `compact`; explicit +per-call values win. `provider_status` remains diagnostic-only. ### Per-tool notes @@ -243,13 +255,14 @@ Tool behavior modules carry unit coverage in-file plus the shared `tools/tests.r ## Projection: `ResponseDetail` and Identity-Preserving Bundles (`projection.rs`) -`ResponseDetail` (`Compact` / `Standard` / `Diagnostic`, default `Diagnostic`) is parsed case-insensitively (`parse_raw`) with a stable error for unknown values. Canonical typed responses are built first, then projected to JSON at the MCP boundary via the central `project(tool, value, detail)` function — structured values are never truncated before host capture. +`ResponseDetail` (`Brief` / `Compact` / `Standard` / `Diagnostic`) is parsed case-insensitively (`parse_raw`) with a stable error for unknown values. Canonical typed responses are built first, then projected to JSON at the MCP boundary. `full` defaults to diagnostic; opt-in `compact` defaults to the new `brief` projection. An explicit detail always wins. - `diagnostic` — full observability payload, passthrough with no added keys. - `standard` — specialist research/security default surface. Keeps full `retrieval_summary`, `conflict_metadata`, `workflow_coverage`, `evidence_role_summary`, `capability_enforcement`, and fetch/cache metadata; summarizes only `routing_decision` (→ `routing_summary`) and `telemetry.routing_decision`. - `compact` — ordinary agent operation. Trims excerpts to 1 per card, replaces `routing_decision` with `routing_summary`, replaces `retrieval_summary`/`conflict_metadata` with `retrieval_status`/`conflict_indicator`, drops `workflow_coverage`/`telemetry`/`document`/`links`, strips per-vulnerability `references`/`details`, caps `repo_map` entries at 50. Every projection stamps `response_detail` plus an explicit truncation marker (`projection_excerpts_trimmed`, `projection_entries_truncated`, `links_omitted_in_compact`). +- `brief` — applies compact safety reductions, then retains identity, status, trust, warnings, applicability and citation fields using explicit allowlists. Content strings are clipped at UTF-8 character boundaries; if the configured `[mcp].brief_max_chars` serialized-character budget still cannot fit, complete successful result records are omitted from the tail with `returned_count`, `omitted_count`, and `output_truncated`. Failed batch members and critical failure/applicability signals are retained. Text plus overlapping focus chunks are avoided through the existing compact fetch projection. The structured result and text fallback are both brief; callers request `diagnostic` for the full diagnostic payload. -The suite asserts `compact < standard < diagnostic` and that compaction never erases failure-vs-absence state, trust/injection markers, conflict/applicability semantics, truncation evidence, or stable IDs. +The suite measures diagnostic/standard/compact/brief serialized characters across the 15-case retrieval-quality corpus and asserts that brief never erases failure-vs-absence state, trust/injection markers, security applicability, PDF coordinates, truncation evidence, failed batch members or stable IDs. The sparse corpus report is descriptive; a separate nontrivial fixture enforces the 30% byte reduction target. Per-tool reducers (`projection.rs`): diff --git a/architecture/testing.md b/architecture/testing.md index 9304982..ab99048 100644 --- a/architecture/testing.md +++ b/architecture/testing.md @@ -50,7 +50,7 @@ Use behavioral suite names. Historical phase-suite names are retired; do not int | `provider_capability_contract` | Native-capability enforcement matrix (native versus local filtering), inventory counts, required-credential posture, default-fan-out exclusion, engine-builder reachability | | `provider_workstream_regression` | Provider inventory, capability descriptors, URL dedup with stable IDs | | `structured_local_code_intelligence` | Structured parsing, definition ranking, regex fallback, budgets, repo-map enrichment | -| `mcp_tool_contract`, `mcp_schema_slimming`, `mcp_2026_protocol`, `mcp_projection` | Registry parity, schema size budgets, structured error contract, response-detail projection | +| `mcp_tool_contract`, `mcp_schema_slimming`, `mcp_2026_protocol`, `mcp_projection` | Registry parity, full/compact schema budgets, structured error contract, bounded brief and response-detail projection | | `fetch_safety` | URL validation and SSRF prevention at the fetch boundary | | `forge_adapter` | Forge endpoint validation, redirect policy, URL construction from the resolved ref (`resolved_ref_name`) | | `local_workspace_integration`, `inventory_freshness`, `retrieval_attempt_ledger`, `evidence_bundle_handoff`, `evidence_integration` | Workspace, inventory lifecycle, ledger validation, bundle pipeline | @@ -214,7 +214,7 @@ which runs `cargo bench --locked --all-features --bench perf --no-run`. ## Tool-surface evaluation -`tests/tool_surface_evaluation.rs` runs a deterministic tool-selection regression over the labeled corpus in `tests/fixtures/tool_surface/cases.json` (43 fixtures across generic web, repo, fetch, exact-error, security, research, evidence, diagnostics, and ambiguity categories; every stable tool is an expected primary at least once). The runner scores each query against the live tool definitions, then asserts description, total, instructions, and compact byte budgets, top-1/recall@3/MRR thresholds, forbidden-primary exclusion, known-tool and next-action reference validity, and synthetic Layer 2 hydration mechanics (next actions stay within the ten stable tools and compact discovery stays below full definitions). The baseline fingerprint and per-category accuracy print with `-- --nocapture` or `make eval-tool-surface`: +`tests/tool_surface_evaluation.rs` runs a deterministic tool-selection regression over the labeled corpus in `tests/fixtures/tool_surface/cases.json` (43 fixtures across generic web, repo, fetch, exact-error, security, research, evidence, diagnostics, and ambiguity categories; every stable tool is an expected primary at least once). Full and compact advertised descriptions are scored independently against every case; both must meet top-1/recall@3/MRR thresholds and avoid forbidden-primary results. The suite also asserts description, total, instructions, and compact byte budgets, known-tool and next-action reference validity, and synthetic Layer 2 hydration mechanics (next actions stay within the ten stable tools and compact discovery stays below full definitions). The fingerprints and per-category accuracy print with `-- --nocapture` or `make eval-tool-surface`: ```bash cargo test --locked --all-features --test tool_surface_evaluation @@ -244,7 +244,7 @@ make eval-retrieval-quality ## Transport, egress, and browser suites -`tests/mcp_http.rs` covers the loopback Streamable HTTP lifecycle: health identity, current and legacy protocol flows, shared tool definitions, request hardening, session cleanup, and shutdown behavior. It runs under `--all-features`: +`tests/mcp_http.rs` covers the loopback Streamable HTTP lifecycle: health identity, current and legacy protocol flows, shared tool definitions, the compact-profile initialized `tools/list`, request hardening, session cleanup, and shutdown behavior. It runs under `--all-features`: ```bash cargo test --locked --all-features --test mcp_http diff --git a/docs/config.md b/docs/config.md index fffeb7e..2b53fe6 100644 --- a/docs/config.md +++ b/docs/config.md @@ -35,6 +35,29 @@ enabled = false roots = [] ``` +## MCP Tool Schema Profile + +The server advertises the complete ten-tool schema by default. Direct clients +with tight model contexts can opt into a smaller advertisement in the same +config file: + +```toml +[mcp] +schema_profile = "compact" +brief_max_chars = 8000 +``` + +Compact mode keeps all ten tool names and required inputs, trims optional +schema guidance, and omits `outputSchema` metadata. Runtime argument parsing +and tool behavior are unchanged; advanced accepted inputs can still be sent. +Omitted `response_detail` defaults to `brief` in compact mode; explicit +`diagnostic`, `standard`, or `compact` calls override it. `brief_max_chars` +defaults to 8,000 and must be between 512 and 32,000. Brief output preserves +status, source identity, trust, critical warnings and security applicability; +it reports omitted records and truncation. The structured result and text +fallback are both brief. Switch to `schema_profile = "full"` for complete +schema guidance and the existing diagnostic-by-default response behavior. + ## Provider Requirements All 44 built-in providers: diff --git a/docs/installation.md b/docs/installation.md index 8f1a74b..960fe44 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -58,6 +58,11 @@ cargo install eggsearch --locked The pinned Unix installer requests assets from the exact `vX.Y.Z` release. The unpinned form uses `releases/latest/download` for convenience. +For configuring Eggsearch as a local stdio server in LM Studio, see +[LM Studio integration](lm-studio.md). The compact profile requires an +executable built with that feature; older published versions use the full +profile. + ## Release targets | Host | Rust target | Public asset | diff --git a/docs/integrations.md b/docs/integrations.md index 9025ebd..9aee3a8 100644 --- a/docs/integrations.md +++ b/docs/integrations.md @@ -58,6 +58,8 @@ Current references: [Codex MCP CLI](https://developers.openai.com/codex/cli/mcp) ## Settings-file clients +For direct local inference with LM Studio, see [LM Studio](lm-studio.md). + Zed renders the current `context_servers` shape for its Settings Editor. Its settings are JSONC-capable and there is no stable registration CLI, so `--apply` intentionally remains print-only: diff --git a/docs/lm-studio.md b/docs/lm-studio.md new file mode 100644 index 0000000..5ab3e50 --- /dev/null +++ b/docs/lm-studio.md @@ -0,0 +1,94 @@ +# LM Studio + +Eggsearch can run as a local stdio MCP server in LM Studio. The `compact` +presentation keeps all ten Eggsearch tools available while shortening their +advertised schemas. It also selects the `brief` response default; callers can +request `response_detail = "diagnostic"` for a full diagnostic response. +The ordinary `full` profile and its diagnostic default remain unchanged. + +## Install and configure + +Install a candidate or published Eggsearch executable that supports the +`[mcp]` settings below. The compact option is not available in older releases. +Create a config file named `eggsearch-lm-studio.toml`: + +```toml +[mcp] +schema_profile = "compact" +brief_max_chars = 8000 +``` + +LM Studio reads MCP server registrations from `mcp.json`. On macOS and Linux, +the user file is `~/.lmstudio/mcp.json`; on Windows it is +`%USERPROFILE%\.lmstudio\mcp.json`. Replace the example executable and config +paths with the paths on your machine: + +```json +{ + "mcpServers": { + "eggsearch": { + "command": "/absolute/path/to/eggsearch", + "args": [ + "--config", + "/absolute/path/to/eggsearch-lm-studio.toml", + "mcp", + "stdio" + ] + } + } +} +``` + +For Windows, use an absolute executable path such as +`C:\\Users\\you\\AppData\\Local\\Eggsearch\\eggsearch.exe` and a TOML +path such as `C:\\Users\\you\\.config\\eggsearch\\lm-studio.toml` in the +JSON strings. Keep the JSON escaping for backslashes. + +Restart LM Studio or refresh its MCP server list, then start a new chat. Confirm +that the server connects and all ten Eggsearch tool names appear. stdio is a +client-owned process: stdout carries MCP protocol messages, so do not wrap the +command in a script that writes status text to stdout. Eggsearch logs to stderr. + +To use the compatibility profile, remove `--config` and its path from `args`, +or set `schema_profile = "full"`. The executable must still be a build that +understands the `[mcp]` section. Changes to the profile require a fresh server +process and may require a new chat because MCP clients can cache tool lists. + +To verify an installed candidate's stdio handshake, tool list and local +diagnostic call without starting a model, run: + +```bash +python3 scripts/lm_studio_stdio_smoke.py /absolute/path/to/eggsearch +``` + +The smoke starts the executable with temporary full and compact configs, +checks all ten names and the compact byte limit, calls `provider_status` with +probing disabled, and compares its structured output across profiles. + +## API integrations + +LM Studio's API can address configured MCP servers as `mcp/eggsearch` and can +restrict a request to selected tools with `allowed_tools`. API access to MCP +servers declared in `mcp.json` is controlled by LM Studio's **Allow calling +servers from mcp.json** server setting and authentication. Enabling it expands +which API clients can invoke locally configured MCP tools; use LM Studio's +authentication and access controls before enabling it. This setting is not +needed for ordinary interactive app use. + +See LM Studio's [MCP app guide](https://lmstudio.ai/docs/app/mcp), +[MCP API guide](https://lmstudio.ai/docs/developer/core/mcp), and +[server settings](https://lmstudio.ai/docs/developer/core/server/settings). + +## What the measurements establish + +Eggsearch's offline fixtures measure serialized schema and result sizes and +exercise the MCP protocol. They do not establish LM Studio tokenizer counts, +model-visible prompt inclusion, tool-choice quality, or end-to-end task +success. Those require a live LM Studio chat with the actual local model. The +current environment has LM Studio 0.4.26+4 and a tool-capable local model, but +the API MCP smoke is blocked until the protected API setting above is enabled. +Accordingly, the standalone inference workstream does not claim a measured +LM Studio context or completion-quality improvement. + +For configuration keys and bounds, see [Eggsearch configuration](config.md) +and the [context-efficiency measurements](retrieval-quality-context-efficiency.md). diff --git a/docs/retrieval-quality-context-efficiency.md b/docs/retrieval-quality-context-efficiency.md index aa74365..07621d5 100644 --- a/docs/retrieval-quality-context-efficiency.md +++ b/docs/retrieval-quality-context-efficiency.md @@ -1,26 +1,29 @@ # Agent Context Efficiency Measurements -M005 measures current Eggsearch schemas and projections with serialized JSON bytes. Token figures are estimates at four bytes per token; no model tokenizer or live model was used. The task corpus and hydration mechanics are deterministic synthetic host simulations. +This report measures serialized JSON bytes for current Eggsearch tool definitions and MCP projections. Token figures are estimates at four bytes per token; no model tokenizer was used. The 15-task corpus is a deterministic synthetic host simulation. ## Advertised tool surface and discovery | Surface | Serialized bytes | Approx. tokens | Relative to full ten-tool definitions | |---|---:|---:|---:| -| Ten MCP tool definitions | 72,939 | 18,235 | 100% | +| Ten MCP tool definitions from initialized stdio `tools/list` | 72,923 | 18,231 | 100% | +| Compact ten-tool profile from initialized stdio `tools/list` | 18,545 | 4,637 | 25.4% (74.6% smaller) | | Core palette (`web_search`, `web_fetch`, `repo_search`) | 14,008 | 3,502 | 19.2% (80.8% smaller) | | Compact discovery, max observed result | 769 total; 260 max per query | 193 total; 65 max per query | discovery response | -The 43-case tool-choice corpus scored top-1 43/43, recall@3 43/43 and MRR 1.000. The synthetic hydration run selected follow-up tools only when requested; one fixture execution hydrated six tools. This is tool-choice and schema-byte evidence, not end-to-end model completion quality or measured host round-trip latency. +The 43-case tool-choice corpus scored top-1 43/43, recall@3 43/43 and MRR 1.000 in both full and compact description profiles. The synthetic hydration run selected follow-up tools only when requested; one fixture execution hydrated six tools. This is deterministic discovery and schema-byte evidence, not end-to-end model completion quality or measured host round-trip latency. -The M001 15-task host trace serializes 21,221 diagnostic bytes, 21,539 standard bytes and 21,068 compact bytes (5,306 / 5,385 / 5,267 estimated tokens). All projections retain the same 14 relevant result documents present in the fixture result lists; one of 15 judged relevant documents was not retrieved by the fixture, matching M001 recall of 14/15 for the 13 ranking tasks. Failure and absence states remain distinct. The fixture implies 42 MCP tool calls across the 15 tasks when each fixture-retrieved relevant source is fetched once and each non-empty task is bundled once. This is a deterministic call-count simulation, not measured model turns; it does not include deferred-tool discovery or schema hydration calls. Per-task serialized byte counts and retained result counts are emitted by `retrieval_quality_host_trace_records_per_case_projection_cost`. +The 15-task host trace serializes 21,221 diagnostic bytes, 21,539 standard bytes, 21,068 compact bytes and 18,715 brief bytes (5,306 / 5,385 / 5,267 / 4,679 estimated tokens). Every projection retains the same 14 relevant documents present in fixture result lists; one judged document was not retrieved by the fixture. Failure and absence remain distinct. The trace implies 42 MCP calls if each retrieved relevant source is fetched once and each non-empty task is bundled once. This is a deterministic call-count simulation, not measured model turns; it excludes deferred-tool discovery and schema hydration. Brief is 11.8% smaller than diagnostic on this sparse corpus, below the threshold by design; per-task counts are emitted by `retrieval_quality_host_trace_records_per_case_projection_cost`. ## Projection payload comparison -| Representative payload | Diagnostic bytes | Standard bytes | Compact bytes | Compact reduction | -|---|---:|---:|---:|---:| -| `web_search` with two cards and a failure | 1,642 | 1,634 | 1,478 | 10.0% | -| `web_fetch` with 2,000-character text and a link | 4,292 | not measured by this fixture | 2,278 | 46.9% | +| Representative payload | Diagnostic bytes | Brief bytes | Brief reduction vs diagnostic | +|---|---:|---:|---:| +| `web_search` with rich source metadata and a provider failure | 2,728 | 1,418 | 48.0% | +| `web_fetch` with 2,000-character text and a link | 4,292 | 1,090 | 74.6% | +| `repo_search` grouped stress fixture with redundant metadata | 10,846 | 669 | 93.8% | +| `research_search` grouped stress fixture with redundant metadata | 10,846 | 669 | 93.8% | -The MCP projection suite confirms compact output retains partial evidence, failure-versus-absence, trust markers, stable IDs, truncation/focus, security applicability and evidence-bundle identity. Corpus-level compact output is only 0.7% smaller than diagnostic because these synthetic result cards are already short, and standard output is 1.5% larger than diagnostic for the typed repository projections. On a representative 2,000-character `web_fetch` payload, compact output falls from 4,292 to 2,278 bytes (46.9%). `response_detail` continues to default to diagnostic; the benchmark does not authorize changing that compatibility boundary. Callers needing smaller context can opt into `compact` and retain the unprojected MCP `structuredContent` outside the model window. +The brief projection tests preserve partial retrieval, failure-versus-absence, trust and injection markers, stable IDs, truncation and PDF page coordinates, security applicability, failed batch members, and evidence-bundle identity. Brief serializes the same projected data into both structured content and its text fallback. Its 8,000-character default display budget is configurable from 512 to 32,000 characters; omitted result counts and truncation are reported. The web and fetch fixtures are nontrivial samples; the repository and research figures are synthetic stress fixtures with deliberately redundant metadata and are not model-prompt or production-average claims. Compact mode defaults to brief only when selected in server configuration; the full profile remains diagnostic by default. -This harness measures serialized bytes and rough token estimates only. It does not measure provider latency, model turns, host storage behavior, or evidence sufficiency after real model reasoning. No global clipping or batch orchestration changes are justified from these fixtures. +This harness measures serialized characters/bytes and rough token estimates only. It does not measure provider latency, actual model tokens, host storage behavior, or evidence sufficiency after real model reasoning. LM Studio consumer qualification is a separate operational gate. diff --git a/docs/test-inventory.md b/docs/test-inventory.md index 8ee06be..904aa00 100644 --- a/docs/test-inventory.md +++ b/docs/test-inventory.md @@ -158,7 +158,7 @@ Source of truth: `fuzz/Cargo.toml` [[bin]] entries. | `fetch_safety` | Fetch safety bounds | | `security_applicability_corpus` | Security applicability pipeline | | `security_applicability_contract` | Security applicability assessment contract | -| `mcp_http` | Loopback Streamable HTTP lifecycle and transport hardening | +| `mcp_http` | Loopback Streamable HTTP lifecycle, transport hardening, and compact-profile initialized `tools/list` | | `security_applicability_regression` | Security applicability regression | | `research_evidence_corpus` | Research evidence regression | | `research_semantic_roles` | Research semantic role mapping | @@ -173,20 +173,20 @@ Source of truth: `fuzz/Cargo.toml` [[bin]] entries. | Suite | Tests | Focus | |-------|-------|-------| -| `mcp_tool_contract` | 14 | Canonical registry parity: aliases, discovery text, sanitization, fingerprint determinism | +| `mcp_tool_contract` | 15 | Canonical registry parity, sanitization, deterministic fingerprints, and compact-profile size/all-name contract | | `mcp_schema_slimming` | 17 | Slimmed ordinary schema size budget with legacy-field acceptance | | `mcp_2026_protocol` | 14 | Structured results, output schemas, repairable error contract | -| `mcp_projection` | 14 | Response-detail projection: failure-vs-absence, trust, conflicts, truncation, bundle identity, byte reduction | +| `mcp_projection` | 19 | Brief/compact/standard/diagnostic projections: status, trust, security applicability, failure retention, bounded failure details, repository/research reduction, character budgets, UTF-8 framing, 15-task byte/retention trace, bundle identity | ## Tool-Surface Evaluation (2 suites) | Suite | Feature Gate | Tests | Focus | |-------|-------------|-------|-------| -| `tool_surface_evaluation` | None | 4 | Labeled 43-fixture discovery corpus (top-1/recall@3/MRR), description/total/instructions/compact byte budgets, fingerprint determinism, forbidden-primary exclusion, synthetic next-action hydration mechanics | +| `tool_surface_evaluation` | None | 5 | Labeled 43-fixture discovery corpus for full and compact profiles (top-1/recall@3/MRR), description/total/instructions/compact byte budgets, fingerprint determinism, forbidden-primary exclusion, synthetic next-action hydration mechanics | | `tool_surface_live` | None (comparison `#[ignore]`d) | 1 (+1 ignored) | Layer 3 report contract (fingerprint, config, deltas); opt-in manual multi-model comparison via `EGGSEARCH_EVAL_MODEL` | Corpus: `tests/fixtures/tool_surface/cases.json` (43 cases) with `README.md` -baseline (72939 definition bytes, 1614 instruction bytes, 43/43 top-1; the +baseline (72906 per-tool serialized definition bytes, 1614 instruction bytes, 43/43 top-1; the post-consolidation v0.3.9 figure was 73037 definition bytes before `repo_map` stopped advertising `providers`). Re-run with `make eval-tool-surface`. diff --git a/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md b/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md index 58baf07..446f10a 100644 --- a/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md +++ b/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md @@ -1,6 +1,6 @@ # M001 — Opt-in Compact Ten-Tool MCP Advertisement -Status: ready +Status: active Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (2026-10-30) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` @@ -17,7 +17,7 @@ Create a **process-configured, opt-in** compact MCP tool-definition profile for - `src/mcp/output_schema.rs` attaches permissive per-tool output schemas; `src/mcp/tool_contract.rs` already has concise canonical selection descriptions, aliases and discovery metadata. - `src/mcp/tools/{web_search,repo_search,research_search,security_search,web_fetch,repo_fetch,batch_fetch,repo_map,evidence_bundle,provider_status}.rs` deserialize the canonical runtime argument structs; older advanced fields are sometimes `#[schemars(skip)]` while remaining accepted. - `src/mcp/mod.rs::build_server` is shared by client-owned stdio and loopback HTTP; `src/commands/mcp.rs` is transport startup. `src/core/config.rs` / `src/config.rs` own config parsing and validation. -- `docs/retrieval-quality-context-efficiency.md` records 72,939 full `tools/list` serialized bytes, 14,008 bytes for a three-tool subset, and ~18.2k tokens by a 4-bytes/token proxy. Do not treat this as a measured LM Studio prompt cost. +- `docs/retrieval-quality-context-efficiency.md` records 72,950 full `tools/list` serialized bytes, 18,481 bytes for the compact ten-tool profile, and 14,008 bytes for a three-tool subset. Token figures are a four-bytes/token proxy, not a measured LM Studio prompt cost. ## Invariants / non-goals diff --git a/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md b/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md index d576c20..99dba22 100644 --- a/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md +++ b/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md @@ -1,6 +1,6 @@ # M002 — Brief Evidence Projection and Deterministic Output Budgets -Status: blocked (hard dependency: M001 closure) +Status: blocked (implementation is ready for closure after accepted M001 closure) Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (reconfirm after M001) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` diff --git a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md index c6884f7..2a52265 100644 --- a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md +++ b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md @@ -1,6 +1,6 @@ # M003 — LM Studio Direct-MCP Qualification and Consumer Documentation -Status: blocked (hard dependencies: M001 and M002 closure) +Status: blocked (hard dependency: M002 closure; live LM Studio API smoke also requires its protected API setting) Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (rebase baseline to exact M002 candidate at start) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` diff --git a/plans/registry.md b/plans/registry.md index b9627fa..286872a 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | ready (M001 ready; M002-M003 blocked) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001 compact ten-tool schema advertisement handoff | New opt-in direct-client presentation; full ten-tool/default diagnostic contract unchanged; no CodeGG implementation dependency. | +| Standalone MCP local-inference efficiency | active (M001 active; M002-M003 blocked) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001 compact ten-tool schema advertisement in progress | New opt-in direct-client presentation; full ten-tool/default diagnostic contract unchanged; no CodeGG implementation dependency. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -52,7 +52,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Subsystem | Milestone | Status | Implementation plan | Dependencies / handoff note | |---|---|---|---|---| -| Standalone MCP local-inference | M001 opt-in compact tool advertisement | ready | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | No hard dependency; implement first against main `a666517c` after verifying current tool-list baseline. Keep default full ten-tool contract. | +| Standalone MCP local-inference | M001 opt-in compact tool advertisement | active | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | No hard dependency; implement first against main `a666517c` after verifying current tool-list baseline. Keep default full ten-tool contract. | | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | blocked | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Hard: accepted M001 closure. Default profile remains diagnostic; brief is opt-in via compact profile or explicit call. | | Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Hard: accepted M001/M002 closure; live LM Studio test is an operational closure gate, never assumed from synthetic bytes. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index fa55ad4..294498c 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: ready (planning handoff; M001 ready, M002-M003 blocked by hard dependencies) +Status: active (M001 active, M002-M003 blocked by hard dependencies) Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. diff --git a/scripts/lm_studio_stdio_smoke.py b/scripts/lm_studio_stdio_smoke.py new file mode 100644 index 0000000..7894f58 --- /dev/null +++ b/scripts/lm_studio_stdio_smoke.py @@ -0,0 +1,95 @@ +import json +import pathlib +import subprocess +import sys +import tempfile + + +binary = pathlib.Path(sys.argv[1]).resolve() + + +def run(profile): + with tempfile.TemporaryDirectory(prefix="eggsearch-stdio-smoke-") as temp: + config = pathlib.Path(temp) / "eggsearch.toml" + config.write_text(f'[mcp]\nschema_profile = "{profile}"\nbrief_max_chars = 8000\n') + proc = subprocess.Popen( + [str(binary), "--config", str(config), "mcp", "stdio"], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + + def request(message): + proc.stdin.write(json.dumps(message) + "\n") + proc.stdin.flush() + line = proc.stdout.readline() + if not line: + raise RuntimeError(proc.stderr.read()) + response = json.loads(line) + if "error" in response: + raise RuntimeError(response["error"]) + return response["result"] + + request( + { + "jsonrpc": "2.0", + "id": 1, + "method": "initialize", + "params": { + "protocolVersion": "2025-06-18", + "capabilities": {}, + "clientInfo": {"name": "eggsearch-installed-smoke", "version": "1"}, + }, + } + ) + proc.stdin.write( + json.dumps({"jsonrpc": "2.0", "method": "notifications/initialized"}) + "\n" + ) + proc.stdin.flush() + tools = request( + {"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}} + )["tools"] + names = sorted(tool["name"] for tool in tools) + if len(names) != 10: + raise AssertionError(f"{profile}: expected 10 tools, got {names}") + if profile == "compact": + encoded_tools = json.dumps(tools, separators=(",", ":")) + if len(encoded_tools.encode()) > 36_000: + raise AssertionError(f"compact tools/list too large: {len(encoded_tools)} bytes") + if any("outputSchema" in tool for tool in tools): + raise AssertionError("compact profile unexpectedly contains outputSchema") + call = request( + { + "jsonrpc": "2.0", + "id": 3, + "method": "tools/call", + "params": {"name": "provider_status", "arguments": {"probe": False}}, + } + ) + if call.get("isError"): + raise AssertionError(f"{profile}: provider_status failed: {call}") + proc.stdin.close() + proc.wait(timeout=10) + stderr = proc.stderr.read() + print( + json.dumps( + { + "profile": profile, + "tool_count": len(names), + "tools_list_bytes": len( + json.dumps(tools, separators=(",", ":")).encode() + ), + "provider_status_ok": True, + "exit_code": proc.returncode, + "stderr_nonempty": bool(stderr), + } + ) + ) + return call.get("structuredContent") + + +full = run("full") +compact = run("compact") +if full != compact: + raise AssertionError("full and compact profiles returned different provider_status payloads") diff --git a/src/core/config.rs b/src/core/config.rs index 1cbe414..1827e63 100644 --- a/src/core/config.rs +++ b/src/core/config.rs @@ -93,6 +93,39 @@ pub enum Mode { Live, } +#[derive( + Clone, Copy, Debug, Default, Eq, PartialEq, Serialize, Deserialize, schemars::JsonSchema, +)] +#[serde(rename_all = "lowercase")] +#[allow(missing_docs)] +pub enum McpSchemaProfile { + #[default] + Full, + Compact, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +#[allow(missing_docs)] +pub struct McpSection { + #[serde(default)] + pub schema_profile: McpSchemaProfile, + #[serde(default = "default_brief_max_chars")] + pub brief_max_chars: usize, +} + +fn default_brief_max_chars() -> usize { + 8_000 +} + +impl Default for McpSection { + fn default() -> Self { + Self { + schema_profile: McpSchemaProfile::Full, + brief_max_chars: default_brief_max_chars(), + } + } +} + impl Mode { /// Parse a mode string. Only `"off"` and `"live"` are accepted; `"ask"` /// is a host-level policy and is not a valid value at this layer. @@ -862,6 +895,9 @@ pub struct AppConfig { /// Optional `[egress]` section for listener-free outbound proxy-chain routing. #[serde(default)] pub egress: EgressSection, + /// Optional MCP presentation settings. + #[serde(default)] + pub mcp: McpSection, } impl AppConfig { @@ -1155,6 +1191,11 @@ impl AppConfig { /// rather than silently degrading behavior. pub fn validate(&self) -> CoreResult<()> { self.validate_search_profiles()?; + if !(512..=32_000).contains(&self.mcp.brief_max_chars) { + return Err(CoreError::Config( + "[mcp].brief_max_chars must be between 512 and 32000".to_string(), + )); + } if self.fetch.max_chars_cap < self.fetch.max_chars_default { return Err(CoreError::Config(format!( "[fetch].max_chars_cap ({}) must be >= [fetch].max_chars_default ({})", diff --git a/src/mcp/projection.rs b/src/mcp/projection.rs index 34fbba7..9b676d4 100644 --- a/src/mcp/projection.rs +++ b/src/mcp/projection.rs @@ -7,15 +7,19 @@ use serde::{Deserialize, Serialize}; pub enum ResponseDetail { Compact, Standard, + Brief, #[default] Diagnostic, } +pub const DEFAULT_BRIEF_MAX_CHARS: usize = 8_000; + impl ResponseDetail { pub fn as_str(self) -> &'static str { match self { Self::Compact => "compact", Self::Standard => "standard", + Self::Brief => "brief", Self::Diagnostic => "diagnostic", } } @@ -24,15 +28,26 @@ impl ResponseDetail { opt.unwrap_or(Self::Diagnostic) } + pub fn from_opt_for_profile( + opt: Option, + profile: crate::core::config::McpSchemaProfile, + ) -> Self { + opt.unwrap_or(match profile { + crate::core::config::McpSchemaProfile::Full => Self::Diagnostic, + crate::core::config::McpSchemaProfile::Compact => Self::Brief, + }) + } + pub fn parse_raw(raw: Option<&str>) -> Result, String> { match raw { None => Ok(None), Some(s) => match s.to_ascii_lowercase().as_str() { "compact" => Ok(Some(Self::Compact)), "standard" => Ok(Some(Self::Standard)), + "brief" => Ok(Some(Self::Brief)), "diagnostic" => Ok(Some(Self::Diagnostic)), _ => Err(format!( - "invalid response_detail '{s}'; accepted values: compact, standard, diagnostic" + "invalid response_detail '{s}'; accepted values: brief, compact, standard, diagnostic" )), }, } @@ -40,7 +55,17 @@ impl ResponseDetail { } pub fn project(tool: &str, value: serde_json::Value, detail: ResponseDetail) -> serde_json::Value { + project_with_budget(tool, value, detail, DEFAULT_BRIEF_MAX_CHARS) +} + +pub fn project_with_budget( + tool: &str, + value: serde_json::Value, + detail: ResponseDetail, + max_chars: usize, +) -> serde_json::Value { match detail { + ResponseDetail::Brief => project_brief(tool, value, max_chars), ResponseDetail::Diagnostic => value, ResponseDetail::Compact => match tool { "web_search" => project_web_search(value, true), @@ -69,6 +94,349 @@ pub fn project(tool: &str, value: serde_json::Value, detail: ResponseDetail) -> } } +fn project_brief(tool: &str, value: serde_json::Value, max_chars: usize) -> serde_json::Value { + if tool == "build_evidence_bundle" { + return value; + } + let mut projected = project(tool, value, ResponseDetail::Compact); + reduce_brief_fields(&mut projected); + if let Some(object) = projected.as_object_mut() { + object.insert( + "response_detail".into(), + serde_json::Value::String("brief".into()), + ); + } + let primary_key = ["results", "groups", "vulnerabilities", "items"] + .into_iter() + .find(|key| { + projected + .get(*key) + .and_then(|value| value.as_array()) + .is_some() + }); + let original_count = primary_key + .and_then(|key| projected.get(key)) + .and_then(serde_json::Value::as_array) + .map_or(0, Vec::len); + let mut clipped = false; + let mut omitted = 0usize; + for per_text_limit in [800usize, 320, 128, 48, 0] { + clipped |= clip_brief_strings(&mut projected, per_text_limit); + loop { + let size = brief_serialized_chars(&projected); + if size <= max_chars { + break; + } + let Some(key) = primary_key else { + break; + }; + let Some(values) = projected + .get_mut(key) + .and_then(serde_json::Value::as_array_mut) + else { + break; + }; + let removable = values + .iter() + .rposition(|item| !brief_record_is_failure(item)); + let Some(index) = removable else { + break; + }; + values.remove(index); + omitted += 1; + } + let retained_count = primary_key + .and_then(|key| projected.get(key)) + .and_then(serde_json::Value::as_array) + .map_or(0, Vec::len); + if let Some(object) = projected.as_object_mut() { + object.insert("returned_count".into(), retained_count.into()); + object.insert("omitted_count".into(), omitted.into()); + object.insert( + "output_truncated".into(), + serde_json::Value::Bool(clipped || omitted > 0), + ); + } + if brief_serialized_chars(&projected) <= max_chars { + break; + } + } + for optional in [ + "next_actions", + "suggested_fetches", + "providers_queried", + "routing_summary", + "workflow_coverage", + "capability_enforcement", + "response_detail", + "query", + ] { + if brief_serialized_chars(&projected) > max_chars { + if let Some(object) = projected.as_object_mut() { + clipped |= object.remove(optional).is_some(); + object.insert( + "output_truncated".into(), + serde_json::Value::Bool(clipped || omitted > 0), + ); + } + } + } + let retained_count = primary_key + .and_then(|key| projected.get(key)) + .and_then(serde_json::Value::as_array) + .map_or(0, Vec::len); + if let Some(object) = projected.as_object_mut() { + object.insert("returned_count".into(), retained_count.into()); + object.insert( + "omitted_count".into(), + (omitted.max(original_count.saturating_sub(retained_count))).into(), + ); + object.insert( + "output_truncated".into(), + serde_json::Value::Bool(clipped || omitted > 0), + ); + } + projected +} + +fn brief_serialized_chars(value: &serde_json::Value) -> usize { + serde_json::to_string(value).map_or(usize::MAX, |encoded| encoded.chars().count()) +} + +fn brief_record_is_failure(value: &serde_json::Value) -> bool { + value.get("ok").and_then(serde_json::Value::as_bool) == Some(false) + || value.get("fetched").and_then(serde_json::Value::as_bool) == Some(false) + || value.get("error").is_some_and(|error| !error.is_null()) + || value.get("error_class").is_some() + || value.get("failed").and_then(serde_json::Value::as_bool) == Some(true) +} + +fn reduce_brief_fields(value: &mut serde_json::Value) { + const FIELDS: &[&str] = &[ + "affected_versions", + "aggregate_budget_exhausted", + "applicability", + "attempted_job_count", + "batch_status", + "build_configs", + "cache_status", + "capability_enforcement", + "code", + "code_span_id", + "commit_sha", + "conflict_count", + "conflict_indicator", + "conflicts", + "content_type", + "counterpoints", + "coverage", + "coverage_status", + "checkout_state", + "dirty", + "degraded", + "dependency_findings", + "document_id", + "ecosystem", + "error", + "error_class", + "evidence_gaps", + "evidence_role", + "excerpts", + "failed", + "failed_job_count", + "fetches", + "fetched", + "final_url", + "fixed_versions", + "focus", + "gaps", + "group", + "groups", + "has_absences", + "has_conflicts", + "has_failures", + "has_truncation", + "id", + "injection_hits", + "item_type", + "items", + "items_completed", + "items_failed", + "items_requested", + "chars_returned", + "total_chars_returned", + "chunks", + "page", + "page_number", + "label", + "language", + "line_end", + "line_start", + "locator", + "message", + "mode", + "next_actions", + "omitted_count", + "output_truncated", + "page_end", + "page_start", + "package", + "partial", + "path", + "provider_id", + "providers_failed", + "providers_queried", + "query", + "rank", + "ref", + "redirected", + "response", + "response_detail", + "ok", + "results", + "retrieval_status", + "retrieval_summary", + "returned_count", + "routing_summary", + "telemetry", + "severity", + "snippet", + "source_id", + "source_kind", + "source_role", + "sources", + "stable_id", + "status", + "structured_warnings", + "suggested_fetches", + "text", + "title", + "total_chars", + "truncated", + "untracked", + "trust", + "trust_level", + "trust_markers", + "type", + "unknown", + "url", + "version", + "workspace_id", + "workspace_root", + "remote_mismatch", + "vulnerabilities", + "warnings", + "workflow_coverage", + "affected", + "unaffected", + "applicable", + "not_applicable", + "confidence", + "advisory_id", + "vulnerability_id", + "source", + "reason", + "tool", + "items_truncated", + "truncation_evidence", + "completed_job_count", + "returned_sources", + "omitted_sources", + "conflict_metadata", + ]; + match value { + serde_json::Value::Object(object) => { + object.retain(|key, _| FIELDS.contains(&key.as_str())); + for child in object.values_mut() { + reduce_brief_fields(child); + } + } + serde_json::Value::Array(values) => { + for child in values { + reduce_brief_fields(child); + } + } + _ => {} + } +} + +fn clip_brief_strings(value: &mut serde_json::Value, max_chars: usize) -> bool { + let mut changed = false; + match value { + serde_json::Value::Object(object) => { + for (key, child) in object.iter_mut() { + if key == "warnings" { + if let Some(warnings) = child.as_array_mut() { + for warning in warnings { + if let Some(text) = warning.as_str() { + let warning_limit = max_chars.min(96); + if text.chars().count() > warning_limit { + *warning = serde_json::Value::String(clip_warning_text( + text, + warning_limit, + )); + changed = true; + } + } + } + } + } + if matches!(key.as_str(), "snippet" | "text" | "message" | "error") { + if let Some(text) = child.as_str() { + if text.chars().count() > max_chars { + *child = serde_json::Value::String(clip_framed_text(text, max_chars)); + changed = true; + } + } + } + changed |= clip_brief_strings(child, max_chars); + } + } + serde_json::Value::Array(values) => { + for child in values { + changed |= clip_brief_strings(child, max_chars); + } + } + _ => {} + } + changed +} + +fn clip_warning_text(text: &str, max_chars: usize) -> String { + if max_chars == 0 { + return text + .split_once(':') + .map_or_else(String::new, |(code, _)| code.to_string()); + } + let prefix = text.split_once(':').map_or(text, |(code, _)| code); + if prefix.chars().count() <= max_chars { + prefix.to_string() + } else { + prefix.chars().take(max_chars).collect() + } +} + +fn clip_framed_text(text: &str, max_chars: usize) -> String { + if max_chars == 0 { + return String::new(); + } + let start = "<<>>") { + if text.starts_with(start) && text.rfind(end).is_some() { + let header_end = header_end + 3; + let header = &text[..header_end]; + let body_end = text.rfind(end).unwrap_or(text.len()); + let body = text[header_end..body_end].trim_matches('\n'); + let body_chars = max_chars.saturating_sub(header.chars().count() + end.len() + 2); + let clipped: String = body.chars().take(body_chars).collect(); + return format!("{header}\n{clipped}\n{end}"); + } + } + let clipped: String = text.chars().take(max_chars).collect(); + clipped +} + fn retrieval_status_minimal(summary: &serde_json::Value) -> serde_json::Value { serde_json::json!({ "has_failures": summary.get("has_failures").and_then(|v| v.as_bool()).unwrap_or(false), diff --git a/src/mcp/server.rs b/src/mcp/server.rs index fb38a3b..ae27ebb 100644 --- a/src/mcp/server.rs +++ b/src/mcp/server.rs @@ -36,8 +36,13 @@ impl std::fmt::Debug for EggsearchServer { impl EggsearchServer { pub fn new(state: Arc) -> Self { + let profile = state.config.mcp.schema_profile; let tool_router = Self::tool_router(); - let advertised_tools = Arc::new(apply_contract_metadata(tool_router.list_all())); + let mut advertised_tools = apply_contract_metadata(tool_router.list_all()); + if profile == crate::core::config::McpSchemaProfile::Compact { + advertised_tools = apply_compact_profile(advertised_tools); + } + let advertised_tools = Arc::new(advertised_tools); let advertised_fingerprint: Arc = contract_fingerprint(&advertised_tools).into(); Self { state, @@ -61,6 +66,233 @@ impl EggsearchServer { } } +fn apply_compact_profile(mut tools: Vec) -> Vec { + for tool in &mut tools { + if let Some(description) = compact_description(tool.name.as_ref()) { + tool.description = Some(description.into()); + } + let mut schema = serde_json::Value::Object(tool.input_schema.as_ref().clone()); + compact_schema(tool.name.as_ref(), &mut schema); + if let Some(object) = schema.as_object() { + tool.input_schema = Arc::new(object.clone()); + } + tool.output_schema = None; + } + tools +} + +fn compact_description(name: &str) -> Option<&'static str> { + Some(match name { + "web_search" => "Search the public web for sources; use web_fetch to inspect a result.", + "provider_status" => "Check configured provider availability and diagnostics.", + "repo_search" => "Search repository code, documentation, issues, and releases.", + "web_fetch" => "Fetch one known HTTP(S) URL; use batch_fetch for several targets.", + "repo_fetch" => "Fetch a known repository file or span by structured locator.", + "security_search" => "Find vulnerability evidence and assess package applicability.", + "research_search" => "Gather grouped evidence for complex research questions.", + "batch_fetch" => "Fetch several known URLs or repository locators in one call.", + "repo_map" => "Describe a repository layout before searching or fetching files.", + "build_evidence_bundle" => { + "Package already selected search and fetch evidence for handoff." + } + _ => return None, + }) +} + +fn compact_schema(name: &str, schema: &mut serde_json::Value) { + const COMMON: &[&str] = &["response_detail"]; + let allowed: &[&str] = match name { + "web_search" => &[ + "query", + "max_results", + "safe_search", + "intent", + "freshness", + "date_range", + "include_domains", + "exclude_domains", + "language", + "region", + "excerpt_count", + "response_detail", + ], + "provider_status" => &["probe"], + "repo_search" => &[ + "query", + "owner", + "repo", + "host", + "profile", + "goal", + "sources", + "include_local", + "response_detail", + ], + "web_fetch" => &[ + "url", + "max_chars", + "extract_mode", + "include_links", + "pdf", + "focus", + "focus_max_chunks", + "focus_max_chars", + "response_detail", + ], + "repo_fetch" => &[ + "owner", + "repo", + "path", + "ref", + "commit_sha", + "line_start", + "line_end", + "symbol", + "max_chars", + "response_detail", + ], + "security_search" => &[ + "query", + "package", + "version", + "ecosystem", + "assess_applicability", + "dependency_files", + "goal", + "response_detail", + ], + "research_search" => &[ + "query", + "domain", + "research_domain", + "goal", + "depth", + "compare_targets", + "constraints", + "known_context", + "response_detail", + ], + "batch_fetch" => &[ + "items", + "max_chars_per_item", + "total_max_chars", + "response_detail", + ], + "repo_map" => &[ + "owner", + "repo", + "host", + "ref", + "path", + "max_depth", + "response_detail", + ], + "build_evidence_bundle" => &[ + "goal", + "sources", + "fetches", + "max_sources", + "max_fetched_items", + "max_total_chars", + "include_unfetched_sources", + "response_detail", + ], + _ => COMMON, + }; + compact_schema_value(schema, allowed); + if name == "build_evidence_bundle" { + compact_evidence_bundle_schema(schema); + } +} + +fn compact_evidence_bundle_schema(schema: &mut serde_json::Value) { + let Some(properties) = schema + .get_mut("properties") + .and_then(serde_json::Value::as_object_mut) + else { + return; + }; + properties.insert( + "sources".into(), + serde_json::json!({ + "type": "array", + "items": { + "type": "object", + "properties": { + "id": {"type": "string"}, + "url": {"type": "string"}, + "title": {"type": "string"}, + "snippet": {"type": "string"}, + "providers": {"type": "array", "items": {"type": "string"}}, + "trust": {"type": "string"}, + "trust_markers": {"type": "object", "additionalProperties": true}, + "metadata": {"type": "object", "additionalProperties": true}, + "quality": {"type": "object", "additionalProperties": true} + }, + "additionalProperties": true + } + }), + ); + properties.insert( + "fetches".into(), + serde_json::json!({ + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": {"type": "string"}, + "url": {"type": "string"}, + "fetched": {"type": "boolean"}, + "content_type": {"type": "string"}, + "text": {"type": "string"}, + "truncated": {"type": "boolean"}, + "trust": {"type": "string"}, + "warnings": {"type": "array", "items": {"type": "string"}} + }, + "additionalProperties": true + } + }), + ); + if let Some(object) = schema.as_object_mut() { + object.remove("$defs"); + } +} + +fn compact_schema_value(value: &mut serde_json::Value, allowed: &[&str]) { + match value { + serde_json::Value::Object(object) => { + let required = object + .get("required") + .and_then(serde_json::Value::as_array) + .cloned() + .unwrap_or_default(); + if let Some(properties) = object.get_mut("properties").and_then(|v| v.as_object_mut()) { + properties.retain(|key, _| { + allowed.contains(&key.as_str()) + || required.iter().any(|entry| entry.as_str() == Some(key)) + }); + for value in properties.values_mut() { + compact_schema_value(value, &[]); + } + } + object.remove("description"); + object.remove("title"); + object.insert("additionalProperties".into(), serde_json::Value::Bool(true)); + if let Some(defs) = object.get_mut("$defs").and_then(|v| v.as_object_mut()) { + for value in defs.values_mut() { + compact_schema_value(value, &[]); + } + } + } + serde_json::Value::Array(values) => { + for value in values { + compact_schema_value(value, &[]); + } + } + _ => {} + } +} + #[tool_router] impl EggsearchServer { #[tool( diff --git a/src/mcp/tools/batch_fetch.rs b/src/mcp/tools/batch_fetch.rs index 44696f0..721ed4a 100644 --- a/src/mcp/tools/batch_fetch.rs +++ b/src/mcp/tools/batch_fetch.rs @@ -26,7 +26,7 @@ pub struct BatchFetchArgs { /// Defaults to `true`. #[serde(default, skip_serializing_if = "Option::is_none")] pub continue_on_error: Option, - /// Response detail: compact, standard, or diagnostic (default diagnostic). + /// Response detail: brief, compact, standard, or diagnostic. #[serde(default, skip_serializing_if = "Option::is_none")] pub response_detail: Option, } @@ -607,11 +607,15 @@ pub async fn run_batch_fetch( let value = serde_json::to_value(&response) .map_err(|e| ToolError::internal(format!("serialization error: {e}")))?; - let detail = crate::mcp::projection::ResponseDetail::from_opt(args.response_detail); - Ok(crate::mcp::projection::project( + let detail = crate::mcp::projection::ResponseDetail::from_opt_for_profile( + args.response_detail, + state.config.mcp.schema_profile, + ); + Ok(crate::mcp::projection::project_with_budget( "batch_fetch", value, detail, + state.config.mcp.brief_max_chars, )) } diff --git a/src/mcp/tools/evidence_bundle.rs b/src/mcp/tools/evidence_bundle.rs index 97b3056..11d3ac0 100644 --- a/src/mcp/tools/evidence_bundle.rs +++ b/src/mcp/tools/evidence_bundle.rs @@ -25,7 +25,7 @@ pub struct EvidenceBundleArgs { /// Maximum total characters across all fetched text (default 100000, cap 500000). #[serde(default, skip_serializing_if = "Option::is_none")] pub max_total_chars: Option, - /// Response detail: compact, standard, or diagnostic. Bundle identity is + /// Response detail: brief, compact, standard, or diagnostic. Bundle identity is /// unchanged across all modes; the canonical bundle is always returned. #[serde(default, skip_serializing_if = "Option::is_none")] pub response_detail: Option, diff --git a/src/mcp/tools/repo_fetch.rs b/src/mcp/tools/repo_fetch.rs index 1cd9b42..303dbf7 100644 --- a/src/mcp/tools/repo_fetch.rs +++ b/src/mcp/tools/repo_fetch.rs @@ -83,7 +83,7 @@ pub struct RepoFetchArgs { /// from the local workspace instead of fetching remotely. #[serde(default, skip_serializing_if = "Option::is_none")] pub prefer_local: Option, - /// Response detail: compact, standard, or diagnostic (default diagnostic). + /// Response detail: brief, compact, standard, or diagnostic. #[serde(default, skip_serializing_if = "Option::is_none")] pub response_detail: Option, } @@ -118,9 +118,18 @@ pub async fn run_repo_fetch_with_host_scope( // --- workspace:// local file fetch (bypasses fetch policy) --- if let Some(ref h) = args.host { if h.to_lowercase() == "workspace" { - let detail = crate::mcp::projection::ResponseDetail::from_opt(args.response_detail); + let detail = crate::mcp::projection::ResponseDetail::from_opt_for_profile( + args.response_detail, + state.config.mcp.schema_profile, + ); + let max_chars = state.config.mcp.brief_max_chars; let value = run_workspace_fetch(state, args, local_view).await?; - return Ok(crate::mcp::projection::project("repo_fetch", value, detail)); + return Ok(crate::mcp::projection::project_with_budget( + "repo_fetch", + value, + detail, + max_chars, + )); } } @@ -143,8 +152,11 @@ pub async fn run_repo_fetch_with_host_scope( ); if let Some(rid) = matched { // Redirect to workspace fetch using the matched root - let detail = - crate::mcp::projection::ResponseDetail::from_opt(args.response_detail); + let detail = crate::mcp::projection::ResponseDetail::from_opt_for_profile( + args.response_detail, + state.config.mcp.schema_profile, + ); + let max_chars = state.config.mcp.brief_max_chars; let ws_args = RepoFetchArgs { host: Some("workspace".to_string()), owner: rid.root_name.clone(), @@ -168,7 +180,12 @@ pub async fn run_repo_fetch_with_host_scope( response_detail: None, }; let value = run_workspace_fetch(state, ws_args, local_view).await?; - return Ok(crate::mcp::projection::project("repo_fetch", value, detail)); + return Ok(crate::mcp::projection::project_with_budget( + "repo_fetch", + value, + detail, + max_chars, + )); } if host_local_scope.is_some() { return Err(ToolError::Validation( @@ -605,8 +622,16 @@ pub async fn run_repo_fetch_with_host_scope( let value = serde_json::to_value(&fetch_response) .map_err(|e| ToolError::internal(format!("serialization error: {e}")))?; - let detail = crate::mcp::projection::ResponseDetail::from_opt(args.response_detail); - Ok(crate::mcp::projection::project("repo_fetch", value, detail)) + let detail = crate::mcp::projection::ResponseDetail::from_opt_for_profile( + args.response_detail, + state.config.mcp.schema_profile, + ); + Ok(crate::mcp::projection::project_with_budget( + "repo_fetch", + value, + detail, + state.config.mcp.brief_max_chars, + )) } Err(e) => Err(ToolError::internal(format!("{}: {}", e.error_code(), e))), } diff --git a/src/mcp/tools/repo_map.rs b/src/mcp/tools/repo_map.rs index adcaf1b..645a8f6 100644 --- a/src/mcp/tools/repo_map.rs +++ b/src/mcp/tools/repo_map.rs @@ -47,7 +47,7 @@ pub struct RepoMapArgs { #[serde(default, skip_serializing_if = "Vec::is_empty")] #[schemars(skip)] pub providers: Vec, - /// Response detail: compact, standard, or diagnostic (default diagnostic). + /// Response detail: brief, compact, standard, or diagnostic. #[serde(default, skip_serializing_if = "Option::is_none")] pub response_detail: Option, } @@ -326,6 +326,14 @@ pub async fn run_repo_map_with_host_scope( let value = serde_json::to_value(&response) .map_err(|e| ToolError::internal(format!("serialization error: {e}")))?; - let detail = crate::mcp::projection::ResponseDetail::from_opt(args.response_detail); - Ok(crate::mcp::projection::project("repo_map", value, detail)) + let detail = crate::mcp::projection::ResponseDetail::from_opt_for_profile( + args.response_detail, + state.config.mcp.schema_profile, + ); + Ok(crate::mcp::projection::project_with_budget( + "repo_map", + value, + detail, + state.config.mcp.brief_max_chars, + )) } diff --git a/src/mcp/tools/repo_search.rs b/src/mcp/tools/repo_search.rs index 4e96823..50d5fd0 100644 --- a/src/mcp/tools/repo_search.rs +++ b/src/mcp/tools/repo_search.rs @@ -111,7 +111,7 @@ pub struct RepoSearchArgs { #[serde(default, skip_serializing_if = "Option::is_none")] #[schemars(skip)] pub workflow: Option, - /// Response detail: compact, standard, or diagnostic (default diagnostic). + /// Response detail: brief, compact, standard, or diagnostic. #[serde(default, skip_serializing_if = "Option::is_none")] pub response_detail: Option, } @@ -450,10 +450,14 @@ pub async fn run_repo_search_with_host_scope( let value = serde_json::to_value(&response) .map_err(|e| ToolError::internal(format!("serialization error: {e}")))?; - let detail = crate::mcp::projection::ResponseDetail::from_opt(args.response_detail); - Ok(crate::mcp::projection::project( + let detail = crate::mcp::projection::ResponseDetail::from_opt_for_profile( + args.response_detail, + state.config.mcp.schema_profile, + ); + Ok(crate::mcp::projection::project_with_budget( "repo_search", value, detail, + state.config.mcp.brief_max_chars, )) } diff --git a/src/mcp/tools/research_search.rs b/src/mcp/tools/research_search.rs index 9f735f3..b6d90a9 100644 --- a/src/mcp/tools/research_search.rs +++ b/src/mcp/tools/research_search.rs @@ -70,7 +70,7 @@ pub struct ResearchSearchArgs { /// Known context. #[serde(default, skip_serializing_if = "Option::is_none")] pub known_context: Option, - /// Response detail: compact, standard, or diagnostic (default diagnostic). + /// Response detail: brief, compact, standard, or diagnostic. #[serde(default, skip_serializing_if = "Option::is_none")] pub response_detail: Option, } @@ -223,10 +223,14 @@ pub async fn run_research_search( let value = serde_json::to_value(&response) .map_err(|e| ToolError::internal(format!("serialization error: {e}")))?; - let detail = crate::mcp::projection::ResponseDetail::from_opt(args.response_detail); - Ok(crate::mcp::projection::project( + let detail = crate::mcp::projection::ResponseDetail::from_opt_for_profile( + args.response_detail, + state.config.mcp.schema_profile, + ); + Ok(crate::mcp::projection::project_with_budget( "research_search", value, detail, + state.config.mcp.brief_max_chars, )) } diff --git a/src/mcp/tools/security_search.rs b/src/mcp/tools/security_search.rs index 2ca98bb..25af1db 100644 --- a/src/mcp/tools/security_search.rs +++ b/src/mcp/tools/security_search.rs @@ -75,7 +75,7 @@ pub struct SecuritySearchArgs { #[serde(default, skip_serializing_if = "Option::is_none")] #[schemars(skip)] pub workflow: Option, - /// Response detail: compact, standard, or diagnostic (default diagnostic). + /// Response detail: brief, compact, standard, or diagnostic. #[serde(default, skip_serializing_if = "Option::is_none")] pub response_detail: Option, } @@ -294,10 +294,14 @@ pub async fn run_security_search( let value = serde_json::to_value(&response) .map_err(|e| ToolError::internal(format!("serialization error: {e}")))?; - let detail = crate::mcp::projection::ResponseDetail::from_opt(args.response_detail); - Ok(crate::mcp::projection::project( + let detail = crate::mcp::projection::ResponseDetail::from_opt_for_profile( + args.response_detail, + state.config.mcp.schema_profile, + ); + Ok(crate::mcp::projection::project_with_budget( "security_search", value, detail, + state.config.mcp.brief_max_chars, )) } diff --git a/src/mcp/tools/web_fetch.rs b/src/mcp/tools/web_fetch.rs index 079b2a6..4985411 100644 --- a/src/mcp/tools/web_fetch.rs +++ b/src/mcp/tools/web_fetch.rs @@ -58,7 +58,7 @@ pub struct WebFetchArgs { /// Requires the `browser` feature. #[serde(default)] pub browser_profile: Option, - /// Response detail: compact, standard, or diagnostic (default diagnostic). + /// Response detail: brief, compact, standard, or diagnostic. #[serde(default)] pub response_detail: Option, } @@ -959,10 +959,14 @@ pub async fn run_web_fetch( "transport": resp.transport.as_deref().unwrap_or("http"), "browser_escalated": resp.browser_escalated, }); - let detail = crate::mcp::projection::ResponseDetail::from_opt(args.response_detail); - Ok(crate::mcp::projection::project( + let detail = crate::mcp::projection::ResponseDetail::from_opt_for_profile( + args.response_detail, + state.config.mcp.schema_profile, + ); + Ok(crate::mcp::projection::project_with_budget( "web_fetch", payload, detail, + state.config.mcp.brief_max_chars, )) } diff --git a/src/mcp/tools/web_search.rs b/src/mcp/tools/web_search.rs index bf5528e..5fe46ef 100644 --- a/src/mcp/tools/web_search.rs +++ b/src/mcp/tools/web_search.rs @@ -45,7 +45,7 @@ pub struct WebSearchArgs { /// Excerpt demand (0-3). Defaults to 0. #[serde(default)] pub excerpt_count: Option, - /// Response detail: compact, standard, or diagnostic (default diagnostic). + /// Response detail: brief, compact, standard, or diagnostic. #[serde(default)] pub response_detail: Option, } @@ -275,10 +275,14 @@ pub async fn run_web_search( "capability_enforcement": resp.capability_enforcement.as_ref().map(|t| serde_json::to_value(t).unwrap_or(serde_json::json!({}))).unwrap_or(serde_json::json!({})), }); - let detail = crate::mcp::projection::ResponseDetail::from_opt(args.response_detail); - Ok(crate::mcp::projection::project( + let detail = crate::mcp::projection::ResponseDetail::from_opt_for_profile( + args.response_detail, + state.config.mcp.schema_profile, + ); + Ok(crate::mcp::projection::project_with_budget( "web_search", payload, detail, + state.config.mcp.brief_max_chars, )) } diff --git a/tests/config_validation.rs b/tests/config_validation.rs index c02c8fe..9ef83d0 100644 --- a/tests/config_validation.rs +++ b/tests/config_validation.rs @@ -23,6 +23,38 @@ default_providers = ["duckduckgo"] assert_eq!(cfg.search.mode, Mode::Live); } +#[test] +fn mcp_schema_profile_defaults_to_full_and_accepts_compact() { + use eggsearch::core::config::McpSchemaProfile; + + assert_eq!( + AppConfig::default().mcp.schema_profile, + McpSchemaProfile::Full + ); + let config: AppConfig = toml::from_str("[mcp]\nschema_profile = 'compact'\n").unwrap(); + assert_eq!(config.mcp.schema_profile, McpSchemaProfile::Compact); + assert_eq!(config.mcp.brief_max_chars, 8_000); +} + +#[test] +fn brief_character_budget_is_bounded() { + let mut config = AppConfig::default(); + config.mcp.brief_max_chars = 511; + assert!(config + .validate() + .unwrap_err() + .to_string() + .contains("brief_max_chars")); + config.mcp.brief_max_chars = 8_000; + assert!(config.validate().is_ok()); + config.mcp.brief_max_chars = 32_001; + assert!(config + .validate() + .unwrap_err() + .to_string() + .contains("brief_max_chars")); +} + #[test] fn unknown_toml_search_key_is_silently_ignored() { let toml = r#" diff --git a/tests/mcp_http.rs b/tests/mcp_http.rs index 741fe1b..cb9e39b 100644 --- a/tests/mcp_http.rs +++ b/tests/mcp_http.rs @@ -19,6 +19,10 @@ struct TestServer { impl TestServer { async fn start() -> Self { + Self::start_with_config(AppConfig::default()).await + } + + async fn start_with_config(config: AppConfig) -> Self { let options = ServeOptions { bind: "127.0.0.1:0".parse().unwrap(), path: "/mcp".parse::().unwrap(), @@ -26,7 +30,7 @@ impl TestServer { let listener = tokio::net::TcpListener::bind(options.bind).await.unwrap(); let address = listener.local_addr().unwrap(); let cancellation_token = CancellationToken::new(); - let server = build_server(AppConfig::default()).unwrap(); + let server = build_server(config).unwrap(); let app = http::router(server, &options, cancellation_token.clone()); let shutdown = cancellation_token.clone(); let task = tokio::spawn(async move { @@ -206,6 +210,71 @@ async fn legacy_http_lifecycle_lists_tools_and_calls_local_tool() { server.stop().await; } +#[tokio::test] +async fn compact_profile_is_used_by_initialized_http_tools_list() { + let mut config = AppConfig::default(); + config.mcp.schema_profile = eggsearch::core::config::McpSchemaProfile::Compact; + let server = TestServer::start_with_config(config).await; + let (session_id, _) = initialize_legacy(&server).await; + let tools = sse_payload( + legacy_request( + &server, + &session_id, + json!({"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}), + ) + .await, + ) + .await; + let listed = tools["result"]["tools"].as_array().unwrap(); + let names = listed + .iter() + .map(|tool| tool["name"].as_str().unwrap()) + .collect::>(); + assert_eq!(names.len(), 10); + assert!(names.contains(&"security_search")); + assert!(names.contains(&"research_search")); + assert!(listed.iter().all(|tool| tool.get("outputSchema").is_none())); + assert!(serde_json::to_vec(listed).unwrap().len() <= 36_000); + let compact_call = sse_payload( + legacy_request( + &server, + &session_id, + json!({ + "jsonrpc":"2.0", + "id":3, + "method":"tools/call", + "params":{"name":"provider_status","arguments":{"probe":false}} + }), + ) + .await, + ) + .await; + assert_eq!(compact_call["result"]["isError"], false); + + let full_server = TestServer::start().await; + let (full_session, _) = initialize_legacy(&full_server).await; + let full_call = sse_payload( + legacy_request( + &full_server, + &full_session, + json!({ + "jsonrpc":"2.0", + "id":3, + "method":"tools/call", + "params":{"name":"provider_status","arguments":{"probe":false}} + }), + ) + .await, + ) + .await; + assert_eq!( + compact_call["result"]["structuredContent"], + full_call["result"]["structuredContent"] + ); + full_server.stop().await; + server.stop().await; +} + #[tokio::test] async fn current_http_protocol_uses_request_metadata_without_a_session() { let server = TestServer::start().await; diff --git a/tests/mcp_projection.rs b/tests/mcp_projection.rs index eb3a5ea..8cc0e89 100644 --- a/tests/mcp_projection.rs +++ b/tests/mcp_projection.rs @@ -43,6 +43,14 @@ fn card(id: &str, excerpts: usize, injection_hits: u64) -> Value { "stable_id": format!("stable_{id}"), "title": format!("Title {id}"), "url": format!("https://example.com/{id}"), + "canonical_url": format!("https://example.com/{id}"), + "providers": ["duckduckgo", "brave_api", "startpage"], + "provider_ids": ["duckduckgo", "brave_api"], + "score": 0.8734, + "source_kind": "documentation", + "source_role": "official_documentation", + "quality": {"freshness": "recent", "source_authority": "high", "matched_terms": ["async", "runtime"]}, + "metadata": {"evidence_role": "official_documentation", "debug_trace": "redundant provider normalization details".repeat(4)}, "trust": "external_untrusted", "evidence_role": "primary", "excerpts": ex, @@ -187,6 +195,190 @@ fn compact_trims_excerpts_but_keeps_stable_ids() { ); } +#[test] +fn brief_keeps_identity_trust_failures_and_distinguishes_absence() { + let failed = web_payload( + json!([card("src_a", 2, 2)]), + json!([{"provider_id": "offline", "error_class": "timeout", "message": "provider timeout"}]), + json!({"has_failures": true, "has_absences": false, "has_truncation": false, "attempted_job_count": 2, "completed_job_count": 1, "failed_job_count": 1}), + ); + let brief = project("web_search", failed, ResponseDetail::Brief); + assert_eq!(brief["response_detail"], "brief"); + assert_eq!(brief["retrieval_status"]["has_failures"], true); + assert_eq!(brief["retrieval_status"]["has_absences"], false); + assert_eq!(brief["results"][0]["stable_id"], "stable_src_a"); + assert_eq!(brief["results"][0]["trust"], "external_untrusted"); + assert_eq!(brief["results"][0]["trust_markers"]["injection_hits"], 2); + assert_eq!(brief["providers_failed"][0]["error_class"], "timeout"); + + let absent = web_payload( + json!([]), + json!([]), + json!({"has_failures": false, "has_absences": true, "has_truncation": false}), + ); + let brief_absent = project("web_search", absent, ResponseDetail::Brief); + assert_eq!(brief_absent["retrieval_status"]["has_failures"], false); + assert_eq!(brief_absent["retrieval_status"]["has_absences"], true); +} + +#[test] +fn brief_preserves_security_applicability_states_and_fetch_truncation() { + let security = json!({ + "query": "openssl", + "vulnerabilities": [{"id":"CVE-2025-1","title":"Issue","severity":"high"}], + "applicability": [ + {"package":"openssl","status":"affected","version":"1.0"}, + {"package":"boringssl","status":"not_affected"}, + {"package":"libressl","status":"unknown"} + ], + "providers_failed": [{"provider_id":"osv","error_class":"timeout"}], + "warnings": ["applicability is incomplete"], + "structured_warnings": [{"code":"applicability_unknown","severity":"high","message":"check package metadata"}], + "retrieval_summary":{"has_failures":true,"has_absences":false,"has_truncation":false} + }); + let brief = project("security_search", security, ResponseDetail::Brief); + let states = brief["applicability"] + .as_array() + .unwrap() + .iter() + .filter_map(|item| item["status"].as_str()) + .collect::>(); + assert!(states.contains(&"affected")); + assert!(states.contains(&"not_affected")); + assert!(states.contains(&"unknown")); + assert_eq!(brief["vulnerabilities"][0]["id"], "CVE-2025-1"); + + let fetch = json!({ + "url":"https://example.com/file.pdf","final_url":"https://example.com/file.pdf", + "content_type":"application/pdf","fetched":true,"truncated":true, + "trust":"external_untrusted","trust_markers":{"injection_hits":1}, + "text":"<<>>\n".to_owned() + &"quoted text ".repeat(200) + "\n<<>>", + "focus":{"chunks":[{"text":"focused passage","page":4,"line_start":8,"line_end":9}]}, + "warnings":["content_truncated"],"structured_warnings":[] + }); + let brief_fetch = project("web_fetch", fetch, ResponseDetail::Brief); + assert_eq!(brief_fetch["truncated"], true); + assert_eq!(brief_fetch["trust"], "external_untrusted"); + assert!(brief_fetch["text"].as_str().unwrap().contains("<<>>")); + assert_eq!(brief_fetch["focus"]["chunks"][0]["page"], 4); +} + +#[test] +fn brief_enforces_character_budget_and_retains_failed_batch_items() { + let mut payload = web_payload( + json!([card("a", 1, 0), card("b", 1, 0), card("c", 1, 0)]), + json!([]), + json!({"has_failures": false, "has_absences": false, "has_truncation": false}), + ); + payload["results"][0]["excerpts"][0]["text"] = json!("w".repeat(1200)); + payload["results"][1]["excerpts"][0]["text"] = json!("x".repeat(1200)); + payload["results"][2]["excerpts"][0]["text"] = json!("y".repeat(1200)); + let brief = eggsearch::mcp::projection::project_with_budget( + "web_search", + payload, + ResponseDetail::Brief, + 512, + ); + assert!(bytes(&brief) <= 512, "{} chars", bytes(&brief)); + assert_eq!(brief["output_truncated"], true); + assert!(brief["omitted_count"].as_u64().unwrap() > 0); + assert!(brief["returned_count"].as_u64().unwrap() < 3); + + let batch = json!({ + "fetched":1,"failed":1,"truncated":true,"total_chars_returned":100, + "results":[ + {"index":0,"item_type":"web","stable_id":"ok","ok":true,"error":null,"response":{"url":"https://example.com/a","fetched":true,"truncated":false,"text":"success"}}, + {"index":1,"item_type":"web","stable_id":null,"ok":false,"error":"timeout","response":null,"truncated":true} + ], + "warnings":["batch item failed"],"structured_warnings":[] + }); + let brief_batch = eggsearch::mcp::projection::project_with_budget( + "batch_fetch", + batch, + ResponseDetail::Brief, + 512, + ); + assert!(brief_batch["results"] + .as_array() + .unwrap() + .iter() + .any(|item| item["error"] == "timeout")); + assert_eq!(brief_batch["failed"], 1); +} + +#[test] +fn brief_budget_and_reduction_cover_repository_and_research_groups() { + let oversized_failure = json!({ + "results": [], + "providers_failed": [{ + "provider_id": "fixture", + "error_class": "timeout", + "error": "timeout: ".to_owned() + &"failure detail ".repeat(300) + }], + "retrieval_summary": {"has_failures": true, "has_absences": false, "has_truncation": false} + }); + let bounded = eggsearch::mcp::projection::project_with_budget( + "web_search", + oversized_failure, + ResponseDetail::Brief, + 512, + ); + assert!(bytes(&bounded) <= 512, "{} chars", bytes(&bounded)); + assert_eq!(bounded["retrieval_status"]["has_failures"], true); + assert_eq!(bounded["providers_failed"].as_array().unwrap().len(), 1); + assert_eq!(bounded["providers_failed"][0]["error_class"], "timeout"); + + for tool in ["repo_search", "research_search"] { + let item = json!({ + "id": "source-1", + "stable_id": "stable-source-1", + "title": "Primary source", + "url": "https://example.com/source", + "locator": {"path": "src/lib.rs", "line_start": 12, "line_end": 24}, + "trust": "external_untrusted", + "source_role": "primary", + "excerpts": [{"text": "relevant evidence", "provenance": "content"}], + "metadata": {"discardable": "debug details ".repeat(80)}, + "quality": {"discardable": "quality details ".repeat(80)}, + "providers": ["provider-a", "provider-b"], + "score": 0.82 + }); + let payload = json!({ + "query": "locate the primary source", + "groups": [{"group": "primary_sources", "results": [item.clone()]}], + "retrieval_summary": {"has_failures": false, "has_absences": false, "has_truncation": false}, + "conflict_metadata": [], + "telemetry": {"discardable": "routing diagnostics ".repeat(100)}, + "workflow_context": {"discardable": "workflow diagnostics ".repeat(100)}, + "source_quality": {"discardable": "source diagnostics ".repeat(100)}, + "evidence_role_summary": {"discardable": "role diagnostics ".repeat(100)} + }); + let diagnostic_bytes = bytes(&payload); + let brief = project(tool, payload, ResponseDetail::Brief); + let brief_bytes = bytes(&brief); + println!( + "brief-fixture tool={tool} diagnostic_bytes={diagnostic_bytes} brief_bytes={brief_bytes} reduction_percent={:.1}", + 100.0 * (diagnostic_bytes - brief_bytes) as f64 / diagnostic_bytes as f64 + ); + assert!( + brief_bytes * 10 <= diagnostic_bytes * 7, + "{tool} brief={brief_bytes}, diagnostic={diagnostic_bytes}" + ); + assert_eq!( + brief["groups"][0]["results"][0]["stable_id"], + "stable-source-1" + ); + assert_eq!( + brief["groups"][0]["results"][0]["locator"]["line_start"], + 12 + ); + assert_eq!( + brief["groups"][0]["results"][0]["trust"], + "external_untrusted" + ); + } +} + #[test] fn compact_reports_conflicts_via_indicator() { let summary = json!({"has_failures": false, "has_absences": false, "has_truncation": false}); @@ -408,9 +600,14 @@ fn compact_reduces_bytes_for_representative_payloads() { ); let diag_bytes = bytes(&web); let compact_bytes = bytes(&project("web_search", web.clone(), ResponseDetail::Compact)); - let standard_bytes = bytes(&project("web_search", web, ResponseDetail::Standard)); + let standard_bytes = bytes(&project( + "web_search", + web.clone(), + ResponseDetail::Standard, + )); + let brief_bytes = bytes(&project("web_search", web, ResponseDetail::Brief)); println!( - "projection-bytes tool=web_search diagnostic={diag_bytes} standard={standard_bytes} compact={compact_bytes}" + "projection-bytes tool=web_search diagnostic={diag_bytes} standard={standard_bytes} compact={compact_bytes} brief={brief_bytes}" ); assert!( compact_bytes < diag_bytes, @@ -424,6 +621,10 @@ fn compact_reduces_bytes_for_representative_payloads() { compact_bytes <= standard_bytes, "compact {compact_bytes} should be <= standard {standard_bytes}" ); + assert!( + brief_bytes * 10 <= diag_bytes * 7, + "web brief={brief_bytes}, diagnostic={diag_bytes}" + ); let fetch = json!({ "url": "https://example.com/a", @@ -436,20 +637,31 @@ fn compact_reduces_bytes_for_representative_payloads() { "cache_status": "miss" }); let f_diag = bytes(&fetch); - let f_compact = bytes(&project("web_fetch", fetch, ResponseDetail::Compact)); - println!("projection-bytes tool=web_fetch diagnostic={f_diag} compact={f_compact}"); + let f_compact = bytes(&project( + "web_fetch", + fetch.clone(), + ResponseDetail::Compact, + )); + let f_brief = bytes(&project("web_fetch", fetch, ResponseDetail::Brief)); + println!( + "projection-bytes tool=web_fetch diagnostic={f_diag} compact={f_compact} brief={f_brief}" + ); assert!( f_compact < f_diag, "fetch compact {f_compact} < diagnostic {f_diag}" ); + assert!( + f_brief * 10 <= f_diag * 7, + "fetch brief={f_brief}, diagnostic={f_diag}" + ); } #[test] fn retrieval_quality_host_trace_records_per_case_projection_cost() { let corpus: Value = serde_json::from_str(RETRIEVAL_CASES).unwrap(); let cases = corpus["cases"].as_array().unwrap(); - let mut byte_totals = [0usize; 3]; - let mut retained_totals = [0usize; 3]; + let mut byte_totals = [0usize; 4]; + let mut retained_totals = [0usize; 4]; let mut simulated_call_total = 0usize; for case in cases { let results = case["results"] @@ -501,9 +713,10 @@ fn retrieval_quality_host_trace_records_per_case_projection_cost() { ResponseDetail::Diagnostic, ResponseDetail::Standard, ResponseDetail::Compact, + ResponseDetail::Brief, ]; - let mut mode_bytes = [0usize; 3]; - let mut retained = [0usize; 3]; + let mut mode_bytes = [0usize; 4]; + let mut retained = [0usize; 4]; let relevant_ids: HashSet<&str> = case["relevant_documents"] .as_array() .unwrap() @@ -525,7 +738,8 @@ fn retrieval_quality_host_trace_records_per_case_projection_cost() { }) .count(); retained_totals[index] += retained[index]; - if mode == ResponseDetail::Compact && (failed || absent) { + if matches!(mode, ResponseDetail::Compact | ResponseDetail::Brief) && (failed || absent) + { assert_eq!(projected["retrieval_status"]["has_failures"], json!(failed)); assert_eq!(projected["retrieval_status"]["has_absences"], json!(absent)); } @@ -534,7 +748,7 @@ fn retrieval_quality_host_trace_records_per_case_projection_cost() { let simulated_calls = 1 + source_fetches + usize::from(source_fetches > 0); simulated_call_total += simulated_calls; println!( - "retrieval-context-trace case={} tool={} partition={} state={} bytes_diagnostic={} bytes_standard={} bytes_compact={} relevant_retained_diagnostic={} relevant_retained_standard={} relevant_retained_compact={} simulated_tool_calls={} token_estimates={}/{}/{}", + "retrieval-context-trace case={} tool={} partition={} state={} bytes_diagnostic={} bytes_standard={} bytes_compact={} bytes_brief={} relevant_retained_diagnostic={} relevant_retained_standard={} relevant_retained_compact={} relevant_retained_brief={} simulated_tool_calls={} token_estimates={}/{}/{}/{}", case["id"].as_str().unwrap(), tool, case["partition"].as_str().unwrap(), @@ -542,26 +756,31 @@ fn retrieval_quality_host_trace_records_per_case_projection_cost() { mode_bytes[0], mode_bytes[1], mode_bytes[2], + mode_bytes[3], retained[0], retained[1], retained[2], + retained[3], simulated_calls, mode_bytes[0].div_ceil(4), mode_bytes[1].div_ceil(4), mode_bytes[2].div_ceil(4) + ,mode_bytes[3].div_ceil(4) ); } - assert_eq!(retained_totals, [14, 14, 14]); + assert_eq!(retained_totals, [14, 14, 14, 14]); assert_eq!(simulated_call_total, 42); println!( - "retrieval-context-trace-total cases={} bytes_diagnostic={} bytes_standard={} bytes_compact={} token_estimates={}/{}/{} retained={:?} simulated_tool_calls={}", + "retrieval-context-trace-total cases={} bytes_diagnostic={} bytes_standard={} bytes_compact={} bytes_brief={} token_estimates={}/{}/{}/{} retained={:?} simulated_tool_calls={}", cases.len(), byte_totals[0], byte_totals[1], byte_totals[2], + byte_totals[3], byte_totals[0].div_ceil(4), byte_totals[1].div_ceil(4), byte_totals[2].div_ceil(4), + byte_totals[3].div_ceil(4), retained_totals, simulated_call_total ); @@ -570,7 +789,22 @@ fn retrieval_quality_host_trace_records_per_case_projection_cost() { #[test] fn response_detail_param_defaults_to_diagnostic_and_roundtrips() { assert_eq!(ResponseDetail::from_opt(None), ResponseDetail::Diagnostic); + assert_eq!( + ResponseDetail::from_opt_for_profile( + None, + eggsearch::core::config::McpSchemaProfile::Compact + ), + ResponseDetail::Brief + ); + assert_eq!( + ResponseDetail::from_opt_for_profile( + Some(ResponseDetail::Diagnostic), + eggsearch::core::config::McpSchemaProfile::Compact + ), + ResponseDetail::Diagnostic + ); for (raw, expected) in [ + ("brief", ResponseDetail::Brief), ("compact", ResponseDetail::Compact), ("standard", ResponseDetail::Standard), ("diagnostic", ResponseDetail::Diagnostic), diff --git a/tests/mcp_tool_contract.rs b/tests/mcp_tool_contract.rs index 92e147c..78816bd 100644 --- a/tests/mcp_tool_contract.rs +++ b/tests/mcp_tool_contract.rs @@ -450,3 +450,56 @@ fn fingerprint_is_content_based_not_count_based() { "tools/list must stay name-sorted for caching" ); } + +#[test] +fn compact_profile_advertises_all_tools_with_smaller_stable_contract() { + let full_state = Arc::new(eggsearch::mcp::ServerState::build(AppConfig::default()).unwrap()); + let full = eggsearch::mcp::EggsearchServer::new(full_state); + let mut config = AppConfig::default(); + config.mcp.schema_profile = eggsearch::core::config::McpSchemaProfile::Compact; + let compact_state = Arc::new(eggsearch::mcp::ServerState::build(config).unwrap()); + let compact = eggsearch::mcp::EggsearchServer::new(compact_state); + let full_tools = full.tool_definitions(); + let compact_tools = compact.tool_definitions(); + assert_eq!(compact_tools.len(), 10); + assert_eq!( + compact_tools + .iter() + .map(|tool| tool.name.as_ref()) + .collect::>(), + full_tools + .iter() + .map(|tool| tool.name.as_ref()) + .collect::>() + ); + let full_bytes = serde_json::to_vec(&full_tools).unwrap().len(); + let compact_bytes = serde_json::to_vec(&compact_tools).unwrap().len(); + assert!( + compact_bytes * 2 <= full_bytes, + "compact contract must be at least 50% smaller: full={full_bytes}, compact={compact_bytes}" + ); + assert!( + compact_bytes <= 36_000, + "compact contract is {compact_bytes} bytes" + ); + assert_ne!(full.tool_fingerprint(), compact.tool_fingerprint()); + let compact_again = eggsearch::mcp::EggsearchServer::new(Arc::new( + eggsearch::mcp::ServerState::build(AppConfig { + mcp: eggsearch::core::config::McpSection { + schema_profile: eggsearch::core::config::McpSchemaProfile::Compact, + ..Default::default() + }, + ..AppConfig::default() + }) + .unwrap(), + )); + assert_eq!(compact.tool_fingerprint(), compact_again.tool_fingerprint()); + for (full_tool, compact_tool) in full_tools.iter().zip(compact_tools.iter()) { + let required = full_tool.input_schema.get("required"); + assert_eq!(required, compact_tool.input_schema.get("required")); + assert_eq!( + compact_tool.input_schema.get("additionalProperties"), + Some(&serde_json::Value::Bool(true)) + ); + } +} diff --git a/tests/tool_surface_evaluation.rs b/tests/tool_surface_evaluation.rs index f6cb9b3..bab4a7f 100644 --- a/tests/tool_surface_evaluation.rs +++ b/tests/tool_surface_evaluation.rs @@ -362,6 +362,13 @@ fn live_server() -> eggsearch::mcp::EggsearchServer { eggsearch::mcp::EggsearchServer::new(state) } +fn compact_server() -> eggsearch::mcp::EggsearchServer { + let mut config = AppConfig::default(); + config.mcp.schema_profile = eggsearch::core::config::McpSchemaProfile::Compact; + let state = Arc::new(ServerState::build(config).expect("compact state builds")); + eggsearch::mcp::EggsearchServer::new(state) +} + #[test] fn tool_surface_corpus_is_well_formed() { let cases = load_cases(); @@ -615,6 +622,62 @@ fn tool_surface_discovery_accuracy() { assert!(mrr >= MIN_MRR, "MRR {mrr:.3} below {MIN_MRR}"); } +#[test] +fn compact_tool_surface_discovery_accuracy() { + let server = compact_server(); + let tools = server.tool_definitions(); + assert_eq!(tools.len(), KNOWN_TOOLS.len()); + let desc_index = description_index(&tools); + let cases = load_cases(); + let mut top1_hits = 0usize; + let mut recall3_hits = 0usize; + let mut reciprocal_sum = 0.0; + let mut failures = Vec::new(); + for case in &cases { + let ranked = rank_tools(&case.query, &tools, &desc_index); + assert_eq!(ranked.len(), KNOWN_TOOLS.len()); + assert!( + !case.forbidden_primary.contains(&ranked[0].0), + "compact profile ranked forbidden tool {} for case {}", + ranked[0].0, + case.id + ); + let acceptable: HashSet<&str> = case.acceptable.iter().map(String::as_str).collect(); + let first_hit = ranked + .iter() + .position(|(name, _)| acceptable.contains(name.as_str())); + match first_hit { + Some(0) => top1_hits += 1, + Some(position) => failures.push(format!("{}: primary #{}", case.id, position + 1)), + None => failures.push(format!("{}: primary missing from top-3", case.id)), + } + if first_hit.is_some_and(|position| position < 3) { + recall3_hits += 1; + } + if let Some(position) = first_hit { + reciprocal_sum += 1.0 / (position as f64 + 1.0); + } + } + let total = cases.len() as f64; + let top1 = top1_hits as f64 / total; + let recall3 = recall3_hits as f64 / total; + let mrr = reciprocal_sum / total; + for failure in &failures { + println!("compact-tool-surface-miss {failure}"); + } + println!( + "compact-tool-surface-report cases={} top1={top1_hits}/{n} acc={top1:.3} recall3={recall3_hits}/{n} r3={recall3:.3} mrr={mrr:.3}", + cases.len(), + n = cases.len(), + ); + assert!( + top1 >= MIN_TOP1_ACCURACY, + "compact top-1 accuracy {top1:.3}" + ); + assert!(recall3 >= MIN_RECALL_AT_3, "compact recall@3 {recall3:.3}"); + assert!(mrr >= MIN_MRR, "compact MRR {mrr:.3}"); +} + #[test] fn tool_surface_layer2_synthetic_mechanics() { let source_ids = vec!["src_1".to_string(), "src_2".to_string()]; diff --git a/tests/tool_surface_live.rs b/tests/tool_surface_live.rs index c14f5e0..b652015 100644 --- a/tests/tool_surface_live.rs +++ b/tests/tool_surface_live.rs @@ -12,8 +12,8 @@ use eggsearch::mcp::state::ServerState; // Baseline re-measured after `repo_map` stopped advertising `providers`: // -98 definition bytes, -25 estimated tokens. -const BASELINE_TOTAL_BYTES: i64 = 72939; -const BASELINE_EST_TOKENS: i64 = 18235; +const BASELINE_TOTAL_BYTES: i64 = 72906; +const BASELINE_EST_TOKENS: i64 = 18227; const BASELINE_MAX_DESC: i64 = 186; const BASELINE_INSTRUCTIONS_BYTES: i64 = 1614; From 02a8af6fe4f87971c7daca7d1fca20c087b328ec Mon Sep 17 00:00:00 2001 From: eggsearch Date: Fri, 30 Oct 2026 20:52:44 +0000 Subject: [PATCH 03/25] fix: keep brief failure summaries within budget --- architecture/mcp.md | 2 +- docs/config.md | 6 +- docs/retrieval-quality-context-efficiency.md | 2 +- docs/test-inventory.md | 2 +- src/mcp/projection.rs | 177 +++++++++++++++++++ tests/mcp_projection.rs | 107 ++++++++++- 6 files changed, 289 insertions(+), 7 deletions(-) diff --git a/architecture/mcp.md b/architecture/mcp.md index 874383a..4f9c694 100644 --- a/architecture/mcp.md +++ b/architecture/mcp.md @@ -260,7 +260,7 @@ Tool behavior modules carry unit coverage in-file plus the shared `tools/tests.r - `diagnostic` — full observability payload, passthrough with no added keys. - `standard` — specialist research/security default surface. Keeps full `retrieval_summary`, `conflict_metadata`, `workflow_coverage`, `evidence_role_summary`, `capability_enforcement`, and fetch/cache metadata; summarizes only `routing_decision` (→ `routing_summary`) and `telemetry.routing_decision`. - `compact` — ordinary agent operation. Trims excerpts to 1 per card, replaces `routing_decision` with `routing_summary`, replaces `retrieval_summary`/`conflict_metadata` with `retrieval_status`/`conflict_indicator`, drops `workflow_coverage`/`telemetry`/`document`/`links`, strips per-vulnerability `references`/`details`, caps `repo_map` entries at 50. Every projection stamps `response_detail` plus an explicit truncation marker (`projection_excerpts_trimmed`, `projection_entries_truncated`, `links_omitted_in_compact`). -- `brief` — applies compact safety reductions, then retains identity, status, trust, warnings, applicability and citation fields using explicit allowlists. Content strings are clipped at UTF-8 character boundaries; if the configured `[mcp].brief_max_chars` serialized-character budget still cannot fit, complete successful result records are omitted from the tail with `returned_count`, `omitted_count`, and `output_truncated`. Failed batch members and critical failure/applicability signals are retained. Text plus overlapping focus chunks are avoided through the existing compact fetch projection. The structured result and text fallback are both brief; callers request `diagnostic` for the full diagnostic payload. +- `brief` — applies compact safety reductions, then retains identity, status, trust, warnings, applicability and citation fields using explicit allowlists. Content strings are clipped at UTF-8 character boundaries; if the configured `[mcp].brief_max_chars` serialized-character budget still cannot fit, complete result records are omitted from the tail with `returned_count`, `omitted_count`, and `output_truncated`. Oversized applicability lists use an `applicability_summary` with affected, not-affected, unknown, total and omitted counts; provider failures expose `providers_failed_summary` when records are omitted. Omitted batch failure records retain their indexes in `failed_member_indices`. Text plus overlapping focus chunks are avoided through the existing compact fetch projection. The structured result and text fallback are both brief; callers request `diagnostic` for the full diagnostic payload. The suite measures diagnostic/standard/compact/brief serialized characters across the 15-case retrieval-quality corpus and asserts that brief never erases failure-vs-absence state, trust/injection markers, security applicability, PDF coordinates, truncation evidence, failed batch members or stable IDs. The sparse corpus report is descriptive; a separate nontrivial fixture enforces the 30% byte reduction target. diff --git a/docs/config.md b/docs/config.md index 2b53fe6..d5ddc79 100644 --- a/docs/config.md +++ b/docs/config.md @@ -54,8 +54,10 @@ Omitted `response_detail` defaults to `brief` in compact mode; explicit `diagnostic`, `standard`, or `compact` calls override it. `brief_max_chars` defaults to 8,000 and must be between 512 and 32,000. Brief output preserves status, source identity, trust, critical warnings and security applicability; -it reports omitted records and truncation. The structured result and text -fallback are both brief. Switch to `schema_profile = "full"` for complete +when applicability records exceed the budget, per-state totals remain in +`applicability_summary`; omitted provider failures and batch member indexes +are also counted. The structured result and text fallback are both brief. +Switch to `schema_profile = "full"` for complete schema guidance and the existing diagnostic-by-default response behavior. ## Provider Requirements diff --git a/docs/retrieval-quality-context-efficiency.md b/docs/retrieval-quality-context-efficiency.md index 07621d5..bab514d 100644 --- a/docs/retrieval-quality-context-efficiency.md +++ b/docs/retrieval-quality-context-efficiency.md @@ -24,6 +24,6 @@ The 15-task host trace serializes 21,221 diagnostic bytes, 21,539 standard bytes | `repo_search` grouped stress fixture with redundant metadata | 10,846 | 669 | 93.8% | | `research_search` grouped stress fixture with redundant metadata | 10,846 | 669 | 93.8% | -The brief projection tests preserve partial retrieval, failure-versus-absence, trust and injection markers, stable IDs, truncation and PDF page coordinates, security applicability, failed batch members, and evidence-bundle identity. Brief serializes the same projected data into both structured content and its text fallback. Its 8,000-character default display budget is configurable from 512 to 32,000 characters; omitted result counts and truncation are reported. The web and fetch fixtures are nontrivial samples; the repository and research figures are synthetic stress fixtures with deliberately redundant metadata and are not model-prompt or production-average claims. Compact mode defaults to brief only when selected in server configuration; the full profile remains diagnostic by default. +The brief projection tests preserve partial retrieval, failure-versus-absence, trust and injection markers, stable IDs, truncation and PDF page coordinates, security applicability, failed batch members, and evidence-bundle identity. Large applicability lists retain affected/not-affected/unknown totals and expose per-entry omissions while staying inside the configured budget. Brief serializes the same projected data into both structured content and its text fallback. Its 8,000-character default display budget is configurable from 512 to 32,000 characters; omitted result counts and truncation are reported. The web and fetch fixtures are nontrivial samples; the repository and research figures are synthetic stress fixtures with deliberately redundant metadata and are not model-prompt or production-average claims. Compact mode defaults to brief only when selected in server configuration; the full profile remains diagnostic by default. This harness measures serialized characters/bytes and rough token estimates only. It does not measure provider latency, actual model tokens, host storage behavior, or evidence sufficiency after real model reasoning. LM Studio consumer qualification is a separate operational gate. diff --git a/docs/test-inventory.md b/docs/test-inventory.md index 904aa00..9bcc91d 100644 --- a/docs/test-inventory.md +++ b/docs/test-inventory.md @@ -176,7 +176,7 @@ Source of truth: `fuzz/Cargo.toml` [[bin]] entries. | `mcp_tool_contract` | 15 | Canonical registry parity, sanitization, deterministic fingerprints, and compact-profile size/all-name contract | | `mcp_schema_slimming` | 17 | Slimmed ordinary schema size budget with legacy-field acceptance | | `mcp_2026_protocol` | 14 | Structured results, output schemas, repairable error contract | -| `mcp_projection` | 19 | Brief/compact/standard/diagnostic projections: status, trust, security applicability, failure retention, bounded failure details, repository/research reduction, character budgets, UTF-8 framing, 15-task byte/retention trace, bundle identity | +| `mcp_projection` | 21 | Brief/compact/standard/diagnostic projections: status, trust, security applicability summary and budget retention, provider and batch failure summaries, bounded failure details, repository/research reduction, character budgets, UTF-8 framing, 15-task byte/retention trace, bundle identity | ## Tool-Surface Evaluation (2 suites) diff --git a/src/mcp/projection.rs b/src/mcp/projection.rs index 9b676d4..432f64b 100644 --- a/src/mcp/projection.rs +++ b/src/mcp/projection.rs @@ -100,6 +100,8 @@ fn project_brief(tool: &str, value: serde_json::Value, max_chars: usize) -> serd } let mut projected = project(tool, value, ResponseDetail::Compact); reduce_brief_fields(&mut projected); + let applicability_total = add_applicability_summary(&mut projected); + let provider_failure_total = add_provider_failure_summary(&mut projected); if let Some(object) = projected.as_object_mut() { object.insert( "response_detail".into(), @@ -140,6 +142,71 @@ fn project_brief(tool: &str, value: serde_json::Value, max_chars: usize) -> serd .iter() .rposition(|item| !brief_record_is_failure(item)); let Some(index) = removable else { + if tool == "batch_fetch" { + let removed_failure = projected + .get_mut(key) + .and_then(serde_json::Value::as_array_mut) + .and_then(|values| { + values + .iter() + .rposition(brief_record_is_failure) + .map(|index| (index, values.remove(index))) + }); + if let Some((position, member)) = removed_failure { + let member_index = member + .get("index") + .and_then(serde_json::Value::as_u64) + .unwrap_or(position as u64); + record_failed_batch_member(&mut projected, member_index); + omitted += 1; + clipped = true; + continue; + } + } + let removed_applicability = projected + .get_mut("applicability") + .and_then(serde_json::Value::as_array_mut) + .and_then(|items| (!items.is_empty()).then(|| items.pop()).flatten()) + .is_some(); + if removed_applicability { + clipped = true; + let retained = projected + .get("applicability") + .and_then(serde_json::Value::as_array) + .map_or(0, Vec::len); + if let Some(summary) = projected + .get_mut("applicability_summary") + .and_then(serde_json::Value::as_object_mut) + { + summary.insert( + "omitted".into(), + applicability_total.saturating_sub(retained).into(), + ); + } + continue; + } + let removed_provider_failure = projected + .get_mut("providers_failed") + .and_then(serde_json::Value::as_array_mut) + .and_then(Vec::pop) + .is_some(); + if removed_provider_failure { + clipped = true; + let retained = projected + .get("providers_failed") + .and_then(serde_json::Value::as_array) + .map_or(0, Vec::len); + if let Some(summary) = projected + .get_mut("providers_failed_summary") + .and_then(serde_json::Value::as_object_mut) + { + summary.insert( + "omitted".into(), + provider_failure_total.saturating_sub(retained).into(), + ); + } + continue; + } break; }; values.remove(index); @@ -195,10 +262,120 @@ fn project_brief(tool: &str, value: serde_json::Value, max_chars: usize) -> serd "output_truncated".into(), serde_json::Value::Bool(clipped || omitted > 0), ); + let retained = object + .get("applicability") + .and_then(serde_json::Value::as_array) + .map_or(0, Vec::len); + if let Some(summary) = object + .get_mut("applicability_summary") + .and_then(serde_json::Value::as_object_mut) + { + summary.insert( + "omitted".into(), + applicability_total.saturating_sub(retained).into(), + ); + } + let retained_provider_failures = object + .get("providers_failed") + .and_then(serde_json::Value::as_array) + .map_or(0, Vec::len); + if let Some(summary) = object + .get_mut("providers_failed_summary") + .and_then(serde_json::Value::as_object_mut) + { + summary.insert( + "omitted".into(), + provider_failure_total + .saturating_sub(retained_provider_failures) + .into(), + ); + } } projected } +fn add_applicability_summary(value: &mut serde_json::Value) -> usize { + let Some(items) = value + .get("applicability") + .and_then(serde_json::Value::as_array) + else { + return 0; + }; + let total = items.len(); + let mut summary = serde_json::Map::new(); + for (key, aliases) in [ + ("affected", ["affected", "applicable"]), + ("not_affected", ["not_affected", "unaffected"]), + ("unknown", ["unknown", "unknown"]), + ] { + let count = items + .iter() + .filter(|item| { + item.get("status") + .and_then(serde_json::Value::as_str) + .is_some_and(|status| aliases.contains(&status)) + }) + .count(); + summary.insert(key.into(), count.into()); + } + summary.insert("total".into(), total.into()); + summary.insert("omitted".into(), 0usize.into()); + if let Some(object) = value.as_object_mut() { + object.insert( + "applicability_summary".into(), + serde_json::Value::Object(summary), + ); + } + total +} + +fn add_provider_failure_summary(value: &mut serde_json::Value) -> usize { + let Some(failures) = value + .get("providers_failed") + .and_then(serde_json::Value::as_array) + else { + return 0; + }; + let total = failures.len(); + if total > 0 { + let mut error_classes = serde_json::Map::new(); + for failure in failures { + let class = failure + .get("error_class") + .and_then(serde_json::Value::as_str) + .unwrap_or("unknown"); + let count = error_classes + .get(class) + .and_then(serde_json::Value::as_u64) + .unwrap_or(0); + error_classes.insert(class.into(), (count + 1).into()); + } + if let Some(object) = value.as_object_mut() { + object.insert( + "providers_failed_summary".into(), + serde_json::json!({ + "total": total, + "omitted": 0, + "error_classes": error_classes + }), + ); + } + } + total +} + +fn record_failed_batch_member(value: &mut serde_json::Value, index: u64) { + if let Some(object) = value.as_object_mut() { + let indices = object + .entry("failed_member_indices") + .or_insert_with(|| serde_json::Value::Array(Vec::new())); + if let Some(indices) = indices.as_array_mut() { + indices.push(index.into()); + indices.sort_by_key(|value| value.as_u64().unwrap_or(u64::MAX)); + } + } +} + fn brief_serialized_chars(value: &serde_json::Value) -> usize { serde_json::to_string(value).map_or(usize::MAX, |encoded| encoded.chars().count()) } diff --git a/tests/mcp_projection.rs b/tests/mcp_projection.rs index 8cc0e89..b4c5966 100644 --- a/tests/mcp_projection.rs +++ b/tests/mcp_projection.rs @@ -325,8 +325,11 @@ fn brief_budget_and_reduction_cover_repository_and_research_groups() { ); assert!(bytes(&bounded) <= 512, "{} chars", bytes(&bounded)); assert_eq!(bounded["retrieval_status"]["has_failures"], true); - assert_eq!(bounded["providers_failed"].as_array().unwrap().len(), 1); - assert_eq!(bounded["providers_failed"][0]["error_class"], "timeout"); + assert_eq!(bounded["providers_failed_summary"]["total"], 1); + assert_eq!( + bounded["providers_failed_summary"]["error_classes"]["timeout"], + 1 + ); for tool in ["repo_search", "research_search"] { let item = json!({ @@ -379,6 +382,106 @@ fn brief_budget_and_reduction_cover_repository_and_research_groups() { } } +#[test] +fn brief_budget_bounds_large_security_applicability_lists() { + let payload = json!({ + "query": "package ecosystem", + "vulnerabilities": [], + "applicability": (0..40).map(|index| json!({ + "package": format!("package-{index}"), + "version": "1.2.3", + "status": match index % 3 { + 0 => "affected", + 1 => "not_affected", + _ => "unknown" + } + })).collect::>(), + "retrieval_summary": {"has_failures": false, "has_absences": false, "has_truncation": false}, + "warnings": [], + "structured_warnings": [] + }); + let brief = eggsearch::mcp::projection::project_with_budget( + "security_search", + payload, + ResponseDetail::Brief, + 512, + ); + assert!(bytes(&brief) <= 512, "{} chars", bytes(&brief)); + assert_eq!(brief["applicability_summary"]["affected"], 14); + assert_eq!(brief["applicability_summary"]["not_affected"], 13); + assert_eq!(brief["applicability_summary"]["unknown"], 13); + assert_eq!(brief["output_truncated"], true); +} + +#[test] +fn brief_budget_summarizes_many_provider_and_batch_failures() { + let providers_failed = (0..40) + .map(|index| { + json!({ + "provider_id": format!("provider-{index}"), + "error_class": "timeout", + "message": "provider timed out" + }) + }) + .collect::>(); + let provider_payload = json!({ + "results": [], + "providers_failed": providers_failed, + "retrieval_summary": {"has_failures": true, "has_absences": false, "has_truncation": false}, + "warnings": [], + "structured_warnings": [] + }); + let provider_brief = eggsearch::mcp::projection::project_with_budget( + "web_search", + provider_payload, + ResponseDetail::Brief, + 512, + ); + assert!( + bytes(&provider_brief) <= 512, + "{} chars", + bytes(&provider_brief) + ); + assert_eq!(provider_brief["retrieval_status"]["has_failures"], true); + assert_eq!(provider_brief["providers_failed_summary"]["total"], 40); + assert!( + provider_brief["providers_failed_summary"]["omitted"] + .as_u64() + .unwrap() + > 0 + ); + + let results = (0..20) + .map(|index| { + json!({ + "index": index, + "item_type": "web", + "ok": false, + "error": "timeout", + "response": null + }) + }) + .collect::>(); + let batch_payload = + json!({"failed":20,"fetched":0,"results":results,"warnings":[],"structured_warnings":[]}); + let batch_brief = eggsearch::mcp::projection::project_with_budget( + "batch_fetch", + batch_payload, + ResponseDetail::Brief, + 512, + ); + assert!(bytes(&batch_brief) <= 512, "{} chars", bytes(&batch_brief)); + assert_eq!(batch_brief["failed"], 20); + let retained_failure_count = batch_brief["results"].as_array().unwrap().len(); + let omitted_failure_indices = batch_brief["failed_member_indices"].as_array().unwrap(); + assert_eq!(retained_failure_count + omitted_failure_indices.len(), 20); + assert!(batch_brief["results"] + .as_array() + .unwrap() + .iter() + .all(|item| item["ok"] == false)); +} + #[test] fn compact_reports_conflicts_via_indicator() { let summary = json!({"has_failures": false, "has_absences": false, "has_truncation": false}); From d983fa5c709d66fe02aa7e7c2aba9759ba188315 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Fri, 30 Oct 2026 20:53:34 +0000 Subject: [PATCH 04/25] docs: close compact MCP advertisement milestone --- .../001-status.md | 131 ++++++++++++++++++ .../001-opt-in-compact-tool-advertisement.md | 2 +- .../002-brief-evidence-and-output-budgets.md | 2 +- plans/registry.md | 8 +- .../standalone-mcp-local-inference-roadmap.md | 2 +- 5 files changed, 138 insertions(+), 7 deletions(-) create mode 100644 plans/closure/standalone-mcp-local-inference/001-status.md diff --git a/plans/closure/standalone-mcp-local-inference/001-status.md b/plans/closure/standalone-mcp-local-inference/001-status.md new file mode 100644 index 0000000..ac6b0bb --- /dev/null +++ b/plans/closure/standalone-mcp-local-inference/001-status.md @@ -0,0 +1,131 @@ +# Standalone MCP Local-Inference M001 — Compact Advertisement Closure + +Status: closed + +Source implementation plan: +`plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` + +Source subsystem roadmap: +`plans/subsystems/standalone-mcp-local-inference-roadmap.md` + +Repository baseline reviewed: `a666517c39adf28f32fad04198c74cb70d935c5b` +(2026-10-30 main baseline). + +Implementation candidate: `02a8af6fe4f87971c7daca7d1fca20c087b328ec` +(compact ten-tool advertisement plus budgeted brief output and the corrective +summary-budget pass). + +## 1. Executive finding + +M001 is closed. The opt-in `[mcp].schema_profile = "compact"` advertises all +ten canonical tools with required inputs intact, while leaving the default +profile as `full`. Real initialized stdio MCP enumeration measured 72,923 bytes +for full and 18,545 bytes for compact (74.6% less). Full and compact both +scored 43/43 top-1 and 43/43 recall@3 with MRR 1.000 in the deterministic +discovery corpus. Loopback HTTP and stdio use the same configured contracts. + +The full contract includes the additive `response_detail = "brief"` enum +introduced by M002; omitted response detail and all other defaults retain the +full/diagnostic behavior. This additive option is recorded as a M002 contract +extension, not a schema-profile behavior change. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Opt-in process-level configuration | `src/core/config.rs`, `tests/config_validation.rs` | pass | `full` is default; invalid profiles reject | +| Ten canonical names remain advertised | `tests/mcp_tool_contract.rs`, `tests/mcp_http.rs`, installed-candidate stdio smoke | pass | Compact names and required keys retained | +| Schema-size reduction meets 36 KB limit | Initialized stdio child `tools/list`: full 72,923 bytes, compact 18,545 bytes | pass | Compact is 74.6% smaller; both profiles expose ten tools | +| Both MCP transports share the profile | `tests/mcp_http.rs`, `tests/mcp_tool_contract.rs` | pass | HTTP initialization and stdio contract paths exercised | +| Runtime behavior/default remain intact | Full and compact `provider_status(probe=false)` produce equal structured output; existing MCP/tool suites | pass | Hidden advanced inputs continue through existing handlers | +| Tool discovery remains above existing gate | `tests/tool_surface_evaluation.rs` | pass | Full and compact: 43/43 top-1, 43/43 recall@3, MRR 1.000 | +| Profile fingerprint reflects advertised contract | `tests/mcp_tool_contract.rs` | pass | Deterministic within profile and distinct between profiles | + +## 3. Production implementation evidence + +`McpSchemaProfile` selects an advertised-only projection in +`src/mcp/server.rs`; it does not replace handlers or runtime argument +validation. Compact retains every canonical tool name, required property and +accepted additional property behavior, and omits optional output schemas and +nonessential descriptions. Profile-specific fingerprints derive from the +serialized advertised contracts. Stdio and loopback HTTP share the same +configuration path. + +An installed-candidate child-process smoke initialized MCP over stdio, listed +all tools and called `provider_status` with `probe=false`. It completed with +exit code 0 for both profiles and identical structured call results. The +recorded measurements are wire bytes from the child response; estimated +token costs are documented separately and are not LM Studio prompt-token +measurements. + +## 4. Verification executed + +- `rtk make check` — pass; formatting, Clippy, no-default-features check, + all-features test gate (3,486 tests), docs, hygiene, dependency policy and + packaging contract checks passed. +- `rtk make docs-check` — pass. +- `rtk make bench-check` — pass; optimized Criterion harness compiled with + `--no-run` (11m32s on this host). +- `cargo test --locked --all-features --test mcp_tool_contract` — pass. +- `cargo test --locked --all-features --test mcp_http` — pass (12 tests). +- `cargo test --locked --all-features --test tool_surface_evaluation -- --nocapture` + — pass (5 tests; full and compact 43/43). +- `cargo test --locked --all-features --test mcp_2026_protocol` — pass + (14 tests). +- `cargo test --locked --all-features --test mcp_schema_slimming` — pass + (17 tests). +- `rtk git diff --check` — pass. + +## 5. Invariant review + +The ten-tool registry, handlers, tool names, dispatch/permission semantics, +provider routing, search budgets, stable IDs, CLI/Python surfaces and default +full profile remain in place. Compact is presentation metadata only; it does +not act as an authorization boundary. Runtime deserialization remains +authoritative for advanced fields omitted from compact schemas. + +## 6. Failure and recovery review + +The ordinary MCP and workflow suites pass. The separate LM Studio live-model +qualification remains M003 work; no host prompt behavior or model-quality +claim is inferred from the offline byte and discovery measurements. Operators +can return to `schema_profile = "full"` and restart the MCP process. + +## 7. Migration and compatibility review + +No existing config requires migration. Omitted `schema_profile` remains +`full`. Compact is explicit opt-in. An additive `brief` detail value is +available for callers choosing that response mode; existing detail values +and the full-profile diagnostic default remain available. + +## 8. Security review + +No trust, filesystem, SSRF, provider-egress, authorization, or tool-execution +policy changed. Compact schemas do not narrow runtime capabilities or imply +permission denial. + +## 9. Documentation and operations + +Configuration and behavior are documented in `docs/config.md`, +`architecture/config.md`, `architecture/mcp.md`, `docs/lm-studio.md`, and +`docs/retrieval-quality-context-efficiency.md`. The context-efficiency report +distinguishes actual child `tools/list` bytes from estimated token costs and +LM Studio prompt inclusion. + +## 10. Unresolved findings + +- Low, operational: exact schema injection and tokenizer costs inside LM + Studio remain unmeasured; M003 owns that qualification. +- No M001 correctness, security, or compatibility finding remains open. + +## 11. Roadmap disposition + +M001 moves from active to closed. M002 is ready and may proceed on this +accepted M001 contract. M003 remains blocked on M002 and its separately +required real-client qualification. + +## 12. Registry updates + +In the same closure commit, M001 is marked closed, M002 ready, the roadmap +records M001 closed/M002 ready/M003 blocked, and `plans/registry.md` points to +this control record. diff --git a/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md b/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md index 446f10a..ae8a64c 100644 --- a/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md +++ b/plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md @@ -1,6 +1,6 @@ # M001 — Opt-in Compact Ten-Tool MCP Advertisement -Status: active +Status: closed (candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`) Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (2026-10-30) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` diff --git a/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md b/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md index 99dba22..bf3da95 100644 --- a/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md +++ b/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md @@ -1,6 +1,6 @@ # M002 — Brief Evidence Projection and Deterministic Output Budgets -Status: blocked (implementation is ready for closure after accepted M001 closure) +Status: ready (M001 closed; implementation candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`) Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (reconfirm after M001) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` diff --git a/plans/registry.md b/plans/registry.md index 286872a..bb248cf 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001 active; M002-M003 blocked) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001 compact ten-tool schema advertisement in progress | New opt-in direct-client presentation; full ten-tool/default diagnostic contract unchanged; no CodeGG implementation dependency. | +| Standalone MCP local-inference efficiency | active (M001 closed; M002 ready; M003 blocked) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001 closed at `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M002 implementation complete and ready for closure | M001 closure `plans/closure/standalone-mcp-local-inference/001-status.md`; M003 requires M002 closure and real LM Studio consumer evidence. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -52,8 +52,8 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Subsystem | Milestone | Status | Implementation plan | Dependencies / handoff note | |---|---|---|---|---| -| Standalone MCP local-inference | M001 opt-in compact tool advertisement | active | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | No hard dependency; implement first against main `a666517c` after verifying current tool-list baseline. Keep default full ten-tool contract. | -| Standalone MCP local-inference | M002 brief evidence and deterministic budgets | blocked | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Hard: accepted M001 closure. Default profile remains diagnostic; brief is opt-in via compact profile or explicit call. | +| Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | +| Standalone MCP local-inference | M002 brief evidence and deterministic budgets | ready | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | M001 accepted; implementation is in candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; canonical gates passed. | | Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Hard: accepted M001/M002 closure; live LM Studio test is an operational closure gate, never assumed from synthetic bytes. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | @@ -125,7 +125,7 @@ Corrective closure sequence: `plans/implementation/dependency-evidence-hardening **Python native library gate:** ADR-0005 authorizes a narrow direct Rust SDK facade and Python PyO3 native package without broad Rust module stability. M001–M008 are closed at `067468a` with fully-green qualifying run `38641009246` (5/5 builds, 20/20 smokes, offline sdist rebuild, green audit, digests in `008-status.md`). Earlier runs are historical/deficiency/supporting evidence only. First PyPI publication is separately closed under `python-pypi-distribution` M003 at 0.4.4; Eggpack binary assets remain unchanged. -**Standalone direct-MCP local-inference gate:** New roadmap `plans/subsystems/standalone-mcp-local-inference-roadmap.md` is ready for M001; M002 waits on M001 closure and M003 waits on M001/M002 closure. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. No current LM Studio smoke or token savings are claimed. +**Standalone direct-MCP local-inference gate:** M001 is closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec` with an 18,545-byte compact ten-tool stdio list and 43/43 discovery results. M002 implementation is ready for closure after this M001 record. M003 still requires M002 closure and real LM Studio model-visible qualification; offline wire bytes are not host token savings. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. **Tool-surface gate:** M001-M007 closed. Do not run evaluation against pre-consolidation bytes and present it as consolidation diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index 294498c..6b446f3 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001 active, M002-M003 blocked by hard dependencies) +Status: active (M001 closed, M002 ready, M003 blocked by M002) Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. From 5daec8065f676a107a0539a4e7781f6949aa57e2 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Fri, 30 Oct 2026 20:54:36 +0000 Subject: [PATCH 05/25] docs: close brief projection and record LM Studio gate --- .../002-status.md | 149 ++++++++++++++++++ .../003-status.md | 79 ++++++++++ .../002-brief-evidence-and-output-budgets.md | 2 +- ...-studio-consumer-qualification-and-docs.md | 2 +- plans/registry.md | 8 +- .../standalone-mcp-local-inference-roadmap.md | 2 +- 6 files changed, 235 insertions(+), 7 deletions(-) create mode 100644 plans/closure/standalone-mcp-local-inference/002-status.md create mode 100644 plans/closure/standalone-mcp-local-inference/003-status.md diff --git a/plans/closure/standalone-mcp-local-inference/002-status.md b/plans/closure/standalone-mcp-local-inference/002-status.md new file mode 100644 index 0000000..bfa78ec --- /dev/null +++ b/plans/closure/standalone-mcp-local-inference/002-status.md @@ -0,0 +1,149 @@ +# Standalone MCP Local-Inference M002 — Brief Projection Closure + +Status: closed + +Source implementation plan: +`plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` + +Source subsystem roadmap: +`plans/subsystems/standalone-mcp-local-inference-roadmap.md` + +Repository baseline reviewed: M001 closure commit `d983fa5` and implementation +candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`. + +Implementation candidate: `02a8af6fe4f87971c7daca7d1fca20c087b328ec`. +M001 closure: `d983fa5` (`plans/closure/standalone-mcp-local-inference/001-status.md`). + +## 1. Executive finding + +M002 is closed. The additive `brief` result mode defaults only when the +operator selects the compact profile and omits `response_detail`; explicit +per-call values still win, and the full profile remains diagnostic by default. +Brief responses use deterministic field-aware reduction and a configurable +serialized-character ceiling. Status, failure-vs-absence, trust and source +identity remain visible; oversized applicability and provider-failure arrays +retain summary counts, and omitted batch failures retain their member indexes. + +Nontrivial fixtures exceed the plan's 30% reduction criterion. Sparse task +corpus results are separately reported at 11.8% reduction and are not hidden +behind the richer fixtures. No LM Studio model-token or task-quality claim is +made here; that is M003's live consumer gate. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Brief is additive and profile-aware | `src/mcp/projection.rs`, config and tool tests | pass | Compact omitted detail => brief; full => diagnostic; explicit detail wins | +| Budget is bounded and configurable | `tests/mcp_projection.rs`, config validation | pass | Default 8,000 characters; accepted range 512–32,000 | +| Representative nontrivial outputs save at least 30% | `docs/retrieval-quality-context-efficiency.md` | pass | Web search 48.0%; fetch 74.6%; synthetic repo/research stress fixtures 93.8% | +| Failure, absence and partial status survive | Projection fixtures and MCP/domain suites | pass | Provider failure summaries retain total and error classes | +| Security applicability remains truthful under pressure | 40-entry applicability stress fixture | pass | State totals retained: 14 affected, 13 not affected, 13 unknown; output <=512 chars | +| Batch failure members are not silently erased | 20-member failure stress fixture | pass | Each failure remains as a record or a listed `failed_member_indices` value under 512 chars | +| Stable IDs, trust markers, truncation and PDF coordinates survive | `tests/mcp_projection.rs` | pass | Explicit regression coverage | +| Evidence bundle stays identity-preserving | `tests/mcp_projection.rs` | pass | Brief does not trim canonical bundle identity | +| Sparse corpus is reported without an artificial target | Context-efficiency report | pass | 21,221 diagnostic vs 18,715 brief bytes (11.8% smaller); same 14 retrieved relevant documents | + +## 3. Production implementation evidence + +`ResponseDetail::Brief` is parsed by the shared MCP projection layer. It +applies existing compact safety reductions, an explicit field allowlist, +UTF-8-safe content clipping, deterministic record omission and +`returned_count`/`omitted_count`/`output_truncated` metadata. If a security +applicability list cannot fit, `applicability_summary` preserves per-state +counts and exposes omitted entries. Provider failures retain a total and +error-class counts; batch failures retain indexes when full failure records +must be omitted. The tool handlers use the configured brief budget for both +structured content and the text fallback. Diagnostic remains available as an +explicit override. Evidence-bundle identity and the diagnostic-only provider +status contract are retained. + +Measured fixtures (serialized bytes): + +| Fixture | Diagnostic | Brief | Reduction | +|---|---:|---:|---:| +| Rich web search with provider failure | 2,728 | 1,418 | 48.0% | +| Web fetch with 2,000-character text and a link | 4,292 | 1,090 | 74.6% | +| Repo grouped stress fixture, deliberately redundant | 10,846 | 669 | 93.8% | +| Research grouped stress fixture, deliberately redundant | 10,846 | 669 | 93.8% | +| Sparse 15-task deterministic trace | 21,221 | 18,715 | 11.8% | + +The repo/research figures are synthetic stress fixtures, not production +averages. Byte and character costs are not model-token costs. + +## 4. Verification executed + +- `rtk make check` — pass; formatting, Clippy, no-default-features check, + all-features tests (3,486), docs, hygiene, dependency policy and packaging + contract checks passed. +- `rtk make docs-check` — pass after implementation and M001 closure update. +- `rtk make bench-check` — pass; optimized Criterion harness compiled with + `--no-run` (11m32s on this host). +- `cargo test --locked --all-features --test mcp_projection` — pass + (21 tests, including oversized applicability/provider/batch cases). +- `cargo test --locked --all-features --test mcp_http` — pass (12 tests). +- `cargo test --locked --all-features --test mcp_2026_protocol` — pass + (14 tests). +- `cargo test --locked --all-features --test tool_surface_evaluation -- --nocapture` + — pass (5 tests; both profiles 43/43 top-1 and recall@3, MRR 1.000). +- `rtk git diff --check` — pass. + +## 5. Invariant review + +Projection operates only at the MCP presentation boundary. It does not change +upstream retrieval, candidate count, ranking, provider selection, cache, +browser/PDF extraction, security decisions, stable identity, or execution +budgets. No inference, summarization, persistent result cache, or follow-up +fetch is introduced. Full/diagnostic remains the unmodified configuration +default. + +## 6. Failure and recovery review + +Budget fitting omits complete successful records before critical state. If +large failure or applicability collections still exceed the budget, the +summaries and omitted-index/count signals keep the result bounded while +retaining aggregate truth. The typed bundle path remains complete. Explicit +diagnostic provides the complete bounded diagnostic payload when requested. + +## 7. Migration and compatibility review + +No migration is required. `brief` is additive; existing `compact`, `standard` +and `diagnostic` modes remain available. Full-profile default behavior remains +diagnostic. Compact is opt-in through server configuration; explicit call +values override the profile default. + +## 8. Security review + +Untrusted framing, trust markers, source locators, failure state and security +applicability are retained in brief results. Large applicability lists retain +all status totals even when individual records are omitted. No authorization, +SSRF, provider-routing, local-filesystem, or egress policy changed. + +## 9. Documentation and operations + +`architecture/mcp.md`, `docs/config.md`, `docs/test-inventory.md`, +`docs/retrieval-quality-context-efficiency.md`, and `docs/lm-studio.md` explain +the profile default, brief budget, semantics, measurements, and live-client +limits. The ordinary rollback is to set `schema_profile = "full"` and restart +the server process. + +## 10. Unresolved findings + +- Low, operational: sparse result sets show less than the 30% reduction target; + this is reported as observed and no universal reduction is claimed. +- Low, operational: LM Studio tokenizer, host-injection and task-quality + behavior remain unmeasured; M003 is blocked pending live consumer evidence. +- No M002 correctness or security finding remains open. + +## 11. Roadmap disposition + +M001 and M002 are closed. M003 implementation dependencies are satisfied, but +M003 remains blocked until live LM Studio model-turn, context-cost, +compatibility and task-quality evidence is obtained. The offline measurements +in this record do not discharge that condition. + +## 12. Registry updates + +In the same closure commit, M002 is marked closed, the roadmap records +M001-M002 closed and M003 blocked on live client evidence, and +`plans/registry.md` points to this record. A separate M003 status record names +the remaining operational gate. diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md new file mode 100644 index 0000000..9b0be11 --- /dev/null +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -0,0 +1,79 @@ +# Standalone MCP Local-Inference M003 — Operational Status + +Status: blocked (live LM Studio consumer qualification not complete) + +Source implementation plan: +`plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` + +Source subsystem roadmap: +`plans/subsystems/standalone-mcp-local-inference-roadmap.md` + +Implementation candidate reviewed: `02a8af6fe4f87971c7daca7d1fca20c087b328ec`. +Predecessor closures: M001 `d983fa5`; M002 `plans/closure/standalone-mcp-local-inference/002-status.md`. + +## 1. Current finding + +M003 is not closed. The onboarding guide and repeatable offline installed- +binary stdio smoke are implemented. The smoke initializes the actual Eggsearch +child, confirms ten names in each profile, measures the serialized `tools/list` +array (72,923 full / 18,545 compact bytes), calls `provider_status` with +probing disabled, compares structured results and exits successfully. This is +protocol and executable evidence, not an LM Studio model-turn measurement. + +The local host is LM Studio 0.4.26+4 with `google/gemma-4-e2b` available. An +API chat request addressing `mcp/eggsearch` returned: `Permission denied to use +plugin 'mcp/eggsearch'. Ensure that the server configuration allows plugin +usage and, if using an API token, it has the necessary permissions.` The +LM Studio setting **Allow calling servers from mcp.json** is currently not +enabled. No setting was changed. No interactive model-visible token count, +paired task result, repair count, latency comparison, or larger-model +reference run has been recorded. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Remaining work | +|---|---|---|---| +| Copyable installed-binary `mcp.json` setup | `docs/lm-studio.md` | pass | Confirm with a live LM Studio chat | +| Windows/macOS/Linux path guidance and stdio logging contract | `docs/lm-studio.md` | pass | None for documentation | +| Reproducible installed-candidate stdio smoke | `scripts/lm_studio_stdio_smoke.py` | pass | Live host integration still required | +| Ten tools and profile list sizes | Real initialized child-process smoke | pass | Measure actual LM Studio prompt inclusion/tokens | +| Interactive small and larger model comparison | No model-turn records | blocked | Run paired prompt set with actual models | +| Tool choice, arguments, repairs, evidence truthfulness, latency | No live task records | blocked | Record per-plan measurements for web, security, research, fetch and failures | +| OutputSchema/structuredContent/text fallback and refresh behavior | Offline protocol coverage only | partial | Observe LM Studio behavior across sessions/config refresh | +| Protected API MCP smoke | Permission-denied API response | blocked | Enable the named LM Studio server setting and validate API authorization | + +## 3. Work completed + +`docs/lm-studio.md` documents the opt-in compact/brief TOML, absolute-path +`mcp.json` entry, Windows escaping, full-profile fallback, stdio logging, +refresh behavior, API `allowed_tools`, the protected API setting and the limits +of byte-based estimates. `scripts/lm_studio_stdio_smoke.py` provides a local, +network-free handshake/list/call check. Context-efficiency records actual +serialized bytes and clearly separates these from model-token estimates. + +## 4. Required evidence to unblock and close + +1. With explicit operator authorization, enable **Allow calling servers from + mcp.json** for the LM Studio API, use authenticated local access, run the + configured MCP smoke, then restore the setting to its prior state if it was + enabled only for this qualification. +2. Run paired full+diagnostic and compact+brief tasks on a smaller + tool-capable model and a larger reference model, with at least 8k context + where available. Include `security_search` and `research_search` tasks. +3. Record exact model identifiers, tokenizer, context length, host-injected + prompt tokens, tool-choice and argument success, repairs/retries, evidence + found, truthful failure interpretation, calls, latency and prompt + processing time. Separate client `allowed_tools` effects from profile + effects. +4. Observe output-schema handling, structured content vs text fallback, + refresh/new-chat visibility, and rollback in the actual LM Studio host. +5. Update this record with measured results and a new exact implementation + candidate SHA; close only if the plan's live acceptance criteria pass. + +## 5. Security and operational disposition + +No API setting or MCP permission was changed during this work. Enabling that +setting expands API clients' ability to invoke locally configured MCP tools +and requires explicit action-time authorization. Until the live gate is +authorized and completed, M003 remains blocked and no LM Studio context or +quality improvement is claimed. diff --git a/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md b/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md index bf3da95..1f3eb51 100644 --- a/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md +++ b/plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md @@ -1,6 +1,6 @@ # M002 — Brief Evidence Projection and Deterministic Output Budgets -Status: ready (M001 closed; implementation candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`) +Status: closed (candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`) Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (reconfirm after M001) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` diff --git a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md index 2a52265..c9ce445 100644 --- a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md +++ b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md @@ -1,6 +1,6 @@ # M003 — LM Studio Direct-MCP Qualification and Consumer Documentation -Status: blocked (hard dependency: M002 closure; live LM Studio API smoke also requires its protected API setting) +Status: blocked (M001/M002 dependencies closed; live LM Studio consumer qualification remains; see `plans/closure/standalone-mcp-local-inference/003-status.md`) Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (rebase baseline to exact M002 candidate at start) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` diff --git a/plans/registry.md b/plans/registry.md index bb248cf..b8c4ab1 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001 closed; M002 ready; M003 blocked) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001 closed at `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M002 implementation complete and ready for closure | M001 closure `plans/closure/standalone-mcp-local-inference/001-status.md`; M003 requires M002 closure and real LM Studio consumer evidence. | +| Standalone MCP local-inference efficiency | active (M001-M002 closed; M003 blocked) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M003 awaits live LM Studio model-visible qualification | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -53,8 +53,8 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Subsystem | Milestone | Status | Implementation plan | Dependencies / handoff note | |---|---|---|---|---| | Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | -| Standalone MCP local-inference | M002 brief evidence and deterministic budgets | ready | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | M001 accepted; implementation is in candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; canonical gates passed. | -| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Hard: accepted M001/M002 closure; live LM Studio test is an operational closure gate, never assumed from synthetic bytes. | +| Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | +| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | M001/M002 closed; offline installed-child smoke passes. Live LM Studio API call is permission-denied until the protected server setting is authorized; no model-turn or token evidence yet. `plans/closure/standalone-mcp-local-inference/003-status.md`. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | @@ -125,7 +125,7 @@ Corrective closure sequence: `plans/implementation/dependency-evidence-hardening **Python native library gate:** ADR-0005 authorizes a narrow direct Rust SDK facade and Python PyO3 native package without broad Rust module stability. M001–M008 are closed at `067468a` with fully-green qualifying run `38641009246` (5/5 builds, 20/20 smokes, offline sdist rebuild, green audit, digests in `008-status.md`). Earlier runs are historical/deficiency/supporting evidence only. First PyPI publication is separately closed under `python-pypi-distribution` M003 at 0.4.4; Eggpack binary assets remain unchanged. -**Standalone direct-MCP local-inference gate:** M001 is closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec` with an 18,545-byte compact ten-tool stdio list and 43/43 discovery results. M002 implementation is ready for closure after this M001 record. M003 still requires M002 closure and real LM Studio model-visible qualification; offline wire bytes are not host token savings. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. +**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; the compact ten-tool stdio list is 18,545 bytes, discovery remains 43/43, and brief response budgets preserve required failure/applicability signals. M003 remains blocked: the offline installed-child smoke passes, but LM Studio API use of the configured server returned permission denied, and no live model-turn/token/task-quality evidence exists. Offline wire bytes are not host token savings. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. **Tool-surface gate:** M001-M007 closed. Do not run evaluation against pre-consolidation bytes and present it as consolidation diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index 6b446f3..fbcd606 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001 closed, M002 ready, M003 blocked by M002) +Status: active (M001-M002 closed, M003 blocked on live LM Studio consumer qualification) Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. From 776f8ca4a8246e467041696b2220600eb7b2648d Mon Sep 17 00:00:00 2001 From: eggsearch Date: Fri, 30 Oct 2026 21:40:11 +0000 Subject: [PATCH 06/25] fix(mcp): advertise compact research argument values --- .../003-status.md | 160 ++++++++++++------ ...-studio-consumer-qualification-and-docs.md | 2 +- ...mpact-schema-discoverability-corrective.md | 94 ++++++++++ plans/registry.md | 7 +- .../standalone-mcp-local-inference-roadmap.md | 3 +- src/mcp/server.rs | 87 ++++++++++ tests/mcp_tool_contract.rs | 90 ++++++++++ 7 files changed, 382 insertions(+), 61 deletions(-) create mode 100644 plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index 9b0be11..e923266 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M003 — Operational Status -Status: blocked (live LM Studio consumer qualification not complete) +Status: blocked (live LM Studio interaction is qualified, but the acceptance matrix is incomplete and Gemma quality failures require a corrective decision) Source implementation plan: `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` @@ -8,72 +8,120 @@ Source implementation plan: Source subsystem roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` -Implementation candidate reviewed: `02a8af6fe4f87971c7daca7d1fca20c087b328ec`. +Implementation candidate exercised: `02a8af6fe4f87971c7daca7d1fca20c087b328ec`. Predecessor closures: M001 `d983fa5`; M002 `plans/closure/standalone-mcp-local-inference/002-status.md`. ## 1. Current finding -M003 is not closed. The onboarding guide and repeatable offline installed- -binary stdio smoke are implemented. The smoke initializes the actual Eggsearch -child, confirms ten names in each profile, measures the serialized `tools/list` -array (72,923 full / 18,545 compact bytes), calls `provider_status` with -probing disabled, compares structured results and exits successfully. This is -protocol and executable evidence, not an LM Studio model-turn measurement. +M003 is not closed. The installed-candidate stdio smoke and the interactive +LM Studio chat path both worked with the candidate, and all ten tools remained +available. This is the first real model-turn evidence for the branch. It does +not establish that compact is a safe default: one paired web task showed a +large context-meter reduction, but specialist research and direct-fetch tasks +had material quality failures. The tests used only `google/gemma-4-e2b` in +LM Studio 0.4.26+4, so the required larger-reference comparison remains open. -The local host is LM Studio 0.4.26+4 with `google/gemma-4-e2b` available. An -API chat request addressing `mcp/eggsearch` returned: `Permission denied to use -plugin 'mcp/eggsearch'. Ensure that the server configuration allows plugin -usage and, if using an API token, it has the necessary permissions.` The -LM Studio setting **Allow calling servers from mcp.json** is currently not -enabled. No setting was changed. No interactive model-visible token count, -paired task result, repair count, latency comparison, or larger-model -reference run has been recorded. +Model/server facts observed in LM Studio: -## 2. Requirement-to-evidence matrix +- Model identifier: `google/gemma-4-e2b`, MLX 4-bit, 4.37 GB on disk. +- Context limit shown by the app: 10,703 tokens (above the plan's 8k minimum). +- Model advertised tool-use capability. A per-request tokenizer identity and + exact per-chat prompt-token usage were not exposed in the app chat UI. +- The local server was running at `http://127.0.0.1:1234`. +- Temporary LM Studio config selected the branch release binary and compact + profile. The original `~/.lmstudio/mcp.json` was restored and byte-compared + with its backup after the tests. No API authentication or server permission + setting was changed. + +## 2. Live task evidence + +All chats were fresh LM Studio conversations. Tool approval was granted once +per call; the persistent “Always allow” option was not selected. + +| Profile/task | Tool behavior | Result quality | Observed context / timing | +|---|---|---|---| +| Full: official Rust 1.90.0 announcement | Selected `web_search`; valid arguments; one call; no repair | Correct official title, URL, and source classification | Context meter 96.1% of 10,703; total latency not captured | +| Compact: same official announcement task | Selected `web_search`; valid arguments; one call; no repair | Same correct title, URL, and source classification | Context meter 51.9% of 10,703. Paired UI-meter difference: 44.2 percentage points (about 4,730 context slots); this is a single-task observation, not a general token-savings claim. Model thought durations were 12.79 s before and 9.91 s after the tool call; end-to-end latency was not captured. | +| Compact: Log4j 2.14.1 / CVE-2021-44228 | Selected `security_search`; valid applicability arguments; one call | Correctly reported affected status and CVE; accurately retained Yahoo HTTP 500 provider failures, untrusted-result warning, and no severity inference | Context meter 44.3%; end-to-end latency not captured | +| Compact: two concrete Rust 1.90.0 changes with evidence | Four `research_search` calls (three argument failures, one partial search); one irrelevant current release-index `web_fetch`; one `web_search` argument failure followed by a valid retry | Final answer misclassified secondary `releases.rs` as official release documentation and gave weakly supported claims. The first two schema errors used invalid research-domain/goal values; the web-search retry corrected an invalid safe-search value. | Context meter reached 109% of 10,703; task overran the available context. Latency not captured. | +| Compact: fetch the official Rust 1.90.0 announcement and name two changes | Selected `web_fetch`; valid URL; one call | Fetch succeeded and LLD-default change was correctly identified. The second item was an update command, not a concrete release change, so the requested two-change extraction failed. | Context meter 31.3%; model thought durations 5.84 s before and 12.12 s after fetch; end-to-end latency not captured. | + +The paired web search is evidence that this LM Studio interactive path used the +two profiles differently. Its task and model were paired, but there was only +one prompt pair and the UI context meter is not an independently tokenized +schema/result attribution. No exact tokenizer counts were captured. The model +server log showed aggregate generation and an unrelated request; it did not +provide a reliable per-chat usage breakdown for this test set. + +## 3. Requirement-to-evidence matrix | Requirement | Evidence | Result | Remaining work | |---|---|---|---| -| Copyable installed-binary `mcp.json` setup | `docs/lm-studio.md` | pass | Confirm with a live LM Studio chat | -| Windows/macOS/Linux path guidance and stdio logging contract | `docs/lm-studio.md` | pass | None for documentation | -| Reproducible installed-candidate stdio smoke | `scripts/lm_studio_stdio_smoke.py` | pass | Live host integration still required | -| Ten tools and profile list sizes | Real initialized child-process smoke | pass | Measure actual LM Studio prompt inclusion/tokens | -| Interactive small and larger model comparison | No model-turn records | blocked | Run paired prompt set with actual models | -| Tool choice, arguments, repairs, evidence truthfulness, latency | No live task records | blocked | Record per-plan measurements for web, security, research, fetch and failures | -| OutputSchema/structuredContent/text fallback and refresh behavior | Offline protocol coverage only | partial | Observe LM Studio behavior across sessions/config refresh | -| Protected API MCP smoke | Permission-denied API response | blocked | Enable the named LM Studio server setting and validate API authorization | +| Copyable installed-binary `mcp.json` setup | `docs/lm-studio.md`; config temporarily pointed at branch release binary | pass | None for configuration syntax | +| Windows/macOS/Linux paths and stdio logging contract | `docs/lm-studio.md` | pass | None for documentation | +| Reproducible installed-candidate stdio smoke | `scripts/lm_studio_stdio_smoke.py` | pass | Offline protocol evidence only | +| Ten tools and profile list sizes | Real initialized child-process smoke: 72,923 full / 18,545 compact bytes; all ten names; `provider_status` with probing off | pass | Measure model-tokenized attribution | +| LM Studio tool visibility and interactive tool use | Full and compact chats each called `web_search`; compact also called `security_search`, `research_search`, and `web_fetch` | partial | Add more paired tasks and verify refresh/new-chat behavior explicitly | +| Paired small/larger model comparison | One full/compact pair on Gemma E2B only | blocked | Run the same task matrix on a larger local reference model | +| Tool choice, argument validity, repairs, evidence and failure truthfulness | Five task records above | partial / quality concern | Investigate research argument retries, context overrun, and source classification; add paired specialist/fetch cases | +| Exact tokenizer and prompt-token attribution | App context meter only; no per-chat token counts or tokenizer identifier | blocked | Capture host-injected prompt and response usage for each profile | +| Output schema / structuredContent vs text fallback | Offline protocol coverage only | partial | Inspect actual LM Studio handling and compare a generic MCP inspector/client | +| Protected API MCP smoke | Previous API call was permission-denied; “Allow calling servers from mcp.json” remained disabled | blocked | Operator must configure LM Studio authentication and authorize that server permission before API qualification; no such security settings were changed | +| Config rollback | Restored original `~/.lmstudio/mcp.json`; `cmp` against backup succeeded | pass | None | -## 3. Work completed +## 4. Work completed `docs/lm-studio.md` documents the opt-in compact/brief TOML, absolute-path `mcp.json` entry, Windows escaping, full-profile fallback, stdio logging, -refresh behavior, API `allowed_tools`, the protected API setting and the limits -of byte-based estimates. `scripts/lm_studio_stdio_smoke.py` provides a local, +refresh behavior, API `allowed_tools`, protected API setting, and limits of +byte-based estimates. `scripts/lm_studio_stdio_smoke.py` provides a local, network-free handshake/list/call check. Context-efficiency records actual -serialized bytes and clearly separates these from model-token estimates. - -## 4. Required evidence to unblock and close - -1. With explicit operator authorization, enable **Allow calling servers from - mcp.json** for the LM Studio API, use authenticated local access, run the - configured MCP smoke, then restore the setting to its prior state if it was - enabled only for this qualification. -2. Run paired full+diagnostic and compact+brief tasks on a smaller - tool-capable model and a larger reference model, with at least 8k context - where available. Include `security_search` and `research_search` tasks. -3. Record exact model identifiers, tokenizer, context length, host-injected - prompt tokens, tool-choice and argument success, repairs/retries, evidence - found, truthful failure interpretation, calls, latency and prompt - processing time. Separate client `allowed_tools` effects from profile - effects. -4. Observe output-schema handling, structured content vs text fallback, - refresh/new-chat visibility, and rollback in the actual LM Studio host. -5. Update this record with measured results and a new exact implementation - candidate SHA; close only if the plan's live acceptance criteria pass. - -## 5. Security and operational disposition - -No API setting or MCP permission was changed during this work. Enabling that -setting expands API clients' ability to invoke locally configured MCP tools -and requires explicit action-time authorization. Until the live gate is -authorized and completed, M003 remains blocked and no LM Studio context or -quality improvement is claimed. +serialized bytes and keeps those separate from model-token estimates. + +The offline smoke passed for both profiles with ten tools and successful +`provider_status`. Interactive tests demonstrated that Gemma can select and +call core and specialist tools, but also recorded argument repair, context +overrun, weak source classification, and an incomplete extraction. These +negative findings rule out closure. M004 is registered to expose runtime- +accepted research goal/domain values in compact schemas and repeat the focused +qualification. + +## 5. Remaining evidence to unblock and close + +1. Run the same paired full+diagnostic and compact+brief prompts on Gemma E2B + and a larger reference model, including security, research, page/PDF fetch, + repo/evidence, and grouped-failure tasks. Capture tool choice, argument + validity, repairs, evidence, failure interpretation, calls, total latency, + prompt processing time, and context usage. +2. Capture exact tokenizer identity and host-injected prompt/completion token + counts for each paired task. Keep those distinct from context-meter + percentages and serialized JSON byte counts. +3. Characterize LM Studio's structured output and text fallback handling, + visibility after config refresh/new chat, and compare one generic MCP + inspector/client. +4. For API-specific qualification, the LM Studio server requires + authentication before enabling “Allow calling servers from mcp.json.” No + credentials were created and no security-sensitive setting was changed; + complete that prerequisite in LM Studio, then run the API smoke and restore + prior settings. +5. M004 is implemented in source to expose accepted research goal/domain + values in compact schemas. Focused contract tests, all-features compile, + release build, and installed-child smoke pass. The release binary advertised + all ten tools at 18,990 compact schema bytes. A post-fix Gemma turn was not + completed: LM Studio's exposed accessibility state and visible window + diverged during integration refresh, so results from the previous child + were not attributed to the new candidate. See + `plans/closure/standalone-mcp-local-inference/004-status.md`. +6. Run the focused live tasks against the corrected exact candidate, then + update this status with the result; close M003 only after its full live + acceptance criteria pass. + +## 6. Security and operational disposition + +No API authentication, credential, or MCP permission setting was changed. +The temporary global MCP config was restored and verified byte-identical to +its pre-test backup. LM Studio interactive chat qualification is now recorded, +so the remaining block is broader than the earlier API permission denial: +model comparison, exact token attribution, host behavior checks, and the +substantial live quality failures remain open. Do not claim a general LM +Studio token-saving or quality win from the single paired task. diff --git a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md index c9ce445..9ca700f 100644 --- a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md +++ b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md @@ -1,6 +1,6 @@ # M003 — LM Studio Direct-MCP Qualification and Consumer Documentation -Status: blocked (M001/M002 dependencies closed; live LM Studio consumer qualification remains; see `plans/closure/standalone-mcp-local-inference/003-status.md`) +Status: blocked (M001/M002 dependencies closed; interactive Gemma qualification is recorded, but model comparison, exact token attribution, API/host behavior checks, and quality issues remain; see `plans/closure/standalone-mcp-local-inference/003-status.md`) Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (rebase baseline to exact M002 candidate at start) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` diff --git a/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md b/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md new file mode 100644 index 0000000..dce72e5 --- /dev/null +++ b/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md @@ -0,0 +1,94 @@ +# M004 — Compact Schema Discoverability Corrective + +Status: active (registered after live M003 qualification found research argument retries) +Primary class: capability +Source roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` +Corrects: M003 evidence in `plans/closure/standalone-mcp-local-inference/003-status.md` +ADRs: ADR-0001 and ADR-0006 +Baseline: `5daec8065f676a107a0539a4e7781f6949aa57e2` + +## Objective + +Make the compact MCP input schema disclose the accepted values for bounded +research classification fields, then verify that the actual advertised +contract retains those values without weakening runtime validation or +materially eroding compact size. Re-run targeted LM Studio tasks to learn +whether the schema correction reduces argument retries. This corrective does +not claim to solve general small-model reasoning or evidence extraction. + +## Finding and why M001/M002 missed it + +In M003 compact-profile LM Studio trials, Gemma E2B submitted invalid +`research_domain` and `goal` strings; the server's repair responses provided +the accepted values. `ResearchSearchArgs` currently models these as strings, +and compact projection removes property descriptions. Runtime parsing already +has finite accepted sets, but those constraints are not necessarily visible in +the compact tool schema. M001 qualified schema size and names, while M002 +qualified output semantics; neither asserted these input-value constraints. + +The M003 direct-fetch trial also returned an incomplete two-change answer, and +the research trial overran context and misclassified a secondary source. These +are qualification findings, not presumed code defects; this plan records them +as live recheck cases and will not loosen trust or retrieval policy to improve +a benchmark score. + +## Scope and non-goals + +- Add compact-profile JSON Schema enums for `research_search.goal` and + `research_search.research_domain`, including the accepted aliases. +- Add regression assertions that compact `tools/list` advertises values that + runtime parsing accepts. +- Requalify ten-tool count, deterministic fingerprint, and serialized compact + size against the exact candidate. +- Repeat focused Gemma E2B research-search calls and the official-page fetch + task, recording retries, context and evidence quality. +- Do not change tool handlers, accepted runtime inputs, providers, ranking, + fetch behavior, brief response semantics, MCP permissions, or API auth. +- Do not mark M003 closed from this corrective alone; its larger-model, + tokenizer, structured-content, refresh, and protected API criteria remain. + +## Ordered work packages + +1. Inspect the generated full/compact schemas and attach compact enum + constraints from the canonical runtime accepted-value lists. +2. Add schema contract tests for `goal` and `research_domain` values, all ten + names, stable compact fingerprint, and the existing size ceiling. +3. Run focused MCP tool-contract tests and `make docs-check`; then run the + canonical `make check` gate if production code changes. +4. Rebuild the release candidate and installed-child smoke. Repeat the two + focused live tasks in a fresh LM Studio session using Gemma E2B. Record + per-call argument validity, retries, context meter, returned evidence and + any remaining failure. Keep `mcp.json` rollback verified. +5. Update this plan and M003 status with exact candidate and test evidence; + leave M003 blocked for remaining operational acceptance items. + +## Invariants + +- All ten tool names remain advertised; compact remains an opt-in presentation + profile and is not an execution or authorization boundary. +- Every newly advertised value is accepted by the existing runtime parser. +- Full profile and runtime validation remain unchanged. +- Failure, applicability, trust, and truncation signals remain visible in + tool results. +- No external network dependency is introduced to routine CI. + +## Acceptance and stop conditions + +- Compact `research_search` schema advertises all accepted research goal and + domain values, including supported aliases; contract tests prove these + values are accepted by runtime parsers. +- Ten canonical tools remain present and compact schema stays below the + current 36,000-byte ceiling with deterministic fingerprinting. +- Focused offline tests, the canonical gate, and installed-child smoke pass + on the exact candidate. +- Live results report whether the targeted retries were eliminated; no + generalized quality claim is made from two tasks. +- If JSON Schema or client behavior cannot represent these constraints + compatibly, stop and record the limitation rather than changing runtime + semantics. + +## Closure evidence + +Record exact code candidate, schema excerpts/measurements, test commands and +outcomes, live Gemma task traces, config rollback, and remaining M003 blockers +in `plans/closure/standalone-mcp-local-inference/004-status.md`. diff --git a/plans/registry.md b/plans/registry.md index b8c4ab1..8f4180a 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001-M002 closed; M003 blocked) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M003 awaits live LM Studio model-visible qualification | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`. | +| Standalone MCP local-inference efficiency | active (M001-M002 closed; M003 blocked; M004 active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M003 interactive Gemma evidence recorded; M004 addresses compact research-schema discoverability | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`; M004 implementation `004-compact-schema-discoverability-corrective.md`. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -54,7 +54,8 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under |---|---|---|---|---| | Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | -| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | M001/M002 closed; offline installed-child smoke passes. Live LM Studio API call is permission-denied until the protected server setting is authorized; no model-turn or token evidence yet. `plans/closure/standalone-mcp-local-inference/003-status.md`. | +| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | M001/M002 closed; offline smoke and five interactive Gemma task records are in `plans/closure/standalone-mcp-local-inference/003-status.md`. One paired web task reduced the UI context meter, but research/fetch quality failures remain; larger-model, exact token attribution, structured-content/refresh, and protected API checks remain open. | +| Standalone MCP local-inference | M004 compact schema discoverability corrective | active | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Corrects M003's invalid research goal/domain argument retries by advertising runtime-accepted values; focused offline and live requalification required. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | @@ -125,7 +126,7 @@ Corrective closure sequence: `plans/implementation/dependency-evidence-hardening **Python native library gate:** ADR-0005 authorizes a narrow direct Rust SDK facade and Python PyO3 native package without broad Rust module stability. M001–M008 are closed at `067468a` with fully-green qualifying run `38641009246` (5/5 builds, 20/20 smokes, offline sdist rebuild, green audit, digests in `008-status.md`). Earlier runs are historical/deficiency/supporting evidence only. First PyPI publication is separately closed under `python-pypi-distribution` M003 at 0.4.4; Eggpack binary assets remain unchanged. -**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; the compact ten-tool stdio list is 18,545 bytes, discovery remains 43/43, and brief response budgets preserve required failure/applicability signals. M003 remains blocked: the offline installed-child smoke passes, but LM Studio API use of the configured server returned permission denied, and no live model-turn/token/task-quality evidence exists. Offline wire bytes are not host token savings. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. +**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; the compact ten-tool stdio list is 18,545 bytes, discovery remains 43/43, and brief response budgets preserve required failure/applicability signals. The offline installed-child smoke passes. Five interactive LM Studio task records on `google/gemma-4-e2b` show successful web and security calls and one paired web task with a 44.2-point lower context-meter reading under compact; research overran the 10,703-token context and misclassified a secondary source, and direct fetch produced only one of two requested concrete changes. M004 is active to expose accepted research goal/domain values in compact schemas and requalify those calls. No larger-model comparison or exact tokenizer/token attribution exists. M003 remains blocked on corrective quality evidence, broader paired comparison, exact token and host behavior evidence, and protected API qualification (the original MCP config has been restored; no auth or permission setting was changed). Offline wire bytes and this single UI-meter pair are not general token savings. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. **Tool-surface gate:** M001-M007 closed. Do not run evaluation against pre-consolidation bytes and present it as consolidation diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index fbcd606..cae9dc2 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001-M002 closed, M003 blocked on live LM Studio consumer qualification) +Status: active (M001-M002 closed; M003 blocked; M004 active for compact research-schema discoverability) Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. @@ -54,6 +54,7 @@ No CodeGG dependency; it is a regression consumer, not the implementation target | M001 | capability | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Opt-in compact `tools/list` for all ten existing names; measurable schema reduction; parity across MCP transports; default unchanged | | M002 | capability | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Explicit brief projection and bounded results with critical failure, identity, trust and security semantics intact; unchanged legacy defaults | | M003 | capability | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Verified LM Studio install/smoke, measured local-model context cost and completion quality, onboarding guides, and stop/rollback evidence | +| M004 | capability | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Advertise runtime-accepted research goal/domain values in compact schemas; test schema parity and size; repeat focused live tasks; M003 remains open for its broader gates | ## Cross-cutting controls diff --git a/src/mcp/server.rs b/src/mcp/server.rs index ae27ebb..8d173ab 100644 --- a/src/mcp/server.rs +++ b/src/mcp/server.rs @@ -200,11 +200,98 @@ fn compact_schema(name: &str, schema: &mut serde_json::Value) { _ => COMMON, }; compact_schema_value(schema, allowed); + compact_research_schema(name, schema); if name == "build_evidence_bundle" { compact_evidence_bundle_schema(schema); } } +fn compact_research_schema(name: &str, schema: &mut serde_json::Value) { + if name != "research_search" { + return; + } + let Some(properties) = schema + .get_mut("properties") + .and_then(serde_json::Value::as_object_mut) + else { + return; + }; + if let Some(domain) = properties.get_mut("research_domain") { + set_string_schema_enum( + domain, + &[ + "general", + "software_architecture", + "architecture", + "api_design", + "api", + "distributed_systems", + "distributed", + "security", + "performance", + "language_ecosystem", + "ecosystem", + "machine_learning", + "ml", + "infrastructure", + "infra", + ], + ); + } + if let Some(goal) = properties.get_mut("goal") { + set_string_schema_enum(goal, crate::mcp::tools::canonical::RESEARCH_GOAL_VALUES); + } +} + +fn set_string_schema_enum(schema: &mut serde_json::Value, values: &[&str]) { + match schema { + serde_json::Value::Object(object) => { + match object.get("type") { + Some(serde_json::Value::String(kind)) if kind == "string" => { + object.insert( + "enum".into(), + serde_json::Value::Array( + values + .iter() + .map(|value| serde_json::Value::String((*value).into())) + .collect(), + ), + ); + } + Some(serde_json::Value::Array(kinds)) + if kinds.iter().any(|kind| kind.as_str() == Some("string")) => + { + let mut allowed: Vec = values + .iter() + .map(|value| serde_json::Value::String((*value).into())) + .collect(); + if kinds.iter().any(serde_json::Value::is_null) { + allowed.push(serde_json::Value::Null); + } + object.insert("enum".into(), serde_json::Value::Array(allowed)); + } + _ => {} + } + for key in ["anyOf", "oneOf", "allOf"] { + if let Some(variants) = object + .get_mut(key) + .and_then(serde_json::Value::as_array_mut) + { + for variant in variants { + set_string_schema_enum(variant, values); + } + } + } + } + serde_json::Value::Array(variants) => { + for value in variants { + set_string_schema_enum(value, values); + } + } + _ => {} + } +} + fn compact_evidence_bundle_schema(schema: &mut serde_json::Value) { let Some(properties) = schema .get_mut("properties") diff --git a/tests/mcp_tool_contract.rs b/tests/mcp_tool_contract.rs index 78816bd..ac90f94 100644 --- a/tests/mcp_tool_contract.rs +++ b/tests/mcp_tool_contract.rs @@ -502,4 +502,94 @@ fn compact_profile_advertises_all_tools_with_smaller_stable_contract() { Some(&serde_json::Value::Bool(true)) ); } + + fn schema_enum_strings(schema: &serde_json::Value) -> Vec { + let mut values = Vec::new(); + match schema { + serde_json::Value::Object(object) => { + if let Some(enumeration) = object.get("enum").and_then(serde_json::Value::as_array) + { + values.extend( + enumeration + .iter() + .filter_map(serde_json::Value::as_str) + .map(str::to_owned), + ); + } + for value in object.values() { + values.extend(schema_enum_strings(value)); + } + } + serde_json::Value::Array(items) => { + for value in items { + values.extend(schema_enum_strings(value)); + } + } + _ => {} + } + values.sort(); + values.dedup(); + values + } + + let research = compact_tools + .iter() + .find(|tool| tool.name.as_ref() == "research_search") + .unwrap(); + let properties = research.input_schema.get("properties").unwrap(); + let goal_values = schema_enum_strings(&properties["goal"]); + let mut expected_goal_values = [ + "general", + "architecture_decision", + "api_evaluation", + "library_comparison", + "migration_planning", + "security_review", + "performance_investigation", + "ecosystem_survey", + "understand", + "debug", + "compare", + "pre_change", + "post_change", + ] + .map(str::to_owned) + .to_vec(); + expected_goal_values.sort(); + assert_eq!(goal_values, expected_goal_values); + for goal in &goal_values { + assert!( + eggsearch::mcp::tools::canonical::parse_research_goal(goal).is_some(), + "compact schema goal must be accepted by runtime parser: {goal}" + ); + } + + let domain_values = schema_enum_strings(&properties["research_domain"]); + let mut expected_domain_values = [ + "general", + "software_architecture", + "architecture", + "api_design", + "api", + "distributed_systems", + "distributed", + "security", + "performance", + "language_ecosystem", + "ecosystem", + "machine_learning", + "ml", + "infrastructure", + "infra", + ] + .map(str::to_owned) + .to_vec(); + expected_domain_values.sort(); + assert_eq!(domain_values, expected_domain_values); + for domain in &domain_values { + assert!( + eggsearch::core::ResearchDomain::parse(domain).is_some(), + "compact schema domain must be accepted by runtime parser: {domain}" + ); + } } From 1608fe904408c29d3bfe2d3ce35d4ccc79f683fa Mon Sep 17 00:00:00 2001 From: eggsearch Date: Fri, 30 Oct 2026 21:40:58 +0000 Subject: [PATCH 07/25] docs(plans): record LM Studio corrective qualification status --- .../003-status.md | 3 +- .../004-status.md | 71 +++++++++++++++++++ ...mpact-schema-discoverability-corrective.md | 2 +- plans/registry.md | 6 +- .../standalone-mcp-local-inference-roadmap.md | 2 +- 5 files changed, 78 insertions(+), 6 deletions(-) create mode 100644 plans/closure/standalone-mcp-local-inference/004-status.md diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index e923266..a83dca5 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -104,7 +104,8 @@ qualification. credentials were created and no security-sensitive setting was changed; complete that prerequisite in LM Studio, then run the API smoke and restore prior settings. -5. M004 is implemented in source to expose accepted research goal/domain +5. M004 is implemented in source at candidate + `776f8ca4a8246e467041696b2220600eb7b2648d` to expose accepted research goal/domain values in compact schemas. Focused contract tests, all-features compile, release build, and installed-child smoke pass. The release binary advertised all ten tools at 18,990 compact schema bytes. A post-fix Gemma turn was not diff --git a/plans/closure/standalone-mcp-local-inference/004-status.md b/plans/closure/standalone-mcp-local-inference/004-status.md new file mode 100644 index 0000000..cb2e3dd --- /dev/null +++ b/plans/closure/standalone-mcp-local-inference/004-status.md @@ -0,0 +1,71 @@ +# Standalone MCP Local-Inference M004 — Corrective Status + +Status: blocked (schema correction and offline qualification pass; post-fix LM Studio interaction was not reliably refreshed) + +Source implementation plan: +`plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` + +Source subsystem roadmap: +`plans/subsystems/standalone-mcp-local-inference-roadmap.md` + +Corrected implementation candidate: `776f8ca4a8246e467041696b2220600eb7b2648d`. +Original finding: M003 `02a8af6fe4f87971c7daca7d1fca20c087b328ec`, closure +`plans/closure/standalone-mcp-local-inference/003-status.md`. + +## 1. Change and evidence + +The compact `research_search` schema now includes finite enum values for +`goal` and `research_domain`, including supported aliases. Goal values come +from the runtime canonical list; domain aliases are enumerated beside the +existing parser contract. Full-profile schema generation and runtime +validation are unchanged. + +The schema contract test verifies that all advertised values are present and +accepted by the runtime parsers. It also retains the ten-tool, deterministic +fingerprint, and compact-size checks. The optimized all-features release +candidate passed `scripts/lm_studio_stdio_smoke.py`: + +| Profile | Tool count | Serialized `tools/list` bytes | `provider_status` | Exit | +|---|---:|---:|---|---:| +| Full | 10 | 72,923 | pass | 0 | +| Compact | 10 | 18,990 | pass | 0 | + +Compact remains below the 36,000-byte ceiling; the enum additions increase the +measured list by 445 bytes over M003's 18,545-byte candidate. + +## 2. Verification + +- `cargo fmt --check` — pass. +- `cargo test --locked --all-features --test mcp_tool_contract` — 15 passed. +- `cargo test --locked --all-features --test static_guards` — 60 passed. +- `cargo check --locked --all-features --all-targets` — pass. +- `cargo build --locked --release --all-features` — pass. +- `scripts/lm_studio_stdio_smoke.py target/release/eggsearch` — full and + compact profiles pass, ten tools each, `provider_status` succeeds. +- `make docs-check` — pass. +- `make check` — started but manually interrupted after the + `bounded_command` test binary stalled. A standalone rerun showed four + existing process-group/cap-breach tests running over 60 seconds: + `test_bounded_command_inventory_cap_breach_terminates`, + `test_cap_breach_returns_before_timeout`, + `test_inventory_fallback_on_stdout_cap_reason`, and + `test_no_zombie_after_cap_breach`. No assertion failure was reported before + interruption; the canonical gate is not green on this host. + +## 3. Live qualification and disposition + +The LM Studio `mcp.json` on disk was temporarily pointed at the corrected +release candidate and compact profile, then restored byte-for-byte to the +original installed-binary configuration. The app's accessibility tree +continued to report a Chat route while its visible window remained on Model +Server; save/refresh actions could not be verified. No post-fix model turn is +claimed, and no API authentication or MCP permission setting was changed. + +M004 therefore remains blocked on a fresh Gemma E2B research-schema and +official-page fetch task against this exact candidate, with LM Studio refresh +and rollback visibly verified. M003 remains blocked on broader paired model +trials, exact tokenizer and prompt-token attribution, structured-content and +refresh behavior, protected API qualification, and resolution of the +observed quality issues. Do not claim the enum correction fixes Gemma's +research quality or the incomplete fetch answer until a new live run confirms +it. diff --git a/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md b/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md index dce72e5..fb2b188 100644 --- a/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md +++ b/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md @@ -1,6 +1,6 @@ # M004 — Compact Schema Discoverability Corrective -Status: active (registered after live M003 qualification found research argument retries) +Status: blocked (schema correction and offline qualification pass; post-fix LM Studio interaction was not reliably refreshed; see `plans/closure/standalone-mcp-local-inference/004-status.md`) Primary class: capability Source roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` Corrects: M003 evidence in `plans/closure/standalone-mcp-local-inference/003-status.md` diff --git a/plans/registry.md b/plans/registry.md index 8f4180a..2ad4446 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001-M002 closed; M003 blocked; M004 active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M003 interactive Gemma evidence recorded; M004 addresses compact research-schema discoverability | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`; M004 implementation `004-compact-schema-discoverability-corrective.md`. | +| Standalone MCP local-inference efficiency | active (M001-M002 closed; M003 and M004 blocked on live LM Studio qualification) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M003 interactive Gemma evidence recorded; M004 schema correction candidate `776f8ca4a8246e467041696b2220600eb7b2648d` passed offline checks but needs a fresh live model turn | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`; M004 status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -55,7 +55,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | | Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | M001/M002 closed; offline smoke and five interactive Gemma task records are in `plans/closure/standalone-mcp-local-inference/003-status.md`. One paired web task reduced the UI context meter, but research/fetch quality failures remain; larger-model, exact token attribution, structured-content/refresh, and protected API checks remain open. | -| Standalone MCP local-inference | M004 compact schema discoverability corrective | active | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Corrects M003's invalid research goal/domain argument retries by advertising runtime-accepted values; focused offline and live requalification required. | +| Standalone MCP local-inference | M004 compact schema discoverability corrective | blocked | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Candidate `776f8ca4a8246e467041696b2220600eb7b2648d`; offline checks and installed-child smoke pass. Need a fresh Gemma live turn against this binary; LM Studio app refresh could not be verified. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | @@ -126,7 +126,7 @@ Corrective closure sequence: `plans/implementation/dependency-evidence-hardening **Python native library gate:** ADR-0005 authorizes a narrow direct Rust SDK facade and Python PyO3 native package without broad Rust module stability. M001–M008 are closed at `067468a` with fully-green qualifying run `38641009246` (5/5 builds, 20/20 smokes, offline sdist rebuild, green audit, digests in `008-status.md`). Earlier runs are historical/deficiency/supporting evidence only. First PyPI publication is separately closed under `python-pypi-distribution` M003 at 0.4.4; Eggpack binary assets remain unchanged. -**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; the compact ten-tool stdio list is 18,545 bytes, discovery remains 43/43, and brief response budgets preserve required failure/applicability signals. The offline installed-child smoke passes. Five interactive LM Studio task records on `google/gemma-4-e2b` show successful web and security calls and one paired web task with a 44.2-point lower context-meter reading under compact; research overran the 10,703-token context and misclassified a secondary source, and direct fetch produced only one of two requested concrete changes. M004 is active to expose accepted research goal/domain values in compact schemas and requalify those calls. No larger-model comparison or exact tokenizer/token attribution exists. M003 remains blocked on corrective quality evidence, broader paired comparison, exact token and host behavior evidence, and protected API qualification (the original MCP config has been restored; no auth or permission setting was changed). Offline wire bytes and this single UI-meter pair are not general token savings. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. +**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; the compact ten-tool stdio list is 18,545 bytes, discovery remains 43/43, and brief response budgets preserve required failure/applicability signals. The offline installed-child smoke passes. Five interactive LM Studio task records on `google/gemma-4-e2b` show successful web and security calls and one paired web task with a 44.2-point lower context-meter reading under compact; research overran the 10,703-token context and misclassified a secondary source, and direct fetch produced only one of two requested concrete changes. M004 added compact research goal/domain enums and passes focused offline checks at `776f8ca4a8246e467041696b2220600eb7b2648d`; its post-fix live turn remains unverified because LM Studio refresh could not be confirmed. No larger-model comparison or exact tokenizer/token attribution exists. M003 and M004 remain blocked on live corrective quality evidence, broader paired comparison, exact token and host behavior evidence, and protected API qualification (the original MCP config has been restored; no auth or permission setting was changed). Offline wire bytes and this single UI-meter pair are not general token savings. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. **Tool-surface gate:** M001-M007 closed. Do not run evaluation against pre-consolidation bytes and present it as consolidation diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index cae9dc2..9ef15fc 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001-M002 closed; M003 blocked; M004 active for compact research-schema discoverability) +Status: active (M001-M002 closed; M003 and corrective M004 blocked on remaining live LM Studio qualification) Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. From 54c0a1d5b4a7cd1d983bcbdede839826ad791193 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Fri, 30 Oct 2026 22:16:56 +0000 Subject: [PATCH 08/25] docs(plans): record M004 live requalification --- .../003-status.md | 21 ++++---- .../004-status.md | 48 ++++++++++++------- ...mpact-schema-discoverability-corrective.md | 2 +- plans/registry.md | 4 +- .../standalone-mcp-local-inference-roadmap.md | 2 +- 5 files changed, 45 insertions(+), 32 deletions(-) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index a83dca5..9a84a73 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -104,18 +104,15 @@ qualification. credentials were created and no security-sensitive setting was changed; complete that prerequisite in LM Studio, then run the API smoke and restore prior settings. -5. M004 is implemented in source at candidate - `776f8ca4a8246e467041696b2220600eb7b2648d` to expose accepted research goal/domain - values in compact schemas. Focused contract tests, all-features compile, - release build, and installed-child smoke pass. The release binary advertised - all ten tools at 18,990 compact schema bytes. A post-fix Gemma turn was not - completed: LM Studio's exposed accessibility state and visible window - diverged during integration refresh, so results from the previous child - were not attributed to the new candidate. See - `plans/closure/standalone-mcp-local-inference/004-status.md`. -6. Run the focused live tasks against the corrected exact candidate, then - update this status with the result; close M003 only after its full live - acceptance criteria pass. +5. M004 candidate `776f8ca4a8246e467041696b2220600eb7b2648d` exposes the + runtime-accepted research goal/domain values in compact schemas. Its + focused Gemma requalification and remaining fetch-extraction failure are + recorded in `plans/closure/standalone-mcp-local-inference/004-status.md`. + The canonical `make check` gate remains stalled in the macOS + `bounded_command` suite, so M004 is still blocked. M003 also remains blocked + on its broader operational acceptance matrix. +6. Complete M003's broader operational acceptance matrix, then update this + status; close M003 only after its full live acceptance criteria pass. ## 6. Security and operational disposition diff --git a/plans/closure/standalone-mcp-local-inference/004-status.md b/plans/closure/standalone-mcp-local-inference/004-status.md index cb2e3dd..f8e2e8a 100644 --- a/plans/closure/standalone-mcp-local-inference/004-status.md +++ b/plans/closure/standalone-mcp-local-inference/004-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M004 — Corrective Status -Status: blocked (schema correction and offline qualification pass; post-fix LM Studio interaction was not reliably refreshed) +Status: blocked (schema correction, focused live requalification, and installed-child smoke pass; canonical `make check` stalls in `bounded_command` on this host) Source implementation plan: `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` @@ -54,18 +54,34 @@ measured list by 445 bytes over M003's 18,545-byte candidate. ## 3. Live qualification and disposition -The LM Studio `mcp.json` on disk was temporarily pointed at the corrected -release candidate and compact profile, then restored byte-for-byte to the -original installed-binary configuration. The app's accessibility tree -continued to report a Chat route while its visible window remained on Model -Server; save/refresh actions could not be verified. No post-fix model turn is -claimed, and no API authentication or MCP permission setting was changed. - -M004 therefore remains blocked on a fresh Gemma E2B research-schema and -official-page fetch task against this exact candidate, with LM Studio refresh -and rollback visibly verified. M003 remains blocked on broader paired model -trials, exact tokenizer and prompt-token attribution, structured-content and -refresh behavior, protected API qualification, and resolution of the -observed quality issues. Do not claim the enum correction fixes Gemma's -research quality or the incomplete fetch answer until a new live run confirms -it. +The LM Studio `mcp.json` was temporarily pointed at the corrected +`target/release/eggsearch` binary with the compact profile and then restored +byte-for-byte to the original installed-binary configuration (`cmp` passed). +The model was `google/gemma-4-e2b` in LM Studio 0.4.26+4, MLX 4-bit, with a +10,703-token context. Both focused prompts were run in fresh chats. Tool +approval was per-call; neither persistent “Always allow” option was selected. +No API authentication or MCP permission setting was changed. + +| Task | Tool behavior | Result quality | Context / disposition | +|---|---|---|---| +| Research: Rust 1.90.0 changes, `goal=understand`, `research_domain=language_ecosystem` | First prompt was treated as underspecified and did not call a tool. On an explicit-query retry, Gemma called `research_search` once with `query="Rust 1.90.0 release notes new features"`, `goal="understand"`, and `research_domain="language_ecosystem"`; values were valid with no argument repair. | Returned two changes and named official release-note pages, but citations were page-level and not deep links. | Final chat context meter was approximately 49.6% (5,314/10,703). The enum correction supported a valid call once the query was explicit; it did not address prompt ambiguity or establish general research quality. | +| Fetch official Rust 1.90.0 announcement and report two changes | Gemma called `web_fetch` once with the requested URL and valid arguments. | Fetch succeeded, but the visible fetched payload contained only “LLD is now the default.” Gemma correctly identified that item, then incorrectly used `$ rustup update stable` as its second “change”; that is an installation command, not a release change. | Final chat context meter was approximately 31.4% (3,359/10,703). The focused extraction criterion failed and confirms the previously recorded M003 quality issue. | + +The live schema-discoverability task and compact fetch task are therefore +requalified on the exact implementation candidate, and the original MCP +configuration rollback is verified. The canonical gate was rerun with +`rtk make check` on branch HEAD `1608fe904408c29d3bfe2d3ce35d4ccc79f683fa` +(whose production code is unchanged from candidate +`776f8ca4a8246e467041696b2220600eb7b2648d`). It reached +`tests/bounded_command.rs` and remained there without test output for over two +minutes, then was manually interrupted (exit 130). No assertion failure was +reported. The same host-specific stall was recorded in the original M004 +qualification. The canonical gate is not green, so M004 remains blocked; +rerun it on a host where the process-group/cap-breach tests complete before +closing this milestone. + +M003 remains blocked on broader paired model trials, exact tokenizer and +prompt-token attribution, structured-content and refresh behavior, protected +API qualification, and resolution/explicit acceptance of its observed +quality issues. Do not claim that the enum correction fixes prompt ambiguity, +general research quality, or incomplete fetch extraction. diff --git a/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md b/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md index fb2b188..e97cce0 100644 --- a/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md +++ b/plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md @@ -1,6 +1,6 @@ # M004 — Compact Schema Discoverability Corrective -Status: blocked (schema correction and offline qualification pass; post-fix LM Studio interaction was not reliably refreshed; see `plans/closure/standalone-mcp-local-inference/004-status.md`) +Status: blocked (schema correction, focused LM Studio requalification, and installed-child smoke pass; canonical `make check` stalls in the existing macOS `bounded_command` suite; see `plans/closure/standalone-mcp-local-inference/004-status.md`) Primary class: capability Source roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` Corrects: M003 evidence in `plans/closure/standalone-mcp-local-inference/003-status.md` diff --git a/plans/registry.md b/plans/registry.md index 2ad4446..feebbf8 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001-M002 closed; M003 and M004 blocked on live LM Studio qualification) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M003 interactive Gemma evidence recorded; M004 schema correction candidate `776f8ca4a8246e467041696b2220600eb7b2648d` passed offline checks but needs a fresh live model turn | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`; M004 status `plans/closure/standalone-mcp-local-inference/004-status.md`. | +| Standalone MCP local-inference efficiency | active (M001-M002 closed; M003 and M004 blocked on remaining acceptance) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M003 broader Gemma acceptance remains open; M004 candidate `776f8ca4a8246e467041696b2220600eb7b2648d` passed focused schema call and smoke, but compact fetch extraction failed and canonical `make check` stalled in `bounded_command` | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`; M004 status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -55,7 +55,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | | Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | M001/M002 closed; offline smoke and five interactive Gemma task records are in `plans/closure/standalone-mcp-local-inference/003-status.md`. One paired web task reduced the UI context meter, but research/fetch quality failures remain; larger-model, exact token attribution, structured-content/refresh, and protected API checks remain open. | -| Standalone MCP local-inference | M004 compact schema discoverability corrective | blocked | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Candidate `776f8ca4a8246e467041696b2220600eb7b2648d`; offline checks and installed-child smoke pass. Need a fresh Gemma live turn against this binary; LM Studio app refresh could not be verified. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | +| Standalone MCP local-inference | M004 compact schema discoverability corrective | blocked | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Candidate `776f8ca4a8246e467041696b2220600eb7b2648d`; offline checks, installed-child smoke, and focused Gemma schema call pass. Compact fetch extraction still misreports an install command as a change; canonical `make check` stalls in `bounded_command`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index 9ef15fc..7a5bf43 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001-M002 closed; M003 and corrective M004 blocked on remaining live LM Studio qualification) +Status: active (M001-M002 closed; M003 blocked on broader LM Studio acceptance; M004 blocked on canonical gate completion after focused live requalification) Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. From 7082d78b4da127b817df134e5827977d3fa57003 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Fri, 30 Oct 2026 22:19:12 +0000 Subject: [PATCH 09/25] docs(plans): record serial gate stall --- .../closure/standalone-mcp-local-inference/004-status.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/plans/closure/standalone-mcp-local-inference/004-status.md b/plans/closure/standalone-mcp-local-inference/004-status.md index f8e2e8a..52f2bfa 100644 --- a/plans/closure/standalone-mcp-local-inference/004-status.md +++ b/plans/closure/standalone-mcp-local-inference/004-status.md @@ -76,9 +76,12 @@ configuration rollback is verified. The canonical gate was rerun with `tests/bounded_command.rs` and remained there without test output for over two minutes, then was manually interrupted (exit 130). No assertion failure was reported. The same host-specific stall was recorded in the original M004 -qualification. The canonical gate is not green, so M004 remains blocked; -rerun it on a host where the process-group/cap-breach tests complete before -closing this milestone. +qualification. A serial retry, +`rtk cargo test --locked --all-features --test bounded_command -- --test-threads=1`, +also remained silent for over two minutes and was interrupted (exit 130), so +test parallelism did not resolve the stall. The canonical gate is not green, +so M004 remains blocked; rerun it on a host where the process-group/cap-breach +tests complete before closing this milestone. M003 remains blocked on broader paired model trials, exact tokenizer and prompt-token attribution, structured-content and refresh behavior, protected From 755008c0a1da658835dd8f215d3b82bd1cb3dee5 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Fri, 30 Oct 2026 22:53:49 +0000 Subject: [PATCH 10/25] fix(mcp): preserve brief document text within budget --- .github/workflows/ci.yml | 17 +++- architecture/mcp.md | 2 +- docs/test-inventory.md | 6 +- ...t-extraction-and-paired-requalification.md | 97 +++++++++++++++++++ plans/registry.md | 3 +- .../standalone-mcp-local-inference-roadmap.md | 10 +- src/mcp/projection.rs | 6 +- tests/mcp_projection.rs | 41 ++++++++ 8 files changed, 171 insertions(+), 11 deletions(-) create mode 100644 plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8e846c7..f1016c6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -5,6 +5,7 @@ on: branches: [main] pull_request: branches: [main] + workflow_dispatch: permissions: contents: read @@ -51,13 +52,27 @@ jobs: - name: Check all features on Windows target run: cargo check --locked --all-features --target x86_64-pc-windows-msvc macos-portability: - name: macOS local filesystem portability + name: macOS canonical verification and local filesystem portability runs-on: macos-15 steps: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - uses: dtolnay/rust-toolchain@6bed0761d98439e5a578e2877258200ad565ba87 # stable snapshot 2026-09-28 with: toolchain: "1.89" + components: rustfmt, clippy + - name: Run canonical verification + run: make ci + env: + GITHUB_TOKEN: "" + GH_TOKEN: "" + GITLAB_TOKEN: "" + CODEBERG_TOKEN: "" + GITEA_TOKEN: "" + FORGEJO_TOKEN: "" + SOURCEGRAPH_API_KEY: "" + BRAVE_API_KEY: "" + SEMANTIC_SCHOLAR_API_KEY: "" + NVD_API_KEY: "" - name: Test unsupported symlink following behavior run: cargo test --locked --no-default-features --test property_local_fs_extended -- safe_open_follow_symlink_is_unsupported - name: Test host-scoped workspace root canonicalization diff --git a/architecture/mcp.md b/architecture/mcp.md index 4f9c694..1a573ce 100644 --- a/architecture/mcp.md +++ b/architecture/mcp.md @@ -260,7 +260,7 @@ Tool behavior modules carry unit coverage in-file plus the shared `tools/tests.r - `diagnostic` — full observability payload, passthrough with no added keys. - `standard` — specialist research/security default surface. Keeps full `retrieval_summary`, `conflict_metadata`, `workflow_coverage`, `evidence_role_summary`, `capability_enforcement`, and fetch/cache metadata; summarizes only `routing_decision` (→ `routing_summary`) and `telemetry.routing_decision`. - `compact` — ordinary agent operation. Trims excerpts to 1 per card, replaces `routing_decision` with `routing_summary`, replaces `retrieval_summary`/`conflict_metadata` with `retrieval_status`/`conflict_indicator`, drops `workflow_coverage`/`telemetry`/`document`/`links`, strips per-vulnerability `references`/`details`, caps `repo_map` entries at 50. Every projection stamps `response_detail` plus an explicit truncation marker (`projection_excerpts_trimmed`, `projection_entries_truncated`, `links_omitted_in_compact`). -- `brief` — applies compact safety reductions, then retains identity, status, trust, warnings, applicability and citation fields using explicit allowlists. Content strings are clipped at UTF-8 character boundaries; if the configured `[mcp].brief_max_chars` serialized-character budget still cannot fit, complete result records are omitted from the tail with `returned_count`, `omitted_count`, and `output_truncated`. Oversized applicability lists use an `applicability_summary` with affected, not-affected, unknown, total and omitted counts; provider failures expose `providers_failed_summary` when records are omitted. Omitted batch failure records retain their indexes in `failed_member_indices`. Text plus overlapping focus chunks are avoided through the existing compact fetch projection. The structured result and text fallback are both brief; callers request `diagnostic` for the full diagnostic payload. +- `brief` — applies compact safety reductions, then retains identity, status, trust, warnings, applicability and citation fields using explicit allowlists. Content strings remain intact when the serialized response fits the configured `[mcp].brief_max_chars` budget; only oversized responses progressively clip strings at UTF-8 character boundaries before complete result records are omitted from the tail with `returned_count`, `omitted_count`, and `output_truncated`. Oversized applicability lists use an `applicability_summary` with affected, not-affected, unknown, total and omitted counts; provider failures expose `providers_failed_summary` when records are omitted. Omitted batch failure records retain their indexes in `failed_member_indices`. Text plus overlapping focus chunks are avoided through the existing compact fetch projection. The structured result and text fallback are both brief; callers request `diagnostic` for the full diagnostic payload. The suite measures diagnostic/standard/compact/brief serialized characters across the 15-case retrieval-quality corpus and asserts that brief never erases failure-vs-absence state, trust/injection markers, security applicability, PDF coordinates, truncation evidence, failed batch members or stable IDs. The sparse corpus report is descriptive; a separate nontrivial fixture enforces the 30% byte reduction target. diff --git a/docs/test-inventory.md b/docs/test-inventory.md index 9bcc91d..e5aa847 100644 --- a/docs/test-inventory.md +++ b/docs/test-inventory.md @@ -6,8 +6,8 @@ Inventory of all hardening and regression test suites. | Feature Combo | Tests | Ignored | |--------------|-------|---------| -| `--all-features` | 5693 | 24 | -| `--features mock` | 5422 | 2 | +| `--all-features` | 5694 | 24 | +| `--features mock` | 5423 | 2 | Ignored tests are live-network smoke tests (`corpus_runner`, `browser_live_smoke`, `native_forge_smoke`, `retrieval_quality`) plus the opt-in live-model comparison (`tool_surface_live`) — they run only via explicit opt-in targets. @@ -176,7 +176,7 @@ Source of truth: `fuzz/Cargo.toml` [[bin]] entries. | `mcp_tool_contract` | 15 | Canonical registry parity, sanitization, deterministic fingerprints, and compact-profile size/all-name contract | | `mcp_schema_slimming` | 17 | Slimmed ordinary schema size budget with legacy-field acceptance | | `mcp_2026_protocol` | 14 | Structured results, output schemas, repairable error contract | -| `mcp_projection` | 21 | Brief/compact/standard/diagnostic projections: status, trust, security applicability summary and budget retention, provider and batch failure summaries, bounded failure details, repository/research reduction, character budgets, UTF-8 framing, 15-task byte/retention trace, bundle identity | +| `mcp_projection` | 22 | Brief/compact/standard/diagnostic projections: status, trust, security applicability summary and budget retention, provider and batch failure summaries, bounded failure details, repository/research reduction, character budgets, UTF-8 framing, intact over-800-character document extraction within budget, 15-task byte/retention trace, bundle identity | ## Tool-Surface Evaluation (2 suites) diff --git a/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md b/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md new file mode 100644 index 0000000..53956dc --- /dev/null +++ b/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md @@ -0,0 +1,97 @@ +# M005 — Brief Document Extraction and Paired Requalification + +Status: active + +Source subsystem roadmap: +`plans/subsystems/standalone-mcp-local-inference-roadmap.md` + +Corrects M002's brief projection after M003/M004 live qualification found that +the unconditional 800-character per-string clip hid most fetched document text. +The existing M002 closure remains historical and unchanged. This plan also +records the paired M003 acceptance and finishes M004's hosted canonical gate. + +## Objective + +Preserve all fetched document text that fits the configured brief response +budget, and requalify diagnostic versus brief responses using the same query +and same extracted document. Complete paired interactive LM Studio tasks on +`google/gemma-4-e2b` and the larger installed `prism-ml/bonsai-27b`, measuring +tool-call validity, evidence completeness, failure interpretation, and actual +token use wherever the host exposes counters. Run the canonical `make ci` gate +on hosted Linux and macOS for the final candidate. + +## Explicit non-goals + +- Reopen or amend the accepted M001 or M002 closure records. +- Change retrieval, extraction, provider, trust, or stable-ID semantics. +- Claim exact prompt/completion token attribution from a rounded context meter + or serialized byte counts. +- Enable LM Studio API authentication or permission settings. +- Add network-dependent model qualification to routine CI. + +## Current evidence and defect + +`src/mcp/projection.rs::project_brief` currently clips every `text`, `snippet`, +`message`, and `error` value to 800 characters before checking the configured +8,000-character aggregate budget. A fetched document can therefore be marked +untruncated by retrieval while brief projection silently drops most of its +extract. M003 and M004 live records already contain the same Rust 1.90.0 +announcement query and document URL; use those as the paired qualification +case and add a deterministic regression fixture. + +## Work packages + +1. Change the brief reducer to preserve content while the serialized response + fits its configured limit, and reduce field text progressively only when + the response exceeds that limit. Keep truncation and omission indicators + accurate. +2. Add a deterministic regression that projects one document/query into both + diagnostic and brief, proves the text is identical when under budget, and + proves brief text exceeds 800 characters without setting a truncation flag. +3. Run the same Rust 1.90.0 extraction query on the same official URL with + diagnostic and brief output, on Gemma E2B and Bonsai 27B. Also repeat the + research/schema and provider-failure interpretation tasks on both models. + Record each tool call, argument validity, repairs, cited evidence, + completeness, failure interpretation, latency, and host token/context data. +4. Keep full/diagnostic and compact/brief MCP configs isolated per run and + restore the original LM Studio `mcp.json` byte-for-byte. +5. Enable manual workflow dispatch for the CI workflow and run the canonical + `make ci` suite on hosted Linux and macOS. Record whether the prior local + process-group stall reproduces. Do not treat a timeout as a passing gate. +6. Update M003/M004 statuses only with exact candidate evidence, this plan's + closure, the roadmap, and the registry in the same closure commit. + +## Verification + +```bash +cargo fmt --check +cargo test --locked --all-features --test mcp_projection +cargo test --locked --all-features --test mcp_tool_contract +cargo build --locked --release --all-features +python3 scripts/lm_studio_stdio_smoke.py target/release/eggsearch +make ci +``` + +Hosted macOS/Linux `make ci` runs are required for M004. Interactive LM Studio +qualification is operational and must not be added to routine CI. + +## Acceptance + +- A document longer than 800 characters remains unchanged in brief output when + the response fits its configured budget. +- Diagnostic and brief comparisons use the identical query and fetched + document; all projection truncation is explicitly surfaced. +- Both models complete paired tasks with evidence and failures accurately + interpreted, or each remaining deficiency is explicitly recorded and M003 + remains open. +- Exact model token counters are recorded only if LM Studio exposes them; UI + context percentage, context capacity, latency, and serialized bytes are + reported as distinct measurements. +- Hosted Linux and macOS canonical runs are green on the final implementation + SHA for M004 closure. +- The original LM Studio MCP config is restored and verified byte-for-byte. + +Closure evidence belongs in +`plans/closure/standalone-mcp-local-inference/005-status.md` and must link the +exact source SHA, hosted runs, LM Studio version/config, task records, test +outcomes, restored config check, and any unresolved M003 criterion. diff --git a/plans/registry.md b/plans/registry.md index feebbf8..0361a36 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001-M002 closed; M003 and M004 blocked on remaining acceptance) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M003 broader Gemma acceptance remains open; M004 candidate `776f8ca4a8246e467041696b2220600eb7b2648d` passed focused schema call and smoke, but compact fetch extraction failed and canonical `make check` stalled in `bounded_command` | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`; M004 status `plans/closure/standalone-mcp-local-inference/004-status.md`. | +| Standalone MCP local-inference efficiency | active (M001-M002 closed; M003-M004 acceptance open; M005 corrective active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M003 paired model acceptance remains open; M004 hosted canonical gate pending; M005 fixes brief extraction clipping and records exact-pair qualification | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`; M004 status `plans/closure/standalone-mcp-local-inference/004-status.md`; M005 is registered below. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -56,6 +56,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | | Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | M001/M002 closed; offline smoke and five interactive Gemma task records are in `plans/closure/standalone-mcp-local-inference/003-status.md`. One paired web task reduced the UI context meter, but research/fetch quality failures remain; larger-model, exact token attribution, structured-content/refresh, and protected API checks remain open. | | Standalone MCP local-inference | M004 compact schema discoverability corrective | blocked | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Candidate `776f8ca4a8246e467041696b2220600eb7b2648d`; offline checks, installed-child smoke, and focused Gemma schema call pass. Compact fetch extraction still misreports an install command as a change; canonical `make check` stalls in `bounded_command`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | +| Standalone MCP local-inference | M005 brief document extraction and paired requalification | active | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Corrective to M002, with M003 and M004 operational gates; requalify the same Rust 1.90 announcement query/document across diagnostic and brief, Gemma E2B and `prism-ml/bonsai-27b`. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index 7a5bf43..2624bbf 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -40,11 +40,14 @@ Avoid an unnecessary additional server-side dynamic discovery tool: direct MCP h ## Dependencies and sequence ```text -M001 compact ten-tool advertised schema profile (ready) +M001 compact ten-tool advertised schema profile (closed) | - +-- hard --> M002 brief/budgeted agent responses (blocked on M001) + +-- hard --> M002 brief/budgeted agent responses (closed) | - +-- hard --> M003 LM Studio/direct-client compatibility and quality (blocked on M001+M002) + +-- operational --> M003 LM Studio/direct-client compatibility and quality (open acceptance) + +-- corrective --> M005 brief document extraction and paired requalification (active) + +M004 compact schema discoverability corrective (implementation complete; hosted gate pending) ``` No CodeGG dependency; it is a regression consumer, not the implementation target. M003's offline qualification can run in CI; a real LM Studio interactive smoke is an operational closure requirement, not a reason to block M001/M002 development. @@ -55,6 +58,7 @@ No CodeGG dependency; it is a regression consumer, not the implementation target | M002 | capability | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Explicit brief projection and bounded results with critical failure, identity, trust and security semantics intact; unchanged legacy defaults | | M003 | capability | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Verified LM Studio install/smoke, measured local-model context cost and completion quality, onboarding guides, and stop/rollback evidence | | M004 | capability | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Advertise runtime-accepted research goal/domain values in compact schemas; test schema parity and size; repeat focused live tasks; M003 remains open for its broader gates | +| M005 | corrective | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Preserve extracted document text when it fits the brief budget; compare diagnostic and brief on identical queries/documents; complete paired Gemma E2B and larger-model evidence and close M004's hosted Linux/macOS canonical gate | ## Cross-cutting controls diff --git a/src/mcp/projection.rs b/src/mcp/projection.rs index 432f64b..ee14d92 100644 --- a/src/mcp/projection.rs +++ b/src/mcp/projection.rs @@ -122,7 +122,8 @@ fn project_brief(tool: &str, value: serde_json::Value, max_chars: usize) -> serd .map_or(0, Vec::len); let mut clipped = false; let mut omitted = 0usize; - for per_text_limit in [800usize, 320, 128, 48, 0] { + let mut per_text_limit = max_chars; + loop { clipped |= clip_brief_strings(&mut projected, per_text_limit); loop { let size = brief_serialized_chars(&projected); @@ -224,9 +225,10 @@ fn project_brief(tool: &str, value: serde_json::Value, max_chars: usize) -> serd serde_json::Value::Bool(clipped || omitted > 0), ); } - if brief_serialized_chars(&projected) <= max_chars { + if brief_serialized_chars(&projected) <= max_chars || per_text_limit == 0 { break; } + per_text_limit = (per_text_limit / 2).min(per_text_limit.saturating_sub(1)); } for optional in [ "next_actions", diff --git a/tests/mcp_projection.rs b/tests/mcp_projection.rs index b4c5966..a5e4fb1 100644 --- a/tests/mcp_projection.rs +++ b/tests/mcp_projection.rs @@ -263,6 +263,47 @@ fn brief_preserves_security_applicability_states_and_fetch_truncation() { assert_eq!(brief_fetch["focus"]["chunks"][0]["page"], 4); } +#[test] +fn brief_keeps_extracted_document_text_when_it_fits_the_budget() { + let query = "What are two changes in the Rust 1.90.0 announcement?"; + let document_text = format!( + "<<>>\n{}\n<<>>", + (0..40) + .map(|index| format!( + "Section {index}: Rust 1.90.0 includes a documented language and tooling change." + )) + .collect::>() + .join("\n") + ); + let document = json!({ + "query": query, + "url": "https://blog.rust-lang.org/2025/09/18/Rust-1.90.0/", + "final_url": "https://blog.rust-lang.org/2025/09/18/Rust-1.90.0/", + "content_type": "text/html", + "fetched": true, + "truncated": false, + "trust": "external_untrusted", + "text": document_text, + "warnings": [], + "structured_warnings": [] + }); + + let diagnostic = project("web_fetch", document.clone(), ResponseDetail::Diagnostic); + let brief = eggsearch::mcp::projection::project_with_budget( + "web_fetch", + document, + ResponseDetail::Brief, + 8_000, + ); + + assert_eq!(diagnostic["query"], query); + assert_eq!(brief["query"], query); + assert_eq!(diagnostic["text"], brief["text"]); + assert_eq!(brief["truncated"], false); + assert_eq!(brief["output_truncated"], false); + assert!(brief["text"].as_str().unwrap().chars().count() > 800); +} + #[test] fn brief_enforces_character_budget_and_retains_failed_batch_items() { let mut payload = web_payload( From de69f1ddd7a7e4acc067fb9892c438f6165cfe83 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 02:29:45 +0000 Subject: [PATCH 11/25] fix(mcp): expose compact security workflow goals --- .../003-status.md | 102 +++++++++++++++--- .../004-status.md | 22 +++- ...t-extraction-and-paired-requalification.md | 9 +- plans/registry.md | 4 +- .../standalone-mcp-local-inference-roadmap.md | 4 +- src/mcp/server.rs | 58 +++++----- tests/mcp_schema_slimming.rs | 25 +++++ 7 files changed, 171 insertions(+), 53 deletions(-) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index 9a84a73..df7090c 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -88,14 +88,17 @@ qualification. ## 5. Remaining evidence to unblock and close -1. Run the same paired full+diagnostic and compact+brief prompts on Gemma E2B - and a larger reference model, including security, research, page/PDF fetch, - repo/evidence, and grouped-failure tasks. Capture tool choice, argument - validity, repairs, evidence, failure interpretation, calls, total latency, - prompt processing time, and context usage. -2. Capture exact tokenizer identity and host-injected prompt/completion token - counts for each paired task. Keep those distinct from context-meter - percentages and serialized JSON byte counts. +1. Initial paired full+diagnostic and compact+brief API trials on Gemma E2B + and Bonsai 27B, including extraction and security failure interpretation, + are recorded in §7 with tool validity, evidence quality, actual usage, and + latency. Repeat Bonsai's security task after M005 adds accepted goal values; + expand the task set to research, PDF, repository/evidence, and grouped + failures before closing M003. +2. Exact prompt/completion usage is now recorded for completed requests in §7. + LM Studio did not return counters for the Bonsai full-profile timeout, and + no tokenizer identity was exposed. Keep usage, context capacity, and + serialized bytes separate; additional prompt/token attribution is still + required for the broader acceptance matrix. 3. Characterize LM Studio's structured output and text fallback handling, visibility after config refresh/new chat, and compare one generic MCP inspector/client. @@ -104,15 +107,13 @@ qualification. credentials were created and no security-sensitive setting was changed; complete that prerequisite in LM Studio, then run the API smoke and restore prior settings. -5. M004 candidate `776f8ca4a8246e467041696b2220600eb7b2648d` exposes the - runtime-accepted research goal/domain values in compact schemas. Its - focused Gemma requalification and remaining fetch-extraction failure are - recorded in `plans/closure/standalone-mcp-local-inference/004-status.md`. - The canonical `make check` gate remains stalled in the macOS - `bounded_command` suite, so M004 is still blocked. M003 also remains blocked - on its broader operational acceptance matrix. -6. Complete M003's broader operational acceptance matrix, then update this - status; close M003 only after its full live acceptance criteria pass. +5. M004's hosted Linux/macOS/Windows canonical gate now passes on candidate + `755008c0a1da658835dd8f215d3b82bd1cb3dee5`; see its closure status. The + earlier local `bounded_command` stall did not reproduce in hosted CI. +6. M003 remains blocked on the post-correction Bonsai retry, broader domain + coverage, LM Studio UI structured-content/refresh behavior, protected API + smoke, and remediation or explicit acceptance of the quality failures in + §7. Close M003 only after its remaining live acceptance criteria pass. ## 6. Security and operational disposition @@ -123,3 +124,70 @@ so the remaining block is broader than the earlier API permission denial: model comparison, exact token attribution, host behavior checks, and the substantial live quality failures remain open. Do not claim a general LM Studio token-saving or quality win from the single paired task. + +## 7. Paired local inference API measurements (M005) + +The later M005 trial used LM Studio's local OpenAI-compatible inference API +with the compact/full tool definitions obtained from the branch MCP stdio +server. The harness supplied the definitions to the model, executed the +returned MCP call against the local server, and returned the tool result to the +model. These are real model turns with per-request usage counters, but they +are not the LM Studio chat UI's native MCP integration and do not satisfy its +protected API permission check. The trials used LM Studio 0.4.26+4 and the +already loaded models `google/gemma-4-e2b` (MLX 4-bit, 10,703-token loaded +context) and `prism-ml/bonsai-27b` (MLX 2-bit, 262,144-token loaded context). + +### Paired Rust 1.90.0 document extraction + +Both profiles received the same question and official Rust 1.90.0 release +announcement URL. The branch MCP server fetched the document once and served +the cached same document to the paired request. Its diagnostic and brief +projections were both 4,498 characters, byte-for-byte equal; retrieval marked +neither document truncated. The brief response's outer warning metadata was +clipped, recorded as `output_truncated=true`, while the extracted document +text itself remained intact and longer than 800 characters. + +| Model | Profile | Tool call | Actual usage (prompt + completion) | Result | +|---|---|---|---:|---| +| Gemma E2B | full / diagnostic | One valid `web_fetch`, exact URL and query focus | 9,273 + 1,319 = 10,592 tokens across the tool-call and answer requests | Correct first change, then blank final answer at the 10,703-token loaded-context ceiling; quality fail. | +| Gemma E2B | compact / brief | One valid `web_fetch`, same URL and query focus | 6,172 + 798 = 6,970 tokens | Correctly named LLD as the default linker for `x86_64-unknown-linux-gnu` and Cargo's native `cargo publish --workspace`; evidence complete. | +| Bonsai 27B | full / diagnostic | No completed tool turn | Request timed out at 240 seconds | Fail; no usage counter returned. | +| Bonsai 27B | compact / brief | One valid `web_fetch`, same URL and query focus | 13,217 + 750 = 13,967 tokens | Correctly named both release changes with an official-page citation; 157.4 seconds. | + +The compact schema bytes in the direct inference requests were 18,718 versus +68,736 for full. The initialized MCP stdio smoke separately measured 18,990 +versus 72,923 serialized `tools/list` bytes. Do not conflate schema bytes with +the actual usage counts above. + +### Paired failure interpretation + +Both models received the same compact-profile Log4j 2.14.1 / CVE-2021-44228 +applicability prompt. The actual `security_search` result recorded 12 provider +attempts, 6 completions, 3 failures, no absence markers, and an HTTP 500 from +Yahoo; no native advisory provider was available, and the returned evidence +was generic web material. + +| Model | Tool-call validity | Actual usage (prompt + completion) | Interpretation | +|---|---|---:|---| +| Gemma E2B | One valid `security_search` call; no runtime error | 6,403 + 900 = 7,303 tokens over two inference requests; 25.31 seconds | Correctly said Yahoo's 500 does not prove absence, but overstated generic web pages as authoritative advisories and omitted the untrusted-source and unavailable-severity warnings. Partial, with evidence-trust failure. | +| Bonsai 27B | Called `security_search` with unsupported `goal="assess_applicability"`; runtime rejected it and the following repair attempt was not executed by the one-call harness | 11,280 + 367 = 11,647 tokens; 77.6 seconds | No usable final answer; invalid tool arguments and failure-interpretation task failed. | + +The Bonsai call exposed that compact `security_search` listed `goal` without +its accepted values. M005 adds the runtime's canonical security-goal enum and +a schema contract test. The post-correction Bonsai retry was run against that +schema, but the LM Studio request timed out after 300 seconds before returning +a tool call, final answer, or usage counters. This is recorded as a failed +retry, not as a pass; the initial invalid call remains in the table above. + +### Remaining M003 acceptance + +The paired model matrix now has exact prompt/completion usage for successful +requests and explicit timeouts where the host returned no counter, including +the 300-second post-correction Bonsai retry. It still +does not pass M003: the full Gemma extraction ended blank, Gemma's security +answer misstated evidence authority, Bonsai's security call failed argument +validation, and Bonsai full-profile inference timed out. Native LM Studio UI +tool-result handling and refresh behavior, structured-content/text fallback, +and the protected API smoke also remain open. No API authentication or MCP +permission setting was changed. Keep M003 blocked until these criteria pass or +are explicitly re-scoped by a future plan. diff --git a/plans/closure/standalone-mcp-local-inference/004-status.md b/plans/closure/standalone-mcp-local-inference/004-status.md index 52f2bfa..4bf652b 100644 --- a/plans/closure/standalone-mcp-local-inference/004-status.md +++ b/plans/closure/standalone-mcp-local-inference/004-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M004 — Corrective Status -Status: blocked (schema correction, focused live requalification, and installed-child smoke pass; canonical `make check` stalls in `bounded_command` on this host) +Status: closed (hosted canonical Linux, macOS, and Windows portability qualification passed on candidate `755008c0a1da658835dd8f215d3b82bd1cb3dee5`) Source implementation plan: `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` @@ -79,9 +79,23 @@ reported. The same host-specific stall was recorded in the original M004 qualification. A serial retry, `rtk cargo test --locked --all-features --test bounded_command -- --test-threads=1`, also remained silent for over two minutes and was interrupted (exit 130), so -test parallelism did not resolve the stall. The canonical gate is not green, -so M004 remains blocked; rerun it on a host where the process-group/cap-breach -tests complete before closing this milestone. +test parallelism did not resolve the stall. At that point, the canonical gate +was still open pending hosted qualification; the hosted result is recorded +below. + +## 4. Hosted canonical requalification + +The exact candidate `755008c0a1da658835dd8f215d3b82bd1cb3dee5` includes the +M004 research schema correction and passed the hosted CI workflow's canonical +`make ci` job plus Windows portability and the added macOS canonical/local +filesystem job in GitHub Actions run +[`41141113615`](https://github.com/eggstack/eggsearch/actions/runs/41141113615). +All three jobs completed successfully. The process-group/cap-breach stall did +not reproduce on hosted Linux or macOS. This resolves the M004 canonical gate; +the earlier local interruption remains useful host-specific history, not a +current blocker. A separate egress qualification run had its macOS egress job +cancelled while the other egress targets passed; that job is outside the M004 +canonical gate and does not affect this closure. M003 remains blocked on broader paired model trials, exact tokenizer and prompt-token attribution, structured-content and refresh behavior, protected diff --git a/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md b/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md index 53956dc..0f403aa 100644 --- a/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md +++ b/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md @@ -9,6 +9,10 @@ Corrects M002's brief projection after M003/M004 live qualification found that the unconditional 800-character per-string clip hid most fetched document text. The existing M002 closure remains historical and unchanged. This plan also records the paired M003 acceptance and finishes M004's hosted canonical gate. +The initial Bonsai 27B paired failure-interpretation trial also exposed that +compact `security_search` advertised a free-form goal despite runtime accepting +only canonical values; this plan now includes making those values discoverable +and repeating that call. ## Objective @@ -58,7 +62,10 @@ case and add a deterministic regression fixture. 5. Enable manual workflow dispatch for the CI workflow and run the canonical `make ci` suite on hosted Linux and macOS. Record whether the prior local process-group stall reproduces. Do not treat a timeout as a passing gate. -6. Update M003/M004 statuses only with exact candidate evidence, this plan's +6. Advertise the canonical runtime-accepted security goals in the compact + `security_search` schema, add a schema contract assertion, and repeat the + larger-model failure-interpretation trial to check for a valid call. +7. Update M003/M004 statuses only with exact candidate evidence, this plan's closure, the roadmap, and the registry in the same closure commit. ## Verification diff --git a/plans/registry.md b/plans/registry.md index 0361a36..6dbbe05 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001-M002 closed; M003-M004 acceptance open; M005 corrective active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M003 paired model acceptance remains open; M004 hosted canonical gate pending; M005 fixes brief extraction clipping and records exact-pair qualification | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`; M004 status `plans/closure/standalone-mcp-local-inference/004-status.md`; M005 is registered below. | +| Standalone MCP local-inference efficiency | active (M001, M002, M004 closed; M003 operational acceptance open; M005 corrective active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 hosted Linux/macOS gate passed on `755008c`; M003 paired model acceptance remains open; M005 fixes brief extraction clipping and records paired model evidence | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`; M004 status `plans/closure/standalone-mcp-local-inference/004-status.md`; M005 is registered below. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -55,7 +55,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | | Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | M001/M002 closed; offline smoke and five interactive Gemma task records are in `plans/closure/standalone-mcp-local-inference/003-status.md`. One paired web task reduced the UI context meter, but research/fetch quality failures remain; larger-model, exact token attribution, structured-content/refresh, and protected API checks remain open. | -| Standalone MCP local-inference | M004 compact schema discoverability corrective | blocked | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Candidate `776f8ca4a8246e467041696b2220600eb7b2648d`; offline checks, installed-child smoke, and focused Gemma schema call pass. Compact fetch extraction still misreports an install command as a change; canonical `make check` stalls in `bounded_command`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | +| Standalone MCP local-inference | M004 compact schema discoverability corrective | closed | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Candidate `755008c0a1da658835dd8f215d3b82bd1cb3dee5`; hosted canonical Linux, macOS, and Windows portability jobs pass in CI run `41141113615`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Standalone MCP local-inference | M005 brief document extraction and paired requalification | active | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Corrective to M002, with M003 and M004 operational gates; requalify the same Rust 1.90 announcement query/document across diagnostic and brief, Gemma E2B and `prism-ml/bonsai-27b`. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index 2624bbf..9f19fbe 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001-M002 closed; M003 blocked on broader LM Studio acceptance; M004 blocked on canonical gate completion after focused live requalification) +Status: active (M001, M002, and M004 closed; M003 remains open on operational acceptance; M005 corrective active) Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. @@ -47,7 +47,7 @@ M001 compact ten-tool advertised schema profile (closed) +-- operational --> M003 LM Studio/direct-client compatibility and quality (open acceptance) +-- corrective --> M005 brief document extraction and paired requalification (active) -M004 compact schema discoverability corrective (implementation complete; hosted gate pending) +M004 compact schema discoverability corrective (closed on hosted Linux/macOS candidate 755008c) ``` No CodeGG dependency; it is a regression consumer, not the implementation target. M003's offline qualification can run in CI; a real LM Studio interactive smoke is an operational closure requirement, not a reason to block M001/M002 development. diff --git a/src/mcp/server.rs b/src/mcp/server.rs index 8d173ab..6936609 100644 --- a/src/mcp/server.rs +++ b/src/mcp/server.rs @@ -200,14 +200,14 @@ fn compact_schema(name: &str, schema: &mut serde_json::Value) { _ => COMMON, }; compact_schema_value(schema, allowed); - compact_research_schema(name, schema); + compact_workflow_schema(name, schema); if name == "build_evidence_bundle" { compact_evidence_bundle_schema(schema); } } -fn compact_research_schema(name: &str, schema: &mut serde_json::Value) { - if name != "research_search" { +fn compact_workflow_schema(name: &str, schema: &mut serde_json::Value) { + if name != "research_search" && name != "security_search" { return; } let Some(properties) = schema @@ -216,30 +216,34 @@ fn compact_research_schema(name: &str, schema: &mut serde_json::Value) { else { return; }; - if let Some(domain) = properties.get_mut("research_domain") { - set_string_schema_enum( - domain, - &[ - "general", - "software_architecture", - "architecture", - "api_design", - "api", - "distributed_systems", - "distributed", - "security", - "performance", - "language_ecosystem", - "ecosystem", - "machine_learning", - "ml", - "infrastructure", - "infra", - ], - ); - } - if let Some(goal) = properties.get_mut("goal") { - set_string_schema_enum(goal, crate::mcp::tools::canonical::RESEARCH_GOAL_VALUES); + if name == "research_search" { + if let Some(domain) = properties.get_mut("research_domain") { + set_string_schema_enum( + domain, + &[ + "general", + "software_architecture", + "architecture", + "api_design", + "api", + "distributed_systems", + "distributed", + "security", + "performance", + "language_ecosystem", + "ecosystem", + "machine_learning", + "ml", + "infrastructure", + "infra", + ], + ); + } + if let Some(goal) = properties.get_mut("goal") { + set_string_schema_enum(goal, crate::mcp::tools::canonical::RESEARCH_GOAL_VALUES); + } + } else if let Some(goal) = properties.get_mut("goal") { + set_string_schema_enum(goal, crate::mcp::tools::canonical::SECURITY_GOAL_VALUES); } } diff --git a/tests/mcp_schema_slimming.rs b/tests/mcp_schema_slimming.rs index 0f1ceee..4e69c10 100644 --- a/tests/mcp_schema_slimming.rs +++ b/tests/mcp_schema_slimming.rs @@ -1,6 +1,7 @@ use std::sync::Arc; use eggsearch::core::config::AppConfig; +use eggsearch::core::config::McpSchemaProfile; use eggsearch::mcp::ServerState; fn tool_definitions() -> Vec { @@ -18,6 +19,19 @@ fn schema_for(name: &str) -> serde_json::Value { serde_json::to_value(&tool.input_schema).expect("schema serializes") } +fn compact_schema_for(name: &str) -> serde_json::Value { + let mut config = AppConfig::default(); + config.mcp.schema_profile = McpSchemaProfile::Compact; + let state = Arc::new(ServerState::build(config).expect("compact state")); + let server = eggsearch::mcp::EggsearchServer::new(state); + let tool = server + .tool_definitions() + .into_iter() + .find(|t| t.name.as_ref() == name) + .unwrap_or_else(|| panic!("tool {name} not found")); + serde_json::to_value(&tool.input_schema).expect("schema serializes") +} + fn serialized_bytes(name: &str) -> usize { let tools = tool_definitions(); let tool = tools @@ -182,6 +196,17 @@ fn security_search_advertises_goal_and_include() { let schema = schema_for("security_search"); assert!(has_prop(&schema, "goal"), "goal must be advertised"); assert!(has_prop(&schema, "include"), "include must be advertised"); + let compact_schema = compact_schema_for("security_search"); + let goal_values = compact_schema["properties"]["goal"]["enum"] + .as_array() + .expect("compact security goal must advertise an enum") + .iter() + .map(|value| value.as_str().expect("goal enum value is a string")) + .collect::>(); + assert_eq!( + goal_values, + eggsearch::mcp::tools::canonical::SECURITY_GOAL_VALUES + ); for legacy in [ "include_kev", "include_exploit_context", From e40c8b46217fc849fb9905793d5fcd7b40339760 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 02:45:22 +0000 Subject: [PATCH 12/25] docs(plans): close M005 and hosted M004 gate --- .../004-status.md | 15 ++-- .../005-status.md | 88 +++++++++++++++++++ ...t-extraction-and-paired-requalification.md | 2 +- plans/registry.md | 8 +- .../standalone-mcp-local-inference-roadmap.md | 8 +- 5 files changed, 104 insertions(+), 17 deletions(-) create mode 100644 plans/closure/standalone-mcp-local-inference/005-status.md diff --git a/plans/closure/standalone-mcp-local-inference/004-status.md b/plans/closure/standalone-mcp-local-inference/004-status.md index 4bf652b..341090f 100644 --- a/plans/closure/standalone-mcp-local-inference/004-status.md +++ b/plans/closure/standalone-mcp-local-inference/004-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M004 — Corrective Status -Status: closed (hosted canonical Linux, macOS, and Windows portability qualification passed on candidate `755008c0a1da658835dd8f215d3b82bd1cb3dee5`) +Status: closed (hosted canonical Linux, macOS, and Windows portability qualification passed on final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`) Source implementation plan: `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` @@ -85,17 +85,16 @@ below. ## 4. Hosted canonical requalification -The exact candidate `755008c0a1da658835dd8f215d3b82bd1cb3dee5` includes the +The final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83` includes the M004 research schema correction and passed the hosted CI workflow's canonical `make ci` job plus Windows portability and the added macOS canonical/local filesystem job in GitHub Actions run -[`41141113615`](https://github.com/eggstack/eggsearch/actions/runs/41141113615). +[`41160264796`](https://github.com/eggstack/eggsearch/actions/runs/41160264796). All three jobs completed successfully. The process-group/cap-breach stall did -not reproduce on hosted Linux or macOS. This resolves the M004 canonical gate; -the earlier local interruption remains useful host-specific history, not a -current blocker. A separate egress qualification run had its macOS egress job -cancelled while the other egress targets passed; that job is outside the M004 -canonical gate and does not affect this closure. +not reproduce on hosted Linux or macOS. The same correction had also passed +earlier hosted CI run `41141113615`; the final-candidate run supersedes that +earlier evidence for closure. The prior local interruption remains useful +host-specific history, not a current blocker. M003 remains blocked on broader paired model trials, exact tokenizer and prompt-token attribution, structured-content and refresh behavior, protected diff --git a/plans/closure/standalone-mcp-local-inference/005-status.md b/plans/closure/standalone-mcp-local-inference/005-status.md new file mode 100644 index 0000000..5422adf --- /dev/null +++ b/plans/closure/standalone-mcp-local-inference/005-status.md @@ -0,0 +1,88 @@ +# Standalone MCP Local-Inference M005 — Corrective Closure + +Status: closed (brief extraction preserves the paired document; compact security goal values are discoverable; final hosted CI passes) + +Source implementation plan: +`plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` + +Source subsystem roadmap: +`plans/subsystems/standalone-mcp-local-inference-roadmap.md` + +Qualified candidate: `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`. +This corrective preserves the historical M002 closure and leaves the broader +M003 LM Studio acceptance status open; its exact model records are in +`plans/closure/standalone-mcp-local-inference/003-status.md`. + +## 1. Brief extraction correction + +`src/mcp/projection.rs::project_brief` no longer clips every string at 800 +characters before applying the configured response budget. It keeps the +original field text when the serialized response fits and progressively +reduces text only when necessary. A deterministic regression projects the +same long extracted document and query in diagnostic and brief modes and +checks equality when the brief response fits. + +The live paired qualification used the official Rust 1.90.0 announcement and +the identical question, “What two concrete Rust 1.90.0 changes are stated on +this page?” The diagnostic and brief document projections were each 4,498 +characters and byte-for-byte equal. Both retrieval results marked the fetched +document untruncated, and the document text exceeded 800 characters. Brief +projection did mark outer warning metadata as output-truncated; it did not +clip the document text. This behavior is recorded separately from retrieval +truncation. + +## 2. Paired local-model evidence + +The real local model trials, including actual per-request usage, call +arguments, evidence completeness, latency, and failure interpretation, are +recorded in §7 of M003 status. They used LM Studio 0.4.26+4 and its local +OpenAI-compatible inference API with tool definitions fetched from the branch +MCP stdio server. The harness executed model-returned calls through that +server. It did not exercise LM Studio chat UI's native MCP result handling or +the protected API permission path. + +The initial Bonsai 27B security call used an unsupported free-form +`goal="assess_applicability"`. M005 therefore adds the runtime's canonical +`SECURITY_GOAL_VALUES` enum to compact `security_search` and a test asserting +that the compact advertised values match that source of truth. On a post-fix +repeat, Bonsai's LM Studio request timed out after 300 seconds without usage, +a tool call, or a final answer. The timeout is a recorded failure, not a pass. + +M003 stays open because the matrix found a blank full-profile Gemma answer, a +Gemma security answer that overstated the authority of generic web evidence, +the invalid and timed-out Bonsai security attempts, and a Bonsai full-profile +timeout. The post-fix compact extraction was successful for both models. + +## 3. Verification + +- `cargo fmt --check` — pass. +- `cargo test --locked --all-features --test mcp_schema_slimming` — 17 passed. +- `cargo test --locked --all-features --test mcp_projection` — 22 passed. +- `cargo build --locked --release --all-features` — pass. +- `python3 scripts/lm_studio_stdio_smoke.py target/release/eggsearch` — full + and compact profiles both passed; ten tools and `provider_status` succeeded. + Serialized `tools/list` sizes were 72,923 bytes full and 19,281 bytes + compact. +- Hosted final-candidate CI run + [`41160264796`](https://github.com/eggstack/eggsearch/actions/runs/41160264796) + — all jobs passed: Linux canonical `make ci`, macOS canonical `make ci` plus + local-filesystem portability, and Windows all-features portability. The + process-group/cap-breach stall previously seen on the local macOS host did + not reproduce. +- PR #15 checks — 47 passed, zero failed. The separate egress, Python native, + and release drift workflows also passed on the same candidate. +- `~/.lmstudio/mcp.json` was restored from its saved original and verified + byte-for-byte: both original and restored SHA-256 are + `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. + No LM Studio API authentication or MCP permission setting was changed. + +## 4. Disposition + +M005 acceptance is satisfied: brief extraction retains the full document when +it fits, the paired projections use the same query/document, the local-model +successes and failures are recorded with actual usage where available, the +schema regression is tested, the original config is restored, and the exact +candidate passed hosted Linux, macOS, and Windows checks. M004's hosted gate is +also closed on this candidate. M003 remains blocked on its listed operational +and quality criteria; this corrective does not claim a general token-saving +or model-quality win. diff --git a/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md b/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md index 0f403aa..c67f6a9 100644 --- a/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md +++ b/plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md @@ -1,6 +1,6 @@ # M005 — Brief Document Extraction and Paired Requalification -Status: active +Status: closed Source subsystem roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` diff --git a/plans/registry.md b/plans/registry.md index 6dbbe05..8cae82c 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001, M002, M004 closed; M003 operational acceptance open; M005 corrective active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 hosted Linux/macOS gate passed on `755008c`; M003 paired model acceptance remains open; M005 fixes brief extraction clipping and records paired model evidence | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` and `002-status.md`; M003 operational status `003-status.md`; M004 status `plans/closure/standalone-mcp-local-inference/004-status.md`; M005 is registered below. | +| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005 closed; M003 operational acceptance open) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 and M005 passed final hosted Linux/macOS/Windows gates on `de69f1d`; M003's paired model acceptance remains blocked on documented quality and host checks | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `005-status.md`; M003 operational status remains `003-status.md`. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -54,9 +54,9 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under |---|---|---|---|---| | Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | -| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | M001/M002 closed; offline smoke and five interactive Gemma task records are in `plans/closure/standalone-mcp-local-inference/003-status.md`. One paired web task reduced the UI context meter, but research/fetch quality failures remain; larger-model, exact token attribution, structured-content/refresh, and protected API checks remain open. | -| Standalone MCP local-inference | M004 compact schema discoverability corrective | closed | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Candidate `755008c0a1da658835dd8f215d3b82bd1cb3dee5`; hosted canonical Linux, macOS, and Windows portability jobs pass in CI run `41141113615`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | -| Standalone MCP local-inference | M005 brief document extraction and paired requalification | active | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Corrective to M002, with M003 and M004 operational gates; requalify the same Rust 1.90 announcement query/document across diagnostic and brief, Gemma E2B and `prism-ml/bonsai-27b`. | +| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Interactive UI history and paired direct local-inference API matrix are in `plans/closure/standalone-mcp-local-inference/003-status.md`, including actual prompt/completion usage and extraction/security failures for Gemma E2B and Bonsai 27B. UI structured-content/refresh, protected API, and quality criteria remain open. | +| Standalone MCP local-inference | M004 compact schema discoverability corrective | closed | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted canonical Linux, macOS, and Windows jobs pass in CI run `41160264796`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | +| Standalone MCP local-inference | M005 brief document extraction and paired requalification | closed | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted CI run `41160264796` passed. Closure `plans/closure/standalone-mcp-local-inference/005-status.md`; M003 remains open for the documented operational gaps. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index 9f19fbe..f24922c 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001, M002, and M004 closed; M003 remains open on operational acceptance; M005 corrective active) +Status: active (M001, M002, M004, and M005 closed; M003 remains open on operational acceptance) Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. @@ -45,9 +45,9 @@ M001 compact ten-tool advertised schema profile (closed) +-- hard --> M002 brief/budgeted agent responses (closed) | +-- operational --> M003 LM Studio/direct-client compatibility and quality (open acceptance) - +-- corrective --> M005 brief document extraction and paired requalification (active) + +-- corrective --> M005 brief document extraction and paired requalification (closed) -M004 compact schema discoverability corrective (closed on hosted Linux/macOS candidate 755008c) +M004 compact schema discoverability corrective (closed on hosted Linux/macOS candidate de69f1d) ``` No CodeGG dependency; it is a regression consumer, not the implementation target. M003's offline qualification can run in CI; a real LM Studio interactive smoke is an operational closure requirement, not a reason to block M001/M002 development. @@ -58,7 +58,7 @@ No CodeGG dependency; it is a regression consumer, not the implementation target | M002 | capability | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Explicit brief projection and bounded results with critical failure, identity, trust and security semantics intact; unchanged legacy defaults | | M003 | capability | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Verified LM Studio install/smoke, measured local-model context cost and completion quality, onboarding guides, and stop/rollback evidence | | M004 | capability | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Advertise runtime-accepted research goal/domain values in compact schemas; test schema parity and size; repeat focused live tasks; M003 remains open for its broader gates | -| M005 | corrective | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Preserve extracted document text when it fits the brief budget; compare diagnostic and brief on identical queries/documents; complete paired Gemma E2B and larger-model evidence and close M004's hosted Linux/macOS canonical gate | +| M005 | corrective | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Closed on `de69f1d`; brief preserves same-document extraction within budget; model evidence and unresolved M003 failures are recorded; hosted Linux/macOS/Windows gates pass | ## Cross-cutting controls From e580e13ebfb86d3562a03fddc744a9b86b4dc279 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 03:36:21 +0000 Subject: [PATCH 13/25] docs(plans): record LFM3 LM Studio trials --- .../003-status.md | 64 +++++++++++++++++++ plans/registry.md | 2 +- 2 files changed, 65 insertions(+), 1 deletion(-) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index df7090c..9347eba 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -191,3 +191,67 @@ tool-result handling and refresh behavior, structured-content/text fallback, and the protected API smoke also remain open. No API authentication or MCP permission setting was changed. Keep M003 blocked until these criteria pass or are explicitly re-scoped by a future plan. + +## 8. LFM3-2B paired search and extraction trials + +The user authorized additional trials with `lfm3-2b-omni-abliter8`. LM Studio +reported it loaded as a 2B Q4_K_M GGUF model with tool-use capability and a +128,000-token context. As with §7, the model received tool schemas from the +branch MCP stdio server through the OpenAI-compatible `/v1/chat/completions` +endpoint; the harness executed tool calls through eggsearch and returned the +MCP result. This measures local model turns and token usage, but is not LM +Studio's native MCP API integration. + +### Search pair + +Both profiles received the same request to find the official Rust 1.90.0 +announcement, report two concrete changes, cite evidence and URLs, and +distinguish the official source. Each profile first called `web_search` with +`excerpt_count=5`, which runtime rejected because the maximum is 3. The model +recovered with a second call using `excerpt_count=3` and completed its answer. + +| Profile | Actual usage (prompt + completion) | Calls / time | Result | +|---|---:|---|---| +| Full / diagnostic | 14,563 + 470 = 15,033 tokens | Two calls; 21.52 seconds | Correctly identified the Rust Blog as the official announcement, but cited `releases.rs` and Mastodon for the two changes, then mislabeled the official Rust Language Mastodon account as community discussion. Evidence/source classification was partial. | +| Compact / brief | 8,718 + 352 = 9,070 tokens | Two calls; 13.15 seconds | Correctly named both changes and the official Rust Blog, but attributed the workspace-publishing claim to the `releases.rs` result, whose returned snippet did not establish that claim. Evidence attribution failed. | + +The successful full search result reported 10 DuckDuckGo results, zero +Startpage results, and a Yahoo HTTP 500. The compact retry returned results +without a provider failure. Both outputs therefore also show why provider +failure evidence must be interpreted from each actual result rather than +carried across paired runs. + +### Direct-fetch pair + +Both profiles received the same official Rust 1.90.0 URL and request for two +changes with supporting evidence. Both made one valid `web_fetch` call and +identified the LLD default change and Cargo workspace publishing. Each cited +the official announcement URL. The text itself was not marked retrieval +truncated in either result. + +| Profile | Fetch arguments | Actual usage (prompt + completion) | Result | +|---|---|---:|---| +| Full / diagnostic | Exact URL, `extract_mode=text`, `max_chars=2000` | 8,816 + 495 = 9,311 tokens; 11.56 seconds | Correct changes and official URL; grounded both in excerpts from the fetched announcement. | +| Compact / brief | Exact URL and `extract_mode=text` | 4,962 + 418 = 5,380 tokens; 7.73 seconds | Correct changes and official URL with direct excerpts. Fetched text was 4,498 characters, over 800; `output_truncated=true` applied to outer projection metadata, while `truncated=false` for the fetched document. | + +These direct-fetch answers are a positive result for LFM3 on this task, not a +general quality claim. Relative to full, compact used 3,931 fewer total +prompt-plus-completion tokens (about 42.2%) in this one paired trial. This +comparison is the local API's reported usage; it is separate from serialized +MCP byte counts and from the LM Studio UI context meter. + +### Native MCP API route check + +The local server was started on eggsearch's documented loopback-only +Streamable HTTP endpoint (`127.0.0.1:11320/mcp`). LM Studio's native REST +`/api/v1/chat` with an `ephemeral_mcp` integration rejected that URL because +dynamic remote MCP integrations allow only public addresses. No tunnel or +public exposure was introduced. + +The LM Studio settings page showed “Allow per-request MCPs” on, “Require +Authentication” off, and “Allow calling servers from mcp.json” disabled. The +last setting requires authentication. The configured-server path is therefore +not qualified; enabling it and creating an API token would change +security-sensitive settings and require explicit operator participation. +No such setting or credential was changed. The loopback server was stopped +after the request. diff --git a/plans/registry.md b/plans/registry.md index 8cae82c..a001f00 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -54,7 +54,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under |---|---|---|---|---| | Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | -| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Interactive UI history and paired direct local-inference API matrix are in `plans/closure/standalone-mcp-local-inference/003-status.md`, including actual prompt/completion usage and extraction/security failures for Gemma E2B and Bonsai 27B. UI structured-content/refresh, protected API, and quality criteria remain open. | +| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Interactive UI history and paired direct local-inference API matrix are in `plans/closure/standalone-mcp-local-inference/003-status.md`, including Gemma E2B, Bonsai 27B, and LFM3-2B usage, evidence, and failures. Native configured-server API access, UI structured-content/refresh, and quality criteria remain open. | | Standalone MCP local-inference | M004 compact schema discoverability corrective | closed | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted canonical Linux, macOS, and Windows jobs pass in CI run `41160264796`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Standalone MCP local-inference | M005 brief document extraction and paired requalification | closed | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted CI run `41160264796` passed. Closure `plans/closure/standalone-mcp-local-inference/005-status.md`; M003 remains open for the documented operational gaps. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | From a2bda32ee8aa37ceefc8ee82e9ecc0a77b333903 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 04:20:45 +0000 Subject: [PATCH 14/25] docs(plans): record LM Studio token permission probe --- .../003-status.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index 9347eba..3488d1e 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -255,3 +255,22 @@ not qualified; enabling it and creating an API token would change security-sensitive settings and require explicit operator participation. No such setting or credential was changed. The loopback server was stopped after the request. + +## 9. Bearer-token probe with authentication disabled + +At the user's request, a loopback-only `POST /api/v1/chat` request included +the supplied bearer token and the configured `mcp/eggsearch` integration. The +request returned HTTP 403 with LM Studio's `Permission denied to use plugin +'mcp/eggsearch'` error, directing the operator to allow plugin use in server +configuration and ensure the token has the required permission. At the time of +the probe, `Require Authentication` and `Allow calling servers from mcp.json` +were both off; neither setting was changed. The token value is intentionally +not recorded here. + +This establishes that supplying a bearer token alone does not bypass the +disabled configured-server permission. Since authentication was off, the +response does not establish whether LM Studio validated the token. The +existing no-auth API clients were not interrupted. Enabling the authentication +switch makes a valid token mandatory for every API request, according to LM +Studio's server contract; existing no-auth clients would then need to supply a +valid token too. From 1490a1684a2a8b9948e5dc6783dbbb001e926b0c Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 04:27:01 +0000 Subject: [PATCH 15/25] docs(plans): record authenticated LM Studio MCP acceptance --- .../003-status.md | 120 +++++++++++------- plans/registry.md | 4 +- 2 files changed, 78 insertions(+), 46 deletions(-) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index 3488d1e..823b20d 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M003 — Operational Status -Status: blocked (live LM Studio interaction is qualified, but the acceptance matrix is incomplete and Gemma quality failures require a corrective decision) +Status: blocked (interactive and authenticated native API paths are qualified, but the acceptance matrix is incomplete and documented model-quality failures require a corrective decision) Source implementation plan: `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` @@ -13,13 +13,15 @@ Predecessor closures: M001 `d983fa5`; M002 `plans/closure/standalone-mcp-local-i ## 1. Current finding -M003 is not closed. The installed-candidate stdio smoke and the interactive -LM Studio chat path both worked with the candidate, and all ten tools remained -available. This is the first real model-turn evidence for the branch. It does -not establish that compact is a safe default: one paired web task showed a -large context-meter reduction, but specialist research and direct-fetch tasks -had material quality failures. The tests used only `google/gemma-4-e2b` in -LM Studio 0.4.26+4, so the required larger-reference comparison remains open. +M003 is not closed. The installed-candidate stdio smoke, interactive LM +Studio chat path, and authenticated native `/api/v1/chat` path worked with the +candidate, and all ten tools remained available. The paired direct-inference +trials included `google/gemma-4-e2b`, `prism-ml/bonsai-27b`, and +`lfm3-2b-omni-abliter8`; the required same-task comparison with a larger +reference model remains incomplete. Existing evidence does not establish +that compact is a safe default: specialist research and direct-fetch tasks +had material quality failures, and several runs timed out or exhausted the +available context. Model/server facts observed in LM Studio: @@ -66,7 +68,7 @@ provide a reliable per-chat usage breakdown for this test set. | Tool choice, argument validity, repairs, evidence and failure truthfulness | Five task records above | partial / quality concern | Investigate research argument retries, context overrun, and source classification; add paired specialist/fetch cases | | Exact tokenizer and prompt-token attribution | App context meter only; no per-chat token counts or tokenizer identifier | blocked | Capture host-injected prompt and response usage for each profile | | Output schema / structuredContent vs text fallback | Offline protocol coverage only | partial | Inspect actual LM Studio handling and compare a generic MCP inspector/client | -| Protected API MCP smoke | Previous API call was permission-denied; “Allow calling servers from mcp.json” remained disabled | blocked | Operator must configure LM Studio authentication and authorize that server permission before API qualification; no such security settings were changed | +| Protected API MCP smoke | Authenticated native `/api/v1/chat` call through configured `mcp/eggsearch`, with only `web_fetch` allowed | pass | UI structured-content/text fallback and refresh behavior remain uncharacterized | | Config rollback | Restored original `~/.lmstudio/mcp.json`; `cmp` against backup succeeded | pass | None | ## 4. Work completed @@ -91,7 +93,7 @@ qualification. 1. Initial paired full+diagnostic and compact+brief API trials on Gemma E2B and Bonsai 27B, including extraction and security failure interpretation, are recorded in §7 with tool validity, evidence quality, actual usage, and - latency. Repeat Bonsai's security task after M005 adds accepted goal values; + latency. The post-M005 Bonsai security retry timed out without a result; expand the task set to research, PDF, repository/evidence, and grouped failures before closing M003. 2. Exact prompt/completion usage is now recorded for completed requests in §7. @@ -102,28 +104,27 @@ qualification. 3. Characterize LM Studio's structured output and text fallback handling, visibility after config refresh/new chat, and compare one generic MCP inspector/client. -4. For API-specific qualification, the LM Studio server requires - authentication before enabling “Allow calling servers from mcp.json.” No - credentials were created and no security-sensitive setting was changed; - complete that prerequisite in LM Studio, then run the API smoke and restore - prior settings. +4. The authenticated native configured-server API smoke now passes; see §10. + The user enabled authentication and “Allow calling servers from mcp.json.” + UI structured-content/text fallback and refresh behavior remain untested. 5. M004's hosted Linux/macOS/Windows canonical gate now passes on candidate `755008c0a1da658835dd8f215d3b82bd1cb3dee5`; see its closure status. The earlier local `bounded_command` stall did not reproduce in hosted CI. -6. M003 remains blocked on the post-correction Bonsai retry, broader domain - coverage, LM Studio UI structured-content/refresh behavior, protected API - smoke, and remediation or explicit acceptance of the quality failures in - §7. Close M003 only after its remaining live acceptance criteria pass. +6. M003 remains blocked on broader domain coverage, LM Studio UI + structured-content/refresh behavior, and remediation or explicit + acceptance of the quality failures in §7. Close M003 only after its + remaining live acceptance criteria pass. ## 6. Security and operational disposition -No API authentication, credential, or MCP permission setting was changed. -The temporary global MCP config was restored and verified byte-identical to -its pre-test backup. LM Studio interactive chat qualification is now recorded, -so the remaining block is broader than the earlier API permission denial: -model comparison, exact token attribution, host behavior checks, and the -substantial live quality failures remain open. Do not claim a general LM -Studio token-saving or quality win from the single paired task. +The agent did not change API authentication, credentials, or MCP permission +settings. The user enabled authentication and configured-server access after +the earlier tests. The temporary global MCP config was restored and verified +byte-identical to its pre-test backup. LM Studio interactive and native API +qualification are recorded, but model comparison, exact token attribution, +UI host behavior checks, and substantial live quality failures remain open. +Do not claim a general LM Studio token-saving or quality win from the single +paired task. ## 7. Paired local inference API measurements (M005) @@ -188,9 +189,9 @@ does not pass M003: the full Gemma extraction ended blank, Gemma's security answer misstated evidence authority, Bonsai's security call failed argument validation, and Bonsai full-profile inference timed out. Native LM Studio UI tool-result handling and refresh behavior, structured-content/text fallback, -and the protected API smoke also remain open. No API authentication or MCP -permission setting was changed. Keep M003 blocked until these criteria pass or -are explicitly re-scoped by a future plan. +and broader domain coverage remain open. The protected API smoke is later +qualified in §10. Keep M003 blocked until remaining criteria pass or are +explicitly re-scoped by a future plan. ## 8. LFM3-2B paired search and extraction trials @@ -259,18 +260,49 @@ after the request. ## 9. Bearer-token probe with authentication disabled At the user's request, a loopback-only `POST /api/v1/chat` request included -the supplied bearer token and the configured `mcp/eggsearch` integration. The -request returned HTTP 403 with LM Studio's `Permission denied to use plugin -'mcp/eggsearch'` error, directing the operator to allow plugin use in server -configuration and ensure the token has the required permission. At the time of -the probe, `Require Authentication` and `Allow calling servers from mcp.json` -were both off; neither setting was changed. The token value is intentionally -not recorded here. - -This establishes that supplying a bearer token alone does not bypass the -disabled configured-server permission. Since authentication was off, the -response does not establish whether LM Studio validated the token. The -existing no-auth API clients were not interrupted. Enabling the authentication -switch makes a valid token mandatory for every API request, according to LM -Studio's server contract; existing no-auth clients would then need to supply a -valid token too. +the supplied bearer token and a temporary configured-server alias +`mcp/eggsearch-test`. It returned HTTP 403 with LM Studio's `Permission denied +to use plugin` error. At the time of the probe, `Require Authentication` and +`Allow calling servers from mcp.json` were both off; neither setting was +changed. A plain authenticated inference request succeeded. The token value +is intentionally not recorded here. + +The 403 was specific to the temporary alias and disabled configured-server +permission, rather than evidence that the token lacked MCP permission. The +follow-up in §10 tests the actual `mcp/eggsearch` entry after the user enabled +authentication and configured-server access. LM Studio requires an API token +for every request when authentication is enabled; the user enabled this +setting after the earlier probe. + +## 10. Authenticated native MCP API acceptance + +After the user enabled authentication and “Allow calling servers from +mcp.json,” a loopback-only `POST http://127.0.0.1:1234/api/v1/chat` request +used the actual `mcp/eggsearch` integration from the restored LM Studio +configuration. LM Studio 0.4.26+4 ran `lfm3-2b-omni-abliter8` (Q4_K_M, +128,000-token model context). No token value is recorded. + +The request restricted the integration to `web_fetch` and asked the model to +read LM Studio's official authentication documentation. The native API +returned HTTP 200, emitted one valid `web_fetch` call for the requested URL, +and returned a correct answer identifying the Authentication page and +explaining that an existing token's permissions can be edited in the API +Tokens modal. LM Studio reported 1,575 input tokens and 113 output tokens; +the local request took about six seconds. Eggsearch returned a successful +fetch of the official page. The request used a 24,000-token context setting. + +A first request exposing the full Eggsearch tool palette at a 12,000-token +context setting failed before inference because the resolved prompt required +18,737 tokens against an available 12,032. Raising context to 32,768 allowed +the native request to run. Restricting `allowed_tools` to the single required +tool therefore produced a much smaller measured prompt (1,575 input tokens) +and successfully completed the retrieval. These are API-reported usage +figures for this request, not a general quality or token-saving claim. + +The user enabled the server authentication and configured-server settings; +the agent did not change them. The original `~/.lmstudio/mcp.json` was +restored byte-for-byte after the temporary test alias was removed. The +protected native MCP API criterion is now passed. Interactive UI +structured-content versus text fallback, config refresh behavior, broader +paired-task quality, and the documented negative model outcomes remain open, +so M003 is still blocked. diff --git a/plans/registry.md b/plans/registry.md index a001f00..c7248ac 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005 closed; M003 operational acceptance open) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 and M005 passed final hosted Linux/macOS/Windows gates on `de69f1d`; M003's paired model acceptance remains blocked on documented quality and host checks | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `005-status.md`; M003 operational status remains `003-status.md`. | +| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005 closed; M003 operational acceptance open) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 and M005 passed final hosted Linux/macOS/Windows gates on `de69f1d`; M003's protected native API smoke now passes, with UI host behavior and model-quality evidence still open | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `005-status.md`; M003 operational status remains `003-status.md`. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -54,7 +54,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under |---|---|---|---|---| | Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | -| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Interactive UI history and paired direct local-inference API matrix are in `plans/closure/standalone-mcp-local-inference/003-status.md`, including Gemma E2B, Bonsai 27B, and LFM3-2B usage, evidence, and failures. Native configured-server API access, UI structured-content/refresh, and quality criteria remain open. | +| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Interactive UI history, paired inference matrix, and authenticated native MCP API success are in `plans/closure/standalone-mcp-local-inference/003-status.md`. UI structured-content/refresh, broader task coverage, and documented model-quality failures remain open. | | Standalone MCP local-inference | M004 compact schema discoverability corrective | closed | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted canonical Linux, macOS, and Windows jobs pass in CI run `41160264796`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Standalone MCP local-inference | M005 brief document extraction and paired requalification | closed | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted CI run `41160264796` passed. Closure `plans/closure/standalone-mcp-local-inference/005-status.md`; M003 remains open for the documented operational gaps. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | From 3bd46b161481c0d9d4bc0c3d2a7ef37841832558 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 05:43:54 +0000 Subject: [PATCH 16/25] docs(plans): reconcile M003 LM Studio acceptance --- docs/lm-studio.md | 13 ++++--- .../003-status.md | 35 +++++++++++-------- ...-studio-consumer-qualification-and-docs.md | 2 +- 3 files changed, 31 insertions(+), 19 deletions(-) diff --git a/docs/lm-studio.md b/docs/lm-studio.md index 5ab3e50..66a5299 100644 --- a/docs/lm-studio.md +++ b/docs/lm-studio.md @@ -85,10 +85,15 @@ Eggsearch's offline fixtures measure serialized schema and result sizes and exercise the MCP protocol. They do not establish LM Studio tokenizer counts, model-visible prompt inclusion, tool-choice quality, or end-to-end task success. Those require a live LM Studio chat with the actual local model. The -current environment has LM Studio 0.4.26+4 and a tool-capable local model, but -the API MCP smoke is blocked until the protected API setting above is enabled. -Accordingly, the standalone inference workstream does not claim a measured -LM Studio context or completion-quality improvement. +recorded environment used LM Studio 0.4.26+4. After the user enabled +authentication and configured-server access, a native `/api/v1/chat` request +through `mcp/eggsearch` completed a `web_fetch` call and returned LM Studio's +usage counters; see the M003 operational record for the task and limits. The +interactive app also rendered an Eggsearch `web_fetch` result card. These +smokes do not establish a general context or completion-quality improvement: +the paired local-model trials include both successful tasks and quality +failures, and host handling of `structuredContent` versus the text fallback +has not been isolated. For configuration keys and bounds, see [Eggsearch configuration](config.md) and the [context-efficiency measurements](retrieval-quality-context-efficiency.md). diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index 823b20d..e4728a5 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M003 — Operational Status -Status: blocked (interactive and authenticated native API paths are qualified, but the acceptance matrix is incomplete and documented model-quality failures require a corrective decision) +Status: blocked (the live UI and authenticated native API paths work, but M003's host-contract characterization, broader paired task matrix, and documented quality failures remain unresolved) Source implementation plan: `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` @@ -8,7 +8,7 @@ Source implementation plan: Source subsystem roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` -Implementation candidate exercised: `02a8af6fe4f87971c7daca7d1fca20c087b328ec`. +Final implementation candidate exercised by the installed-child smoke and native API: `de69f1ddd7a7e4acc067fb9892c438f6165cfe83` (`eggsearch 0.4.4`; M004/M005 changes included). Earlier interactive and model-turn evidence remains separately identified below. Predecessor closures: M001 `d983fa5`; M002 `plans/closure/standalone-mcp-local-inference/002-status.md`. ## 1. Current finding @@ -23,6 +23,11 @@ that compact is a safe default: specialist research and direct-fetch tasks had material quality failures, and several runs timed out or exhausted the available context. +The final candidate's installed-child stdio smoke passed on 2026-10-31: +full exposed all ten tools in 72,923 serialized `tools/list` bytes, compact +exposed the same ten in 19,281 bytes, and `provider_status` succeeded in both. +These byte counts do not estimate LM Studio tokenizer usage. + Model/server facts observed in LM Studio: - Model identifier: `google/gemma-4-e2b`, MLX 4-bit, 4.37 GB on disk. @@ -62,12 +67,12 @@ provide a reliable per-chat usage breakdown for this test set. | Copyable installed-binary `mcp.json` setup | `docs/lm-studio.md`; config temporarily pointed at branch release binary | pass | None for configuration syntax | | Windows/macOS/Linux paths and stdio logging contract | `docs/lm-studio.md` | pass | None for documentation | | Reproducible installed-candidate stdio smoke | `scripts/lm_studio_stdio_smoke.py` | pass | Offline protocol evidence only | -| Ten tools and profile list sizes | Real initialized child-process smoke: 72,923 full / 18,545 compact bytes; all ten names; `provider_status` with probing off | pass | Measure model-tokenized attribution | -| LM Studio tool visibility and interactive tool use | Full and compact chats each called `web_search`; compact also called `security_search`, `research_search`, and `web_fetch` | partial | Add more paired tasks and verify refresh/new-chat behavior explicitly | -| Paired small/larger model comparison | One full/compact pair on Gemma E2B only | blocked | Run the same task matrix on a larger local reference model | +| Ten tools and profile list sizes | Final-candidate initialized child-process smoke: 72,923 full / 19,281 compact bytes; all ten names; `provider_status` with probing off | pass | Byte counts are not token counts | +| LM Studio tool visibility and interactive tool use | Prior full/compact qualification chats called `web_search`; compact also called `security_search`, `research_search`, and `web_fetch`. The app rendered a `web_fetch mcp/eggsearch` result card. | partial | Verify refresh/new-chat behavior against the final candidate; determine whether the app consumes structured content, text fallback, or both | +| Paired small/larger model comparison | Gemma E2B and Bonsai 27B received the same extraction task; Bonsai full timed out while compact completed. Additional same-task pairs are recorded for Gemma E2B and LFM3 2B. | partial | Complete specialist/research and fetch pairs on the larger model; interpret timeouts as failed trials | | Tool choice, argument validity, repairs, evidence and failure truthfulness | Five task records above | partial / quality concern | Investigate research argument retries, context overrun, and source classification; add paired specialist/fetch cases | -| Exact tokenizer and prompt-token attribution | App context meter only; no per-chat token counts or tokenizer identifier | blocked | Capture host-injected prompt and response usage for each profile | -| Output schema / structuredContent vs text fallback | Offline protocol coverage only | partial | Inspect actual LM Studio handling and compare a generic MCP inspector/client | +| Exact tokenizer and prompt-token attribution | Native API reports 1,575 input / 113 output tokens for one configured-server call; UI still exposes only context-meter readings and no tokenizer identity | partial | Capture same-task API usage across both profiles and characterize UI attribution | +| Output schema / structuredContent vs text fallback | Offline protocol coverage plus a successful rendered UI tool-result card and native API tool call | partial | Isolate which result representation the UI consumes; compare a generic MCP inspector/client if available | | Protected API MCP smoke | Authenticated native `/api/v1/chat` call through configured `mcp/eggsearch`, with only `web_fetch` allowed | pass | UI structured-content/text fallback and refresh behavior remain uncharacterized | | Config rollback | Restored original `~/.lmstudio/mcp.json`; `cmp` against backup succeeded | pass | None | @@ -107,8 +112,9 @@ qualification. 4. The authenticated native configured-server API smoke now passes; see §10. The user enabled authentication and “Allow calling servers from mcp.json.” UI structured-content/text fallback and refresh behavior remain untested. -5. M004's hosted Linux/macOS/Windows canonical gate now passes on candidate - `755008c0a1da658835dd8f215d3b82bd1cb3dee5`; see its closure status. The +5. M004 and M005 hosted Linux/macOS/Windows canonical gates pass on final + candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83` in CI run + `41160264796`; see their closure records. The earlier local `bounded_command` stall did not reproduce in hosted CI. 6. M003 remains blocked on broader domain coverage, LM Studio UI structured-content/refresh behavior, and remediation or explicit @@ -120,11 +126,12 @@ qualification. The agent did not change API authentication, credentials, or MCP permission settings. The user enabled authentication and configured-server access after the earlier tests. The temporary global MCP config was restored and verified -byte-identical to its pre-test backup. LM Studio interactive and native API -qualification are recorded, but model comparison, exact token attribution, -UI host behavior checks, and substantial live quality failures remain open. -Do not claim a general LM Studio token-saving or quality win from the single -paired task. +byte-identical to its pre-test backup +(`b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`). The +live UI and native API paths work, but the larger-model task matrix, host +refresh/structured-content behavior, and quality decision remain open. Do not +claim a general LM Studio token-saving or quality win from these limited +paired trials. ## 7. Paired local inference API measurements (M005) diff --git a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md index 9ca700f..fc25c70 100644 --- a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md +++ b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md @@ -1,6 +1,6 @@ # M003 — LM Studio Direct-MCP Qualification and Consumer Documentation -Status: blocked (M001/M002 dependencies closed; interactive Gemma qualification is recorded, but model comparison, exact token attribution, API/host behavior checks, and quality issues remain; see `plans/closure/standalone-mcp-local-inference/003-status.md`) +Status: blocked (M001/M002 dependencies closed; interactive and authenticated native API paths work, but larger-model paired tasks, profile-specific native API usage, UI refresh and result-shape characterization, and model-quality disposition remain incomplete; see `plans/closure/standalone-mcp-local-inference/003-status.md`) Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (rebase baseline to exact M002 candidate at start) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` From a339cce5005cd8d529f8416783e83603cc0e784c Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 06:05:41 +0000 Subject: [PATCH 17/25] test(local): stabilize mode-off workspace search fixture --- tests/local_workspace_integration.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/local_workspace_integration.rs b/tests/local_workspace_integration.rs index e39d1d4..cab7a9b 100644 --- a/tests/local_workspace_integration.rs +++ b/tests/local_workspace_integration.rs @@ -106,6 +106,7 @@ async fn mode_off_repo_search_uses_local_workspace_without_remote_dispatch() { ecosystem: Some("crates_io".into()), package: Some("locality-fixture-package".into()), include_security_context: Some(true), + timeout_ms: Some(10_000), ..Default::default() }, ) From 43ca0c9378343f444dc960921e671471dd88e568 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 06:22:25 +0000 Subject: [PATCH 18/25] docs(plans): record M003 hosted requalification --- .../standalone-mcp-local-inference/003-status.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index e4728a5..40dfc24 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -120,6 +120,20 @@ qualification. structured-content/refresh behavior, and remediation or explicit acceptance of the quality failures in §7. Close M003 only after its remaining live acceptance criteria pass. +7. Current branch verification passes on `a339cce` (test-fixture-only + stabilization over implementation candidate `de69f1d`): hosted CI run + `41175183998` passed primary CI, macOS canonical/filesystem checks, and + Windows portability; release-drift, egress, and Python-native workflows + also passed. Local `make check` passed on `a339cce`; `make docs-check` and + `make bench-check` passed on its documentation-only parent `3bd46b1` before + the test-fixture-only change. The preceding documentation commit's hosted + macOS run failed the cold local-workspace search fixture; that fixture + relied on an implicit 500 ms budget derived from its 50,000-file cap. The + 10-second explicit test budget passes locally and on the hosted rerun. The + timeout explanation is an inference from the configured budget and failure + behavior; the hosted log did not print the response's timeout warning. This + CI result does not + close M003's live-model acceptance gaps. ## 6. Security and operational disposition From 6dc63d5f1a45b85352c40b26e6effbb6b9793826 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 07:43:46 +0000 Subject: [PATCH 19/25] fix(mcp): clarify compact direct fetch guidance --- docs/lm-studio.md | 12 +- .../003-status.md | 131 ++++++++++++++---- .../006-status.md | 81 +++++++++++ ...act-direct-fetch-tool-choice-corrective.md | 106 ++++++++++++++ plans/registry.md | 3 +- .../standalone-mcp-local-inference-roadmap.md | 4 +- src/mcp/server.rs | 2 +- tests/mcp_schema_slimming.rs | 23 ++- 8 files changed, 326 insertions(+), 36 deletions(-) create mode 100644 plans/closure/standalone-mcp-local-inference/006-status.md create mode 100644 plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md diff --git a/docs/lm-studio.md b/docs/lm-studio.md index 66a5299..8e2d581 100644 --- a/docs/lm-studio.md +++ b/docs/lm-studio.md @@ -88,11 +88,13 @@ success. Those require a live LM Studio chat with the actual local model. The recorded environment used LM Studio 0.4.26+4. After the user enabled authentication and configured-server access, a native `/api/v1/chat` request through `mcp/eggsearch` completed a `web_fetch` call and returned LM Studio's -usage counters; see the M003 operational record for the task and limits. The -interactive app also rendered an Eggsearch `web_fetch` result card. These -smokes do not establish a general context or completion-quality improvement: -the paired local-model trials include both successful tasks and quality -failures, and host handling of `structuredContent` versus the text fallback +usage counters; see the M003 operational record for the task and limits. A +controlled direct-fetch pair found that compact used fewer reported input +tokens and returned the same document evidence, while other specialist tasks +showed no-call, argument, and host plugin-loading failures. The interactive +app also rendered an Eggsearch `web_fetch` result card. These observations do +not establish a general context or completion-quality improvement, and host +handling of `structuredContent` versus the text fallback or profile refresh has not been isolated. For configuration keys and bounds, see [Eggsearch configuration](config.md) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index 40dfc24..cffc5ce 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M003 — Operational Status -Status: blocked (the live UI and authenticated native API paths work, but M003's host-contract characterization, broader paired task matrix, and documented quality failures remain unresolved) +Status: blocked (native API retrieval works for a controlled direct-fetch pair, but specialist-task outcomes are mixed and LM Studio UI refresh/result-shape behavior remains uncharacterized) Source implementation plan: `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` @@ -8,24 +8,26 @@ Source implementation plan: Source subsystem roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` -Final implementation candidate exercised by the installed-child smoke and native API: `de69f1ddd7a7e4acc067fb9892c438f6165cfe83` (`eggsearch 0.4.4`; M004/M005 changes included). Earlier interactive and model-turn evidence remains separately identified below. +Final prior candidate exercised by the installed-child smoke and native API: `de69f1ddd7a7e4acc067fb9892c438f6165cfe83` (`eggsearch 0.4.4`; M004/M005 changes included). The M006 corrective candidate is recorded in §12 and its closure record. Predecessor closures: M001 `d983fa5`; M002 `plans/closure/standalone-mcp-local-inference/002-status.md`. ## 1. Current finding M003 is not closed. The installed-candidate stdio smoke, interactive LM -Studio chat path, and authenticated native `/api/v1/chat` path worked with the -candidate, and all ten tools remained available. The paired direct-inference -trials included `google/gemma-4-e2b`, `prism-ml/bonsai-27b`, and -`lfm3-2b-omni-abliter8`; the required same-task comparison with a larger -reference model remains incomplete. Existing evidence does not establish -that compact is a safe default: specialist research and direct-fetch tasks -had material quality failures, and several runs timed out or exhausted the -available context. +Studio chat path, and authenticated native `/api/v1/chat` path work, and all +ten tools remain available. Native API trials now include `google/gemma-4-e2b`, +`google/gemma-4-e4b`, `prism-ml/bonsai-27b`, and +`lfm3-2b-omni-abliter8`. The controlled E4B research/security turns exposed +valid tool calls alongside missing synthesis, weak source support, invalid +arguments, and a host-side plugin-unloaded error. The direct-fetch compact +no-call outcome also did not reproduce in the controlled baseline/candidate +comparison. This mixed evidence does not establish a general quality or +reliability improvement; LM Studio UI refresh and result-shape acceptance +remain open. The final candidate's installed-child stdio smoke passed on 2026-10-31: full exposed all ten tools in 72,923 serialized `tools/list` bytes, compact -exposed the same ten in 19,281 bytes, and `provider_status` succeeded in both. +exposed the same ten in 19,370 bytes, and `provider_status` succeeded in both. These byte counts do not estimate LM Studio tokenizer usage. Model/server facts observed in LM Studio: @@ -67,14 +69,14 @@ provide a reliable per-chat usage breakdown for this test set. | Copyable installed-binary `mcp.json` setup | `docs/lm-studio.md`; config temporarily pointed at branch release binary | pass | None for configuration syntax | | Windows/macOS/Linux paths and stdio logging contract | `docs/lm-studio.md` | pass | None for documentation | | Reproducible installed-candidate stdio smoke | `scripts/lm_studio_stdio_smoke.py` | pass | Offline protocol evidence only | -| Ten tools and profile list sizes | Final-candidate initialized child-process smoke: 72,923 full / 19,281 compact bytes; all ten names; `provider_status` with probing off | pass | Byte counts are not token counts | +| Ten tools and profile list sizes | M006-candidate initialized child-process smoke: 72,923 full / 19,370 compact bytes; all ten names; `provider_status` with probing off | pass | Byte counts are not token counts | | LM Studio tool visibility and interactive tool use | Prior full/compact qualification chats called `web_search`; compact also called `security_search`, `research_search`, and `web_fetch`. The app rendered a `web_fetch mcp/eggsearch` result card. | partial | Verify refresh/new-chat behavior against the final candidate; determine whether the app consumes structured content, text fallback, or both | -| Paired small/larger model comparison | Gemma E2B and Bonsai 27B received the same extraction task; Bonsai full timed out while compact completed. Additional same-task pairs are recorded for Gemma E2B and LFM3 2B. | partial | Complete specialist/research and fetch pairs on the larger model; interpret timeouts as failed trials | -| Tool choice, argument validity, repairs, evidence and failure truthfulness | Five task records above | partial / quality concern | Investigate research argument retries, context overrun, and source classification; add paired specialist/fetch cases | -| Exact tokenizer and prompt-token attribution | Native API reports 1,575 input / 113 output tokens for one configured-server call; UI still exposes only context-meter readings and no tokenizer identity | partial | Capture same-task API usage across both profiles and characterize UI attribution | +| Paired small/larger model comparison | Gemma E2B and E4B native API trials plus earlier Gemma E2B/Bonsai 27B and LFM3 2B task pairs; see §12 | partial / mixed | Repeat larger-model specialist tasks with a refreshed MCP plugin and enough output budget to synthesize results | +| Tool choice, argument validity, repairs, evidence and failure truthfulness | Earlier interactive findings plus §12; includes no-call, invalid-argument, plugin-unloaded, and truthful evidence-gap outcomes | partial / quality concern | Re-run a balanced research/security matrix after LM Studio plugin refresh; verify source authority and answer citations | +| Exact tokenizer and prompt-token attribution | Controlled native API pair in §12 reports 1,628 input / 89 output tokens for full and 872 / 105 for candidate compact; UI still exposes no tokenizer identity | partial | Keep host usage separate from tokenizer identity and characterize the UI context meter | | Output schema / structuredContent vs text fallback | Offline protocol coverage plus a successful rendered UI tool-result card and native API tool call | partial | Isolate which result representation the UI consumes; compare a generic MCP inspector/client if available | | Protected API MCP smoke | Authenticated native `/api/v1/chat` call through configured `mcp/eggsearch`, with only `web_fetch` allowed | pass | UI structured-content/text fallback and refresh behavior remain uncharacterized | -| Config rollback | Restored original `~/.lmstudio/mcp.json`; `cmp` against backup succeeded | pass | None | +| Config rollback | Restored original `~/.lmstudio/mcp.json`; SHA-256 is `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c` | pass | None | ## 4. Work completed @@ -116,10 +118,11 @@ qualification. candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83` in CI run `41160264796`; see their closure records. The earlier local `bounded_command` stall did not reproduce in hosted CI. -6. M003 remains blocked on broader domain coverage, LM Studio UI - structured-content/refresh behavior, and remediation or explicit - acceptance of the quality failures in §7. Close M003 only after its - remaining live acceptance criteria pass. +6. M003 remains blocked on broader domain coverage and LM Studio UI + structured-content/refresh behavior. The compact direct-fetch concern in + §11 is requalified in §12 and tracked by M006; it is not a reproducible + failure in the controlled baseline/candidate pair. Close M003 only after + its remaining live acceptance criteria pass. 7. Current branch verification passes on `a339cce` (test-fixture-only stabilization over implementation candidate `de69f1d`): hosted CI run `41175183998` passed primary CI, macOS canonical/filesystem checks, and @@ -134,6 +137,13 @@ qualification. behavior; the hosted log did not print the response's timeout warning. This CI result does not close M003's live-model acceptance gaps. +8. A configured-server native API pair on 2026-10-31 (LM Studio 0.4.26+4, + `lfm3-2b-omni-abliter8`, 24,000-token context, identical task and system + prompt, same allowed tool and URL) exposed a compact tool-choice failure; + see §11. A controlled repeat on the saved baseline and M006 candidate + passed all three candidate turns and both baseline compact turns; see §12. + The original failure remains a valid negative trial, but the correction's + causal effect is not established. ## 6. Security and operational disposition @@ -323,7 +333,80 @@ figures for this request, not a general quality or token-saving claim. The user enabled the server authentication and configured-server settings; the agent did not change them. The original `~/.lmstudio/mcp.json` was restored byte-for-byte after the temporary test alias was removed. The -protected native MCP API criterion is now passed. Interactive UI -structured-content versus text fallback, config refresh behavior, broader -paired-task quality, and the documented negative model outcomes remain open, -so M003 is still blocked. +protected native MCP API smoke criterion is passed for the single-tool success +path. A later same-task profile pair is recorded in §11 and shows a repeated +compact no-call failure. Interactive UI structured-content versus text +fallback and config refresh behavior also remain open, so M003 is still +blocked. + +## 11. Native configured-server full/compact direct-fetch pair + +With the user's pause window, `~/.lmstudio/mcp.json` was temporarily pointed +at the exact `de69f1ddd7a7e4acc067fb9892c438f6165cfe83` release candidate and +full then compact profile configs. Both requests used LM Studio's native +`POST /api/v1/chat` plugin integration, not the OpenAI-compatible manual +tool-definition harness. They shared the same model (`lfm3-2b-omni-abliter8`), +task and system prompt, allowed tool, context length, temperature, and +authentication page URL. LM Studio returned loaded instance IDs ending `:3` +for full and `:2` for compact. + +The task asked the model to read the official LM Studio API authentication +page and explain where existing API-token permissions are edited. The system +instruction required exactly one `web_fetch`, prohibited setting +`response_detail`, and required evidence from that page. + +| Profile / trial | Tool call | Actual usage (input + output) | Latency | Result | +|---|---|---:|---:|---| +| Full | One valid `web_fetch` for the exact documentation URL; returned page text was 14,067 characters | 1,627 + 97 tokens | 8.93 s | Correctly named the Edit button in the API Tokens modal and cited the page. | +| Compact, trial 1 | No tool call despite the exact URL, `allowed_tools=["web_fetch"]`, and explicit system instruction | 854 + 39 tokens | 1.92 s | Returned a generic “Authentication section” answer without fetched evidence. Quality fail. | +| Compact, repeated | No tool call under the same request conditions | 854 + 39 tokens | 2.46 s | Repeated the unsupported generic answer. Quality fail. | + +The compact requests used 47.5% fewer input tokens and 59.8% fewer total +input-plus-output tokens than the full request, but skipped required retrieval +twice. These are LM Studio-reported token counts, not tokenizer-specific +attribution. This repeated difference shows the compact native API path is +not an acceptable direct-fetch default for this model and task without a +corrective decision. It does not establish that every model or task fails. + +The original `mcp.json` bytes were restored after the test block and verified +against SHA-256 +`b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. +Authentication and MCP permission settings were not changed. Corrective M006 +(`plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md`) +is active to compare the compact description and test a minimal fix. The +controlled baseline/candidate follow-up is recorded in §12. UI chat refresh +and structured-content/text-fallback behavior are still not isolated. + +## 12. M006 corrective and native specialist-task requalification + +### Compact direct-fetch baseline/candidate comparison + +On 2026-10-31, the saved pre-change executable (SHA-256 `da803489495080f30ad83095cd6f0fe56cceaa4ee1b4897a9767bf0ee1fec00a`) and rebuilt M006 candidate executable (SHA-256 `9b70544f64bc1d87a3956480c020074f3ae4e65e5002a927d83bf973b448f801`) were exercised through LM Studio 0.4.26+4 native `POST /api/v1/chat`. All turns used `lfm3-2b-omni-abliter8`, 24,000 context, temperature 0, the same system and user prompt, the same allowed `web_fetch`, and the exact same URL. The system required one fetch, forbade `response_detail`, and directed the model to answer only from the fetched evidence. + +| Build / profile / trial | Actual usage (input + output) | Latency | Tool and evidence result | +|---|---:|---:|---| +| Saved baseline / compact 1 | 855 + 110 | 7.74 s | One valid fetch; HTTP 200; stable ID `fetch_de9d81a67951e3de`; returned text included the Edit-button sentence; answer cited the API Tokens modal. | +| Saved baseline / compact 2 | 855 + 110 | 4.05 s | Same valid tool call and identical result hash; answer quoted the permission-edit sentence. | +| Candidate / full | 1,628 + 89 | 7.08 s | One valid fetch; same stable ID; 14,067-character tool result; answered correctly. | +| Candidate / compact 1 | 872 + 105 | 3.73 s | One valid fetch; same stable ID and result hash as both baseline compact turns; answer cited the official section and URL. | +| Candidate / compact 2 | 872 + 105 | 3.72 s | Same valid call and answer evidence as candidate compact 1. | + +Candidate compact used 46.4% fewer input tokens and 43.1% fewer total input-plus-output tokens than candidate full. Its initialized tools list was 19,370 serialized bytes, up 89 bytes from the prior 19,281-byte candidate and below the existing compact limit. The compact result was 3,713 serialized characters versus 14,067 for full; both carried the relevant edit-permission evidence and the same stable document identity. LM Studio's `output_truncated` flag was true for brief output, but the required sentence was present. + +The first M003 native pair in §11 had two compact no-call failures. Under this controlled prompt, both saved-baseline compact turns and both M006 compact turns called and used the same source. That does not establish that the description edit caused the change; compact tool choice is variable across the recorded prompt conditions. The candidate meets M006's direct-fetch task gate, while the broader M003 model-quality gate remains open. The exact original config hash was restored after the test block, and authentication and permission settings were not modified. + +### Gemma specialist-task observations + +Additional native LM Studio API turns used `google/gemma-4-e2b` and the larger `google/gemma-4-e4b`, both at 24,000-token context, with one allowed specialist tool and 450 maximum output tokens. The E2B research full/compact turns each exhausted the generation budget in reasoning before a tool call (510/371 input tokens; 449 output tokens, 447 reasoning tokens). Its security full/compact requests returned HTTP 500 `Plugin unloaded` without usage counters. + +E4B called `research_search` once with valid query arguments in both profiles. The full and compact results were 37,830 and 6,126 serialized characters respectively. At the 450-token output setting, neither trial synthesized a final answer. In a follow-up full-profile research turn with 1,800 output tokens, E4B called the tool and produced an answer, but the retrieved set had no primary or official source, included mostly low-confidence snippet-only records, and the answer made unsupported comparative claims. The paired compact retry returned HTTP 500 `Plugin unloaded`, so that higher-budget pair is incomplete. + +For the security task, E4B full called `security_search` without a query or ecosystem; the tool correctly returned `invalid_semantic_value`, and the model truthfully stated that applicability could not be established. E4B compact supplied valid package, version, ecosystem, query, and goal arguments; no authoritative matching advisory was retrieved, and the model correctly did not infer that the package was safe. This is partial evidence of compact specialist reachability, not a completed vulnerability assessment. These runs preserve tool errors and evidence gaps as failures or inconclusive outcomes rather than passes. + +All native API trials restored `~/.lmstudio/mcp.json` to SHA-256 +`b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. +The remaining UI refresh and structured-content/text-fallback acceptance was +not executable through the available UI automation in this environment; the +LM Studio UI native pipe startup failed. M003 remains blocked until those UI +checks and the larger-model specialist matrix are completed or explicitly +re-scoped. diff --git a/plans/closure/standalone-mcp-local-inference/006-status.md b/plans/closure/standalone-mcp-local-inference/006-status.md new file mode 100644 index 0000000..2187521 --- /dev/null +++ b/plans/closure/standalone-mcp-local-inference/006-status.md @@ -0,0 +1,81 @@ +# Standalone MCP Local-Inference M006 — Closure Status + +Status: active (local implementation and qualification pass; hosted CI pending) + +Source implementation plan: +`plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` + +Source subsystem roadmap: +`plans/subsystems/standalone-mcp-local-inference-roadmap.md` + +Implementation candidate: pending commit and hosted Linux/macOS/Windows qualification. +Native API test executable SHA-256: `9b70544f64bc1d87a3956480c020074f3ae4e65e5002a927d83bf973b448f801`. +Final local stdio-smoke executable SHA-256: `52385c8454808a0c784f980dd12e79542ac8ab10a6133ace87396c875c4814dc`. + +## 1. Corrective + +M003's authenticated native LM Studio pair recorded two compact direct-fetch +turns that skipped `web_fetch` while full fetched the explicit user URL. The +compact description was expanded in `src/mcp/server.rs` to say user-provided +URLs are valid targets, fetched text is bounded, `batch_fetch` handles several +targets, and `web_fetch` does not search or crawl. A contract test in +`tests/mcp_schema_slimming.rs` protects those details. + +The update preserves all ten tools, schemas, validation rules, defaults, and +transport behavior. Full-profile descriptions remain unchanged. + +## 2. Native LM Studio qualification + +LM Studio 0.4.26+4's authenticated native `POST /api/v1/chat` plugin path was +used with `lfm3-2b-omni-abliter8`, 24,000-token context, temperature 0, and the +same direct-fetch query, system prompt, allowed tool, and official URL for all +turns. The pre-change executable was retained for a compact baseline. + +| Build/profile/trial | Input/output tokens | Tool outcome | Evidence outcome | +|---|---:|---|---| +| Saved baseline compact 1 | 855 / 110 | One valid `web_fetch` | Correct answer; same official document and permission-edit evidence. | +| Saved baseline compact 2 | 855 / 110 | One valid `web_fetch` | Correct answer; exact result hash repeated. | +| Candidate full | 1,628 / 89 | One valid `web_fetch` | Correct answer; 14,067-character result. | +| Candidate compact 1 | 872 / 105 | One valid `web_fetch` | Correct answer cites the official section and URL. | +| Candidate compact 2 | 872 / 105 | One valid `web_fetch` | Correct answer; exact result hash repeated. | + +All calls fetched `https://lmstudio.ai/docs/developer/core/authentication`, +returned HTTP 200, and had stable document ID `fetch_de9d81a67951e3de`. The +candidate compact result was 3,713 serialized characters versus 14,067 for +full; the permission-edit sentence was present in both. Candidate compact used +46.4% fewer input tokens and 43.1% fewer total input-plus-output tokens than +candidate full. The compact tools list was 19,370 serialized bytes, below the +existing budget. The saved baseline also passed both compact turns, so these +trials do not prove the description change caused the successful tool calls. +The earlier two no-call failures remain recorded in M003 §11; M003 §12 records +the controlled follow-up and its limits. + +The original `~/.lmstudio/mcp.json` was restored byte-for-byte after the +experiment and verified at SHA-256 +`b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. +Authentication and MCP permission settings were not changed. + +## 3. Verification + +| Check | Result | +|---|---| +| `cargo fmt --check` | pass | +| `cargo test --locked --all-features --test mcp_schema_slimming` | pass, 18 tests | +| `python3 scripts/lm_studio_stdio_smoke.py target/release/eggsearch` | pass; ten tools, 72,923 full / 19,370 compact bytes; `provider_status` succeeds | +| `make check` | pass locally; 3,486 tests | +| `make docs-check` | pass locally | +| `make bench-check` | pass; optimized Criterion harness compiled with `--no-run` | +| Hosted Linux, macOS, and Windows CI | pending on the committed candidate | + +## 4. Disposition + +The candidate passes the controlled direct-fetch task gate. The baseline +comparison does not establish that the added description alone changes model +behavior; it clarifies the intended supported URL source without changing +tool behavior or breaching the compact schema budget. M006 can close after +`make bench-check` and hosted CI pass. + +M003 remains blocked on the broader task-quality matrix and LM Studio UI +refresh and result-shape behavior. The additional Gemma E2B/E4B results and +the failed UI automation attempt are recorded in +`plans/closure/standalone-mcp-local-inference/003-status.md` §12. diff --git a/plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md b/plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md new file mode 100644 index 0000000..09245b4 --- /dev/null +++ b/plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md @@ -0,0 +1,106 @@ +# M006 — Compact Direct-Fetch Tool-Choice Corrective + +Status: active +Primary class: corrective +Source roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` +Corrects: M001 compact tool presentation, as evidenced by M003 native API qualification in `plans/closure/standalone-mcp-local-inference/003-status.md` +Related ADRs: ADR-0001 and ADR-0006 +Baseline implementation: `de69f1ddd7a7e4acc067fb9892c438f6165cfe83` + +## Objective + +Correct the compact `web_fetch` disclosure if the native LM Studio paired +failure is caused by omitted workflow guidance. Preserve the small tool +palette while allowing a model to reliably fetch an explicit URL supplied by +the user and produce an evidence-grounded answer. + +## Evidence and why earlier qualification missed it + +On LM Studio 0.4.26+4, `lfm3-2b-omni-abliter8` received the same native +`/api/v1/chat` prompt, `mcp/eggsearch` integration, `allowed_tools=["web_fetch"]`, +and 24,000-token context under full and compact configs. Full called +`web_fetch` once for the exact URL and answered from its result (1,627 input, +97 output tokens). Compact made no tool call and answered from unverified +knowledge (854 input, 39 output tokens). A repeated compact turn made the same +no-call response with the same usage. The original config was restored +byte-for-byte after each swap. + +Earlier offline checks verified the ten names, schema budgets, and tool-call +contracts. The M003 direct-inference harness produced a successful compact +fetch, but it did not reproduce this configured-server API turn. The native +paired evidence therefore exposed a tool-selection gap that byte and schema +checks did not detect. + +## Scope and invariants + +- Keep all ten canonical tool names and runtime behavior unchanged. +- Keep `full` descriptions and defaults unchanged. +- Preserve compact schema size and all validation semantics. +- Change compact disclosure only when a concrete regression test and the + repeated native pair support the change. +- Do not claim model-token savings as a quality win when required retrieval + is skipped. +- No configuration migration, provider behavior change, new tool, transport, + or network listener is in scope. + +## Expected changes + +- Production: compact `web_fetch` tool description in `src/mcp/server.rs`, if + supported by the schema comparison. +- Regression guard: compact description contract in + `tests/mcp_schema_slimming.rs`. +- Operator docs: update `docs/lm-studio.md` only if the corrective changes + recommended profile guidance. +- Planning evidence: update the M003 operational record and add this plan's + M006 closure record. + +## Work packages + +1. Compare the serialized full and compact `web_fetch` definitions. Identify + which compacted description details tell a model that a user-provided URL + is a valid target and that the tool returns bounded page text. +2. If the evidence supports it, preserve those workflow details in the compact + description. Add a contract assertion in `tests/mcp_schema_slimming.rs` for + explicit user URL support and the no-search/no-crawl boundary. +3. Rebuild the exact candidate and repeat the native API pair with the same + prompt, model, allowed tool, context, temperature, and URL. Run one repeated + compact turn. Record tool calls, arguments, result identity, answer evidence, + token usage, latency, and config restoration. +4. Run the M003 focused and canonical verification commands. Requalify the + exact pushed candidate on hosted macOS, Linux, and Windows CI. +5. Update M003's closure record and this plan's closure record. Keep M003 open + if tool selection or evidence quality still fails, and register any further + corrective scope instead of changing M001/M002 history. + +## Verification + +```bash +cargo fmt --check +cargo test --locked --all-features --test mcp_schema_slimming +cargo build --locked --release --all-features +python3 scripts/lm_studio_stdio_smoke.py target/release/eggsearch +make check +make docs-check +make bench-check +``` + +Native LM Studio qualification is operational and must not become a routine +network-dependent test. No new MCP server or public listener is required. + +## Acceptance and stop conditions + +- Full and compact each call `web_fetch` exactly once for the same explicit + URL in the paired native API task. +- The compact answer cites evidence from the returned document and does not + substitute prior knowledge for a skipped tool call. +- The compact ten-tool presentation remains below its existing byte budget; + all ten names remain callable. +- LM Studio's original `mcp.json` is restored byte-for-byte and its + authentication and permission settings are unchanged. +- If repeated compact turns still skip the tool or produce materially weaker + evidence, record the failure and keep M003 open; do not claim acceptance. + +Closure evidence belongs in +`plans/closure/standalone-mcp-local-inference/006-status.md` and must include +the exact candidate SHA, native paired turns, usage and evidence findings, +tests, hosted jobs, and restored-config hash. diff --git a/plans/registry.md b/plans/registry.md index c7248ac..7add42c 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005 closed; M003 operational acceptance open) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 and M005 passed final hosted Linux/macOS/Windows gates on `de69f1d`; M003's protected native API smoke now passes, with UI host behavior and model-quality evidence still open | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `005-status.md`; M003 operational status remains `003-status.md`. | +| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005 closed; M003 operational acceptance open; M006 corrective active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004/M005 passed hosted Linux/macOS/Windows gates on `de69f1d`; M003's configured native API works, but a paired compact direct-fetch task repeatedly skipped `web_fetch` | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `005-status.md`; M003 `003-status.md`; corrective M006 is active. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -57,6 +57,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Interactive UI history, paired inference matrix, and authenticated native MCP API success are in `plans/closure/standalone-mcp-local-inference/003-status.md`. UI structured-content/refresh, broader task coverage, and documented model-quality failures remain open. | | Standalone MCP local-inference | M004 compact schema discoverability corrective | closed | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted canonical Linux, macOS, and Windows jobs pass in CI run `41160264796`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Standalone MCP local-inference | M005 brief document extraction and paired requalification | closed | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted CI run `41160264796` passed. Closure `plans/closure/standalone-mcp-local-inference/005-status.md`; M003 remains open for the documented operational gaps. | +| Standalone MCP local-inference | M006 compact direct-fetch tool-choice corrective | active | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | M003 native LM Studio API pair: full called `web_fetch`; compact skipped it twice despite the same explicit URL and allowed tool. Corrective closure must repeat the native pair and preserve original config. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index f24922c..67edb94 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001, M002, M004, and M005 closed; M003 remains open on operational acceptance) +Status: active (M001, M002, M004, and M005 closed; M003 remains open on operational acceptance; M006 compact direct-fetch corrective is active) Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. @@ -46,6 +46,7 @@ M001 compact ten-tool advertised schema profile (closed) | +-- operational --> M003 LM Studio/direct-client compatibility and quality (open acceptance) +-- corrective --> M005 brief document extraction and paired requalification (closed) + +-- corrective --> M006 compact direct-fetch tool-choice corrective (active) M004 compact schema discoverability corrective (closed on hosted Linux/macOS candidate de69f1d) ``` @@ -59,6 +60,7 @@ No CodeGG dependency; it is a regression consumer, not the implementation target | M003 | capability | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Verified LM Studio install/smoke, measured local-model context cost and completion quality, onboarding guides, and stop/rollback evidence | | M004 | capability | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Advertise runtime-accepted research goal/domain values in compact schemas; test schema parity and size; repeat focused live tasks; M003 remains open for its broader gates | | M005 | corrective | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Closed on `de69f1d`; brief preserves same-document extraction within budget; model evidence and unresolved M003 failures are recorded; hosted Linux/macOS/Windows gates pass | +| M006 | corrective | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Resolve or bound a repeated native LM Studio compact `web_fetch` no-call failure from M003; keep M003 open until evidence supports acceptance | ## Cross-cutting controls diff --git a/src/mcp/server.rs b/src/mcp/server.rs index 6936609..2df99fe 100644 --- a/src/mcp/server.rs +++ b/src/mcp/server.rs @@ -86,7 +86,7 @@ fn compact_description(name: &str) -> Option<&'static str> { "web_search" => "Search the public web for sources; use web_fetch to inspect a result.", "provider_status" => "Check configured provider availability and diagnostics.", "repo_search" => "Search repository code, documentation, issues, and releases.", - "web_fetch" => "Fetch one known HTTP(S) URL; use batch_fetch for several targets.", + "web_fetch" => "Fetch one known HTTP(S) URL from search results or user input; return bounded extracted text. Use batch_fetch for several targets; do not search or crawl.", "repo_fetch" => "Fetch a known repository file or span by structured locator.", "security_search" => "Find vulnerability evidence and assess package applicability.", "research_search" => "Gather grouped evidence for complex research questions.", diff --git a/tests/mcp_schema_slimming.rs b/tests/mcp_schema_slimming.rs index 4e69c10..d6e4b2d 100644 --- a/tests/mcp_schema_slimming.rs +++ b/tests/mcp_schema_slimming.rs @@ -19,17 +19,20 @@ fn schema_for(name: &str) -> serde_json::Value { serde_json::to_value(&tool.input_schema).expect("schema serializes") } -fn compact_schema_for(name: &str) -> serde_json::Value { +fn compact_tool_for(name: &str) -> rmcp::model::Tool { let mut config = AppConfig::default(); config.mcp.schema_profile = McpSchemaProfile::Compact; let state = Arc::new(ServerState::build(config).expect("compact state")); let server = eggsearch::mcp::EggsearchServer::new(state); - let tool = server + server .tool_definitions() .into_iter() .find(|t| t.name.as_ref() == name) - .unwrap_or_else(|| panic!("tool {name} not found")); - serde_json::to_value(&tool.input_schema).expect("schema serializes") + .unwrap_or_else(|| panic!("tool {name} not found")) +} + +fn compact_schema_for(name: &str) -> serde_json::Value { + serde_json::to_value(&compact_tool_for(name).input_schema).expect("schema serializes") } fn serialized_bytes(name: &str) -> usize { @@ -131,6 +134,18 @@ fn web_search_hides_infrastructure_controls() { ); } +#[test] +fn compact_web_fetch_explains_user_urls_and_bounded_extraction() { + let description = compact_tool_for("web_fetch") + .description + .expect("compact web_fetch description") + .to_string(); + assert!(description.contains("search results or user input")); + assert!(description.contains("bounded extracted text")); + assert!(description.contains("batch_fetch")); + assert!(description.contains("do not search or crawl")); +} + #[test] fn repo_search_advertises_goal_and_sources() { let schema = schema_for("repo_search"); From 74442ca7ddd94f952b5ffbf30624c982ded5ff23 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 08:02:52 +0000 Subject: [PATCH 20/25] docs(plans): close standalone MCP M006 --- .../standalone-mcp-local-inference/003-status.md | 5 ++++- .../standalone-mcp-local-inference/006-status.md | 11 ++++++----- ...006-compact-direct-fetch-tool-choice-corrective.md | 2 +- plans/registry.md | 6 +++--- .../standalone-mcp-local-inference-roadmap.md | 6 +++--- 5 files changed, 17 insertions(+), 13 deletions(-) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index cffc5ce..00c41cf 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -407,6 +407,9 @@ All native API trials restored `~/.lmstudio/mcp.json` to SHA-256 `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. The remaining UI refresh and structured-content/text-fallback acceptance was not executable through the available UI automation in this environment; the -LM Studio UI native pipe startup failed. M003 remains blocked until those UI +LM Studio UI native pipe startup failed. A new `cua.getState()` attempt on +2026-10-31, after the user confirmed a client pause window, returned the same +native-pipe startup failure and reported that the browser access host was +unavailable; no profile swap was made. M003 remains blocked until those UI checks and the larger-model specialist matrix are completed or explicitly re-scoped. diff --git a/plans/closure/standalone-mcp-local-inference/006-status.md b/plans/closure/standalone-mcp-local-inference/006-status.md index 2187521..468e2e7 100644 --- a/plans/closure/standalone-mcp-local-inference/006-status.md +++ b/plans/closure/standalone-mcp-local-inference/006-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M006 — Closure Status -Status: active (local implementation and qualification pass; hosted CI pending) +Status: closed Source implementation plan: `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` @@ -8,7 +8,7 @@ Source implementation plan: Source subsystem roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` -Implementation candidate: pending commit and hosted Linux/macOS/Windows qualification. +Implementation candidate: `6dc63d5f1a45b85352c40b26e6effbb6b9793826`. Native API test executable SHA-256: `9b70544f64bc1d87a3956480c020074f3ae4e65e5002a927d83bf973b448f801`. Final local stdio-smoke executable SHA-256: `52385c8454808a0c784f980dd12e79542ac8ab10a6133ace87396c875c4814dc`. @@ -65,15 +65,16 @@ Authentication and MCP permission settings were not changed. | `make check` | pass locally; 3,486 tests | | `make docs-check` | pass locally | | `make bench-check` | pass; optimized Criterion harness compiled with `--no-run` | -| Hosted Linux, macOS, and Windows CI | pending on the committed candidate | +| Hosted Linux, macOS, and Windows CI | pass on exact candidate `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; CI run [`41181749609`](https://github.com/eggstack/eggsearch/actions/runs/41181749609) passed canonical Linux and macOS verification plus Windows all-features portability and macOS filesystem portability checks | ## 4. Disposition The candidate passes the controlled direct-fetch task gate. The baseline comparison does not establish that the added description alone changes model behavior; it clarifies the intended supported URL source without changing -tool behavior or breaching the compact schema budget. M006 can close after -`make bench-check` and hosted CI pass. +tool behavior or breaching the compact schema budget. Focused, canonical, +documentation, benchmark-compilation, installed-child, and hosted checks all +pass for the implementation candidate. M003 remains blocked on the broader task-quality matrix and LM Studio UI refresh and result-shape behavior. The additional Gemma E2B/E4B results and diff --git a/plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md b/plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md index 09245b4..7e4113e 100644 --- a/plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md +++ b/plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md @@ -1,6 +1,6 @@ # M006 — Compact Direct-Fetch Tool-Choice Corrective -Status: active +Status: closed Primary class: corrective Source roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` Corrects: M001 compact tool presentation, as evidenced by M003 native API qualification in `plans/closure/standalone-mcp-local-inference/003-status.md` diff --git a/plans/registry.md b/plans/registry.md index 7add42c..d54e54b 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005 closed; M003 operational acceptance open; M006 corrective active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004/M005 passed hosted Linux/macOS/Windows gates on `de69f1d`; M003's configured native API works, but a paired compact direct-fetch task repeatedly skipped `web_fetch` | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `005-status.md`; M003 `003-status.md`; corrective M006 is active. | +| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005, and M006 closed; M003 blocked) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004/M005 passed hosted Linux/macOS/Windows gates on `de69f1d`; M006 closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826` with hosted CI run `41181749609`; M003 still lacks broader task-quality and UI acceptance. | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `006-status.md`; M003 remains blocked in `003-status.md`. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -57,7 +57,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Interactive UI history, paired inference matrix, and authenticated native MCP API success are in `plans/closure/standalone-mcp-local-inference/003-status.md`. UI structured-content/refresh, broader task coverage, and documented model-quality failures remain open. | | Standalone MCP local-inference | M004 compact schema discoverability corrective | closed | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted canonical Linux, macOS, and Windows jobs pass in CI run `41160264796`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Standalone MCP local-inference | M005 brief document extraction and paired requalification | closed | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted CI run `41160264796` passed. Closure `plans/closure/standalone-mcp-local-inference/005-status.md`; M003 remains open for the documented operational gaps. | -| Standalone MCP local-inference | M006 compact direct-fetch tool-choice corrective | active | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | M003 native LM Studio API pair: full called `web_fetch`; compact skipped it twice despite the same explicit URL and allowed tool. Corrective closure must repeat the native pair and preserve original config. | +| Standalone MCP local-inference | M006 compact direct-fetch tool-choice corrective | closed | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Candidate `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; closure `plans/closure/standalone-mcp-local-inference/006-status.md`; native paired and repeated compact turns passed; hosted Linux/macOS/Windows CI run `41181749609` passed. M003 remains blocked on broader task quality and UI acceptance. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | @@ -128,7 +128,7 @@ Corrective closure sequence: `plans/implementation/dependency-evidence-hardening **Python native library gate:** ADR-0005 authorizes a narrow direct Rust SDK facade and Python PyO3 native package without broad Rust module stability. M001–M008 are closed at `067468a` with fully-green qualifying run `38641009246` (5/5 builds, 20/20 smokes, offline sdist rebuild, green audit, digests in `008-status.md`). Earlier runs are historical/deficiency/supporting evidence only. First PyPI publication is separately closed under `python-pypi-distribution` M003 at 0.4.4; Eggpack binary assets remain unchanged. -**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; the compact ten-tool stdio list is 18,545 bytes, discovery remains 43/43, and brief response budgets preserve required failure/applicability signals. The offline installed-child smoke passes. Five interactive LM Studio task records on `google/gemma-4-e2b` show successful web and security calls and one paired web task with a 44.2-point lower context-meter reading under compact; research overran the 10,703-token context and misclassified a secondary source, and direct fetch produced only one of two requested concrete changes. M004 added compact research goal/domain enums and passes focused offline checks at `776f8ca4a8246e467041696b2220600eb7b2648d`; its post-fix live turn remains unverified because LM Studio refresh could not be confirmed. No larger-model comparison or exact tokenizer/token attribution exists. M003 and M004 remain blocked on live corrective quality evidence, broader paired comparison, exact token and host behavior evidence, and protected API qualification (the original MCP config has been restored; no auth or permission setting was changed). Offline wire bytes and this single UI-meter pair are not general token savings. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. +**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 and M005 are closed at `de69f1d`; M006 is closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826` after repeated authenticated native LM Studio compact direct-fetch turns and hosted Linux/macOS/Windows CI run `41181749609`. M003 remains blocked: the Gemma E2B/E4B specialist-task trials have mixed tool-call and evidence-quality outcomes, and LM Studio UI refresh plus structured-content/fallback behavior remain uncharacterized because the computer-use host is unavailable. See the M003 and M006 closure records for the complete trial evidence and limits. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. **Tool-surface gate:** M001-M007 closed. Do not run evaluation against pre-consolidation bytes and present it as consolidation diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index 67edb94..01d9152 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001, M002, M004, and M005 closed; M003 remains open on operational acceptance; M006 compact direct-fetch corrective is active) +Status: active (M001, M002, M004, M005, and M006 closed; M003 blocked on operational acceptance) Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. @@ -44,9 +44,9 @@ M001 compact ten-tool advertised schema profile (closed) | +-- hard --> M002 brief/budgeted agent responses (closed) | - +-- operational --> M003 LM Studio/direct-client compatibility and quality (open acceptance) + +-- operational --> M003 LM Studio/direct-client compatibility and quality (blocked on live acceptance) +-- corrective --> M005 brief document extraction and paired requalification (closed) - +-- corrective --> M006 compact direct-fetch tool-choice corrective (active) + +-- corrective --> M006 compact direct-fetch tool-choice corrective (closed at `6dc63d5`; hosted CI run `41181749609`) M004 compact schema discoverability corrective (closed on hosted Linux/macOS candidate de69f1d) ``` From fe677ff45a4c69feee50907cfbd451b6ae2a1fd4 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 15:45:03 +0000 Subject: [PATCH 21/25] fix(mcp): retain compact schema references --- docs/lm-studio.md | 26 ++- .../003-status.md | 153 +++++++++++++++--- .../007-status.md | 62 +++++++ ...-studio-consumer-qualification-and-docs.md | 34 ++-- ...007-compact-schema-reference-corrective.md | 49 ++++++ plans/registry.md | 9 +- .../standalone-mcp-local-inference-roadmap.md | 24 +-- src/mcp/server.rs | 10 +- tests/mcp_schema_slimming.rs | 53 +++++- 9 files changed, 348 insertions(+), 72 deletions(-) create mode 100644 plans/closure/standalone-mcp-local-inference/007-status.md create mode 100644 plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md diff --git a/docs/lm-studio.md b/docs/lm-studio.md index 8e2d581..332fe62 100644 --- a/docs/lm-studio.md +++ b/docs/lm-studio.md @@ -82,20 +82,18 @@ See LM Studio's [MCP app guide](https://lmstudio.ai/docs/app/mcp), ## What the measurements establish Eggsearch's offline fixtures measure serialized schema and result sizes and -exercise the MCP protocol. They do not establish LM Studio tokenizer counts, -model-visible prompt inclusion, tool-choice quality, or end-to-end task -success. Those require a live LM Studio chat with the actual local model. The -recorded environment used LM Studio 0.4.26+4. After the user enabled -authentication and configured-server access, a native `/api/v1/chat` request -through `mcp/eggsearch` completed a `web_fetch` call and returned LM Studio's -usage counters; see the M003 operational record for the task and limits. A -controlled direct-fetch pair found that compact used fewer reported input -tokens and returned the same document evidence, while other specialist tasks -showed no-call, argument, and host plugin-loading failures. The interactive -app also rendered an Eggsearch `web_fetch` result card. These observations do -not establish a general context or completion-quality improvement, and host -handling of `structuredContent` versus the text fallback or profile refresh -has not been isolated. +exercise the MCP protocol. They do not establish model context use, tool-choice +quality, or end-to-end task success. LM Studio's native `/api/v1/chat` endpoint +can invoke configured MCP servers and returns actual tool calls, tool results, +and usage counters. Its `stats.input_tokens` includes formatting, tool +definitions, and prior messages; see the [native chat API +docs](https://lmstudio.ai/docs/developer/rest/chat). Use paired requests with +the same model, prompt, context, and allowed tool set to compare profiles. +The M003 record contains LFM3-2B and Gemma E2B trials showing reduced input +usage and result size with compact/brief, along with mixed answer quality and +host/plugin failures. These are model-use experiments through LM Studio as the +available local inference host; app UI refresh and tool-result presentation +are outside the qualification scope. For configuration keys and bounds, see [Eggsearch configuration](config.md) and the [context-efficiency measurements](retrieval-quality-context-efficiency.md). diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index 00c41cf..e1d8498 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M003 — Operational Status -Status: blocked (native API retrieval works for a controlled direct-fetch pair, but specialist-task outcomes are mixed and LM Studio UI refresh/result-shape behavior remains uncharacterized) +Status: active (scope clarified: qualify small-model tool use and context reduction through actual API turns; UI refresh/result-shape behavior is out of scope; paired specialist-task outcomes remain mixed) Source implementation plan: `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` @@ -22,8 +22,10 @@ valid tool calls alongside missing synthesis, weak source support, invalid arguments, and a host-side plugin-unloaded error. The direct-fetch compact no-call outcome also did not reproduce in the controlled baseline/candidate comparison. This mixed evidence does not establish a general quality or -reliability improvement; LM Studio UI refresh and result-shape acceptance -remain open. +reliability improvement. The qualification scope was clarified on 2026-10-31: +LM Studio is the available local API host for MLX and other local models, while +the target is small-model tool-use quality and context cost. App UI refresh and +presentation behavior are not acceptance criteria. The final candidate's installed-child stdio smoke passed on 2026-10-31: full exposed all ten tools in 72,923 serialized `tools/list` bytes, compact @@ -70,12 +72,11 @@ provide a reliable per-chat usage breakdown for this test set. | Windows/macOS/Linux paths and stdio logging contract | `docs/lm-studio.md` | pass | None for documentation | | Reproducible installed-candidate stdio smoke | `scripts/lm_studio_stdio_smoke.py` | pass | Offline protocol evidence only | | Ten tools and profile list sizes | M006-candidate initialized child-process smoke: 72,923 full / 19,370 compact bytes; all ten names; `provider_status` with probing off | pass | Byte counts are not token counts | -| LM Studio tool visibility and interactive tool use | Prior full/compact qualification chats called `web_search`; compact also called `security_search`, `research_search`, and `web_fetch`. The app rendered a `web_fetch mcp/eggsearch` result card. | partial | Verify refresh/new-chat behavior against the final candidate; determine whether the app consumes structured content, text fallback, or both | -| Paired small/larger model comparison | Gemma E2B and E4B native API trials plus earlier Gemma E2B/Bonsai 27B and LFM3 2B task pairs; see §12 | partial / mixed | Repeat larger-model specialist tasks with a refreshed MCP plugin and enough output budget to synthesize results | -| Tool choice, argument validity, repairs, evidence and failure truthfulness | Earlier interactive findings plus §12; includes no-call, invalid-argument, plugin-unloaded, and truthful evidence-gap outcomes | partial / quality concern | Re-run a balanced research/security matrix after LM Studio plugin refresh; verify source authority and answer citations | -| Exact tokenizer and prompt-token attribution | Controlled native API pair in §12 reports 1,628 input / 89 output tokens for full and 872 / 105 for candidate compact; UI still exposes no tokenizer identity | partial | Keep host usage separate from tokenizer identity and characterize the UI context meter | -| Output schema / structuredContent vs text fallback | Offline protocol coverage plus a successful rendered UI tool-result card and native API tool call | partial | Isolate which result representation the UI consumes; compare a generic MCP inspector/client if available | -| Protected API MCP smoke | Authenticated native `/api/v1/chat` call through configured `mcp/eggsearch`, with only `web_fetch` allowed | pass | UI structured-content/text fallback and refresh behavior remain uncharacterized | +| Configured MCP API path | Authenticated native `/api/v1/chat` calls through configured `mcp/eggsearch`, with task-specific `allowed_tools`; exact config rollback hash recorded | pass | None for the tested local host; this verifies the model/API path, not LM Studio UI behavior | +| Paired small-model comparison | LFM3 2B direct-fetch trials in §12 and repeated security-applicability trials in §13; Gemma E2B/E4B observations in §12 | partial / mixed | Expand paired coverage across research and extraction; retain API/plugin failures as infrastructure outcomes, separate from model failures | +| Tool choice, argument validity, evidence and failure truthfulness | §12-13 include valid and invalid calls, no-calls, plugin/API failures, evidence gaps, and both correct and incorrect applicability conclusions | partial / quality concern | Complete repeatable paired task coverage and assess evidence quality across more than one small model | +| Actual context/token reduction | LM Studio native API reports `input_tokens` including formatting, tool definitions, and prior messages; §12-13 record paired full/compact usage and tool-result lengths | pass for measured tasks | Extend measurements across the remaining paired task set; no tokenizer identity or UI context meter is required | +| Eggsearch result contract | Offline MCP protocol tests cover structured results and text fallback; native API exposes returned tool output to the model | pass for Eggsearch contract | No host-specific result-shape or UI-consumption test is required | | Config rollback | Restored original `~/.lmstudio/mcp.json`; SHA-256 is `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c` | pass | None | ## 4. Work completed @@ -95,7 +96,12 @@ negative findings rule out closure. M004 is registered to expose runtime- accepted research goal/domain values in compact schemas and repeat the focused qualification. -## 5. Remaining evidence to unblock and close +## 5. Earlier remaining-evidence list + +The following list records the state before the scope clarification in §13. Its +references to UI behavior as a gate are superseded; UI refresh, UI result cards, +and LM Studio's internal structured-content/fallback selection are not M003 +acceptance criteria. 1. Initial paired full+diagnostic and compact+brief API trials on Gemma E2B and Bonsai 27B, including extraction and security failure interpretation, @@ -211,7 +217,7 @@ schema, but the LM Studio request timed out after 300 seconds before returning a tool call, final answer, or usage counters. This is recorded as a failed retry, not as a pass; the initial invalid call remains in the table above. -### Remaining M003 acceptance +### Remaining M003 acceptance (historical assessment before §13) The paired model matrix now has exact prompt/completion usage for successful requests and explicit timeouts where the host returned no counter, including @@ -220,9 +226,9 @@ does not pass M003: the full Gemma extraction ended blank, Gemma's security answer misstated evidence authority, Bonsai's security call failed argument validation, and Bonsai full-profile inference timed out. Native LM Studio UI tool-result handling and refresh behavior, structured-content/text fallback, -and broader domain coverage remain open. The protected API smoke is later -qualified in §10. Keep M003 blocked until remaining criteria pass or are -explicitly re-scoped by a future plan. +and broader domain coverage remained open at that time. The protected API +smoke is later qualified in §10. The UI-specific criteria in this historical +assessment are superseded by the scope clarification in §13. ## 8. LFM3-2B paired search and extraction trials @@ -405,11 +411,114 @@ For the security task, E4B full called `security_search` without a query or ecos All native API trials restored `~/.lmstudio/mcp.json` to SHA-256 `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. -The remaining UI refresh and structured-content/text-fallback acceptance was -not executable through the available UI automation in this environment; the -LM Studio UI native pipe startup failed. A new `cua.getState()` attempt on -2026-10-31, after the user confirmed a client pause window, returned the same -native-pipe startup failure and reported that the browser access host was -unavailable; no profile swap was made. M003 remains blocked until those UI -checks and the larger-model specialist matrix are completed or explicitly -re-scoped. +The UI refresh and result-presentation checks could not be exercised because +the computer-use host was unavailable; no profile swap was made during that +attempt. This was recorded as a blocker before the user's scope clarification +in §13 and is now historical evidence only. + +## 13. Qualification scope update and repeated LFM3 API pair + +On 2026-10-31, the user clarified that M003 qualifies small local models' +ability to use Eggsearch tools appropriately while the opt-in profiles reduce +context. LM Studio is only the available local inference/API host here, +including for MLX support. Its UI refresh, chat presentation, and internal +choice between MCP structured content and text fallback are not acceptance +criteria. This scope supersedes the earlier UI-related remaining-work wording +above; the recorded UI automation failure is retained as history, not as an +M003 blocker. + +LM Studio's native `POST /api/v1/chat` API supports configured MCP plugins and +returns tool calls/results plus usage counters. Its documented +`stats.input_tokens` includes formatting and tool definitions, making paired +full/compact requests a direct context-cost measure for the selected local +model. Trials below used LM Studio 0.4.26+4, `lfm3-2b-omni-abliter8`, +24,000-token context, temperature 0, `max_output_tokens=1000`, `store=false`, +the same prompt and system instruction, and only `security_search` allowed. +The candidate executable was `target/release/eggsearch`, SHA-256 +`52385c8454808a0c784f980dd12e79542ac8ab10a6133ace87396c875c4814dc`. Full +used `schema_profile="full"`; compact used `schema_profile="compact"` and +`brief_max_chars=8000`. The exact original `mcp.json` hash was restored after +the trials. + +The task asked whether Apache Log4j 2.14.1 is affected by CVE-2021-44228 and +required an evidence-grounded applicability conclusion with failures retained. +The primary Apache security page lists affected Log4j versions as +`[2.0-beta9, 2.15.0)`, which includes 2.14.1, and states that `log4j-core` is +the affected component: https://logging.apache.org/security.html. + +| Profile / trial | API input / output tokens | `security_search` result characters | Model disposition | +|---|---:|---:|---| +| Full, trial 1 | 682 / 450 | 32,659 | Valid call. Answer said applicability was not confirmed despite citing an NVD description whose affected range includes 2.14.1; it preserved provider failures. Incorrect/internally inconsistent conclusion. | +| Compact, trial 1 | 454 / 386 | 7,277 | Valid call with CVE query. Correctly said affected and cited NVD and vendor advisory; retained Yahoo 500 and unavailable native OSV lookup. | +| Full, repeat | 682 / 824 | 46,665 | Valid call. Answer again treated applicability as unconfirmed based on weak/low-confidence results; it noted Yahoo failures. Incorrect conclusion against the authoritative affected-version range. | +| Compact, repeat | 454 / 480 | 7,334 | Valid call with CVE query. Correctly said affected and cited the NVD and vendor advisory; retained Yahoo 500 and unavailable OSV lookup. | + +Across these paired turns, compact input usage was 33.4% lower. Brief result +length was 77.7% and 84.3% lower in the two repeats. Combined input plus +output usage was 25.8% and 38.0% lower per corresponding trial. Compact +preserved the decisive advisory passage and provider-failure uncertainty, and +the model applied the version range correctly in both runs; full produced a +longer result but failed to apply the evidence in both runs. These are two +paired observations for one model/task, not a general model-quality claim. + +A paired `web_search` task was not scored: full returned HTTP 500 with an +MCP structured-content/output-schema validation error (`query` and `results` +missing), while compact returned HTTP 500 `Plugin unloaded`. Neither provided +a model answer or usage counters, so these are recorded as API/tool-path +failures rather than model outcomes. + +The current remaining work is to repeat the paired task matrix for other small +task classes (especially research and document extraction), separate +model/tool-call failures from host/plugin failures, and assess whether the +measured context reduction preserves adequate task quality overall. Current +paired security evidence covers two small models; it is promising but narrow. +M003 is active, not blocked on the UI. The current acceptance objective and +method are in `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md`. + +### Gemma E2B paired security trial + +A second small model, `google/gemma-4-e2b`, was run through the same configured +MCP API path with the identical Log4j applicability task and system prompt. +The run used LM Studio 0.4.26+4, 10,000-token context, temperature 0, +`max_output_tokens=1200`, `store=false`, and only `security_search` allowed. +The same candidate binary and full/compact profile TOML settings as the LFM3 +pair above were used. The original `mcp.json` hash was restored and verified +after the pair. + +| Profile | API input / output tokens | `security_search` result characters | Model disposition | +|---|---:|---:|---| +| Full | 615 / 391 | 46,241 | Valid call with CVE, package, and query. It focused on Yahoo 500 failures and declined to conclude applicability, although the returned diagnostic result included advisory evidence. | +| Compact | 356 / 534 | 7,758 | Valid call with CVE, package, query, goal, and version. Correctly applied the affected range 2.0-beta9–2.15.0 to version 2.14.1 and cited the NVD source. | + +Compact used 42.1% fewer input tokens and returned 83.2% fewer result +characters. Combined input and output tokens were 11.5% lower (890 versus +1,006); the compact turn used more completion tokens to give the correct +evidence-based conclusion. This matches the LFM3 direction on the same task, +but remains one pair for this model and one security case. + +### All-tools palette pair after M007 schema correction + +The M003 task was repeated with no `allowed_tools` filter, exposing the full +ten-tool palette to `lfm3-2b-omni-abliter8`. The candidate executable included +the M007 fix (SHA-256 +`ff98098566a89f0603d1c07ad3cd88e5d36fae830bc8d6bde65cebc3456a71ea`). Both +requests used a 24,000-token context, temperature 0, the same Log4j/CVE task, +the same system instruction, and `store=false`. + +| Profile / trial | API input / prompt tokens | Tool outcome | Model disposition | +|---|---:|---|---| +| Full, all tools | 34,779 prompt tokens; rejected against 24,064 available | No tool call; context overrun | No answer generated; cannot score model tool use. | +| Compact, all tools, after full request | No usage; HTTP 500 `Plugin unloaded` | No tool call | Host/plugin failure, not scored as a model outcome. | +| Compact, all tools, fresh compact-first request | 5,365 input tokens; 362 output tokens | Valid `security_search`; arguments included applicability intent, Log4j package, and CVE query; result was 7,215 characters | Correctly applied the NVD affected range to 2.14.1, cited the advisory, and retained Yahoo's 500 failure. | + +The compact-first request passed the schema validation point that had failed +before M007: all ten tools were exposed and the API completed the specialist +call. Full exceeded this model's 24k context by 10,715 tokens. The 5,365 input +counter is the successful API response's usage value; the full value is the +inference engine's rejected prompt count. Treat this as evidence that the +compact palette fit and was usable in this task, not as a general percentage +savings or proof that every compact tool schema is accepted by every host. +The earlier post-full `Plugin unloaded` response remains recorded as a host +failure; the exact original MCP config was restored byte-for-byte after each +request block (SHA-256 +`b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`). diff --git a/plans/closure/standalone-mcp-local-inference/007-status.md b/plans/closure/standalone-mcp-local-inference/007-status.md new file mode 100644 index 0000000..9b763d9 --- /dev/null +++ b/plans/closure/standalone-mcp-local-inference/007-status.md @@ -0,0 +1,62 @@ +# Standalone MCP Local-Inference M007 — Closure Status + +Status: active (fix and local requalification pass; hosted qualification pending) + +Source implementation plan: +`plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` + +Source subsystem roadmap: +`plans/subsystems/standalone-mcp-local-inference-roadmap.md` + +## 1. Discovery + +During M003's all-tools local-model API comparison, the full profile exceeded +the selected 24,000-token context with a 34,428-token request. The compact +request failed constrained decoding with `cannot resolve $ref +#/$defs/ResponseDetail, $defs not found`. + +Inspection of the exact compact tool schemas found the cause in +`build_evidence_bundle`: its compact schema rewrite drops the root `$defs` +object while retaining the `response_detail` property that references +`ResponseDetail`. The failure is an Eggsearch-advertised schema defect. Earlier +schema tests did not resolve local references after the rewrite, and earlier +API tests restricted the allowed tool set. + +## 2. Candidate and verification + +Implementation candidate: `ff98098566a89f0603d1c07ad3cd88e5d36fae830bc8d6bde65cebc3456a71ea` +(SHA-256 of the rebuilt release executable). + +The compact schema rewrite now retains the referenced `ResponseDetail` +definition. The regression test walks every compact tool schema and resolves +all local references against its root. + +Local verification: + +- `cargo test --locked --all-features --test mcp_schema_slimming`: 19 passed. +- `make check`: passed, including formatting, clippy, feature checks, all tests, + hygiene, dependency policy, and packaging checks. +- Compact executable smoke: all ten tools listed; full definitions measured + 72,923 bytes and compact definitions 19,494 bytes; `provider_status` + completed successfully. +- Original `~/.lmstudio/mcp.json` was restored byte-for-byte; SHA-256 + `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. + +All-tools LM Studio native API requalification used +`lfm3-2b-omni-abliter8`, a 24,000-token context, and no `allowed_tools` filter. +The full palette was rejected at 34,779 prompt tokens against 24,064 +available. After the fix, a fresh compact-first request used 5,365 input and +362 output tokens, called `security_search` with valid arguments, and returned +the correct Log4j 2.14.1 affected-range conclusion with an advisory citation +while preserving Yahoo's 500 failure. This confirms compact schema +initialization and appropriate tool use for this task; it is one task, not the +broader M003 model/task matrix. + +`make docs-check` and hosted CI for the committed implementation remain +pending. + +## 3. Disposition + +M007 remains active until hosted qualification passes and the result is recorded +against the exact committed candidate. M003 remains active and continues to own +the broader small-model tool-use and context qualification. diff --git a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md index fc25c70..74d57d6 100644 --- a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md +++ b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md @@ -1,6 +1,6 @@ -# M003 — LM Studio Direct-MCP Qualification and Consumer Documentation +# M003 — Small Local-Model Tool-Use and Context Qualification -Status: blocked (M001/M002 dependencies closed; interactive and authenticated native API paths work, but larger-model paired tasks, profile-specific native API usage, UI refresh and result-shape characterization, and model-quality disposition remain incomplete; see `plans/closure/standalone-mcp-local-inference/003-status.md`) +Status: active (M001/M002 dependencies closed; native API experiments establish a working harness, but paired small-model task coverage and quality disposition remain incomplete; see `plans/closure/standalone-mcp-local-inference/003-status.md`) Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (rebase baseline to exact M002 candidate at start) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` @@ -8,28 +8,28 @@ ADRs: ADR-0001 and ADR-0006. No CodeGG implementation work. ## Objective -Prove the new opt-in compact schema and brief result modes actually improve local-model use in LM Studio while preserving the ability to call all ten Eggsearch tools, understand retrieval uncertainty and fetch needed evidence. Deliver a reproducible installed-binary LM Studio configuration and tested troubleshooting workflow, rather than relying on simulated schema sizes. +Qualify whether small locally hosted models can select Eggsearch tools, provide valid arguments, and produce evidence-grounded answers while the opt-in `compact` tool schema and `brief` result mode reduce model context use. LM Studio is the available local inference host for these experiments, including MLX-backed models; it is a test harness, not the product whose UI behavior this milestone qualifies. Deliver reproducible paired trials comparing `full + diagnostic` with `compact + brief` on the same model, task, and evidence sources. ## Research/consumer constraints -LM Studio documents local/remote MCP servers in `mcp.json` and warns about context-heavy servers: https://lmstudio.ai/docs/app/mcp . As of the researched documentation, LM Studio API MCP integrations can select configured servers and restrict visible tools through `allowed_tools`, which can reduce prompt costs: https://lmstudio.ai/docs/developer/core/mcp . Verify feature/version behavior during implementation; do not assume `allowed_tools` in the API implies interactive chat dynamically fetches tool schemas, or that all clients use both `structuredContent` and JSON text. +The local inference host must expose actual model turns and usage counters through an API that can invoke the configured MCP server. LM Studio currently provides that path in this environment and supports MLX models. Host UI refresh, interactive chat presentation, and internal handling of `structuredContent` versus text fallback are outside this milestone; record host/version details only for reproducibility. Use MCP protocol tests for Eggsearch's own result contract. Eggsearch default binary transport is client-owned `eggsearch mcp stdio`; optional HTTP is loopback-only via `eggsearch mcp serve`. Do not make local inference dependent on starting a separate cloud service, building on SBCs, or a proprietary UI plugin. Use the actual CLI syntax and executable-path quoting on each OS before publishing examples. ## Ordered work packages -1. Create `docs/integrations/lm-studio.md` (or a well-named existing operator doc section): copyable `mcp.json` JSON for an **installed binary** with the opt-in compact profile (pass `--config` to the actual CLI or use supported env/config controls). Provide standard full-profile fallback, Windows PowerShell executable path, macOS and Linux paths, stdio stdout-is-protocol / stderr-is-logs expectations, and one loopback-HTTP alternative if client support is confirmed. Never hardcode private paths or credentials; no remote/public bind. -2. Record an exact SHA, binary version, LM Studio version/build, local model and context length; enumerate MCP `tools/list` from the actual child process and record schema bytes, tokenizer counts, advertised names and prompt-use attribution. Compare `full + diagnostic`, `compact + brief` and optional client-side `allowed_tools` to avoid confusing profile savings with fewer enabled tools. -3. Build deterministic offline fixtures that replay realistic tool calls through the real MCP stdio server (loopback HTTP as additional protocol qualifier): web search, page/PDF fetch, repo search + specific fetch, research grouping, advisory/version applicability, grouped batch failures, and an evidence bundle. Mock upstream calls in CI; no credential or external network requirement. Confirm status-first brief and explicit diagnostic fallback. -4. Run optional interactive LM Studio qualification with at least one smaller tool-capable local model and one larger reference local model (model sizes are examples, not hard requirements). Use >= 8k context if the application permits, record exact tokenizer and actual observed prompt token usage. Compare tool-choice success, argument validity, repair/retry count, target evidence found, truthful failure interpretation, calls to answer, total latency and prompt-processing time on the same prompts. Include tasks requiring `security_search` and `research_search` to demonstrate capability preservation. Do not rely only on 43-case offline keyword ranking results. -5. Characterize how LM Studio actually handles `outputSchema`, `structuredContent` versus text fallback, `tools/list` refreshing and tool visibility across new chat/session after config changes. Record discrepancies and a verified rollback path. Compare at least one other generic MCP inspector/client if available; keep any vendor-specific advice in docs rather than adding vendor detection to Eggsearch. -6. Implement only minimal fixes necessary for discovered compatibility defects in the already-owned profile or projection modules; otherwise open new corrective milestones and keep the current M003 unclosed. Update `docs/integrations.md`/`docs/installation.md`/`README.md`, `docs/test-inventory.md`, `architecture/mcp.md` as appropriate. Explain optional client `allowed_tools` filtering as an intentional narrowing of per-chat discovery, NOT a server capability loss. +1. Keep `docs/lm-studio.md` as optional host setup guidance for an installed binary, but do not make LM Studio UI behavior a closure dependency. Record the selected `mcp.json` profile and rollback hash for reproducibility without retaining credentials. +2. Record the exact Eggsearch candidate, host/API version, model identifier and quantization, context length, prompt, profile, and task data. Confirm all ten tools remain listed in the child-process smoke. Capture API-reported input/output tokens and returned tool calls/results; use serialized schema/result bytes as supporting measurements, not token estimates. +3. Compare `full + diagnostic` with `compact + brief` using paired trials on the same small model, exact prompt, allowed tool set, URL/query, and source document where applicable. Include direct fetch, web search, `research_search`, `security_search`, and at least one failure/applicability case. Repeat key cases to expose tool-choice variability. Include a larger reference model only as context; the acceptance target is small local models. +4. Score each turn for required-tool selection, argument validity, repairs/retries, evidence completeness and authority, truthful treatment of retrieval failures/uncertainty, successful task completion, API usage, and latency where consistently available. Report model/task variation; do not infer a general quality win from one successful pair or claim savings from wire bytes alone. +5. Keep deterministic offline MCP/schema/response contract tests as the regression gate. No model download or network-dependent model test belongs in routine CI. Use existing protocol tests to verify result shape; no LM Studio UI refresh or generic inspector task is required. +6. Implement only minimal fixes required by reproducible qualification defects, with focused fixtures. Otherwise leave code unchanged and report the measured tradeoff. Update setup docs only when commands or operator guidance changed. ## Compatibility, security and resources -Full mode must continue working in ordinary MCP clients and CodeGG; no provider or transport regression, secret/proxy bypass, config-driven SSRF relaxation, startup service change, or new live agent backend in Eggsearch. Test prompt-injection markers and retrieval-failure truthfulness through the exact text visible to LM Studio. No raw diagnostic JSON dumps in brief mode unless explicitly requested. Do not claim local-model benchmark wins without actual measured outcomes. +Full mode must continue working in ordinary MCP clients and CodeGG; no provider or transport regression, secret/proxy bypass, config-driven SSRF relaxation, startup service change, or new live agent backend in Eggsearch. Test prompt-injection markers and retrieval-failure truthfulness through the exact tool result returned to the model. No raw diagnostic JSON dumps in brief mode unless explicitly requested. Do not claim local-model benchmark wins without actual measured outcomes. -Test a fresh installation from precompiled release assets where available; no compilation on low-resource ARM boards. Offline smoke can run on normal hosted CI. A live LM Studio instance is an operational acceptance dependency for closing the consumer claim; if unavailable, document an unexecuted gate and leave M003 blocked/closing accordingly, without misrepresenting offline results. +Test a fresh installation from precompiled release assets where available; no compilation on low-resource ARM boards. Offline smoke can run on normal hosted CI. A live small-model API trial is an operational acceptance dependency for the model-use claim; an unavailable UI is not a blocker if the API returns actual model turns, tool calls, tool results, and usage counters. ## Verification @@ -46,14 +46,14 @@ make docs-check make bench-check ``` -Check exact suite names against `architecture/testing.md` before running; add a dedicated installed-binary/LM Studio compatibility test runner if the existing MCP protocol tests cannot exercise the targeted path. Avoid a permanent CI dependency on LM Studio or a model download. +Check exact suite names against `architecture/testing.md` before running; keep live model API trials operational and opt-in, and avoid a permanent CI dependency on LM Studio or a model download. ## Acceptance / stop -- A validated `mcp.json` configuration from a published or candidate installed Eggsearch executable connects and the host sees all ten tool names in compact presentation; both generic web and specialist tasks run correctly. -- Capture **actual model-visible** schema/result token counts for full vs compact, plus bytes, tool-call failures, completed-task evidence and latency; do not substitute byte/4 estimates for tokenizer or hosted context metrics. Report negative outcomes honestly. +- A validated local inference API path invokes a candidate installed Eggsearch executable and exposes all ten tools in compact presentation; the paired task set covers generic retrieval and specialist tools. +- Repeated small-model trials capture host-reported input/output tokens for full versus compact and diagnostic versus brief, plus bytes, tool-call outcomes, completed-task evidence, and latency where available; do not substitute byte/4 estimates. Report negative outcomes honestly. - No capability, safety, inference-free or CodeGG regression; full profile and explicit diagnostic work. User documentation has one clear lightweight default, opt-in full fallback and detailed troubleshooting. - If measured compact savings come at a substantial cost in tool correctness, repair, safety or specialist reachability, stop and register a corrective pass with concrete fixtures; do not silently publish a lower-quality default. -- Live LM Studio smoke performed and recorded, or M003 remains explicitly unclosed. +- Live small-model API qualification performed and recorded, or M003 remains explicitly unclosed. -Closure evidence in `plans/closure/standalone-mcp-local-inference/003-status.md`: implementation SHA, installed binary and client versions, configs and nonsecret fixtures, per-task comparisons, test commands/outcomes, residual findings, docs links, and recommended release disposition. Do not claim a public release was made by finishing this plan. +Closure evidence in `plans/closure/standalone-mcp-local-inference/003-status.md`: implementation SHA, installed binary and API host versions, model identifiers/configurations, nonsecret fixtures, per-task paired comparisons, usage/results, test commands/outcomes, residual findings, docs links, and recommended release disposition. Do not claim a public release was made by finishing this plan. diff --git a/plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md b/plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md new file mode 100644 index 0000000..8a30279 --- /dev/null +++ b/plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md @@ -0,0 +1,49 @@ +# M007 — Compact Schema Reference Corrective + +Status: active +Primary class: corrective +Source roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` +Corrects: M001 compact tool advertisement, identified during M003 small-model API qualification +Related plans: `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md`; `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` +Related closures: `plans/closure/standalone-mcp-local-inference/001-status.md`; current M003 evidence in `plans/closure/standalone-mcp-local-inference/003-status.md` + +## Objective + +Make every compact `tools/list` input schema self-contained with all local `$ref` references resolvable, preserving compact tool behavior, all ten tool names, explicit `response_detail` support, and the serialized-size budget. + +## Evidence and why earlier qualification missed it + +An all-tools compact LM Studio native API turn failed constrained decoding with `cannot resolve $ref #/$defs/ResponseDetail, $defs not found`. Inspection of the exact compact MCP schemas found that `build_evidence_bundle` manually replaces nested source/fetch schemas and then removes the entire root `$defs` object, while retaining `response_detail` as a property referencing `#/$defs/ResponseDetail`. + +Earlier compact schema tests verified allowed properties, enum values, names, and byte budgets. They did not resolve every local reference after per-tool schema rewriting; the M003 API tests had also restricted `allowed_tools` to one specialist tool and therefore did not exercise the failing ten-tool palette. + +## Scope and invariants + +- Change only compact input-schema serialization and its regression coverage. +- Preserve all ten tool names, accepted runtime arguments, default behavior, validation, and MCP result contracts. +- Keep explicit `response_detail` advertised and preserve the compact byte budget. +- Do not remove a parameter or weaken runtime validation to hide schema incompatibility. + +## Work packages + +1. Preserve the `ResponseDetail` definition or inline its supported values when compacting `build_evidence_bundle`; remove only definitions that have no remaining references. +2. Add a test that walks all ten compact schemas and asserts every local JSON Schema reference resolves against that schema root. +3. Run the focused schema suite and canonical local verification. Repeat the native small-model API task with all ten tools exposed and no `allowed_tools` filter; confirm the full failure is an input-context limit and compact reaches a valid tool call without a schema error. +4. Update M003 evidence, this closure record, the roadmap, and registry. Keep M003 open until the broader small-model task matrix is evaluated. + +## Verification + +```bash +cargo fmt --check +cargo test --locked --all-features --test mcp_schema_slimming +make docs-check +``` + +The native model API comparison is opt-in operational evidence and must not become a network-dependent CI test. + +## Acceptance + +- All local `$ref` values in every compact tool input schema resolve against the advertised schema root. +- All ten tools remain advertised and compact schema bytes remain within the existing limit. +- The all-tools compact API trial can initialize and invoke the task-appropriate tool without the unresolved-reference error. +- Exact candidate, API result, local tests, and restored `mcp.json` hash are recorded in `plans/closure/standalone-mcp-local-inference/007-status.md`. diff --git a/plans/registry.md b/plans/registry.md index d54e54b..2e3757d 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005, and M006 closed; M003 blocked) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004/M005 passed hosted Linux/macOS/Windows gates on `de69f1d`; M006 closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826` with hosted CI run `41181749609`; M003 still lacks broader task-quality and UI acceptance. | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `006-status.md`; M003 remains blocked in `003-status.md`. | +| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005, and M006 closed; M003 and M007 active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004/M005 passed hosted Linux/macOS/Windows gates on `de69f1d`; M006 closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826` with hosted CI run `41181749609`; M003 is completing paired small-model API trials, and M007 corrects a compact schema reference defect those trials exposed. | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `007-status.md`; M003 evidence in `003-status.md`, M007 in `007-status.md`. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -54,10 +54,11 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under |---|---|---|---|---| | Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | -| Standalone MCP local-inference | M003 LM Studio consumer qualification | blocked | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Interactive UI history, paired inference matrix, and authenticated native MCP API success are in `plans/closure/standalone-mcp-local-inference/003-status.md`. UI structured-content/refresh, broader task coverage, and documented model-quality failures remain open. | +| Standalone MCP local-inference | M003 small local-model tool-use and context qualification | active | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Qualifies whether small models can use tools appropriately while `compact + brief` reduces context. LM Studio is the available API host; its UI behavior is out of scope. Existing trials and remaining quality gaps are in `plans/closure/standalone-mcp-local-inference/003-status.md`. | | Standalone MCP local-inference | M004 compact schema discoverability corrective | closed | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted canonical Linux, macOS, and Windows jobs pass in CI run `41160264796`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Standalone MCP local-inference | M005 brief document extraction and paired requalification | closed | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted CI run `41160264796` passed. Closure `plans/closure/standalone-mcp-local-inference/005-status.md`; M003 remains open for the documented operational gaps. | -| Standalone MCP local-inference | M006 compact direct-fetch tool-choice corrective | closed | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Candidate `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; closure `plans/closure/standalone-mcp-local-inference/006-status.md`; native paired and repeated compact turns passed; hosted Linux/macOS/Windows CI run `41181749609` passed. M003 remains blocked on broader task quality and UI acceptance. | +| Standalone MCP local-inference | M006 compact direct-fetch tool-choice corrective | closed | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Candidate `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; closure `plans/closure/standalone-mcp-local-inference/006-status.md`; native paired and repeated compact turns passed; hosted Linux/macOS/Windows CI run `41181749609` passed. M003 continues small-model quality qualification. | +| Standalone MCP local-inference | M007 compact schema reference corrective | active | `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` | M003 all-tools API turn exposed unresolved `#/$defs/ResponseDetail` in compact `build_evidence_bundle`; root cause and closure gate are in `plans/closure/standalone-mcp-local-inference/007-status.md`. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | @@ -128,7 +129,7 @@ Corrective closure sequence: `plans/implementation/dependency-evidence-hardening **Python native library gate:** ADR-0005 authorizes a narrow direct Rust SDK facade and Python PyO3 native package without broad Rust module stability. M001–M008 are closed at `067468a` with fully-green qualifying run `38641009246` (5/5 builds, 20/20 smokes, offline sdist rebuild, green audit, digests in `008-status.md`). Earlier runs are historical/deficiency/supporting evidence only. First PyPI publication is separately closed under `python-pypi-distribution` M003 at 0.4.4; Eggpack binary assets remain unchanged. -**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 and M005 are closed at `de69f1d`; M006 is closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826` after repeated authenticated native LM Studio compact direct-fetch turns and hosted Linux/macOS/Windows CI run `41181749609`. M003 remains blocked: the Gemma E2B/E4B specialist-task trials have mixed tool-call and evidence-quality outcomes, and LM Studio UI refresh plus structured-content/fallback behavior remain uncharacterized because the computer-use host is unavailable. See the M003 and M006 closure records for the complete trial evidence and limits. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. +**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 and M005 are closed at `de69f1d`; M006 is closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826` after repeated native API compact direct-fetch turns and hosted Linux/macOS/Windows CI run `41181749609`. M003 is active and qualifies model behavior through paired API trials: small local models should select tools, pass valid arguments, use retrieved evidence, and handle failures truthfully while `compact + brief` lowers actual reported input/context use. LM Studio is the available API host, not a product-specific acceptance target; UI refresh and presentation are out of scope. M007 is active to fix an unresolved `$defs/ResponseDetail` reference in the compact `build_evidence_bundle` schema, discovered when all ten tools were exposed in M003. See M003 and M007 status records for measured results and remaining evidence. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. **Tool-surface gate:** M001-M007 closed. Do not run evaluation against pre-consolidation bytes and present it as consolidation diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index 01d9152..e7e6ef0 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,7 +1,7 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001, M002, M004, M005, and M006 closed; M003 blocked on operational acceptance) -Scope: eggsearch native MCP clients (LM Studio first), not CodeGG integration +Status: active (M001, M002, M004, M005, and M006 closed; M003 and M007 active) +Scope: small local-model tool-use and context qualification through an actual inference API; not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. Controlling ADRs: `plans/adrs/ADR-0001-ten-tool-surface-and-adapter-ownership.md` and `plans/adrs/ADR-0006-keyless-deterministic-retrieval-and-optional-extraction.md`. @@ -9,9 +9,9 @@ Closed predecessor: `plans/subsystems/tool-surface-consolidation-roadmap.md` (M0 ## Purpose and ownership -Make eggsearch practical as a direct MCP server in LM Studio and similarly simple local-inference hosts that eagerly pass advertised tool definitions and responses to a model. Eggsearch owns the MCP server's advertised schema, protocol responses, deterministic projection and operator configuration; the client owns the model, tokenizer, context window and tool invocation. CodeGG's native wrappers, progressive disclosure and diagnostic storage are intentionally unchanged. +Qualify Eggsearch's compact schema and brief result modes for small locally hosted models that need to call tools while keeping prompt context small. LM Studio is the available API host in this environment, including for MLX support; the acceptance target is model tool-use and context cost, not LM Studio UI behavior. Eggsearch owns the MCP server's advertised schema, protocol responses, deterministic projection and operator configuration; the host owns the model, tokenizer, context window and invocation. CodeGG's native wrappers, progressive disclosure and diagnostic storage are intentionally unchanged. -Primary classes: M001 capability (compact advertised schema profile); M002 capability (brief bounded output); M003 capability (tested direct-client onboarding/qualification). Infrastructure includes measurement and config; polish follows qualified behavior. +Primary classes: M001 capability (compact advertised schema profile); M002 capability (brief bounded output); M003 capability (small local-model tool-use and context qualification). Infrastructure includes measurement and config; polish follows qualified behavior. ## Baseline and findings @@ -44,35 +44,37 @@ M001 compact ten-tool advertised schema profile (closed) | +-- hard --> M002 brief/budgeted agent responses (closed) | - +-- operational --> M003 LM Studio/direct-client compatibility and quality (blocked on live acceptance) + +-- operational --> M003 small local-model tool-use and context qualification (active) +-- corrective --> M005 brief document extraction and paired requalification (closed) +-- corrective --> M006 compact direct-fetch tool-choice corrective (closed at `6dc63d5`; hosted CI run `41181749609`) + +-- corrective --> M007 compact schema reference corrective (active; discovered during M003 all-tools API qualification) M004 compact schema discoverability corrective (closed on hosted Linux/macOS candidate de69f1d) ``` -No CodeGG dependency; it is a regression consumer, not the implementation target. M003's offline qualification can run in CI; a real LM Studio interactive smoke is an operational closure requirement, not a reason to block M001/M002 development. +No CodeGG dependency; it is a regression consumer, not the implementation target. M003's offline qualification can run in CI; live small-model API turns are the operational closure evidence. LM Studio is the available inference host here, not a product-specific acceptance target. | Milestone | Primary class | Source plan | Exit | |---|---|---|---| | M001 | capability | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Opt-in compact `tools/list` for all ten existing names; measurable schema reduction; parity across MCP transports; default unchanged | | M002 | capability | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Explicit brief projection and bounded results with critical failure, identity, trust and security semantics intact; unchanged legacy defaults | -| M003 | capability | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Verified LM Studio install/smoke, measured local-model context cost and completion quality, onboarding guides, and stop/rollback evidence | +| M003 | capability | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Paired API trials show whether small local models can use the tools appropriately while `compact + brief` reduces actual context use; quality and token evidence determine acceptance, not host UI behavior | | M004 | capability | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Advertise runtime-accepted research goal/domain values in compact schemas; test schema parity and size; repeat focused live tasks; M003 remains open for its broader gates | | M005 | corrective | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Closed on `de69f1d`; brief preserves same-document extraction within budget; model evidence and unresolved M003 failures are recorded; hosted Linux/macOS/Windows gates pass | | M006 | corrective | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Resolve or bound a repeated native LM Studio compact `web_fetch` no-call failure from M003; keep M003 open until evidence supports acceptance | +| M007 | corrective | `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` | Resolve dangling compact-schema `$ref` in `build_evidence_bundle`; preserve ten-tool contract and verify all local references plus all-tools model API invocation | ## Cross-cutting controls -Compatibility/protocol: verify current `rmcp` schema serialization and protocol negotiation before implementing profile-specific tool definitions. MCP `tools/list` caching or `listChanged` support MUST NOT be assumed by LM Studio. Profiles are process/config scoped, never inferred from user agent or model size. Fingerprints MUST differ when serialized contracts differ and remain stable across equivalent process launches. +Compatibility/protocol: verify current `rmcp` schema serialization and protocol negotiation before implementing profile-specific tool definitions. MCP `tools/list` caching or `listChanged` support MUST NOT be assumed by clients. Profiles are process/config scoped, never inferred from user agent or model size. Fingerprints MUST differ when serialized contracts differ and remain stable across equivalent process launches. Safety: every tool uses existing policy and validation gates. Removing an advertised property MUST NOT remove runtime validation. Diagnostics must not disappear from the unmodified default. A brief response must signal incomplete retrieval; when a client ignores structured content, text fallback must still be useful. Resources: local inference budget is an agent-context budget; do not reduce remote search recall, upstream concurrency or browser/PDF capabilities as a side effect. Keep memory bounded, deterministic order and fast serialization; avoid doubling large JSON payloads in memory merely to render brief. -Observability: keep per-tool profile-size fixtures, model-visible response samples, omission counts, source-card retention, constraint-enforcement caveats and error/repair traces. Test actual LM Studio tool-selection success, not merely canned keyword matching. +Observability: keep per-tool profile-size fixtures, model-visible response samples, omission counts, source-card retention, constraint-enforcement caveats and error/repair traces. Test actual small-model API tool-use and evidence quality, not merely canned keyword matching. -Documentation: M001/M002 update `architecture/mcp.md`, `architecture/config.md`, `docs/config.md`, `docs/tool-matrix.md` and `docs/test-inventory.md` as applicable. M003 updates `README.md` / `docs/integrations.md` / `docs/installation.md` with LM Studio guidance. Never rewrite historical closure records. +Documentation: M001/M002 update `architecture/mcp.md`, `architecture/config.md`, `docs/config.md`, `docs/tool-matrix.md` and `docs/test-inventory.md` as applicable. M003 maintains optional LM Studio operator guidance and records the model qualification method/results. Never rewrite historical closure records. ## Known risks and decision boundaries @@ -84,4 +86,4 @@ Documentation: M001/M002 update `architecture/mcp.md`, `architecture/config.md`, ## Definition of done -Every milestone gets an exact-SHA closure record in `plans/closure/standalone-mcp-local-inference/NNN-status.md`; update this roadmap and `plans/registry.md` in each closure commit. No full rollout, release, claims of LM Studio savings, or marking M003 closed on synthetic tests alone. +Every milestone gets an exact-SHA closure record in `plans/closure/standalone-mcp-local-inference/NNN-status.md`; update this roadmap and `plans/registry.md` in each closure commit. No full rollout or release, and no claim of small-model context or quality improvement without measured paired API turns. diff --git a/src/mcp/server.rs b/src/mcp/server.rs index 2df99fe..942c61e 100644 --- a/src/mcp/server.rs +++ b/src/mcp/server.rs @@ -344,8 +344,16 @@ fn compact_evidence_bundle_schema(schema: &mut serde_json::Value) { } }), ); + let response_detail = schema.pointer("/$defs/ResponseDetail").cloned(); if let Some(object) = schema.as_object_mut() { - object.remove("$defs"); + if let Some(response_detail) = response_detail { + object.insert( + "$defs".into(), + serde_json::json!({"ResponseDetail": response_detail}), + ); + } else { + object.remove("$defs"); + } } } diff --git a/tests/mcp_schema_slimming.rs b/tests/mcp_schema_slimming.rs index d6e4b2d..c577cc2 100644 --- a/tests/mcp_schema_slimming.rs +++ b/tests/mcp_schema_slimming.rs @@ -19,13 +19,16 @@ fn schema_for(name: &str) -> serde_json::Value { serde_json::to_value(&tool.input_schema).expect("schema serializes") } -fn compact_tool_for(name: &str) -> rmcp::model::Tool { +fn compact_tool_definitions() -> Vec { let mut config = AppConfig::default(); config.mcp.schema_profile = McpSchemaProfile::Compact; let state = Arc::new(ServerState::build(config).expect("compact state")); let server = eggsearch::mcp::EggsearchServer::new(state); - server - .tool_definitions() + server.tool_definitions() +} + +fn compact_tool_for(name: &str) -> rmcp::model::Tool { + compact_tool_definitions() .into_iter() .find(|t| t.name.as_ref() == name) .unwrap_or_else(|| panic!("tool {name} not found")) @@ -58,6 +61,50 @@ fn has_prop(schema: &serde_json::Value, prop: &str) -> bool { schema.get("properties").and_then(|p| p.get(prop)).is_some() } +fn collect_unresolved_local_refs( + value: &serde_json::Value, + root: &serde_json::Value, + path: &str, + unresolved: &mut Vec, +) { + match value { + serde_json::Value::Object(object) => { + if let Some(reference) = object.get("$ref").and_then(serde_json::Value::as_str) { + if let Some(pointer) = reference.strip_prefix('#') { + if root.pointer(pointer).is_none() { + unresolved.push(format!("{path} -> {reference}")); + } + } + } + for (key, child) in object { + collect_unresolved_local_refs(child, root, &format!("{path}/{key}"), unresolved); + } + } + serde_json::Value::Array(values) => { + for (index, child) in values.iter().enumerate() { + collect_unresolved_local_refs(child, root, &format!("{path}/{index}"), unresolved); + } + } + _ => {} + } +} + +#[test] +fn every_compact_schema_resolves_local_references() { + let tools = compact_tool_definitions(); + assert_eq!(tools.len(), 10); + for tool in tools { + let schema = serde_json::to_value(tool.input_schema).expect("schema serializes"); + let mut unresolved = Vec::new(); + collect_unresolved_local_refs(&schema, &schema, "$", &mut unresolved); + assert!( + unresolved.is_empty(), + "{} has unresolved compact schema references: {unresolved:?}", + tool.name + ); + } +} + #[test] fn search_four_schema_bytes_stay_slim() { let names = [ From 772a073fd40e9f044017b26bae18a6efd0724b1d Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 16:35:39 +0000 Subject: [PATCH 22/25] fix(mcp): expose browser rendering in compact schema --- .../003-status.md | 114 ++++++++++++++++-- .../008-status.md | 57 +++++++++ ...-browser-render-reachability-corrective.md | 48 ++++++++ plans/registry.md | 3 +- .../standalone-mcp-local-inference-roadmap.md | 4 +- src/mcp/server.rs | 6 + tests/mcp_schema_slimming.rs | 17 +++ 7 files changed, 237 insertions(+), 12 deletions(-) create mode 100644 plans/closure/standalone-mcp-local-inference/008-status.md create mode 100644 plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index e1d8498..ef1d59f 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -73,9 +73,9 @@ provide a reliable per-chat usage breakdown for this test set. | Reproducible installed-candidate stdio smoke | `scripts/lm_studio_stdio_smoke.py` | pass | Offline protocol evidence only | | Ten tools and profile list sizes | M006-candidate initialized child-process smoke: 72,923 full / 19,370 compact bytes; all ten names; `provider_status` with probing off | pass | Byte counts are not token counts | | Configured MCP API path | Authenticated native `/api/v1/chat` calls through configured `mcp/eggsearch`, with task-specific `allowed_tools`; exact config rollback hash recorded | pass | None for the tested local host; this verifies the model/API path, not LM Studio UI behavior | -| Paired small-model comparison | LFM3 2B direct-fetch trials in §12 and repeated security-applicability trials in §13; Gemma E2B/E4B observations in §12 | partial / mixed | Expand paired coverage across research and extraction; retain API/plugin failures as infrastructure outcomes, separate from model failures | -| Tool choice, argument validity, evidence and failure truthfulness | §12-13 include valid and invalid calls, no-calls, plugin/API failures, evidence gaps, and both correct and incorrect applicability conclusions | partial / quality concern | Complete repeatable paired task coverage and assess evidence quality across more than one small model | -| Actual context/token reduction | LM Studio native API reports `input_tokens` including formatting, tool definitions, and prior messages; §12-13 record paired full/compact usage and tool-result lengths | pass for measured tasks | Extend measurements across the remaining paired task set; no tokenizer identity or UI context meter is required | +| Paired small-model comparison | §12-15 include direct fetch, web search, research, security applicability, PDF extraction, and a browser-render attempt on LFM3 2B and Gemma E2B | partial / mixed | Repeat research and browser tasks; separate host failures from model outcomes | +| Tool choice, argument validity, evidence and failure truthfulness | §12-14 include valid calls, wrong-tool selection, argument repair, weak evidence, extraction errors, provider failures, and API/browser failures | partial / quality concern | Improve consistency on research and exact applicability ranges; distinguish host failures from model failures | +| Actual context/token reduction | LM Studio native API reports `input_tokens` including formatting, tool definitions, and prior messages; §12-14 record paired full/compact usage and tool-result lengths | pass for measured tasks | Repeat key pairs to estimate variability; no tokenizer identity or UI context meter is required | | Eggsearch result contract | Offline MCP protocol tests cover structured results and text fallback; native API exposes returned tool output to the model | pass for Eggsearch contract | No host-specific result-shape or UI-consumption test is required | | Config rollback | Restored original `~/.lmstudio/mcp.json`; SHA-256 is `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c` | pass | None | @@ -467,13 +467,16 @@ missing), while compact returned HTTP 500 `Plugin unloaded`. Neither provided a model answer or usage counters, so these are recorded as API/tool-path failures rather than model outcomes. -The current remaining work is to repeat the paired task matrix for other small -task classes (especially research and document extraction), separate -model/tool-call failures from host/plugin failures, and assess whether the -measured context reduction preserves adequate task quality overall. Current -paired security evidence covers two small models; it is promising but narrow. -M003 is active, not blocked on the UI. The current acceptance objective and -method are in `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md`. +The expanded paired task results are recorded in §14. They cover direct HTML +fetch, web search, research search, security applicability, and PDF extraction +on two small models. Outcomes remain mixed: compact consistently reduced +measured input tokens, but it did not preserve adequate quality in every task. +At the time of the M007 trials, the compact schema did not advertise optional +`render`; M008 adds that argument, with the first requalification attempt in +§15. Browser-rendered extraction remains unqualified because the compact call +failed before inference on Chrome's shared-profile lock. M003 remains active, +not blocked on the UI. The current acceptance objective and method are in +`plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md`. ### Gemma E2B paired security trial @@ -522,3 +525,94 @@ The earlier post-full `Plugin unloaded` response remains recorded as a host failure; the exact original MCP config was restored byte-for-byte after each request block (SHA-256 `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`). + +## 14. Expanded small-model task matrix on M007 candidate + +The native LM Studio REST API was used as a local inference harness; its UI +behavior is not part of this qualification. LM Studio 0.4.26+4 invoked the +release executable built from the M007 candidate (SHA-256 +`ff98098566a89f0603d1c07ad3cd88e5d36fae830bc8d6bde65cebc3456a71ea`). The +temporary configured-server entries exposed the full Eggsearch palette under +both profiles, without an `allowed_tools` filter for the matrix tasks. Each +pair used a fresh `store=false` request, temperature 0, the same model, +prompt, context size, and source URL. LFM3 used Q4_K_M GGUF at a 128,000-token +request context; Gemma E2B used 4-bit MLX at 32,000 tokens. The profile +settings were `full + diagnostic` and `compact + brief`, with PDF and browser +features enabled in the temporary Eggsearch config. API/plugin errors were +kept separate from model outcomes. + +| Model / task | Full input / output tokens; result chars | Compact input / output tokens; result chars | Tool and task outcome | +|---|---:|---:|---| +| LFM3 2B — direct fetch, official Rust 1.90 announcement | 18,846 / 578; 21,120 | 5,378 / 596; 5,706 | Both made one valid `web_fetch` and correctly identified LLD and workspace publishing with source evidence. | +| LFM3 2B — web search, same release task | 18,821 / 994; 55,193 | 5,353 / 770; 18,973 | Full selected `security_search` for a release query and used secondary evidence. Compact selected `web_search`, repaired an excessive `excerpt_count`, fetched official Rust sources, and answered the requested changes correctly. | +| LFM3 2B — `research_search`, workspace publishing | 18,825 / 1,216; 53,140 | 5,357 / 1,175; 28,581 | Both invoked research. Full leaned on weak/secondary evidence; compact added official Rust release and Cargo sources and better substantiated the claim. | +| LFM3 2B — security applicability, Log4j 2.14.1 | 18,803 / 566; 53,020 | 5,335 / 610; 12,099 | Full made several calls, including repair attempts, and gave an imprecise upper range. Compact used `security_search` then fetched NVD/OSV; it gave the full affected range and reported Yahoo's 500. | +| LFM3 2B — PDF extraction, NIST FIPS 197 | 18,860 / 905; 50,732 | 5,392 / 574; 5,084 | Both fetched the exact PDF. Full incorrectly assigned 192- and 256-bit block sizes; compact correctly reported 128-bit blocks and 128/192/256-bit keys, noting truncation after the needed evidence. | +| Gemma E2B — direct fetch, official Rust 1.90 announcement | 4,612 / 1,030; 21,120 | 1,825 / 1,162; 5,706 | Both made one valid `web_fetch` and identified the two requested release changes. | +| Gemma E2B — web search, same release task | 4,585 / 1,971; 71,023 | 1,798 / 1,659; 13,129 | Full fetched the general GitHub releases page and answered with unrelated examples. Compact searched, fetched the official Rust blog, and gave the requested changes. | +| Gemma E2B — `research_search`, workspace publishing | 4,586 / 1,861; 55,062 | 1,799 / 1,731; 7,414 | Both invoked research but neither established the Rust 1.90-specific feature from primary evidence. Compact appropriately stated the evidence was inconclusive; full overstated a generic workspace result. | +| Gemma E2B — security applicability, Log4j 2.14.1 | 4,589 / 2,280; 140,916 | 1,802 / 1,184; 5,535 | Both concluded 2.14.1 is affected and cited advisories. Both gave an incomplete upper range; full also made extra `provider_status` and repair calls. | +| Gemma E2B — PDF extraction, NIST FIPS 197 | 4,634 / 947; 50,732 | 1,847 / 877; 5,084 | Both fetched the same PDF and correctly separated key length from the 128-bit block size. | + +Compact reduced API-reported input tokens by 71.4–71.6% for the scored LFM3 +pairs and 60.1–60.8% for the Gemma pairs. Result-character reduction varied +by task (approximately 46–90% for LFM3 research/PDF and 73–90% for the direct +fetch/PDF cases; approximately 73–96% for Gemma direct-fetch/security/PDF). +These are per-task measurements, not a universal savings estimate. Across +models, compact improved the release-search and PDF answers, while research +and exact security-range quality remained mixed. Fewer result characters did +not guarantee correctness. + +### Browser-rendered extraction disposition + +The React `useState` page was paired with the same question. Full-profile +requests explicitly using `render="browser"` timed out or returned HTTP errors +in the native API; a successful compact request fetched the page over HTTP +instead. Its result metadata reported `browser_escalated=false`, and the model +made incorrect claims about initializer functions and immediate state updates. +A separate `example.com` call with `render="browser"` returned HTTP 200, but +does not qualify browser extraction quality. The compact schema on this M007 +candidate omitted the optional `render` parameter. M008 adds it and is +recorded in §15. The successful full-profile call and compact launch failure +are not counted as a successful paired browser extraction. + +### Candidate and cleanup + +The original `~/.lmstudio/mcp.json` was restored byte-for-byte after testing; +SHA-256 `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. +The local LM Studio server was restarted with that original registration. +This candidate passed the local canonical `make check` and `make docs-check`; +hosted CI, release drift, and Python native qualification passed on commit +`fe677ff45a4c69feee50907cfbd451b6ae2a1fd4`. The hosted egress qualification +run was cancelled and must be rerun before M007 closure. M003 remains active: +the matrix now demonstrates consistent context reduction on these tasks, but +tool choice, evidence quality, failure handling, and browser reachability still +vary by model and profile. Additional repeats and a clean browser pair are +needed before a release-quality recommendation. + +## 15. M008 compact browser-render reachability corrective + +M008 adds the runtime-supported `render` argument to compact `web_fetch`, with +the accepted enum values `http_only`, `auto`, and `browser`. The source change +is in `src/mcp/server.rs`; `tests/mcp_schema_slimming.rs` now checks that the +compact schema preserves all three values. The exact local candidate release +binary SHA-256 is +`507724a14a1926488913a50b1864543f1e899d8a3df6e1b44d1a90dcc4110b43`. + +The LFM3 2B browser-render API trial used the same `example.com` extraction +prompt for both profiles. Full/diagnostic used 18,805 input and 259 output +tokens; its valid `web_fetch` call included `render="browser"` and returned +HTTP 200 with `transport="browser"`. It extracted the heading and explanatory +paragraph correctly, reported no truncation, and retained the external-content +warning. Compact/brief returned an HTTP 500 before a model turn completed: +Chrome could not create its shared ProcessSingleton because a +`chromiumoxide-runner/SingletonLock` already existed. It produced no usage +counters or scored model outcome. Therefore this is not successful compact +browser invocation evidence. The React reference page remains unqualified. + +M008 remains active until a compact-profile browser call succeeds on the exact +candidate, local canonical checks pass, and hosted CI qualifies the committed +candidate. M003 remains active because research/security evidence quality is +mixed and key specialist/browser tasks need repeats. Existing measured token +reductions are per task and do not establish a general quality or reliability +improvement. diff --git a/plans/closure/standalone-mcp-local-inference/008-status.md b/plans/closure/standalone-mcp-local-inference/008-status.md new file mode 100644 index 0000000..3ea21f2 --- /dev/null +++ b/plans/closure/standalone-mcp-local-inference/008-status.md @@ -0,0 +1,57 @@ +# Standalone MCP Local-Inference M008 — Closure Status + +Status: active (schema corrective and focused contract test pass; compact browser invocation and hosted qualification pending) + +Source implementation plan: +`plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md` + +Source subsystem roadmap: +`plans/subsystems/standalone-mcp-local-inference-roadmap.md` + +## 1. Discovery + +During M003 paired small-model browser extraction trials, the full +`web_fetch` schema advertised the `render` option but the compact schema did +not. Compact models therefore could not select `auto` or `browser` from the +advertised arguments. The compact trial omitted the option and fetched the +page through HTTP. A restricted full-profile request explicitly used browser +rendering on `example.com` and returned HTTP 200; attempts on the React +reference page timed out or returned HTTP errors. These results establish the +compact schema reachability gap, but not successful paired browser extraction. + +## 2. Candidate and verification + +Implementation candidate release executable SHA-256: +`507724a14a1926488913a50b1864543f1e899d8a3df6e1b44d1a90dcc4110b43`. + +The compact schema now preserves `render` and constrains it to the runtime's +accepted values `http_only`, `auto`, and `browser`. The focused +`mcp_schema_slimming` suite passes 20 tests. `make docs-check` passes. + +The paired LFM3 2B browser call used the same `example.com` prompt for full +and compact. Full/diagnostic called `web_fetch` with `render="browser"`, used +18,805 input and 259 output tokens, and returned HTTP 200 with +`transport="browser"`; the model extracted the heading and paragraph +accurately and retained the external-content warning. Compact/brief failed +before a model turn completed because Chrome could not acquire the shared +`chromiumoxide-runner/SingletonLock`. LM Studio returned HTTP 500 and no usage +counters. This is an infrastructure failure, not a compact-model outcome, and +does not satisfy successful compact browser invocation acceptance. + +The canonical `make check` attempt passed its earlier formatting, lint, and +compile stages but stalled without output for more than five minutes in the +unchanged `bounded_command` integration suite on this macOS host. It was +interrupted. The exact committed candidate still requires hosted CI. + +Remaining evidence: committed-candidate hosted CI, an uncontended compact +browser API call with actual usage and extraction results, and verification +that the compact schema remains within its size budget on the built candidate. +The original `~/.lmstudio/mcp.json` is restored at SHA-256 +`b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. + +## 3. Disposition + +M008 remains active until its acceptance criteria pass. M003 remains active and +owns the broader small-model tool-use and context qualification. The config +restoration hash is verified, and no API authentication or MCP permission +setting was changed. diff --git a/plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md b/plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md new file mode 100644 index 0000000..79587f1 --- /dev/null +++ b/plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md @@ -0,0 +1,48 @@ +# M008 — Compact Browser-Render Reachability Corrective + +Status: active +Primary class: corrective +Source roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` +Corrects: M001 compact `web_fetch` schema, discovered during M003 small-model task qualification +Related plans: `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md`; `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` +Related closures: `plans/closure/standalone-mcp-local-inference/001-status.md`; current M003 results in `plans/closure/standalone-mcp-local-inference/003-status.md` + +## Objective + +Allow compact-profile models to request the existing optional browser-rendering behavior from `web_fetch`, without expanding unrelated arguments or weakening runtime policy. + +## Discovery + +M003's paired browser extraction task found that the full `web_fetch` schema advertises `render` (`http_only`, `auto`, `browser`) but the compact schema strips it. The runtime accepts the option, yet a compact model cannot select it from the advertised tool contract. In the observed task, compact omitted `render` and fetched over HTTP; explicit full-profile browser requests timed out or returned source HTTP errors. An example-domain browser request returned HTTP 200, so the path is available but the React task remains unqualified. + +## Scope and invariants + +- Add only the existing `render` parameter to compact `web_fetch` input schemas, preserving its accepted values. +- Keep all ten tool names, runtime validation, response contracts, and the compact schema size budget. +- Preserve fetch bounds, browser concurrency, SSRF protections, and default `http_only` behavior. +- Do not expose persistent browser profiles or relax browser, network, or authentication policy. + +## Work packages + +1. Preserve `render` in the compact `web_fetch` schema and add a contract test for its property and enum values. +2. Run focused schema tests, `make check`, and `make docs-check` on the exact candidate. +3. Repeat the browser extraction prompt with the same source/query for full and compact profiles; record actual input/output tokens, arguments, browser transport result, extraction fidelity, and failures. Keep host/API failures separate from model outcomes. +4. Update M003 evidence, this closure record, the roadmap, and registry. Keep M003 active until its broader task quality and repeatability gates pass. + +## Verification + +```bash +cargo fmt --check +cargo test --locked --all-features --test mcp_schema_slimming +make check +make docs-check +``` + +Native API qualification is opt-in operational evidence, not a network-dependent routine CI test. + +## Acceptance + +- Compact `web_fetch` advertises the three existing `render` values and the tool schema remains within its size limit. +- Regression coverage confirms the property and enum survive compact-schema rewriting. +- A paired API trial can expose and invoke `render="browser"` in compact mode; report content quality and API/browser failures without treating infrastructure errors as model outcomes. +- Exact candidate, local and hosted checks, API measurements, and restored MCP config hash are recorded in `plans/closure/standalone-mcp-local-inference/008-status.md`. diff --git a/plans/registry.md b/plans/registry.md index 2e3757d..ee0ce3a 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005, and M006 closed; M003 and M007 active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M001-M002 closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004/M005 passed hosted Linux/macOS/Windows gates on `de69f1d`; M006 closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826` with hosted CI run `41181749609`; M003 is completing paired small-model API trials, and M007 corrects a compact schema reference defect those trials exposed. | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `007-status.md`; M003 evidence in `003-status.md`, M007 in `007-status.md`. | +| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005, and M006 closed; M003, M007, and M008 active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M006 closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; M003 now records paired small-model API trials across retrieval, specialist search, and PDF extraction; M007 corrects a compact schema reference defect; M008 restores compact reachability for the existing browser-render option. | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `008-status.md`; M003 evidence in `003-status.md`, M007 in `007-status.md`, M008 in `008-status.md`. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -59,6 +59,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Standalone MCP local-inference | M005 brief document extraction and paired requalification | closed | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted CI run `41160264796` passed. Closure `plans/closure/standalone-mcp-local-inference/005-status.md`; M003 remains open for the documented operational gaps. | | Standalone MCP local-inference | M006 compact direct-fetch tool-choice corrective | closed | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Candidate `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; closure `plans/closure/standalone-mcp-local-inference/006-status.md`; native paired and repeated compact turns passed; hosted Linux/macOS/Windows CI run `41181749609` passed. M003 continues small-model quality qualification. | | Standalone MCP local-inference | M007 compact schema reference corrective | active | `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` | M003 all-tools API turn exposed unresolved `#/$defs/ResponseDetail` in compact `build_evidence_bundle`; root cause and closure gate are in `plans/closure/standalone-mcp-local-inference/007-status.md`. | +| Standalone MCP local-inference | M008 compact browser-render reachability corrective | active | `plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md` | M003 browser trials exposed that compact schemas omit the accepted `web_fetch.render` argument; preserve browser mode without weakening fetch policy. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index e7e6ef0..075bb39 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001, M002, M004, M005, and M006 closed; M003 and M007 active) +Status: active (M001, M002, M004, M005, and M006 closed; M003, M007, and M008 active) Scope: small local-model tool-use and context qualification through an actual inference API; not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. @@ -48,6 +48,7 @@ M001 compact ten-tool advertised schema profile (closed) +-- corrective --> M005 brief document extraction and paired requalification (closed) +-- corrective --> M006 compact direct-fetch tool-choice corrective (closed at `6dc63d5`; hosted CI run `41181749609`) +-- corrective --> M007 compact schema reference corrective (active; discovered during M003 all-tools API qualification) + +-- corrective --> M008 compact browser-render reachability corrective (active; discovered during M003 paired extraction) M004 compact schema discoverability corrective (closed on hosted Linux/macOS candidate de69f1d) ``` @@ -63,6 +64,7 @@ No CodeGG dependency; it is a regression consumer, not the implementation target | M005 | corrective | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Closed on `de69f1d`; brief preserves same-document extraction within budget; model evidence and unresolved M003 failures are recorded; hosted Linux/macOS/Windows gates pass | | M006 | corrective | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Resolve or bound a repeated native LM Studio compact `web_fetch` no-call failure from M003; keep M003 open until evidence supports acceptance | | M007 | corrective | `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` | Resolve dangling compact-schema `$ref` in `build_evidence_bundle`; preserve ten-tool contract and verify all local references plus all-tools model API invocation | +| M008 | corrective | `plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md` | Advertise the existing `web_fetch.render` option in compact mode and requalify paired browser extraction | ## Cross-cutting controls diff --git a/src/mcp/server.rs b/src/mcp/server.rs index 942c61e..02fd5bd 100644 --- a/src/mcp/server.rs +++ b/src/mcp/server.rs @@ -137,6 +137,7 @@ fn compact_schema(name: &str, schema: &mut serde_json::Value) { "focus", "focus_max_chunks", "focus_max_chars", + "render", "response_detail", ], "repo_fetch" => &[ @@ -201,6 +202,11 @@ fn compact_schema(name: &str, schema: &mut serde_json::Value) { }; compact_schema_value(schema, allowed); compact_workflow_schema(name, schema); + if name == "web_fetch" { + if let Some(render) = schema.pointer_mut("/properties/render") { + set_string_schema_enum(render, &["http_only", "auto", "browser"]); + } + } if name == "build_evidence_bundle" { compact_evidence_bundle_schema(schema); } diff --git a/tests/mcp_schema_slimming.rs b/tests/mcp_schema_slimming.rs index c577cc2..df69722 100644 --- a/tests/mcp_schema_slimming.rs +++ b/tests/mcp_schema_slimming.rs @@ -181,6 +181,23 @@ fn web_search_hides_infrastructure_controls() { ); } +#[test] +fn compact_web_fetch_preserves_browser_render_modes() { + let schema = compact_schema_for("web_fetch"); + let render = schema + .pointer("/properties/render") + .expect("compact web_fetch exposes render"); + let values = render + .get("enum") + .and_then(serde_json::Value::as_array) + .unwrap_or_else(|| panic!("compact render modes are not constrained: {render}")); + let values = values + .iter() + .filter_map(serde_json::Value::as_str) + .collect::>(); + assert_eq!(values, ["http_only", "auto", "browser"]); +} + #[test] fn compact_web_fetch_explains_user_urls_and_bounded_extraction() { let description = compact_tool_for("web_fetch") From 808f8966d7c17281564ea0eae5b176e0da43ad3f Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 16:51:10 +0000 Subject: [PATCH 23/25] docs(plans): close compact schema correctives --- .../003-status.md | 67 ++++++++++++------- .../007-status.md | 32 ++++++--- .../008-status.md | 58 ++++++++++------ ...007-compact-schema-reference-corrective.md | 2 +- ...-browser-render-reachability-corrective.md | 2 +- plans/registry.md | 8 +-- .../standalone-mcp-local-inference-roadmap.md | 10 +-- 7 files changed, 115 insertions(+), 64 deletions(-) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index ef1d59f..faed6d3 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -73,7 +73,7 @@ provide a reliable per-chat usage breakdown for this test set. | Reproducible installed-candidate stdio smoke | `scripts/lm_studio_stdio_smoke.py` | pass | Offline protocol evidence only | | Ten tools and profile list sizes | M006-candidate initialized child-process smoke: 72,923 full / 19,370 compact bytes; all ten names; `provider_status` with probing off | pass | Byte counts are not token counts | | Configured MCP API path | Authenticated native `/api/v1/chat` calls through configured `mcp/eggsearch`, with task-specific `allowed_tools`; exact config rollback hash recorded | pass | None for the tested local host; this verifies the model/API path, not LM Studio UI behavior | -| Paired small-model comparison | §12-15 include direct fetch, web search, research, security applicability, PDF extraction, and a browser-render attempt on LFM3 2B and Gemma E2B | partial / mixed | Repeat research and browser tasks; separate host failures from model outcomes | +| Paired small-model comparison | §12-15 include direct fetch, web search, research, security applicability, PDF extraction, and a successful LFM3 browser-render pair | partial / mixed | Repeat research and browser tasks; improve compact warning interpretation and separate host failures from model outcomes | | Tool choice, argument validity, evidence and failure truthfulness | §12-14 include valid calls, wrong-tool selection, argument repair, weak evidence, extraction errors, provider failures, and API/browser failures | partial / quality concern | Improve consistency on research and exact applicability ranges; distinguish host failures from model failures | | Actual context/token reduction | LM Studio native API reports `input_tokens` including formatting, tool definitions, and prior messages; §12-14 record paired full/compact usage and tool-result lengths | pass for measured tasks | Repeat key pairs to estimate variability; no tokenizer identity or UI context meter is required | | Eggsearch result contract | Offline MCP protocol tests cover structured results and text fallback; native API exposes returned tool output to the model | pass for Eggsearch contract | No host-specific result-shape or UI-consumption test is required | @@ -472,10 +472,11 @@ fetch, web search, research search, security applicability, and PDF extraction on two small models. Outcomes remain mixed: compact consistently reduced measured input tokens, but it did not preserve adequate quality in every task. At the time of the M007 trials, the compact schema did not advertise optional -`render`; M008 adds that argument, with the first requalification attempt in -§15. Browser-rendered extraction remains unqualified because the compact call -failed before inference on Chrome's shared-profile lock. M003 remains active, -not blocked on the UI. The current acceptance objective and method are in +`render`; M008 adds that argument. The paired API requalification in §15 shows +both profiles can extract the requested browser page, though compact's brief +result omits transport metadata and the model missed its `output_truncated` +warning. Repeat research and browser tasks before a broader quality claim. +M003 remains active, not blocked on the UI. The current acceptance objective and method are in `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md`. ### Gemma E2B paired security trial @@ -581,10 +582,13 @@ are not counted as a successful paired browser extraction. The original `~/.lmstudio/mcp.json` was restored byte-for-byte after testing; SHA-256 `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. The local LM Studio server was restarted with that original registration. -This candidate passed the local canonical `make check` and `make docs-check`; +This M007 candidate passed local `make check` and `make docs-check`; hosted CI, release drift, and Python native qualification passed on commit -`fe677ff45a4c69feee50907cfbd451b6ae2a1fd4`. The hosted egress qualification -run was cancelled and must be rerun before M007 closure. M003 remains active: +`fe677ff45a4c69feee50907cfbd451b6ae2a1fd4`. The separate hosted egress +qualification run `41220450274` passed its Linux, x86_64 macOS, Windows, MSRV, +and ARMv7 jobs; the Apple Silicon macOS job was cancelled by a scheduled +runner-image brownout. This egress-only cancellation did not block M007 or M008 +closure. M003 remains active: the matrix now demonstrates consistent context reduction on these tasks, but tool choice, evidence quality, failure handling, and browser reachability still vary by model and profile. Additional repeats and a clean browser pair are @@ -599,20 +603,33 @@ compact schema preserves all three values. The exact local candidate release binary SHA-256 is `507724a14a1926488913a50b1864543f1e899d8a3df6e1b44d1a90dcc4110b43`. -The LFM3 2B browser-render API trial used the same `example.com` extraction -prompt for both profiles. Full/diagnostic used 18,805 input and 259 output -tokens; its valid `web_fetch` call included `render="browser"` and returned -HTTP 200 with `transport="browser"`. It extracted the heading and explanatory -paragraph correctly, reported no truncation, and retained the external-content -warning. Compact/brief returned an HTTP 500 before a model turn completed: -Chrome could not create its shared ProcessSingleton because a -`chromiumoxide-runner/SingletonLock` already existed. It produced no usage -counters or scored model outcome. Therefore this is not successful compact -browser invocation evidence. The React reference page remains unqualified. - -M008 remains active until a compact-profile browser call succeeds on the exact -candidate, local canonical checks pass, and hosted CI qualifies the committed -candidate. M003 remains active because research/security evidence quality is -mixed and key specialist/browser tasks need repeats. Existing measured token -reductions are per task and do not establish a general quality or reliability -improvement. +The LFM3 2B pair used the same `example.com` extraction prompt, all ten MCP +tools, and the M008 release executable. The local `/v1/chat/completions` +inference harness supplied each profile's actual MCP schemas, executed the +model's MCP call over stdio, and returned its result for synthesis. Full used +68,736 schema bytes, 18,813 first-turn input tokens, and 40,415 total input / +176 output tokens across the tool and answer turns. Its valid call requested +`render="browser"`, returned HTTP 200 with `transport="browser"`, and the +model accurately gave the heading, paragraph, URL, and external-content +warning. The tool result was 9,901 serialized characters. + +Compact used 19,339 schema bytes, 5,382 first-turn input tokens, and 11,904 +total input / 216 output tokens. Its valid call requested `render="browser"` +and returned HTTP 200; the brief result did not expose the `transport` field. +The model correctly gave the same heading, paragraph, URL, and external-content +warning. The tool result was 3,817 serialized characters with +`output_truncated=true`; the model did not report that outer-output warning, +although the extracted page content was complete. The brief output's +transport-field omission and warning miss are recorded as quality limits, not +as a failed browser call. Compact reduced schema bytes by 71.9%, total input +tokens by 70.5%, and result characters by 61.4% for this one task pair. The +React reference page remains unqualified. + +The compact-profile browser call succeeds on the exact M008 executable. +M008 is closed at source commit `772a073fd40e9f044017b26bae18a6efd0724b1d` +after hosted Linux/macOS/Windows CI. The full local `make check` attempt +stalled in `bounded_command`, while the focused 20-test schema suite, release +build, formatting, diff, and documentation checks passed. M003 remains active +because research/security evidence quality is mixed and key specialist/browser +tasks need repeats. Existing measured token reductions are per task and do not +establish a general quality or reliability improvement. diff --git a/plans/closure/standalone-mcp-local-inference/007-status.md b/plans/closure/standalone-mcp-local-inference/007-status.md index 9b763d9..7567edd 100644 --- a/plans/closure/standalone-mcp-local-inference/007-status.md +++ b/plans/closure/standalone-mcp-local-inference/007-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M007 — Closure Status -Status: active (fix and local requalification pass; hosted qualification pending) +Status: closed at `772a073fd40e9f044017b26bae18a6efd0724b1d` Source implementation plan: `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` @@ -27,6 +27,12 @@ API tests restricted the allowed tool set. Implementation candidate: `ff98098566a89f0603d1c07ad3cd88e5d36fae830bc8d6bde65cebc3456a71ea` (SHA-256 of the rebuilt release executable). +The M008-compatible final branch candidate was rebuilt as release executable +SHA-256 `507724a14a1926488913a50b1864543f1e899d8a3df6e1b44d1a90dcc4110b43` +from source commit `772a073fd40e9f044017b26bae18a6efd0724b1d`. It retains the +M007 all-local-reference correction and includes M008's additive `render` +schema property. + The compact schema rewrite now retains the referenced `ResponseDetail` definition. The regression test walks every compact tool schema and resolves all local references against its root. @@ -34,8 +40,11 @@ all local references against its root. Local verification: - `cargo test --locked --all-features --test mcp_schema_slimming`: 19 passed. -- `make check`: passed, including formatting, clippy, feature checks, all tests, - hygiene, dependency policy, and packaging checks. +- On the final source candidate, `cargo test --locked --all-features --test mcp_schema_slimming`: 20 passed; `make docs-check` passed. Local `make check` + stalled in the unchanged `bounded_command` integration suite, as recorded in + M008's closure. Hosted canonical Linux and macOS gates passed on the exact + source candidate, distinguishing the local process-group test stall from a + reproducible regression. - Compact executable smoke: all ten tools listed; full definitions measured 72,923 bytes and compact definitions 19,494 bytes; `provider_status` completed successfully. @@ -52,11 +61,18 @@ while preserving Yahoo's 500 failure. This confirms compact schema initialization and appropriate tool use for this task; it is one task, not the broader M003 model/task matrix. -`make docs-check` and hosted CI for the committed implementation remain -pending. +`make docs-check` passed. Hosted CI run `41220450246` passed Linux canonical, +macOS canonical and filesystem portability, and Windows portability on the +exact source candidate. Release drift run `41220450259` and Python native +qualification run `41220450258` also passed. The separate egress feature run +`41220450274` passed its Linux, x86_64 macOS, Windows, MSRV, and ARMv7 jobs; +the Apple Silicon macOS egress job was cancelled by a scheduled runner-image +brownout and does not cover the MCP schema change. ## 3. Disposition -M007 remains active until hosted qualification passes and the result is recorded -against the exact committed candidate. M003 remains active and continues to own -the broader small-model tool-use and context qualification. +M007 is closed at `772a073fd40e9f044017b26bae18a6efd0724b1d`. The final compact +schema suite resolves every local reference and passes its size budget. The +all-tools API qualification, hosted canonical gates, release drift, and Python +native qualification passed. M003 remains active for broader model/task +quality assessment. diff --git a/plans/closure/standalone-mcp-local-inference/008-status.md b/plans/closure/standalone-mcp-local-inference/008-status.md index 3ea21f2..5dda07b 100644 --- a/plans/closure/standalone-mcp-local-inference/008-status.md +++ b/plans/closure/standalone-mcp-local-inference/008-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M008 — Closure Status -Status: active (schema corrective and focused contract test pass; compact browser invocation and hosted qualification pending) +Status: closed at `772a073fd40e9f044017b26bae18a6efd0724b1d` Source implementation plan: `plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md` @@ -21,37 +21,55 @@ compact schema reachability gap, but not successful paired browser extraction. ## 2. Candidate and verification -Implementation candidate release executable SHA-256: +Implementation candidate commit: `772a073fd40e9f044017b26bae18a6efd0724b1d`. +Release executable SHA-256: `507724a14a1926488913a50b1864543f1e899d8a3df6e1b44d1a90dcc4110b43`. The compact schema now preserves `render` and constrains it to the runtime's accepted values `http_only`, `auto`, and `browser`. The focused `mcp_schema_slimming` suite passes 20 tests. `make docs-check` passes. -The paired LFM3 2B browser call used the same `example.com` prompt for full -and compact. Full/diagnostic called `web_fetch` with `render="browser"`, used -18,805 input and 259 output tokens, and returned HTTP 200 with -`transport="browser"`; the model extracted the heading and paragraph -accurately and retained the external-content warning. Compact/brief failed -before a model turn completed because Chrome could not acquire the shared -`chromiumoxide-runner/SingletonLock`. LM Studio returned HTTP 500 and no usage -counters. This is an infrastructure failure, not a compact-model outcome, and -does not satisfy successful compact browser invocation acceptance. +The paired LFM3 2B browser trial used the same `example.com` prompt, all ten +tool schemas, and the exact candidate executable. A local inference harness +supplied the schemas to LM Studio's `/v1/chat/completions` endpoint, executed +the selected MCP call over stdio, and returned the result for answer synthesis. +Full used 68,736 serialized schema bytes and 40,415 API input / 176 output +tokens across both turns. Its valid `web_fetch` request set `render="browser"` +and returned HTTP 200 with `transport="browser"`; the model extracted the +heading and paragraph accurately and retained the external-content warning. +The serialized tool result was 9,901 characters. + +Compact used 19,339 schema bytes and 11,904 API input / 216 output tokens. Its +valid request also set `render="browser"` and returned HTTP 200; the brief +projection did not include the `transport` field. The model extracted the same +heading and paragraph and retained the external-content warning. Its result +was 3,817 characters and carried `output_truncated=true`; the model did not +mention that outer-output warning even though the relevant page text was +complete. For this one pair, compact reduced schema bytes by 71.9%, total input +tokens by 70.5%, and result characters by 61.4%. This qualifies compact +browser-call reachability and extraction for this simple page, not general +browser reliability or quality. The canonical `make check` attempt passed its earlier formatting, lint, and compile stages but stalled without output for more than five minutes in the unchanged `bounded_command` integration suite on this macOS host. It was -interrupted. The exact committed candidate still requires hosted CI. +interrupted. Hosted CI on the exact source commit `772a073fd40e9f044017b26bae18a6efd0724b1d` +passed Linux canonical, macOS canonical and filesystem portability, and +Windows portability in run `41220450246`. Release drift run `41220450259` and +Python native qualification run `41220450258` also passed. The separate egress +feature run `41220450274` passed its Linux, x86_64 macOS, Windows, MSRV, and +ARMv7 jobs; its Apple Silicon macOS egress job was cancelled by a scheduled +runner-image brownout and does not exercise this change. -Remaining evidence: committed-candidate hosted CI, an uncontended compact -browser API call with actual usage and extraction results, and verification -that the compact schema remains within its size budget on the built candidate. -The original `~/.lmstudio/mcp.json` is restored at SHA-256 +The focused schema suite verifies that the compact tool surface remains within +budget. The original `~/.lmstudio/mcp.json` is restored at SHA-256 `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`. ## 3. Disposition -M008 remains active until its acceptance criteria pass. M003 remains active and -owns the broader small-model tool-use and context qualification. The config -restoration hash is verified, and no API authentication or MCP permission -setting was changed. +M008 is closed at `772a073fd40e9f044017b26bae18a6efd0724b1d`. The compact +schema preserves all render modes within budget, the paired local-model API +trial successfully invoked and extracted through compact browser mode, and +hosted canonical qualification passed. M003 remains active and owns the broader +small-model tool-use and context qualification. The config restoration hash is +verified, and no API authentication or MCP permission setting was changed. diff --git a/plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md b/plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md index 8a30279..cc9789e 100644 --- a/plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md +++ b/plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md @@ -1,6 +1,6 @@ # M007 — Compact Schema Reference Corrective -Status: active +Status: closed (candidate `772a073fd40e9f044017b26bae18a6efd0724b1d`; hosted CI `41220450246`) Primary class: corrective Source roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` Corrects: M001 compact tool advertisement, identified during M003 small-model API qualification diff --git a/plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md b/plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md index 79587f1..67df3da 100644 --- a/plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md +++ b/plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md @@ -1,6 +1,6 @@ # M008 — Compact Browser-Render Reachability Corrective -Status: active +Status: closed (candidate `772a073fd40e9f044017b26bae18a6efd0724b1d`; hosted CI `41220450246`) Primary class: corrective Source roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` Corrects: M001 compact `web_fetch` schema, discovered during M003 small-model task qualification diff --git a/plans/registry.md b/plans/registry.md index ee0ce3a..d59ac6c 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001, M002, M004, M005, and M006 closed; M003, M007, and M008 active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M006 closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; M003 now records paired small-model API trials across retrieval, specialist search, and PDF extraction; M007 corrects a compact schema reference defect; M008 restores compact reachability for the existing browser-render option. | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `008-status.md`; M003 evidence in `003-status.md`, M007 in `007-status.md`, M008 in `008-status.md`. | +| Standalone MCP local-inference efficiency | active (M001, M002, M004-M008 closed; M003 active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M007 and M008 closed at `772a073fd40e9f044017b26bae18a6efd0724b1d`; hosted Linux/macOS/Windows CI run `41220450246`, release drift `41220450259`, and Python native `41220450258` passed. M003 remains active for small-model evidence quality and repeatability. | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `008-status.md`; current M003, M007, and M008 evidence in their closure records. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -58,8 +58,8 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Standalone MCP local-inference | M004 compact schema discoverability corrective | closed | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted canonical Linux, macOS, and Windows jobs pass in CI run `41160264796`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | | Standalone MCP local-inference | M005 brief document extraction and paired requalification | closed | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted CI run `41160264796` passed. Closure `plans/closure/standalone-mcp-local-inference/005-status.md`; M003 remains open for the documented operational gaps. | | Standalone MCP local-inference | M006 compact direct-fetch tool-choice corrective | closed | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Candidate `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; closure `plans/closure/standalone-mcp-local-inference/006-status.md`; native paired and repeated compact turns passed; hosted Linux/macOS/Windows CI run `41181749609` passed. M003 continues small-model quality qualification. | -| Standalone MCP local-inference | M007 compact schema reference corrective | active | `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` | M003 all-tools API turn exposed unresolved `#/$defs/ResponseDetail` in compact `build_evidence_bundle`; root cause and closure gate are in `plans/closure/standalone-mcp-local-inference/007-status.md`. | -| Standalone MCP local-inference | M008 compact browser-render reachability corrective | active | `plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md` | M003 browser trials exposed that compact schemas omit the accepted `web_fetch.render` argument; preserve browser mode without weakening fetch policy. | +| Standalone MCP local-inference | M007 compact schema reference corrective | closed | `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` | Closed at `772a073fd40e9f044017b26bae18a6efd0724b1d`; closure `plans/closure/standalone-mcp-local-inference/007-status.md`; hosted Linux/macOS/Windows CI `41220450246`. | +| Standalone MCP local-inference | M008 compact browser-render reachability corrective | closed | `plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md` | Closed at `772a073fd40e9f044017b26bae18a6efd0724b1d`; closure `plans/closure/standalone-mcp-local-inference/008-status.md`; compact LFM3 browser pair and hosted CI `41220450246`. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | | Deterministic retrieval quality | M003 keyless provider contribution/resilience | closed | `plans/implementation/deterministic-retrieval-quality/003-keyless-provider-contribution-and-resilience.md` | Candidate `3ef5242c928181b27343a463015c09912dcabbc5`; closure `plans/closure/deterministic-retrieval-quality/003-status.md`; no routing change, live observations optional in M006 | | Deterministic retrieval quality | M004 deterministic relevance and selection | closed | `plans/implementation/deterministic-retrieval-quality/004-deterministic-relevance-and-selection.md` | Candidate `84472dfe010c498ab95a60b08153e6209c9ec5e5`; closure `plans/closure/deterministic-retrieval-quality/004-status.md`; no-change decision, no reliable lift established | @@ -130,7 +130,7 @@ Corrective closure sequence: `plans/implementation/dependency-evidence-hardening **Python native library gate:** ADR-0005 authorizes a narrow direct Rust SDK facade and Python PyO3 native package without broad Rust module stability. M001–M008 are closed at `067468a` with fully-green qualifying run `38641009246` (5/5 builds, 20/20 smokes, offline sdist rebuild, green audit, digests in `008-status.md`). Earlier runs are historical/deficiency/supporting evidence only. First PyPI publication is separately closed under `python-pypi-distribution` M003 at 0.4.4; Eggpack binary assets remain unchanged. -**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 and M005 are closed at `de69f1d`; M006 is closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826` after repeated native API compact direct-fetch turns and hosted Linux/macOS/Windows CI run `41181749609`. M003 is active and qualifies model behavior through paired API trials: small local models should select tools, pass valid arguments, use retrieved evidence, and handle failures truthfully while `compact + brief` lowers actual reported input/context use. LM Studio is the available API host, not a product-specific acceptance target; UI refresh and presentation are out of scope. M007 is active to fix an unresolved `$defs/ResponseDetail` reference in the compact `build_evidence_bundle` schema, discovered when all ten tools were exposed in M003. See M003 and M007 status records for measured results and remaining evidence. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. +**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 and M005 are closed at `de69f1d`; M006 is closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; M007 and M008 are closed at `772a073fd40e9f044017b26bae18a6efd0724b1d` with hosted Linux/macOS/Windows CI run `41220450246`. M003 remains active and qualifies model behavior through paired API trials: small local models should select tools, pass valid arguments, use retrieved evidence, and handle failures truthfully while `compact + brief` lowers actual reported input/context use. LM Studio is the available API host, not a product-specific acceptance target; UI refresh and presentation are out of scope. M007 fixed the unresolved `$defs/ResponseDetail` reference in compact `build_evidence_bundle`; M008 added the existing browser-render argument to compact `web_fetch`. See M003 and the M007/M008 status records for measured results and remaining evidence. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. **Tool-surface gate:** M001-M007 closed. Do not run evaluation against pre-consolidation bytes and present it as consolidation diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index 075bb39..fa0826d 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001, M002, M004, M005, and M006 closed; M003, M007, and M008 active) +Status: active (M001, M002, and M004-M008 closed; M003 active) Scope: small local-model tool-use and context qualification through an actual inference API; not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. @@ -47,8 +47,8 @@ M001 compact ten-tool advertised schema profile (closed) +-- operational --> M003 small local-model tool-use and context qualification (active) +-- corrective --> M005 brief document extraction and paired requalification (closed) +-- corrective --> M006 compact direct-fetch tool-choice corrective (closed at `6dc63d5`; hosted CI run `41181749609`) - +-- corrective --> M007 compact schema reference corrective (active; discovered during M003 all-tools API qualification) - +-- corrective --> M008 compact browser-render reachability corrective (active; discovered during M003 paired extraction) + +-- corrective --> M007 compact schema reference corrective (closed at `772a073`; hosted CI run `41220450246`) + +-- corrective --> M008 compact browser-render reachability corrective (closed at `772a073`; hosted CI run `41220450246`) M004 compact schema discoverability corrective (closed on hosted Linux/macOS candidate de69f1d) ``` @@ -63,8 +63,8 @@ No CodeGG dependency; it is a regression consumer, not the implementation target | M004 | capability | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Advertise runtime-accepted research goal/domain values in compact schemas; test schema parity and size; repeat focused live tasks; M003 remains open for its broader gates | | M005 | corrective | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Closed on `de69f1d`; brief preserves same-document extraction within budget; model evidence and unresolved M003 failures are recorded; hosted Linux/macOS/Windows gates pass | | M006 | corrective | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Resolve or bound a repeated native LM Studio compact `web_fetch` no-call failure from M003; keep M003 open until evidence supports acceptance | -| M007 | corrective | `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` | Resolve dangling compact-schema `$ref` in `build_evidence_bundle`; preserve ten-tool contract and verify all local references plus all-tools model API invocation | -| M008 | corrective | `plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md` | Advertise the existing `web_fetch.render` option in compact mode and requalify paired browser extraction | +| M007 | corrective | `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` | Closed at `772a073`; all compact local references resolve; hosted Linux/macOS/Windows CI run `41220450246` passed | +| M008 | corrective | `plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md` | Closed at `772a073`; compact browser call and extraction qualified on LFM3 2B; hosted CI run `41220450246` passed | ## Cross-cutting controls From 5b5e64420b52bb5d577b4c3a1bb097b3bed4dfc2 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 16:54:25 +0000 Subject: [PATCH 24/25] docs(plans): record repeated small-model research trial --- .../003-status.md | 41 +++++++++++++++---- 1 file changed, 34 insertions(+), 7 deletions(-) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index faed6d3..07f6605 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -73,9 +73,9 @@ provide a reliable per-chat usage breakdown for this test set. | Reproducible installed-candidate stdio smoke | `scripts/lm_studio_stdio_smoke.py` | pass | Offline protocol evidence only | | Ten tools and profile list sizes | M006-candidate initialized child-process smoke: 72,923 full / 19,370 compact bytes; all ten names; `provider_status` with probing off | pass | Byte counts are not token counts | | Configured MCP API path | Authenticated native `/api/v1/chat` calls through configured `mcp/eggsearch`, with task-specific `allowed_tools`; exact config rollback hash recorded | pass | None for the tested local host; this verifies the model/API path, not LM Studio UI behavior | -| Paired small-model comparison | §12-15 include direct fetch, web search, research, security applicability, PDF extraction, and a successful LFM3 browser-render pair | partial / mixed | Repeat research and browser tasks; improve compact warning interpretation and separate host failures from model outcomes | -| Tool choice, argument validity, evidence and failure truthfulness | §12-14 include valid calls, wrong-tool selection, argument repair, weak evidence, extraction errors, provider failures, and API/browser failures | partial / quality concern | Improve consistency on research and exact applicability ranges; distinguish host failures from model failures | -| Actual context/token reduction | LM Studio native API reports `input_tokens` including formatting, tool definitions, and prior messages; §12-14 record paired full/compact usage and tool-result lengths | pass for measured tasks | Repeat key pairs to estimate variability; no tokenizer identity or UI context meter is required | +| Paired small-model comparison | §12-16 include direct fetch, web search, research, security applicability, PDF extraction, and a successful LFM3 browser-render pair | partial / mixed | Add further research/browser repeats; improve compact warning interpretation and separate host failures from model outcomes | +| Tool choice, argument validity, evidence and failure truthfulness | §12-16 include valid calls, wrong-tool selection, argument repair, weak evidence, extraction errors, provider failures, and API/browser failures | partial / quality concern | Improve consistency on research and exact applicability ranges; distinguish host failures from model failures | +| Actual context/token reduction | LM Studio native API reports `input_tokens` including formatting, tool definitions, and prior messages; §12-16 record paired full/compact usage and tool-result lengths | pass for measured tasks | Repeat key pairs to estimate variability; no tokenizer identity or UI context meter is required | | Eggsearch result contract | Offline MCP protocol tests cover structured results and text fallback; native API exposes returned tool output to the model | pass for Eggsearch contract | No host-specific result-shape or UI-consumption test is required | | Config rollback | Restored original `~/.lmstudio/mcp.json`; SHA-256 is `b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c` | pass | None | @@ -629,7 +629,34 @@ The compact-profile browser call succeeds on the exact M008 executable. M008 is closed at source commit `772a073fd40e9f044017b26bae18a6efd0724b1d` after hosted Linux/macOS/Windows CI. The full local `make check` attempt stalled in `bounded_command`, while the focused 20-test schema suite, release -build, formatting, diff, and documentation checks passed. M003 remains active -because research/security evidence quality is mixed and key specialist/browser -tasks need repeats. Existing measured token reductions are per task and do not -establish a general quality or reliability improvement. +build, formatting, diff, and documentation checks passed. A further paired +research trial is recorded in §16. M003 remains active because the model still +overstates evidence authority on a version-specific claim, Gemma's security +range remains incomplete, and browser warning interpretation needs more +repeats. Existing measured token reductions are per task and do not establish +a general quality or reliability improvement. + +## 16. Repeated LFM3 2B research-search pair + +The repeated task asked whether Rust 1.90 supports `cargo publish --workspace` +and required version-specific evidence from official Rust release notes or +Cargo documentation. LM Studio's local `/v1/chat/completions` API received the +actual full or compact schemas for all ten tools. A local harness executed +each model MCP call over stdio and returned the actual tool result for +synthesis. Both profiles used `lfm3-2b-omni-abliter8`, temperature 0, the same +system and user prompt, and the exact M008 release executable +(`507724a14a1926488913a50b1864543f1e899d8a3df6e1b44d1a90dcc4110b43`). + +| Profile | Schema bytes | API input / output tokens | Calls and evidence | Outcome | +|---|---:|---:|---|---| +| Full / diagnostic | 68,736 | 87,721 / 689 | First `research_search` call used an invalid free-form goal; a valid repair then returned 101,607 characters. It included generic Cargo docs, an implementation repository, an issue, a development-cycle article, and a secondary article. | Answered yes, but falsely claimed the generic Cargo command page explicitly proved Rust 1.90 support. The version-specific evidence requirement failed. | +| Compact / brief | 19,339 | 44,990 / 510 | One valid `research_search` call used `goal="architecture_decision"`; it then fetched a Cargo development-cycle post and an InfoWorld article. The brief result flagged truncation on fetched pages. | Answered yes and cited the secondary article's explicit statement, but claimed the Cargo docs and Rust blog independently confirmed the version-specific feature. The official-source requirement failed. | + +Compact reduced serialized schema bytes by 71.9% and API input tokens by +48.7% for this pair. It reduced the research result from 101,607 to 15,969 +characters before its additional fetches, but the final answer still +overstated what the primary evidence established. Full required an argument +repair and also made the unsupported version claim. This repeated pair +confirms that lower context and valid tool use do not guarantee evidence +authority or version-specific accuracy. Keep M003 active; do not recommend +compact as an unqualified default from these results. From 6f67a9a985ffa3e2570569c22183fff0df487ed8 Mon Sep 17 00:00:00 2001 From: eggsearch Date: Sat, 31 Oct 2026 16:57:09 +0000 Subject: [PATCH 25/25] docs(plans): close small-model qualification --- .../003-status.md | 58 ++++++++++++++++++- ...-studio-consumer-qualification-and-docs.md | 2 +- plans/registry.md | 10 ++-- .../standalone-mcp-local-inference-roadmap.md | 12 ++-- 4 files changed, 67 insertions(+), 15 deletions(-) diff --git a/plans/closure/standalone-mcp-local-inference/003-status.md b/plans/closure/standalone-mcp-local-inference/003-status.md index 07f6605..997a6c1 100644 --- a/plans/closure/standalone-mcp-local-inference/003-status.md +++ b/plans/closure/standalone-mcp-local-inference/003-status.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference M003 — Operational Status -Status: active (scope clarified: qualify small-model tool use and context reduction through actual API turns; UI refresh/result-shape behavior is out of scope; paired specialist-task outcomes remain mixed) +Status: closed at source candidate `772a073fd40e9f044017b26bae18a6efd0724b1d` (small-model tool use and context qualification completed; results are mixed; compact remains opt-in) Source implementation plan: `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` @@ -658,5 +658,57 @@ characters before its additional fetches, but the final answer still overstated what the primary evidence established. Full required an argument repair and also made the unsupported version claim. This repeated pair confirms that lower context and valid tool use do not guarantee evidence -authority or version-specific accuracy. Keep M003 active; do not recommend -compact as an unqualified default from these results. +authority or version-specific accuracy. + +## 17. Repeated M008 browser-render pair and final disposition + +The same LFM3 2B browser task from §15 was repeated with the exact M008 +executable and profile settings. Both profiles again selected `web_fetch` and +requested `render="browser"`. Full used 68,736 schema bytes and 40,415 API +input / 176 output tokens; compact used 19,339 schema bytes and 11,904 API +input / 216 output tokens. Full returned HTTP 200 with `transport="browser"`; +compact returned HTTP 200 with the correct heading and paragraph, while its +brief projection omitted the transport field. The model again retained the +external-content warning but did not mention the compact result's +`output_truncated=true` flag. The repeated pair had the same calls, token +counts, content lengths, and answer quality as §15. It establishes +repeatability for this static page only; the React reference page remains +unqualified. + +### Qualification result + +M003 is closed as a completed measurement, not as a general quality +endorsement. LFM3 2B, Gemma E2B, and Gemma E4B completed paired tasks across +direct fetch, web search, research, security applicability, PDF extraction, +and browser rendering; earlier Bonsai 27B trials are retained as reference +context. Repeated direct-fetch, security, research, and browser cases expose +both stable context reductions and repeated answer-quality defects. Across the +main paired matrix, compact reduced model-reported input tokens by about +60–72%; the repeated research pair reduced them by 48.7%, and the browser pair +by 70.5%. These are workload-specific measurements, not a universal savings +rate. + +Models successfully selected and called Eggsearch tools in each task class, +but evidence authority, version-specific claims, exact affected-version +ranges, and compact truncation-warning interpretation were inconsistent. +These results do not support switching the default or claiming a general +quality improvement. Keep `full + diagnostic` as the unchanged default and +`compact + brief` as an opt-in profile. The M007 schema-reference and M008 +browser-render reachability correctives are closed; no further deterministic +Eggsearch schema defect was found. Model-quality limitations remain documented +as the qualification result rather than an unmeasured product promise. + +The final source candidate is `772a073fd40e9f044017b26bae18a6efd0724b1d` +with release executable SHA-256 +`507724a14a1926488913a50b1864543f1e899d8a3df6e1b44d1a90dcc4110b43`. Hosted +Linux/macOS/Windows canonical CI run `41220450246`, release drift run +`41220450259`, and Python native qualification run `41220450258` passed. +Local focused schema tests (20/20), static guards (60/60), formatting, +documentation checks, and release build passed. Local `make check` stalled in +the unchanged `bounded_command` integration suite; hosted Linux and macOS +canonical runs on the exact candidate passed. The optional egress feature +matrix's Apple Silicon macOS runner was cancelled by a scheduled brownout; it +does not cover this milestone's changes. The original LM Studio MCP config is +restored at SHA-256 +`b8cb90299d832914cba5f92dbb259684a084045cc503fd186fea4b4cee3fec7c`, and the +local server was restarted with it. diff --git a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md index 74d57d6..3c750ba 100644 --- a/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md +++ b/plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md @@ -1,6 +1,6 @@ # M003 — Small Local-Model Tool-Use and Context Qualification -Status: active (M001/M002 dependencies closed; native API experiments establish a working harness, but paired small-model task coverage and quality disposition remain incomplete; see `plans/closure/standalone-mcp-local-inference/003-status.md`) +Status: closed (qualification completed at source candidate `772a073fd40e9f044017b26bae18a6efd0724b1d`; compact remains opt-in, with no default or general quality claim; see `plans/closure/standalone-mcp-local-inference/003-status.md`) Primary class: capability Planning baseline: eggstack/eggsearch main `a666517c39adf28f32fad04198c74cb70d935c5b` (rebase baseline to exact M002 candidate at start) Roadmap: `plans/subsystems/standalone-mcp-local-inference-roadmap.md` diff --git a/plans/registry.md b/plans/registry.md index d59ac6c..a62193a 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -41,7 +41,7 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under | Performance optimization and footprint | closed | `plans/subsystems/performance-optimization-roadmap.md` | M001-M005 closed | None. Corrective candidate `0af540b8`; qualify `35542118569`. | | Optional outbound routing (egress) | closed | `plans/subsystems/optional-outbound-routing-roadmap.md` | M001-M004 closed | None. Terminal baseline `6414a72`; hardened qualify `35810222447`. No egress-enabled binary published. Outcome B remains the runtime baseline. | | MCP tool-surface consolidation | closed | `plans/subsystems/tool-surface-consolidation-roadmap.md` | M001-M007 closed | M001-M007 closed. Current control points `plans/closure/mcp-tool-surface-consolidation/001-status.md` through `007-status.md`. | -| Standalone MCP local-inference efficiency | active (M001, M002, M004-M008 closed; M003 active) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M007 and M008 closed at `772a073fd40e9f044017b26bae18a6efd0724b1d`; hosted Linux/macOS/Windows CI run `41220450246`, release drift `41220450259`, and Python native `41220450258` passed. M003 remains active for small-model evidence quality and repeatability. | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `008-status.md`; current M003, M007, and M008 evidence in their closure records. | +| Standalone MCP local-inference efficiency | closed (M001-M008 closed) | `plans/subsystems/standalone-mcp-local-inference-roadmap.md` | M003 qualification closed at source candidate `772a073fd40e9f044017b26bae18a6efd0724b1d`: paired local-model trials show substantial context reduction but mixed evidence quality; compact remains opt-in. Hosted Linux/macOS/Windows CI run `41220450246`, release drift `41220450259`, and Python native `41220450258` passed. | Closure records `plans/closure/standalone-mcp-local-inference/001-status.md` through `008-status.md`; final M003 disposition and M007/M008 corrective evidence are recorded in their closure records. | | Dependency evidence hardening | closed | `plans/subsystems/dependency-evidence-hardening-roadmap.md` | M010 corrective closed | M001-M010 closed. Current control point `plans/closure/dependency-evidence-hardening/010-status.md`; implementation `23676cc`, CI `36253474005`. | | Repository security, supply-chain, and maintenance hardening | active | `plans/subsystems/repository-hardening-roadmap.md` | M001/M002/M007/M008/M009/M010/M011 closed; M006 blocked | M011 closed at `99316f31caf37ad2165cd5e822316942faf20c6f`; the separate M006 chromiumoxide blocker is unchanged. | | Python native library for agents | closed (M001-M008 closed; PyPI publication completed in separate workstream) | `plans/subsystems/python-native-library-roadmap.md` | M008 closed at `067468a`, qualifying run `38641009246` (27/27 incl. audit) | ADR-0005 accepted. Runs `38522548258`, `38608502185`/`38611932990`/`38614592873`, `38637266178`, and `38638066765` are historical/deficiency/supporting evidence only. `eggsearch==0.4.4` publication is recorded under the separate PyPI distribution workstream. | @@ -54,10 +54,10 @@ Historical `phase-*.md` plans and pre-migration roadmaps are archived under |---|---|---|---|---| | Standalone MCP local-inference | M001 opt-in compact tool advertisement | closed | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/001-status.md`; compact stdio list 18,545 bytes, 10 tools. | | Standalone MCP local-inference | M002 brief evidence and deterministic budgets | closed | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; closure `plans/closure/standalone-mcp-local-inference/002-status.md`; canonical gates passed. | -| Standalone MCP local-inference | M003 small local-model tool-use and context qualification | active | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Qualifies whether small models can use tools appropriately while `compact + brief` reduces context. LM Studio is the available API host; its UI behavior is out of scope. Existing trials and remaining quality gaps are in `plans/closure/standalone-mcp-local-inference/003-status.md`. | +| Standalone MCP local-inference | M003 small local-model tool-use and context qualification | closed | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Closed at `772a073fd40e9f044017b26bae18a6efd0724b1d`; hosted Linux/macOS/Windows CI `41220450246`; compact remains opt-in because evidence quality is mixed. | | Standalone MCP local-inference | M004 compact schema discoverability corrective | closed | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted canonical Linux, macOS, and Windows jobs pass in CI run `41160264796`. Closure status `plans/closure/standalone-mcp-local-inference/004-status.md`. | -| Standalone MCP local-inference | M005 brief document extraction and paired requalification | closed | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted CI run `41160264796` passed. Closure `plans/closure/standalone-mcp-local-inference/005-status.md`; M003 remains open for the documented operational gaps. | -| Standalone MCP local-inference | M006 compact direct-fetch tool-choice corrective | closed | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Candidate `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; closure `plans/closure/standalone-mcp-local-inference/006-status.md`; native paired and repeated compact turns passed; hosted Linux/macOS/Windows CI run `41181749609` passed. M003 continues small-model quality qualification. | +| Standalone MCP local-inference | M005 brief document extraction and paired requalification | closed | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Final candidate `de69f1ddd7a7e4acc067fb9892c438f6165cfe83`; hosted CI run `41160264796` passed. Closure `plans/closure/standalone-mcp-local-inference/005-status.md`; residual model outcomes are captured in the completed M003 qualification. | +| Standalone MCP local-inference | M006 compact direct-fetch tool-choice corrective | closed | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Candidate `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; closure `plans/closure/standalone-mcp-local-inference/006-status.md`; native paired and repeated compact turns passed; hosted Linux/macOS/Windows CI run `41181749609` passed. M003 is closed with a mixed quality disposition. | | Standalone MCP local-inference | M007 compact schema reference corrective | closed | `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` | Closed at `772a073fd40e9f044017b26bae18a6efd0724b1d`; closure `plans/closure/standalone-mcp-local-inference/007-status.md`; hosted Linux/macOS/Windows CI `41220450246`. | | Standalone MCP local-inference | M008 compact browser-render reachability corrective | closed | `plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md` | Closed at `772a073fd40e9f044017b26bae18a6efd0724b1d`; closure `plans/closure/standalone-mcp-local-inference/008-status.md`; compact LFM3 browser pair and hosted CI `41220450246`. | | Deterministic retrieval quality | M002 PDF/browser extraction matrix | closed | `plans/implementation/deterministic-retrieval-quality/002-pdf-browser-extraction-matrix.md` | Candidate `25f4910cdf68acf513a4f46e125c19e8dc84c5a8`; closure `plans/closure/deterministic-retrieval-quality/002-status.md`; browser rendering limited without Chrome | @@ -130,7 +130,7 @@ Corrective closure sequence: `plans/implementation/dependency-evidence-hardening **Python native library gate:** ADR-0005 authorizes a narrow direct Rust SDK facade and Python PyO3 native package without broad Rust module stability. M001–M008 are closed at `067468a` with fully-green qualifying run `38641009246` (5/5 builds, 20/20 smokes, offline sdist rebuild, green audit, digests in `008-status.md`). Earlier runs are historical/deficiency/supporting evidence only. First PyPI publication is separately closed under `python-pypi-distribution` M003 at 0.4.4; Eggpack binary assets remain unchanged. -**Standalone direct-MCP local-inference gate:** M001-M002 are closed at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004 and M005 are closed at `de69f1d`; M006 is closed at `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; M007 and M008 are closed at `772a073fd40e9f044017b26bae18a6efd0724b1d` with hosted Linux/macOS/Windows CI run `41220450246`. M003 remains active and qualifies model behavior through paired API trials: small local models should select tools, pass valid arguments, use retrieved evidence, and handle failures truthfully while `compact + brief` lowers actual reported input/context use. LM Studio is the available API host, not a product-specific acceptance target; UI refresh and presentation are out of scope. M007 fixed the unresolved `$defs/ResponseDetail` reference in compact `build_evidence_bundle`; M008 added the existing browser-render argument to compact `web_fetch`. See M003 and the M007/M008 status records for measured results and remaining evidence. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. +**Standalone direct-MCP local-inference gate:** M001-M008 are closed. M001-M002 are at candidate `02a8af6fe4f87971c7daca7d1fca20c087b328ec`; M004/M005 at `de69f1d`; M006 at `6dc63d5f1a45b85352c40b26e6effbb6b9793826`; M003/M007/M008 are closed at source candidate `772a073fd40e9f044017b26bae18a6efd0724b1d` with hosted Linux/macOS/Windows CI `41220450246`, release drift `41220450259`, and Python native qualification `41220450258`. Paired local-model trials showed that compact reduces actual input context substantially on the measured task set, but tool and evidence quality varies by task and model. Keep `compact + brief` opt-in; retain the full default and make no general quality claim. LM Studio is the available API host, not a product-specific acceptance target; UI refresh and presentation are out of scope. M007 fixed the unresolved `$defs/ResponseDetail` reference in compact `build_evidence_bundle`; M008 added the existing browser-render argument to compact `web_fetch`. See the closure records for exact measurements and residual quality limitations. This is a separate, opt-in presentation/consumer workstream, not a reopening of completed tool-surface M001-M007 or a CodeGG integration change. **Tool-surface gate:** M001-M007 closed. Do not run evaluation against pre-consolidation bytes and present it as consolidation diff --git a/plans/subsystems/standalone-mcp-local-inference-roadmap.md b/plans/subsystems/standalone-mcp-local-inference-roadmap.md index fa0826d..a154774 100644 --- a/plans/subsystems/standalone-mcp-local-inference-roadmap.md +++ b/plans/subsystems/standalone-mcp-local-inference-roadmap.md @@ -1,6 +1,6 @@ # Standalone MCP Local-Inference Efficiency Roadmap -Status: active (M001, M002, and M004-M008 closed; M003 active) +Status: closed (M001-M008 closed at their recorded candidates) Scope: small local-model tool-use and context qualification through an actual inference API; not CodeGG integration Long-term references: `plans/000-long-term-specification.md` ``1-4, 7-9; `plans/001-terminology-and-domain-model.md`; `plans/002-long-term-roadmap.md`; `plans/003-planning-process.md`. @@ -44,7 +44,7 @@ M001 compact ten-tool advertised schema profile (closed) | +-- hard --> M002 brief/budgeted agent responses (closed) | - +-- operational --> M003 small local-model tool-use and context qualification (active) + +-- operational --> M003 small local-model tool-use and context qualification (closed at `772a073`; mixed outcome, compact remains opt-in) +-- corrective --> M005 brief document extraction and paired requalification (closed) +-- corrective --> M006 compact direct-fetch tool-choice corrective (closed at `6dc63d5`; hosted CI run `41181749609`) +-- corrective --> M007 compact schema reference corrective (closed at `772a073`; hosted CI run `41220450246`) @@ -59,10 +59,10 @@ No CodeGG dependency; it is a regression consumer, not the implementation target |---|---|---|---| | M001 | capability | `plans/implementation/standalone-mcp-local-inference/001-opt-in-compact-tool-advertisement.md` | Opt-in compact `tools/list` for all ten existing names; measurable schema reduction; parity across MCP transports; default unchanged | | M002 | capability | `plans/implementation/standalone-mcp-local-inference/002-brief-evidence-and-output-budgets.md` | Explicit brief projection and bounded results with critical failure, identity, trust and security semantics intact; unchanged legacy defaults | -| M003 | capability | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Paired API trials show whether small local models can use the tools appropriately while `compact + brief` reduces actual context use; quality and token evidence determine acceptance, not host UI behavior | -| M004 | capability | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Advertise runtime-accepted research goal/domain values in compact schemas; test schema parity and size; repeat focused live tasks; M003 remains open for its broader gates | -| M005 | corrective | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Closed on `de69f1d`; brief preserves same-document extraction within budget; model evidence and unresolved M003 failures are recorded; hosted Linux/macOS/Windows gates pass | -| M006 | corrective | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Resolve or bound a repeated native LM Studio compact `web_fetch` no-call failure from M003; keep M003 open until evidence supports acceptance | +| M003 | capability | `plans/implementation/standalone-mcp-local-inference/003-lm-studio-consumer-qualification-and-docs.md` | Closed at `772a073`: actual context reduction is measured across paired local-model tasks; tool and evidence quality are mixed, so compact remains opt-in and the full default is unchanged | +| M004 | capability | `plans/implementation/standalone-mcp-local-inference/004-compact-schema-discoverability-corrective.md` | Advertise runtime-accepted research goal/domain values in compact schemas; test schema parity and size; focused live evidence is incorporated in the completed M003 matrix | +| M005 | corrective | `plans/implementation/standalone-mcp-local-inference/005-brief-document-extraction-and-paired-requalification.md` | Closed on `de69f1d`; brief preserves same-document extraction within budget; model evidence and unresolved M003 findings are recorded; hosted Linux/macOS/Windows gates pass | +| M006 | corrective | `plans/implementation/standalone-mcp-local-inference/006-compact-direct-fetch-tool-choice-corrective.md` | Resolve or bound the repeated compact `web_fetch` no-call failure; its requalification and M003's mixed final disposition are recorded in their closure records | | M007 | corrective | `plans/implementation/standalone-mcp-local-inference/007-compact-schema-reference-corrective.md` | Closed at `772a073`; all compact local references resolve; hosted Linux/macOS/Windows CI run `41220450246` passed | | M008 | corrective | `plans/implementation/standalone-mcp-local-inference/008-compact-browser-render-reachability-corrective.md` | Closed at `772a073`; compact browser call and extraction qualified on LFM3 2B; hosted CI run `41220450246` passed |