Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `web_fetch` and `batch_fetch` web items accept opt-in `include_screenshot` to capture a bounded 1280×720 viewport PNG (maximum 4 MiB) as native MCP image content. Structured JSON remains unchanged; recognized PDF responses are skipped, screenshot requests bypass caches, and screenshot-bearing batches run synchronously with images associated by original item index and label.
- `batch_fetch` web items accept the same PDF page-selection options as `web_fetch`, with page syntax prevalidated before item fetching. `repo_fetch` extracts explicitly named PDFs under configured local workspace roots by extension or `%PDF-` prefix, under `[local].pdf_max_file_bytes` and the existing PDF feature/config gates; PDFs remain excluded from local indexing and repo maps.
- `batch_fetch` web items accept `pdf`, `render`, and `browser_profile` with the same routing and cache semantics as `web_fetch`; repo items accept `symbol`, `symbol_kind`, `match_text`, `expand_to_block`, `max_block_lines`, and `prefer_local`. Host-local scope metadata now restricts batch workspace items to the same authorized view as `repo_fetch`. Item order, per-item failures, stable IDs, budgets, and telemetry are unchanged.
- `web_fetch` accepts the field names used by the reference MCP fetch server and common
search/fetch servers as compatibility aliases: `max_length` for `max_chars`, `start_index` for
`offset`, and `question` or `prompt` for `focus`. Aliases are wire-level spellings, not separate
Expand Down
51 changes: 46 additions & 5 deletions architecture/fetch.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,29 @@ validate_fetch_target (SSRF)
-> sanitize (core/sanitize.rs, three tiers)
```

### Typed target routing

The MCP tools keep their distinct request and response contracts. Their shared
internal `FetchLocator` resolution distinguishes the route before handing work
to its owning handler:

| Locator | Owning path | Boundary |
|---|---|---|
| `WebUrl` | `web_fetch` or a batch web item through the bounded fetch path | HTTP(S) validation, `FetchClient` SSRF/redirect policy, content detection, and existing PDF/HTML/browser behavior remain authoritative. |
| `Repo` | `repo_fetch` remote-source path | Structured host/owner/repo/ref/path and source-span semantics remain authoritative; a detected PDF retains the fetch document outcome and is not decoded as source code. |
| `Local` | `repo_fetch` workspace path | Root name and relative path only; authorized workspace resolution and `safe_read_file` remain the read boundary. Local PDF extraction belongs to PDF M008. |

Resolution is classification, not a transport shortcut: it cannot bypass the
owning handler's validation or security policy, and it does not expose a generic
MCP fetch schema. Batch preflight uses the same typed resolution before any
fetch starts. Web items carry `pdf`, `render`, and `browser_profile` controls
through the shared web-fetch path, alongside per-item cache and focus options;
repo items carry source selection through the repo-fetch handler. The trusted
`codegg/host-local-scope` metadata is forwarded unchanged to repo items, so
workspace reads retain the same authorized root and root-relative path boundary
as `repo_fetch`. Batch order, per-item failures, and aggregate budgets remain
owned by the batch envelope.

1. **Validate.** `limits::validate_url` (sync shape check) runs first;
`limits::validate_fetch_target` / `validate_fetch_target_with_resolved_addrs`
(async full check + DNS) runs per attempt and per redirect hop.
Expand Down Expand Up @@ -173,10 +196,11 @@ per-tier byte budgets with exact accounting (drift self-heals in `stats()`).
- `DerivedCacheKey { scope, raw_content_hash, extraction_key }` where the hash
is `xxh3_64(body)` (`build_raw_response_hash`) and `ExtractionCacheKey`
covers `extract_mode`, exact `max_chars`, `include_links`, `pdf_pages`,
`pdf_ocr`, `include_media`, `renderer_version: 4`, and `sanitize_output`.
Version 3 isolates entries derived before PDF page classification stopped
depending on `include_media`; version 2 isolated entries written before
representation negotiation existed.
`pdf_ocr`, `include_media`, `renderer_version: 5`, and `sanitize_output`.
Version 5 isolates document-scoped extraction; version 4 isolates the
encrypted-PDF refusal behavior; version 3 isolates entries derived before
PDF page classification stopped depending on `include_media`; version 2
isolated entries written before representation negotiation existed.
- `DerivedDocumentCacheEntry` stores the extracted title/description/text,
`raw_text`, links, truncation flags, the structured `FetchDocument` (blocks,
outline, chunks), trust markers, and transport tag.
Expand Down Expand Up @@ -434,7 +458,7 @@ An `/Encrypt` document is refused before decryption is ever considered:
`Document::authenticate_password` validated a password but never replaced
ciphertext, and ciphertext cannot be measured by the scan. `pdf_password` stays
in the tool schema for wire compatibility and is documented as unsupported.
That outcome narrowing is why the derived cache `renderer_version` is 4.
That outcome narrowing advanced the derived-cache renderer version to 4; the current version is 5 after document-scoped extraction.

---

Expand Down Expand Up @@ -561,6 +585,23 @@ IPv4-compatible IPv6 via the shared `limits::classify_ip`), and a 3 s DNS
resolution check rejecting private resolutions. It does not take
`allow_localhost` / `allow_private_network` overrides.

### Viewport screenshot capture

`include_screenshot = true` on `web_fetch` or a `batch_fetch` web item forces
`RenderPolicy::Browser`, even when the configured policy is `http_only`; an
explicit per-call `render = "http_only"` is a pre-I/O conflict. Capture runs
after normal DOM extraction and before the page closes, using the existing
1280×720 viewport. It requests PNG with no full-page extension, scrolling, input,
or other interaction. The PNG signature, IHDR dimensions, and encoded size are
checked before result assembly; width and height may not exceed 1280×720 and the
PNG may not exceed 4 MiB. Capture failures are explicit errors, not text-only
successes. Recognized PDF main responses skip capture and add a warning; repo
and local workspace targets never enter this browser path.

Screenshot requests bypass both cache reads and writes, regardless of the
requested cache policy. Image bytes travel outside `WebFetchResponse` and the
cache structures. The ordinary JSON response and its output schema are unchanged.

### Challenge pages on the HTTP path

Plain HTTP responses are classified before they are returned. `src/fetch/challenge.rs` recognises
Expand Down
17 changes: 12 additions & 5 deletions architecture/local-workspace.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@
## Overview

CodeGG-managed Eggsearch 0.4.4+ can receive an authenticated private request
scope over stdio. For each repository tool call, CodeGG supplies one
authorized workspace root or an explicit `root: null` local-disabled scope.
scope over stdio. For repository-oriented calls, including repo items in
`batch_fetch`, CodeGG supplies one authorized workspace root or an explicit
`root: null` local-disabled scope.
Eggsearch authenticates that metadata with the per-process
`CODEGG_EGGSEARCH_HOST_SCOPE_TOKEN`; missing or invalid authentication is an
error and never falls back to configured roots. The selected backend and
Expand Down Expand Up @@ -38,7 +39,8 @@ and no native plugins are loaded.
startup):

- `enabled`, `roots`
- `max_file_bytes` (default 1_048_576) — per-file read cap
- `max_file_bytes` (default 1_048_576) — per-file read cap for non-PDF workspace fetches
- `pdf_max_file_bytes` (default 16_777_216, range 1..=268_435_456) — separate cap for local PDF reads
- `max_indexed_files` (default 50_000) — bounded scan
- `include_hidden` (default false), `respect_gitignore` (default true),
`follow_symlinks` (default false)
Expand Down Expand Up @@ -229,8 +231,13 @@ the `[local]` budgets; the scan never executes workspace code.

Local `repo_fetch` validates the requested path, then reads it with
`safe_read_file()` relative to the selected workspace root; it never reopens
the canonicalized pathname. Local `repo_map` structure enrichment uses the
same bounded opener for manifest, source, and test contents. Its structural
the canonicalized pathname. It identifies PDFs by case-insensitive `.pdf`
extension or a safely read `%PDF-` prefix. PDF bytes use `pdf_max_file_bytes`
and the existing PDF feature plus `[fetch].pdf_enabled` gates, then flow through
the shared byte-slice extractor. `repo_fetch` and batch repo items do not accept
PDF page selection. PDF remains in `BINARY_EXTENSIONS`, so local indexing and
repo-map filtering are unchanged. Local `repo_map` structure enrichment uses
the same bounded opener for manifest, source, and test contents. Its structural
walk skips symlink entries even when local search permits following symlinks,
so map enrichment does not enumerate or read a linked tree outside the
workspace.
Expand Down
14 changes: 11 additions & 3 deletions architecture/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,10 +124,10 @@ Registration lives in `server.rs`; behavior lives in `tools/`; semantics live in
|------|---------|---------------|--------------|
| `web_search` | Live metasearch returning source cards, never page text or a synthesized answer | `query`, `max_results`, `intent`, `freshness`/`date_range`, `include_domains`/`exclude_domains`, `language`/`region`, `safe_search`, `excerpt_count` | `MetadataSearchAdapter::web_search` (+ `resolve_provider_routing` selection) |
| `web_fetch` | Bounded extraction of one explicit HTTP(S) URL | `url`, `max_chars`, `timeout_ms`, `extract_mode`, `include_links`, `pdf`, `cache_policy`, `focus`/`focus_max_chunks`/`focus_max_chars`, `render`, `browser_profile` | `FetchClient` (+ cache tiers, `OriginController`, optional browser transport) |
| `batch_fetch` | Bounded fan-out over explicit URLs or repo locators, per-item results | `items` (web + repo, each with `focus` projection), `max_items`, `max_chars_per_item`, `max_total_chars`, `timeout_ms`, `continue_on_error` | `FetchClient` per web item + repo-fetch handler per repo item; deterministic `project_*` per payload |
| `batch_fetch` | Bounded fan-out over explicit URLs or repo locators, per-item results | `items`: web entries accept `extract_mode`, `include_links`, `pdf` (including per-URL `pages`), `render`, `browser_profile`, cache, and focus controls; repo entries accept locator/ref, line/context, symbol/match-text/block-expansion, `prefer_local`, and focus controls; plus `max_items`, `max_chars_per_item`, `max_total_chars`, `timeout_ms`, `continue_on_error` | Shared web-fetch handler per web item + repo-fetch handler per repo item; deterministic `project_*` per payload |
| `provider_status` | Diagnostic provider/capability report with optional bounded live probe | `probe`, `recipe_detail` (`none`/`summary`/`full`) | `adapter.provider_status()` + health registry; probe path uses the shared probe service |
| `repo_search` | Structured repository evidence discovery with grouped bundles | `query`, `goal`, `sources`, repo locator/package fields, `include_local` | `MetadataSearchAdapter::repo_search` (+ `LocalWorkspaceBackend` when local-only or mixed) |
| `repo_fetch` | Repository file/span fetch by structured locator | `locator` (`owner`/`repo`/`path`/`ref`), `line_start`/`line_end`, `symbol`, `test_fetch_url` (test-only override) | `workspace://` local path bypasses fetch policy; remote path uses forge raw URL via shared `FetchClient` |
| `repo_fetch` | Repository file/span fetch by structured locator | `locator` (`owner`/`repo`/`path`/`ref`), `line_start`/`line_end`, context windows, `symbol`/`symbol_kind`, `match_text`, block expansion, `prefer_local`, `test_fetch_url` (test-only override) | `workspace://` local path bypasses fetch policy and extracts named PDFs through the bounded PDF reader; remote path uses forge raw URL via shared `FetchClient` |
| `repo_map` | Repository layout discovery without file contents | `owner`, `repo`, `ref`, `path`, security-policy toggles | `forge_adapter::fetch_tree` for supported hosts, `repo_mapper::build_fallback_response` otherwise; local backend when workspace-scoped |
| `security_search` | Vulnerability/advisory retrieval with applicability context | `query`, `identifiers` (CVE/GHSA/OSV), `package`, `ecosystem`, `goal`, `include` (`kev`, `exploit_context`, `defensive_guidance`, `vendor_advisories`) | `MetadataSearchAdapter` security path (advisory engines + KEV client) |
| `research_search` | Multi-source evidence discovery with grouped bundles, no synthesis | `query`, `depth` (`quick`/`standard`/`deep`), `domain`, `goal`, `include` (`counterpoints`, `primary_sources`, `recent_discussion`, `security_considerations`) | `MetadataSearchAdapter::research_search` |
Expand All @@ -144,7 +144,7 @@ per-call values win. `provider_status` remains diagnostic-only.

**`web_fetch`** — single-URL bounded inspection. `url` must be `http`/`https` and non-empty; `validate_fetch_target()` blocks SSRF vectors before any I/O. Supports `max_chars`, per-call `timeout_ms` override, `extract_mode` (`text`/`markdown`/`metadata_only`), `include_links`, PDF options, `cache_policy`/`max_cache_age_seconds`, deterministic `focus` chunk selection over already-extracted content (no extra traversal), and `render`/`browser_profile` escalation (feature-gated). Raw + derived cache tiers are consulted before any network fetch. Returns the fetch envelope (`url`, `final_url`, `status`, `fetched`, `truncated`, `text`, warnings, trust markers, transport metadata).

**`batch_fetch`** — fan-out over `items` (web URLs and structured repo locators, each accepting `focus`/`focus_max_chunks`/`focus_max_chars`). `max_items`, `max_chars_per_item`, and aggregate `max_total_chars` bound the call; `timeout_ms` builds a one-off widened client only when longer than the shared default, otherwise the shared transport is reused. `continue_on_error` defaults to true so each item reports its own result. Web items run the `FetchClient` path; repo items delegate to the repo-fetch handler. Focus is a deterministic projection of extracted content and is excluded from stable-ID computation. Returns per-item results plus aggregate telemetry (requested/completed/failed/truncated, focused items/chunks/chars, budget exhaustion, cache hit/revalidated/miss/bypassed/not-cacheable).
**`batch_fetch`** — fan-out over web URLs and structured repo locators. Web items accept `extract_mode`, `include_links`, `pdf` (including the `web_fetch` page syntax per URL), `render`, `browser_profile`, cache policy/age, and deterministic focus controls, and run through the shared web-fetch handler so PDF derivation, browser authorization, cache scope, and profile locking match `web_fetch`. Invalid page syntax is rejected before any item fetch; page-count-dependent errors remain per-item. Repo items accept line/context windows, symbol or match-text selection, block expansion, `prefer_local`, and focus, and delegate to the repo-fetch handler. Workspace repo items retain the authorized root plus root-relative path semantics; the private `codegg/host-local-scope` metadata is forwarded unchanged from the server wrapper, never widened. Local PDFs are extracted only through that workspace path and do not gain page-selection options. `max_items`, `max_chars_per_item`, and aggregate `max_total_chars` bound the call; `timeout_ms` builds a one-off widened client only when longer than the shared default, otherwise the shared transport is reused. `continue_on_error` defaults to true. Focus is a deterministic projection and is excluded from stable-ID computation. Results preserve request order and per-item failures alongside aggregate telemetry (requested/completed/failed/truncated, focused items/chunks/chars, budget exhaustion, cache hit/revalidated/miss/bypassed/not-cacheable).

**`provider_status`** — diagnostics, never a first research step. Cheap process-local mode by default (`probe = false`): descriptors with `routable`/`skip_code`, health snapshots/views, `code_hosts`, `server_capabilities`, `tool_capabilities`. `probe = true` runs bounded concurrent liveness checks through the shared probe service (the same service as CLI `doctor --probe`); non-routable providers report `skipped` with a stable skip code rather than a network failure. `recipe_detail` controls workflow-recipe verbosity (`none`/`summary`/`full`). Patches the `local_workspace` descriptor from `local_backend` presence. The only tool without `response_detail`.

Expand Down Expand Up @@ -179,6 +179,14 @@ Every tool module follows one pipeline:
- `ToolError::InvalidRequest` → JSON-RPC `invalid_params`. Reserved for shapes that cannot be interpreted at all.
- `ToolError::Internal` → JSON-RPC `internal_error` without stack traces. Reserved for server faults.

For the opt-in screenshot path, `web_fetch` and `batch_fetch` use an
image-capable `ToolOutcome` at the server seam. Its structured JSON stays the
ordinary tool response; validated PNG bytes are appended as native `image/png`
content blocks and never enter `structuredContent`. In batch results, a sanitized
text association names the original zero-based item index and label before each
image, and attachments preserve input order. Screenshot-bearing batches are kept
synchronous because the deferred task result path carries JSON only.

rmcp input-schema decode failures surface as `isError` tool errors in the pinned release; only true protocol failures (malformed JSON-RPC, missing negotiation headers) remain JSON-RPC errors on the wire.

### Shared helpers (`tools/common.rs`)
Expand Down
2 changes: 1 addition & 1 deletion architecture/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ Ordinary agent schemas are slimmed: canonical `goal` (`understand`, `architectur
|------|---------|----------------|
| `web_search` | Live metasearch over configured providers | `query`, `max_results`, `intent`, `freshness`/`date_range`, `include_domains`/`exclude_domains`, `language`/`region`, `excerpt_count` |
| `web_fetch` | Bounded extraction of one HTTP(S) URL | `url`, `max_chars`, `extract_mode` |
| `batch_fetch` | Batch fetch over URLs or repo locators with per-item focus | `items` (per-item focus), `max_chars_per_item`, `max_total_chars`, telemetry |
| `batch_fetch` | Batch fetch over URLs or repo locators with per-target controls | Web-item PDF/render/profile/cache/focus options; repo-item span/source-selection/`prefer_local`/focus options; `max_chars_per_item`, `max_total_chars`, telemetry |
| `provider_status` | Diagnostic provider configuration report with optional bounded live probe | (none; `probe`, `recipe_detail`) |
| `repo_search` | Structured repository evidence discovery | `query`, `goal`, `sources`, repo locator/package fields, `include_local` |
| `repo_fetch` | Repository file fetch by locator | `locator`, `line_start`, `line_end`, `symbol` |
Expand Down
Loading
Loading