Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
2ebcab1
plans: establish deterministic retrieval quality roadmap and baseline…
dbowm91 Oct 29, 2026
2c34789
plans: register ranking, provider, context and qualification milestones
dbowm91 Oct 29, 2026
29a8201
benchmark: add retrieval quality contract and corpus
Oct 29, 2026
3289c7c
benchmark: tighten retrieval report provenance
Oct 29, 2026
df78a9e
benchmark: report retrieval train and holdout scores
Oct 29, 2026
310611b
benchmark: enforce temporal task constraints
Oct 29, 2026
520f191
plans: close retrieval benchmark milestone
Oct 29, 2026
25f4910
docs: record optional extraction comparison matrix
Oct 29, 2026
d781440
plans: close retrieval extraction matrix milestone
Oct 29, 2026
3ef5242
bench: add keyless provider contribution fixture
Oct 29, 2026
780ef02
plans: close keyless provider contribution milestone
Oct 29, 2026
84472df
docs: record deterministic relevance ablation result
Oct 29, 2026
8c3f735
plans: close deterministic relevance experiment
Oct 29, 2026
5c1f0d2
bench: measure retrieval context efficiency and handoff
Oct 29, 2026
fb378a3
plans: close context efficiency and CodeGG handoff
Oct 29, 2026
10b06ab
plan: qualify comparative retrieval evidence
Oct 29, 2026
8710f85
plans: close comparative retrieval qualification
Oct 29, 2026
99316f3
fix: fail closed on unsupported symlink following
Oct 29, 2026
074c29c
plans: close macOS retrieval qualification corrective
Oct 29, 2026
92b0a27
ci: run local filesystem portability tests on macOS
Oct 29, 2026
39e4a20
docs: extend retrieval quality live evidence
Oct 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,15 @@ jobs:
targets: x86_64-pc-windows-msvc
- name: Check all features on Windows target
run: cargo check --locked --all-features --target x86_64-pc-windows-msvc
macos-portability:
name: macOS local filesystem portability
runs-on: macos-15
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: dtolnay/rust-toolchain@6bed0761d98439e5a578e2877258200ad565ba87 # stable snapshot 2026-09-28
with:
toolchain: "1.89"
- name: Test unsupported symlink following behavior
run: cargo test --locked --no-default-features --test property_local_fs_extended -- safe_open_follow_symlink_is_unsupported
- name: Test host-scoped workspace root canonicalization
run: cargo test --locked --lib host_scopes_authenticate_and_cache_distinct_roots
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: check ci fmt clippy feature-check test hygiene dependency-policy packaging-check producer-drift release-check release-candidate-check docs-check release-build publish-check bench-check fuzz-smoke live-smoke eval-tool-surface native-forge-smoke-github native-forge-smoke-gitlab native-forge-smoke-codeberg native-forge-smoke-gitea native-forge-smoke-all
.PHONY: check ci fmt clippy feature-check test hygiene dependency-policy packaging-check producer-drift release-check release-candidate-check docs-check release-build publish-check bench-check fuzz-smoke live-smoke eval-tool-surface eval-retrieval-quality native-forge-smoke-github native-forge-smoke-gitlab native-forge-smoke-codeberg native-forge-smoke-gitea native-forge-smoke-all

check: fmt clippy feature-check test hygiene dependency-policy packaging-check

Expand Down Expand Up @@ -72,6 +72,9 @@ live-smoke:
eval-tool-surface:
cargo test --locked --all-features --test tool_surface_evaluation -- --nocapture

eval-retrieval-quality:
cargo test --locked --features mock --test retrieval_quality -- --nocapture

native-forge-smoke-github:
cargo test --locked --features live-smoke --test native_forge_smoke -- --ignored native_github

Expand Down
2 changes: 1 addition & 1 deletion architecture/hardening.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Field order is insignificant by design (one `write_str` per field under the enti

## Local filesystem containment

`validate_local_fetch_path()` (`src/core/local.rs`) rejects empty, absolute, traversal (`..`), hidden (unless `include_hidden`), skipped-directory (`target`, `node_modules`, `.git`), binary-extension, oversized, and non-file targets. `safe_open_relative()` (`src/meta/safe_open.rs`) then walks component-wise with NUL rejection: on Linux descriptor-relative `openat2` with kernel-enforced `RESOLVE_BENEATH | RESOLVE_NO_MAGICLINKS` (plus `RESOLVE_NO_SYMLINKS` when `follow_symlinks = false`), falling back to `openat` with `O_NOFOLLOW` only where `openat2` is unavailable; on non-Unix every component is `symlink_metadata`-checked at walk time and links are rejected with `SymlinkDetected` when following is disabled. Symlink escapes, intermediate symlinks, loops, and root replacement are rejected fail-closed on every platform; the exact variant is platform-specific (Windows reports `SymlinkDetected` from the walk-time check, macOS reports `Io` where `O_NOFOLLOW` surfaces `ENOTDIR`, Linux rejects from the untouched `openat2`/`openat` path with a non-`SymlinkDetected` variant). The regression test pins fail-closed on unix and `SymlinkDetected` on Windows, never success. Reads after open enforce a hard byte cap. Only Linux (and the non-Linux Unix `O_NOFOLLOW` fallback) eliminates the validation-to-open race atomically: on Windows a check-to-open swap window remains, contained by the user-owned jail-root trust boundary and recorded in the owning closure record. Never bypass these two functions with a direct filesystem read for operator-supplied paths.
`validate_local_fetch_path()` (`src/core/local.rs`) rejects empty, absolute, traversal (`..`), hidden (unless `include_hidden`), skipped-directory (`target`, `node_modules`, `.git`), binary-extension, oversized, and non-file targets. `safe_open_relative()` (`src/meta/safe_open.rs`) then walks components with descriptor-relative opens and NUL rejection. Linux uses `openat2` with kernel-enforced `RESOLVE_BENEATH | RESOLVE_NO_MAGICLINKS` (plus `RESOLVE_NO_SYMLINKS` when `follow_symlinks = false`), falling back to `openat` with `O_NOFOLLOW` only when `openat2` is unavailable and following is disabled. Other Unix platforms use component-wise `openat` with `O_NOFOLLOW` when following is disabled and return `SafeSymlinkFollowingUnsupported` when it is enabled. Non-Unix platforms check each component with `symlink_metadata` at walk time, reject links with `SymlinkDetected` when following is disabled, and return `SafeSymlinkFollowingUnsupported` when enabled. Linux enforces symlink containment in-kernel while following; platforms without a beneath-root following primitive reject that mode instead of approximating it. Reads after open enforce a hard byte cap. Linux and Unix `O_NOFOLLOW` component walks prevent symlink-component swaps during open; Windows retains a check-to-open race window within the user-owned jail-root trust boundary. Never bypass these functions with a direct filesystem read for operator-supplied paths.

`SafeOpenError` taxonomy (`Empty`, `AbsolutePath`, `PathTraversal`, `NullByte`, `NotFound`, `SymlinkDetected`, `NotAFile`, `FileTooLarge`, `FileContentLimitExceeded`, `SafeSymlinkFollowingUnsupported`, `RootOpenFailed`, `Io`) keeps every rejection machine-readable: callers map variants to retrieval attempts and absence kinds instead of string-matching messages. On non-Linux Unix and non-Unix platforms, `follow_symlinks = true` returns `SafeSymlinkFollowingUnsupported` rather than attempting containment without a race-safe primitive.

Expand Down
17 changes: 16 additions & 1 deletion architecture/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Per-suite counts live in `docs/test-inventory.md`; suite placement rules live in

## Suite count and layout

`tests/*.rs` holds 78 suites (high-70s to low-80s as suites are added or merged; treat `docs/test-inventory.md` as the count source of truth). Layout:
`tests/*.rs` holds 79 suites (high-70s to low-80s as suites are added or merged; treat `docs/test-inventory.md` as the count source of truth). Layout:

| Location | Purpose |
|----------|---------|
Expand All @@ -21,6 +21,7 @@ Per-suite counts live in `docs/test-inventory.md`; suite placement rules live in
| `tests/docs_*.rs`, `tests/static_guards.rs`, `tests/schema_identity_registry.rs` | Documentation, hygiene, and identity contracts |
| `tests/mcp_tool_contract.rs`, `mcp_schema_slimming.rs`, `mcp_2026_protocol.rs`, `mcp_projection.rs` | Tool consolidation contracts |
| `tests/tool_surface_evaluation.rs`, `tests/tool_surface_live.rs` | Tool-selection corpus + opt-in live-model comparison |
| `tests/retrieval_quality.rs` | Retrieval relevance metrics over the synthetic reference corpus + ignored live observation |
| `tests/mcp_http.rs` | Loopback Streamable HTTP lifecycle and transport hardening |
| `packaging/test-install.sh`, `packaging/test-install.ps1` | Installer behavior (not `tests/`) |
| `src/**/mod.rs` unit tests | Private-function coverage at the bottom of the source file |
Expand Down Expand Up @@ -227,6 +228,20 @@ make eval-tool-surface
cargo test --locked --all-features --test tool_surface_live -- --ignored
```

`tests/retrieval_quality.rs` separately evaluates retrieved evidence against the
versioned synthetic corpus at `tests/fixtures/retrieval_quality/cases.json`.
It reports ranked relevance, passage hits, diversity, evidence coverage, and
status correctness with per-category denominators. The default evaluator is
offline. Its ignored live observation uses the existing `web_search` path and
requires explicit query/provider allowlists plus separate billing
acknowledgement; methodology and limitations are in
`docs/retrieval-quality-benchmark.md`.

```bash
cargo test --locked --features mock --test retrieval_quality
make eval-retrieval-quality
```

## Transport, egress, and browser suites

`tests/mcp_http.rs` covers the loopback Streamable HTTP lifecycle: health identity, current and legacy protocol flows, shared tool definitions, request hardening, session cleanup, and shutdown behavior. It runs under `--all-features`:
Expand Down
17 changes: 10 additions & 7 deletions docs/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,16 @@ the published release contains the exact 16-asset contract. Unix installer and
updater checks passed locally, and native Windows PowerShell latest/pinned
installer plus `update --check` smoke passed in run `34660182644`.

`v0.4.2` is the latest published release as of preparing the `v0.4.3` candidate.
It was staged by the Eggpack producer with the current 19-asset contract,
attested while draft, and published on 2026-10-07. `v0.4.1` remains the first
binary release carrying the frozen CodeGG parity surface; `v0.4.0` is crate +
tag only, with no published binaries. Corrections after a crates.io publication
always require a new version, a new changelog entry, and full re-qualification;
see `plans/closure/codegg-legacy-search-parity/005-status.md` and the current
`v0.4.3` is the latest published binary release. It was qualified and published
from the exact source recorded in `plans/closure/mode-off-repository-locality-corrective/002-status.md`.
The separate Python distribution `eggsearch==0.4.4` was published to PyPI on
2026-10-28; it does not represent a new binary release. `v0.4.2` was staged by
the Eggpack producer with the 19-asset contract, attested while draft, and
published on 2026-10-07. `v0.4.1` remains the first binary release carrying the
frozen CodeGG parity surface; `v0.4.0` is crate + tag only, with no published
binaries. Corrections after a crates.io publication always require a new
version, a new changelog entry, and full re-qualification; see
`plans/closure/codegg-legacy-search-parity/005-status.md` and the current
Mode-Off corrective plan.

## Preparation
Expand Down
144 changes: 144 additions & 0 deletions docs/retrieval-quality-benchmark.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
# Retrieval quality benchmark

The retrieval-quality harness measures ranked evidence relevance. It is
separate from `tool_surface_evaluation`, which measures whether a query points
to the right MCP tool.

## Offline contract

`tests/fixtures/retrieval_quality/cases.json` is schema version 1 and contains
15 synthetic cases across coding errors, repository symbols/docs/issues/releases,
security advisories and applicability, package registries, multi-source
research, generic web, freshness/date constraints, and adversarial injection.
All result envelopes and passages were authored for this fixture. No provider
text, credentials, or personal queries are tracked. The corpus is auditable but
not an unbiased estimate of live relevance: it has one annotation author and
no independent adjudication. Six cases are explicitly held out from tuning.

The Rust evaluator reports macro precision@5, recall@5, MRR@5, nDCG@5,
passage hit within a 1,200-character budget, source diversity, evidence
coverage, and expected-state accuracy. Passage-hit and status metrics include
Wilson 95% intervals and trial counts; small corpus categories are descriptive.
Precision and nDCG exclude unjudged
results when judgments are incomplete; date-window tasks only match results
with an in-window publication date. Denominators are included. URL aliases
use Eggsearch canonicalization for scheme/host case, `www`, fragments, default
ports, percent encoding, and trailing slashes while preserving query strings.
Corpus validation rejects canonical URL aliases that point to multiple gold
judgments, so ambiguous matches cannot silently select the first label.
Grades are 0–3, with zero reserved for judged irrelevant results. Per-category
metrics are available from the evaluator. A deliberately degraded ranking
must fail its MRR/nDCG check.

Run the offline suite with:

```bash
cargo test --locked --features mock --test retrieval_quality
cargo test --locked --features mock --test retrieval_quality -- --nocapture
make eval-retrieval-quality
```

The corpus fingerprint and metric output are deterministic. The fixture replay
does not perform HTTP, call an inference service, or use Python. The fixtures
characterize evaluator behavior; they are not provider search snapshots and
must not be reported as a measured Eggsearch quality baseline.

### Frozen synthetic evaluator baseline

M001 candidate SHA: `310611b733c2067ddeab184618156ef072934a6e`.
Pre-M001 production baseline SHA: `7ad8e81b2d8f7036d35dcb2772a1778c73d08c3b`.
Corpus: `retrieval-quality-v1`, fingerprint `fnv1a64:b882ce3d844e8ace`.
Evaluator: `retrieval-quality-evaluator-v1`, schema version 1, mock feature.
The emitted JSON report includes per-case result hashes and per-category
denominators. Aggregate oracle-fixture metrics:

| Measure | Value | Denominator |
|---------|------:|------------:|
| Precision@5, macro | 0.3385 | 57 judged rank slots across 13 ranking cases |
| Recall@5, macro | 0.9615 | 13 ranking cases / 15 gold documents |
| MRR@5, macro | 0.9615 | 13 ranking cases |
| nDCG@5, macro | 0.9552 | 13 ranking cases |
| Passage hit@1,200 characters | 0.9333 | 14 of 15 judged passages |
| Source diversity@5, macro | 0.8462 | 13 nonempty-result cases |
| Expected-state accuracy | 1.0000 | 15 labeled outcomes |

Partition results keep tuning and holdout separate:

| Partition | Cases (ranking) | Precision@5 | Recall@5 | MRR@5 | nDCG@5 | Passage hit |
|-----------|----------------:|------------:|---------:|-------:|-------:|------------:|
| Train | 9 (9) | 0.3778 | 0.9444 | 0.9444 | 0.9353 | 0.9000 (9/10) |
| Held out | 6 (4) | 0.2500 | 1.0000 | 1.0000 | 1.0000 | 1.0000 (5/5) |

This baseline freezes evaluator and fixture behavior only. It establishes no
live retrieval quality, cross-provider winner, latency, cost, or superiority
claim. Production implementation is unchanged from the specified M001
planning baseline; the new source changes add only the evaluator, fixtures,
tests, and documentation.

## Opt-in live observations

The ignored `retrieval_quality_live_observation` test calls the existing
`web_search` handler. It sends only explicitly selected corpus queries and
provider IDs, with ceilings of 20 queries, 8 providers, 160 provider-query
attempts, and a 10-second tool timeout. No live test runs in routine CI. It
scores returned cards against the corpus and prints bounded metadata, result
hashes, response byte counts, timestamps, p50/p95 latency, and aggregate
metrics; result content, credentials, and raw URLs are not put in the report.

An operator must set all of the following explicitly:

```bash
EGGSEARCH_RETRIEVAL_LIVE=1 \
EGGSEARCH_BENCH_SOURCE_SHA=<40-hex-source-sha> \
EGGSEARCH_RETRIEVAL_PROVIDERS=searxng \
EGGSEARCH_RETRIEVAL_PAID_PROVIDERS= \
EGGSEARCH_RETRIEVAL_LOCALE=en-US \
EGGSEARCH_RETRIEVAL_CASES=generic-web-001,freshness-date-001 \
cargo test --locked --features mock --test retrieval_quality -- --ignored --nocapture
```

Paid providers must appear in both `EGGSEARCH_RETRIEVAL_PROVIDERS` and
`EGGSEARCH_RETRIEVAL_PAID_PROVIDERS`, and require the separate
`EGGSEARCH_RETRIEVAL_ALLOW_BILLING=1` acknowledgement. Ambient credentials
alone never select or bill a provider. Each provider must already be configured
in Eggsearch. Reference providers are observed one provider at a time through
the production adapter; SearXNG is supported when configured as a provider.
To measure a direct provider path, select that provider alone in Eggsearch;
to measure the credentialed Eggsearch path, select its configured credentialed
provider and declare it in the paid-provider list. The runner adds no providers.
Use separate runs and preserve the same case IDs and task constraints.

The live tool response does not expose a cache-state control, so the report
records that limitation rather than claiming a cache condition. It also does
not assert source-level passage equivalence where the live response omits an
excerpt. Record the exact source SHA, provider configuration, machine, locale,
time window, and whether an operator repeated a cohort alongside any retained
report. Do not commit raw upstream content. A 429, timeout, provider error, or
zero-result outcome is an observed provider outcome and never evidence of
expected absence.

## External independently judged cohort

The separate M007 corrective adds 300 SciFact test claims and 339 published
positive evidence judgments in
`tests/fixtures/retrieval_quality/scifact_test_queries.jsonl` and
`tests/fixtures/retrieval_quality/scifact_test_qrels.jsonl`, with query and
paper-title attribution and no abstract text. Its bounded, opt-in repeated
runner is `scripts/run_retrieval_quality_external_live.py`. It reports title
matched hit@k/recall over known positive judgments and leaves unjudged results
out of precision claims. Successful zero-result trials count as misses;
provider errors and timeouts remain separate outcomes. The first small repeated
DuckDuckGo cohort and the current SearXNG/provider limitations are recorded in
`docs/retrieval-quality-external-live.md`; they are not a broad quality ranking.
This is a new follow-up to closed M006, not a reinterpretation of its synthetic
contract corpus or historical comparison.

## Handoff contract

M002–M005 can add versioned cases and separate recorded envelopes only when
redistribution rights are documented. Keep the held-out partition fixed;
candidate heuristics must not use held-out judgments. Keep retrieval metrics
separate from latency, bytes, estimated token cost, and paid request cost.
Missing annotations, unavailable connectors, and unsupported task classes are
reported as unscorable or skipped with denominators, never assigned a zero
quality score.
24 changes: 24 additions & 0 deletions docs/retrieval-quality-codegg-handoff.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Retrieval Quality CodeGG Consumer Handoff

This is an Eggsearch-side handoff from the deterministic context experiment. CodeGG consumer implementation and acceptance remain owned by the CodeGG repository.

## Contract identity

- Eggsearch package version: `0.4.4`.
- Ten canonical MCP tools remain unchanged.
- Tool-surface fingerprint: `eggsearch-0.4.4|tools=batch_fetch:150,build_evidence_bundle:144,provider_status:148,repo_fetch:150,repo_map:146,repo_search:186,research_search:160,security_search:159,web_fetch:159,web_search:160|bytes=72939`.
- Full serialized tool definitions: 72,939 bytes. Core palette definitions: 14,008 bytes. Compact discovery: at most 260 bytes per observed query in the 43-case corpus.
- Projection modes: `compact`, `standard`, `diagnostic`. Omitted `response_detail` remains `diagnostic`.

## Recommended consumer behavior

1. Advertise the core palette initially: `web_search`, `web_fetch`, `repo_search`.
2. Use the existing tool discovery contract to find deferred tools, then hydrate only tools needed for the current task. Treat `next_actions` as advisory known-tool hints.
3. Request `response_detail=compact` only when a smaller model-visible payload is useful. Keep full returned `structuredContent` in host memory and attach it to later evidence-bundle construction as required.
4. On an unsupported response mode or a malformed/uncertain compact response, retry with the existing `standard` or `diagnostic` mode and preserve current tool-error repair behavior. Use `diagnostic` for failure investigation.
5. Key hydrated-schema caches by the full tool-surface fingerprint. Invalidate on any fingerprint change; do not key by package version alone.
6. Preserve source IDs, provenance/trust, truncation, warnings, partial-failure and absence indicators when selecting evidence. Never treat compact prose as the authoritative result when full structured content is available.

## Consumer acceptance tests

Register these as CodeGG-owned tests: initial core palette is callable; one deferred-tool lookup can hydrate a requested tool; hydration cache invalidates after fingerprint change; compact mode lowers serialized bytes while stable IDs, trust, truncation and failure/absence survive; full structured results remain available for evidence-bundle construction; unsupported compact mode falls back to standard/diagnostic; search-to-fetch-to-bundle trace does not fetch a source twice solely because of disclosure. Record serialized bytes and host round trips in CodeGG; the Eggsearch tests do not claim those consumer measurements.
Loading
Loading