Skip to content

docs: polish guidance and record blocked M003 qualification - #16

Merged
dbowm91 merged 42 commits into
mainfrom
feat/auto-20261110-0a674cdc
Nov 16, 2026
Merged

dbowm91 merged 42 commits into
mainfrom
feat/auto-20261110-0a674cdc

Conversation

@dbowm91

@dbowm91 dbowm91 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Polish user-facing guidance, including scraping/SearXNG and browser/PDF distribution details.
  • Record exact-candidate M003 qualification evidence and the blocked disposition.

Validation

  • Local planning consistency, JSON parsing, git diff --check, and cargo test --locked --all-features --test static_guards passed.
  • CI passed on qualification candidate 62387208ac8bc20efb545b68f0704f466bab7295 (run 43736811317); this PR head still needs its own checks.

M003 status

M003 remains blocked. Python wheel smoke fails the typed-request/schema assertion; the realistic-mix browser escalation is unqualified; the hosted macOS arm64 runner configuration is unsupported; and exact-candidate live MCP plus standard installer/release wrapper checks remain outstanding. No production behavior was changed during qualification.

eggsearch added 30 commits November 11, 2026 00:03
…shot lane, derived intent

M001 scraper fidelity: finalize_scrape enforces results / explicit empty /
structural break, adds EngineError::Blocked and error_class "blocked",
captured fixtures under tests/fixtures/engines, engine_parse_conformance with
a seeded-drift control, and parse_shape in provider_status.

M002 search-result cache: process-local LRU keyed by an FNV-1a digest plus
broadcast single-flight. Partial or deadline-interrupted fan-outs bypass the
cache; hits skip health recording and replay the post-fan-out pipeline so
payloads stay byte-identical. Adds [search].cache_ttl_seconds and
cache_max_entries plus a search_cache diagnostics block.

M003 markdown-first fetch: default-on Accept negotiation for HTML-ish targets,
accept_policy cache dimension, Vary: Accept, one bounded 406/415/empty
fallback, markdown link extraction, and representation/token metadata.

M004 candidate snapshot lane: extraction of adapter/selection.rs, a mock-gated
replay harness, tier-1/tier-2 corpora, an untracked tier-3 recorder, three
seeded negative controls, and duplicate-slot/domain/authoritative-survival
metrics. No ranking behaviour change.

M005 deterministic intent derivation: a pure 11-class classifier with fixed
precedence and reason codes, derived_intent reporting, unchanged planner text,
and applied reranking backed by recorded tier-1 evidence.

Also converts five mcp/tools unit tests that reached live providers to a
mock-backed state, and syncs architecture/, docs/, skills/, plans/, and
CHANGELOG.
Closes M001 scraper fidelity contract, M002 search-result cache and
coalescing, M003 markdown-first fetch negotiation, M004 candidate snapshot
ranking benchmark, and M005 deterministic intent derivation at candidate
c9912fc.

Closure records record the candidate SHA, verification counts, measured
deltas, unchanged safety evidence, and residual risk. M004's snapshot lane is
the measurement gate for later ranking and routing work; M006/M007 are
dependency-ready without plans, M008/M009 stay proposed.
…cutor entry

Upgrade rmcp 3.2.0 -> 3.9.0, which fixes the OAuth SSRF filter, unbounded
tool schema cache, and unbounded response body advisories in the MCP
transports. Feature names, MSRV (1.89), and MCP 2026-07-28 behaviour are
unchanged; the only code change is the deprecated `ServerInfo` alias renamed
to `ServerConfig`.

Upgrade the dev-only async-executor 1.14.0 -> 1.14.1 lock entry, clearing the
yanked-crate failure. Add a documented, dated waiver for the dev-only
pin-utils unmaintained notice (reached only through httpmock -> async-std,
never in a shipped binary), following the existing scoped-waiver policy.

make dependency-policy now passes; the all-features suite is unchanged at
5,848 passed / 26 ignored / 0 failed.
…conomy, presets, pagination

Closes agentic-retrieval-integrity M007-M012; M006 lands conditionally closed.

Challenge solving (ADR-0009): one shared solver in src/meta/challenge with
four operator ceilings under [search.challenge], disabled by default. ALTCHA
PBKDF2/SHA-256 restores mojeek; Anubis fast/preact restores startpage. Pass
cookies are process-local and never logged. Interactive CAPTCHA, OCR, solving
services, proxy rotation, and user-agent spoofing stay prohibited.

startpage now extracts the served React payload (mainline regions) with the
div.result selectors as a fallback, so a solved SERP yields rows. Its live
challenge publishes no redir field, so the redirect target is derived from the
request URL; the solver previously refused every real challenge as malformed.

Intent-aware routing adds specialist providers by derived intent under
[search.routing]. It only adds, never removes or reorders, and each entry must
satisfy the same capability predicate its warning uses. intent_mode ships off
because the tier-1 lane does not model the shipped default provider set;
rustsec is excluded from the security row because it serves advisory-ID lookup
only and would always fail a keyword query.

One response-wide excerpt budget with deterministic shingle suppression for
compact and brief, reporting excerpt_economy counts and per-card
excerpts_suppressed. Advertisement presets filter tools/list while every tool
stays callable and next_actions are filtered with drops counted. Stateless
512-byte replay cursors, bounded token-gated progress, and a capability-gated
Tasks path.

Gate: fmt, clippy -D warnings, no-default build, 6150 tests (0 failed),
hygiene, planning consistency, dependency policy. packaging-check remains
sandbox-blocked by mavis-trash on this host.
ADR-0010 supersedes ADR-0009 on two clauses only: solver coverage follows
the published Anubis registry (fast, slow as its deprecated alias, preact,
metarefresh) rather than a hardcoded name list, and the nonce search stays
single-threaded by policy so the operator ceiling remains the binding control
on solve rate. Every ADR-0009 prohibition stands unchanged.

Registered M013-M018 from the 2026-11-12 live review:

- M013 challenge solver conformance corrections (base prefix, slow alias,
  spent, pre-flight projection, redir scheme/origin validation)
- M014 pass-cookie lifetime from the published attributes, clamped by config,
  with the egress interaction documented and eviction on a rejected token
- M015 challenge solvability observability ledger in provider_status
- M016 Anubis metarefresh solver under ADR-0010
- M017 routing shipped-set evidence plus a keyword-query predicate that pins
  the rustsec regression
- M018 pagination consistency across pages

elapsedTime and the ALTCHA solution time stay fixed constants: ADR-0009's
determinism bullet governs fields the upstream does not verify, and the live
difficulty-6 solve succeeded with elapsedTime=1. Recorded as explicit
non-goals rather than changed.

planning-consistency, repo-hygiene, static_guards, docs_tool_names and
docs_config_snippets all pass. 18 cross-referenced plan paths, none dangling.
ADR-0011 replaces the unreleased opaque replay cursor with offset
addressing on web_fetch, batch_fetch, and research_search. The cursor
never shipped (crate 0.4.4, next_cursor only under Unreleased), so this
is a replacement rather than a deprecation and the schema never offers
two ways to address one tool.

M019 is the prerequisite: the document is extracted at max_chars_cap
while the caller's max_chars bounds only the returned window, truncation
splits into document-truncated versus output-clipped, and the derived
cache key stops varying by caller budget. M020 then emits text_total on
the pre-sanitization basis, replaces next_cursor with next_offset, moves
collections to index offsets, and retires the codec.

M020 registers blocked, not ready: its hard dependency M019 has not
landed, and offset addressing over a document still truncated at the
caller's budget is correct arithmetic over the wrong total.

M009 and M018 carry scoped "superseded in part" pointers and no body
text was rewritten. Planning-consistency, repository hygiene, static
guards, docs name, and docs snippet checks pass.
M019 of the agentic-retrieval-integrity roadmap. Extraction now runs at
the document budget (max_chars_cap) while a caller's max_chars bounds
only the rendered window, so a fetched document can be larger than any
single call. Truncation reports two distinct facts:
document.document_truncated (extraction stopped at the budget) and
document.text_truncated (the caller received less than the document
holds). Neither can now be false while the caller holds less than the
document contains.

New src/fetch/budget.rs resolves FetchBudgets once per call. New
src/fetch/projection.rs owns project_document_text, and the block
truncation loop is extracted from render_blocks into the shared
truncate_blocks_at_budget, so the cold path, the cache-hit path, and the
browser path cannot drift apart. HTML derived cache entries are now
document-scoped and shared across caller windows; renderer_version moves
to 5 so pre-change entries miss rather than serve a stale shape. Non-HTML
documents and PDF keep a caller-scoped key, because their text cannot be
re-projected from rendered blocks without changing it.

Evidence: cargo fmt, clippy -D warnings, no-default-features check,
6,192 all-features tests (27 ignored) and 5,837 mock tests green; the new
mcp_tools::document_budget suite proves the document spans more than one
window, chunk coverage is contiguous and exhaustive, the addressable
total is identical at 2k/8k/12k windows, and a cache hit returns the
bytes a cold fetch of the same window returns. Closure record and status
flips follow in the next commit.
Records the M019 closure in plans/closure/agentic-retrieval-integrity/
019-status.md: what changed, how each invariant was proven, the measured
test evidence, the deviations taken deliberately (HTML-only cache
sharing, window-bounded browser text, projection-derived
text_chars_returned), the measurements that were not run and why, and
the residual risks.

M019 flips to closed in the registry and the roadmap. M020 becomes
dependency-ready: its work package 0 requires exhaustive contiguous
chunk coverage and no pre-truncation at the caller's window, and both
now hold and are tested, so text_total over the chunk join is the true
addressable total rather than an echo of the caller's own truncation.
Replace the unreleased replay cursor with offset windows on `web_fetch`,
`batch_fetch`, and `research_search` (ADR-0011, M020).

- `src/mcp/cursor.rs` (499 lines) is deleted; `src/mcp/plan_key.rs` keeps
  `PlanKey`/`PlanTool` and drops the codec and its tests.
- `web_fetch` gains `offset`/`limit` over the chunk join and reports
  `text_from`/`text_to`/`text_total`/`doc_id` plus `focus_scope: "document"`;
  an unaddressed read keeps the chunk-aligned first page byte-identically.
- `batch_fetch` addresses items (`item_from`/`item_to`/`item_total`) and
  `research_search` non-empty groups (`group_from`, `aggregate_scope`,
  `total_groups`, `returned_groups`), with `next_offset` while units remain.
- `next_cursor`/`cursor_status` and `cursor_mismatch` are removed;
  `plan_mismatch` and `document_unavailable` are added, and a terminal fetch
  failure is `upstream_failed` rather than `internal`.
- `workflow_coverage.found_roles` keeps card order so two identical windows
  report identical bytes.
- Advertisement fingerprints re-pinned: full+full `8f2f6953a1fa6eb2`
  (74,422 bytes), full+compact `32f58c1c4cc33de3` (20,196 bytes); the
  schema-slimming budgets move by the recorded delta of the two new args.
- First-slice corpus `tests/fixtures/fetch_windows/`: recall 7/11 (63.6%),
  median chars-to-answer 4,200, recorded rather than asserted.
- Docs, changelog, property-suite inventory, and the tool-surface measurement
  are synced; `docs/config.md` and `architecture/mcp.md` describe the offset
  contract instead of the retired token.
Record `plans/closure/agentic-retrieval-integrity/020-status.md`: the work
package 0 M019 evidence, the gate results, the compatibility-gate and boundary
transcripts, the UTF-8/framing evidence, the distinguishable-code table, the
collection exhaustiveness counts, the cursor-absence guards, the re-pinned
`tools/list` fingerprints, the first-slice measurement, and the residual risks
plus five recorded deviations from the plan.

Flip the plan header, the three registry rows, and the roadmap to
`conditionally closed`. Named condition: `text_total` is the true
document-budget total for HTML and browser-rendered documents only, because the
PDF and non-HTML extraction paths still bound at the caller window (M019
deviation 1) and this plan's non-goals exclude `src/fetch/**` changes.

Also corrects M019's plan header from `ready` to `closed`, which its own
closure commit left unchanged.
Three verifications that were recorded as gaps now carry measured evidence.

- Compatibility gate measured across versions: a throwaway dump test built the
  same fixed-port fixture at `8bb5c8a` (pre-M020) and in the candidate tree. The
  single-window payload is deeply identical with no key dropped; the
  multi-window payload is deeply identical after removing exactly `next_cursor`
  and `cursor_status` (11 chunks, 11,791 scalars, `text_truncated: true` in
  both). The temporary test and worktree are removed.
- Boundary transcript replaced with one worked example's real observations:
  `text_total = 38,055` for a 11,791-scalar first page, `offset=5_000` with no
  `limit` returning the 12,000-scalar default window, `offset=99_999` clamping to
  an empty window, `limit=0` rejected, focused addressed read carrying
  `focus_scope: "document"`, `metadata_only` returning `document_unavailable`.
- The opt-in live two-window test was run against the real network and passed:
  RFC 9110, `doc_id=doc_bdacf44c950f8e6d`, `text_total=11981`, windows
  `[0,4000)` and `[5000,9000)`, 4,000 scalars each. It is also the live evidence
  for the named condition, because a plain-text document of several hundred
  kilobytes reports `text_total` as the caller-window bound.
Discharge M020's named condition, with the measurement that showed the
condition was half wrong.

- The non-HTML document arm in `src/fetch/client.rs` renders its block and
  chunk set from the document-bounded text (`raw_text`) instead of the
  window-bounded `pre_framing_text`; `text_chars` stays window-scoped because
  it describes the legacy `text` field. `text/plain`, CSV, code, notebooks and
  markdown source now report `text_total` as the extracted document and are
  addressable past the first window, matching HTML.
- Measured on a 20,031-byte `text/plain` body: `text_total` 11,821 -> 19,733,
  `next_offset` now offered, `offset = 13_000` reaches the tail, and the
  addressed and unaddressed reads agree on `text_truncated`. On the live
  RFC 9110 read `text_total` moves 11,981 -> 49,852.
- Correct M020's record and close it: PDF was never window-scoped (a six-page
  fixture reports `text_total` 12,599 against a 12,000 window, because blocks
  are bounded by `PdfLimits` and the chunk is built from all of them), so the
  claim that `extract_pdf_text`'s `max_chars` limited addressability was wrong.
- Six integration tests that asserted block text is bounded by `max_chars` are
  retargeted to the corrected contract through one helper; add non-HTML
  caller-independence and tail-composition tests, and a PDF guard in
  `tests/pdf_fidelity.rs` (the guard lives there because `src/fetch/pdf.rs` is
  at a 70,000-byte ratchet ceiling).
- Record, for the PDF ingestion subsystem, the findings this exposed: flag
  conflation, the hardcoded `chunk_0`, the silent `max_pages` cut, and the
  whole-chunk-per-window payload. Ready PDF ingestion M005 would re-pin the
  caller window and must reconcile with this record.
- Record an unrelated pre-existing counterexample in
  `property_excerpt_economy::suppression_is_monotone_in_budget`.
… challenge pages

M013: honour anubis_base_prefix, solve slow as the fast alias, refuse a spent
challenge, project the expected 16^difficulty cost against max_attempts and the
remaining wall clock before hashing, and accept a published redir only as a
same-origin absolute URL or a safe root-relative path.

M016: add the metarefresh time-challenge solver. It echoes only published
values after the published meta-refresh wait, performs no hashing, consumes no
attempt, and refuses a wait or target the budget cannot honour.

M022: classify Anubis, ALTCHA, interactive, and verification challenge pages on
the plain HTTP fetch path. A detected page keeps its bounded body but carries
manual_interaction_required and one fetch_challenge_page warning; render=auto
now escalates a detected wall instead of returning it as an ordinary stub.

Also reconcile the stale M010-M012 registry rows and plan headers with their
closure records, and record the M013/M016/M022 closures.

Gate: fmt, clippy -D warnings, no-default check, 6225 all-features tests
(28 ignored), planning-consistency, repo-hygiene, dependency-policy.
M014: derive the pass-cookie reuse window from the upstream-published
Max-Age/Expires, clamped by cookie_ttl_seconds, fail closed on malformed or
past attributes, evict a rejected cookie, and warn when a routed egress meets
cookie reuse. A 21-minute live run re-solved after the published expiry instead
of replaying a dead token.

M015: add the bounded process-local SolveLedger and always report a
provider_status.challenge_solver block (ten-key refusal vocabulary, cookie
hits/misses, last candidates and wall clock).

M017: add the keyword-query allowlist and keyword_unsupported disposition,
measure the shipped default provider set across seven cases twice, and ship
[search.routing].intent_mode = "apply"; M006's named condition is discharged.
rustsec can no longer count as native coverage for a keyword query.

M018: closed by discharge under ADR-0011/M020 (offset windows plus the
focus_scope/aggregate_scope labels), with fresh acceptance evidence.

M023: when [search.challenge] is enabled, web_fetch solves a detected Anubis
challenge once with the existing bounded solver and replays once with
spchal-auth. Live Startpage fetch returned the SERP in 2.32 s with no
challenge warning; disabled, refused, and still-challenged pages stay marked.

Gate: fmt, clippy -D warnings, no-default check, 6291 all-features tests
(28 ignored), planning-consistency, repo-hygiene, dependency-policy,
docs-check. The agentic retrieval integrity workstream closes with M001-M023.
The plan's live acceptance requires a two-arm solve-count comparison. Startpage
publishes a lifetime longer than the shipped 240 s ceiling, so at default the
clamp binds and no solve-count reduction is expected; the measured benefit is
the avoided dead-token request, and the derived arm was observed re-solving
after the published expiry. The comparative arm is the named condition.
Replay the pass-challenge cookie pair set instead of a fixed name, accept
both metarefresh presentations, and fail closed on unpublished extensions
until css-load and WASM sha256 are authorized by ADR-0012 and ADR-0013.

- cookie.rs: parse_issued_pairs()/IssuedCookies join non-deletion pairs,
  dedupe by name, bound the header at 4 KiB, and derive the reuse window
  from the longest usable lifetime; parse_issued() stays for ALTCHA.
- anubis.rs: opaque pass-cookie replay; metarefresh randomData echo and
  synthesized pass-challenge URL; bounded extensions with an attributable
  refusal; one same-origin css-load GET; WASM sha256 with u32 nonce,
  last-byte endianness, leading-zero-bit difficulty and ceil(bits/4)
  ceiling mapping. argon2id and hashx remain unsupported.
- startpage/mojeek/web_fetch: replay the cached header verbatim; ALTCHA
  emits chllg=<value> with its Max-Age=0 semantics unchanged.
- plans: ADR-0012/ADR-0013 accepted, M024-M028 closed with records, and
  registry/roadmap/architecture/docs/CHANGELOG updated.

Verification: fmt, clippy -D warnings, no-default-features check,
dependency-policy, repo-hygiene, planning-consistency, and
cargo test --all-features --no-fail-fast (6306 passed, 0 failed,
28 ignored). Live protocol evidence: 2026-11-13 anubis.techaro.lol
v1.28.3 accepted challenge=<randomData> and issued
techaro.lol-anubis-auth-347ddb4a. css-load and sha256 have no observed
deployment and are recorded as synthetic-only.
The shared provider id is also used by the unknown-method tests, which
record unsupported scheme concurrently, so an exact +1 delta is racy.
Assert monotonic growth instead; the exact ten-key mapping stays pinned
by the deterministic ledger unit test.
CI's unlucky proptest draws exposed two test-only assumptions, not
product defects.

- The www and numeric subdomain properties generated the private TLD
  `lan`, which validate_url intentionally rejects. Both now use the same
  private-TLD prop_assume! guard as the deep-subdomain property; the list
  is shared in one PRIVATE_TLDS constant.
- suppression_is_monotone_in_budget asserted admitted passage count
  monotonicity, but a larger budget can admit a long earlier snippet and
  leave fewer passages admitted while admitting more characters. Keep
  the redundancy direction and assert chars_after monotonicity instead.

Verified with 10k and 5k proptest cases for the two suites, plus the
default runs and clippy.
Make the tool surface callable unchanged by agents trained on the
reference fetch server and the common search/fetch MCPs, without adding
schema properties or changing behavior.

- web_fetch accepts max_length for max_chars, start_index for offset, and
  question or prompt for focus as serde wire aliases. The canonical field
  descriptions name the aliases; sending a canonical field and its alias
  together is a duplicate-field invalid-arguments error, not a silent
  precedence rule.
- All ten tools now advertise readOnlyHint=true, destructiveHint=false,
  idempotentHint=true, and an accurate openWorldHint, instead of relying
  on spec defaults for the two omitted hints.
- web_search states that it returns source cards, never a synthesized
  answer, matching the evidence-only contract.
- Tests cover alias deserialization and an end-to-end aliased fetch, the
  duplicate-field rejection, and the four hints; pinned advertisement
  fingerprints, byte budgets, and the tool-surface baseline are updated
  with the documented deltas.
- Docs: tool matrix (new Annotations and Compatibility aliases sections),
  architecture/mcp.md, docs/config.md, docs/codegg-integration.md, README,
  and CHANGELOG.
ADR-0014 selects one default artifact with pdf and browser, a bounded render lifecycle, and a guard-bounded chromiumoxide reqwest edge. Register the browser-artifact-availability workstream with three milestone plans, supersede repository-hardening M006, and keep the planning guards consistent.
…fecycle

Default features now include pdf and browser. PDF and browser config defaults resolve from the compiled-in feature state; browser policy defaults to auto with HTTP still the hot path. The shared browser now performs an idle teardown (90 s default, 5 s floor) and enforces max_requests per session, with lifecycle counters in provider_status. Explicit browser requests report browser_not_compiled_in, browser_disabled, or browser_unavailable instead of silently using HTTP, and auto degradation emits a browser_unavailable warning. The dependency-shape guard allows reqwest only through chromiumoxide in both graphs. PDF pre-parse scanning resolves indirect /Length references instead of refusing filtered streams, and auto escalation classifies JavaScript shells from document text rather than framing or source length. The Python wheel is decoupled to pdf only.
eggsearch added 12 commits November 14, 2026 07:00
Close render lifecycle bounds and default artifact availability with their requirement-to-evidence matrices, record M003 as conditionally closed with four named conditions, and mark PDF M006 closing because its production change landed with the shared default set. Add the render-gate concurrency test.
…he render gate

A browser that does not acknowledge close previously kept the lifecycle write lock and blocked every later render and teardown. The close is now capped at five seconds; the handler is aborted and the lifecycle released regardless.
macOS x86_64 and Windows aarch64 completed qualification with the same ~35% size delta as the other targets. Only aarch64-apple-darwin remains, after the runner cancelled its build job; name that and the remaining independent conditions explicitly.
… test

A closed ephemeral port can be reassigned or report a different client error; the test now asserts the endpoint is not Healthy instead of pinning the Refused classification.
Hosted run 43567644057 passed all three jobs after the macOS rerun. Record the bounded browser close and the final candidate run for the bounded-close HEAD.
@dbowm91
dbowm91 merged commit aea34ee into main Nov 16, 2026
306 of 317 checks passed
dbowm91 pushed a commit that referenced this pull request Nov 16, 2026
Close the M003 records on fb99a43, record PR #16 as merged in aea34ee, and remove the stale open/blocked merge statements. No source changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant