Repository navigation
docs: polish guidance and record blocked M003 qualification - #16
Merged
Merged
Conversation
added 30 commits
November 11, 2026 00:03
…shot lane, derived intent M001 scraper fidelity: finalize_scrape enforces results / explicit empty / structural break, adds EngineError::Blocked and error_class "blocked", captured fixtures under tests/fixtures/engines, engine_parse_conformance with a seeded-drift control, and parse_shape in provider_status. M002 search-result cache: process-local LRU keyed by an FNV-1a digest plus broadcast single-flight. Partial or deadline-interrupted fan-outs bypass the cache; hits skip health recording and replay the post-fan-out pipeline so payloads stay byte-identical. Adds [search].cache_ttl_seconds and cache_max_entries plus a search_cache diagnostics block. M003 markdown-first fetch: default-on Accept negotiation for HTML-ish targets, accept_policy cache dimension, Vary: Accept, one bounded 406/415/empty fallback, markdown link extraction, and representation/token metadata. M004 candidate snapshot lane: extraction of adapter/selection.rs, a mock-gated replay harness, tier-1/tier-2 corpora, an untracked tier-3 recorder, three seeded negative controls, and duplicate-slot/domain/authoritative-survival metrics. No ranking behaviour change. M005 deterministic intent derivation: a pure 11-class classifier with fixed precedence and reason codes, derived_intent reporting, unchanged planner text, and applied reranking backed by recorded tier-1 evidence. Also converts five mcp/tools unit tests that reached live providers to a mock-backed state, and syncs architecture/, docs/, skills/, plans/, and CHANGELOG.
Closes M001 scraper fidelity contract, M002 search-result cache and coalescing, M003 markdown-first fetch negotiation, M004 candidate snapshot ranking benchmark, and M005 deterministic intent derivation at candidate c9912fc. Closure records record the candidate SHA, verification counts, measured deltas, unchanged safety evidence, and residual risk. M004's snapshot lane is the measurement gate for later ranking and routing work; M006/M007 are dependency-ready without plans, M008/M009 stay proposed.
…cutor entry Upgrade rmcp 3.2.0 -> 3.9.0, which fixes the OAuth SSRF filter, unbounded tool schema cache, and unbounded response body advisories in the MCP transports. Feature names, MSRV (1.89), and MCP 2026-07-28 behaviour are unchanged; the only code change is the deprecated `ServerInfo` alias renamed to `ServerConfig`. Upgrade the dev-only async-executor 1.14.0 -> 1.14.1 lock entry, clearing the yanked-crate failure. Add a documented, dated waiver for the dev-only pin-utils unmaintained notice (reached only through httpmock -> async-std, never in a shipped binary), following the existing scoped-waiver policy. make dependency-policy now passes; the all-features suite is unchanged at 5,848 passed / 26 ignored / 0 failed.
…conomy, presets, pagination Closes agentic-retrieval-integrity M007-M012; M006 lands conditionally closed. Challenge solving (ADR-0009): one shared solver in src/meta/challenge with four operator ceilings under [search.challenge], disabled by default. ALTCHA PBKDF2/SHA-256 restores mojeek; Anubis fast/preact restores startpage. Pass cookies are process-local and never logged. Interactive CAPTCHA, OCR, solving services, proxy rotation, and user-agent spoofing stay prohibited. startpage now extracts the served React payload (mainline regions) with the div.result selectors as a fallback, so a solved SERP yields rows. Its live challenge publishes no redir field, so the redirect target is derived from the request URL; the solver previously refused every real challenge as malformed. Intent-aware routing adds specialist providers by derived intent under [search.routing]. It only adds, never removes or reorders, and each entry must satisfy the same capability predicate its warning uses. intent_mode ships off because the tier-1 lane does not model the shipped default provider set; rustsec is excluded from the security row because it serves advisory-ID lookup only and would always fail a keyword query. One response-wide excerpt budget with deterministic shingle suppression for compact and brief, reporting excerpt_economy counts and per-card excerpts_suppressed. Advertisement presets filter tools/list while every tool stays callable and next_actions are filtered with drops counted. Stateless 512-byte replay cursors, bounded token-gated progress, and a capability-gated Tasks path. Gate: fmt, clippy -D warnings, no-default build, 6150 tests (0 failed), hygiene, planning consistency, dependency policy. packaging-check remains sandbox-blocked by mavis-trash on this host.
ADR-0010 supersedes ADR-0009 on two clauses only: solver coverage follows the published Anubis registry (fast, slow as its deprecated alias, preact, metarefresh) rather than a hardcoded name list, and the nonce search stays single-threaded by policy so the operator ceiling remains the binding control on solve rate. Every ADR-0009 prohibition stands unchanged. Registered M013-M018 from the 2026-11-12 live review: - M013 challenge solver conformance corrections (base prefix, slow alias, spent, pre-flight projection, redir scheme/origin validation) - M014 pass-cookie lifetime from the published attributes, clamped by config, with the egress interaction documented and eviction on a rejected token - M015 challenge solvability observability ledger in provider_status - M016 Anubis metarefresh solver under ADR-0010 - M017 routing shipped-set evidence plus a keyword-query predicate that pins the rustsec regression - M018 pagination consistency across pages elapsedTime and the ALTCHA solution time stay fixed constants: ADR-0009's determinism bullet governs fields the upstream does not verify, and the live difficulty-6 solve succeeded with elapsedTime=1. Recorded as explicit non-goals rather than changed. planning-consistency, repo-hygiene, static_guards, docs_tool_names and docs_config_snippets all pass. 18 cross-referenced plan paths, none dangling.
ADR-0011 replaces the unreleased opaque replay cursor with offset addressing on web_fetch, batch_fetch, and research_search. The cursor never shipped (crate 0.4.4, next_cursor only under Unreleased), so this is a replacement rather than a deprecation and the schema never offers two ways to address one tool. M019 is the prerequisite: the document is extracted at max_chars_cap while the caller's max_chars bounds only the returned window, truncation splits into document-truncated versus output-clipped, and the derived cache key stops varying by caller budget. M020 then emits text_total on the pre-sanitization basis, replaces next_cursor with next_offset, moves collections to index offsets, and retires the codec. M020 registers blocked, not ready: its hard dependency M019 has not landed, and offset addressing over a document still truncated at the caller's budget is correct arithmetic over the wrong total. M009 and M018 carry scoped "superseded in part" pointers and no body text was rewritten. Planning-consistency, repository hygiene, static guards, docs name, and docs snippet checks pass.
M019 of the agentic-retrieval-integrity roadmap. Extraction now runs at the document budget (max_chars_cap) while a caller's max_chars bounds only the rendered window, so a fetched document can be larger than any single call. Truncation reports two distinct facts: document.document_truncated (extraction stopped at the budget) and document.text_truncated (the caller received less than the document holds). Neither can now be false while the caller holds less than the document contains. New src/fetch/budget.rs resolves FetchBudgets once per call. New src/fetch/projection.rs owns project_document_text, and the block truncation loop is extracted from render_blocks into the shared truncate_blocks_at_budget, so the cold path, the cache-hit path, and the browser path cannot drift apart. HTML derived cache entries are now document-scoped and shared across caller windows; renderer_version moves to 5 so pre-change entries miss rather than serve a stale shape. Non-HTML documents and PDF keep a caller-scoped key, because their text cannot be re-projected from rendered blocks without changing it. Evidence: cargo fmt, clippy -D warnings, no-default-features check, 6,192 all-features tests (27 ignored) and 5,837 mock tests green; the new mcp_tools::document_budget suite proves the document spans more than one window, chunk coverage is contiguous and exhaustive, the addressable total is identical at 2k/8k/12k windows, and a cache hit returns the bytes a cold fetch of the same window returns. Closure record and status flips follow in the next commit.
Records the M019 closure in plans/closure/agentic-retrieval-integrity/ 019-status.md: what changed, how each invariant was proven, the measured test evidence, the deviations taken deliberately (HTML-only cache sharing, window-bounded browser text, projection-derived text_chars_returned), the measurements that were not run and why, and the residual risks. M019 flips to closed in the registry and the roadmap. M020 becomes dependency-ready: its work package 0 requires exhaustive contiguous chunk coverage and no pre-truncation at the caller's window, and both now hold and are tested, so text_total over the chunk join is the true addressable total rather than an echo of the caller's own truncation.
Replace the unreleased replay cursor with offset windows on `web_fetch`, `batch_fetch`, and `research_search` (ADR-0011, M020). - `src/mcp/cursor.rs` (499 lines) is deleted; `src/mcp/plan_key.rs` keeps `PlanKey`/`PlanTool` and drops the codec and its tests. - `web_fetch` gains `offset`/`limit` over the chunk join and reports `text_from`/`text_to`/`text_total`/`doc_id` plus `focus_scope: "document"`; an unaddressed read keeps the chunk-aligned first page byte-identically. - `batch_fetch` addresses items (`item_from`/`item_to`/`item_total`) and `research_search` non-empty groups (`group_from`, `aggregate_scope`, `total_groups`, `returned_groups`), with `next_offset` while units remain. - `next_cursor`/`cursor_status` and `cursor_mismatch` are removed; `plan_mismatch` and `document_unavailable` are added, and a terminal fetch failure is `upstream_failed` rather than `internal`. - `workflow_coverage.found_roles` keeps card order so two identical windows report identical bytes. - Advertisement fingerprints re-pinned: full+full `8f2f6953a1fa6eb2` (74,422 bytes), full+compact `32f58c1c4cc33de3` (20,196 bytes); the schema-slimming budgets move by the recorded delta of the two new args. - First-slice corpus `tests/fixtures/fetch_windows/`: recall 7/11 (63.6%), median chars-to-answer 4,200, recorded rather than asserted. - Docs, changelog, property-suite inventory, and the tool-surface measurement are synced; `docs/config.md` and `architecture/mcp.md` describe the offset contract instead of the retired token.
Record `plans/closure/agentic-retrieval-integrity/020-status.md`: the work package 0 M019 evidence, the gate results, the compatibility-gate and boundary transcripts, the UTF-8/framing evidence, the distinguishable-code table, the collection exhaustiveness counts, the cursor-absence guards, the re-pinned `tools/list` fingerprints, the first-slice measurement, and the residual risks plus five recorded deviations from the plan. Flip the plan header, the three registry rows, and the roadmap to `conditionally closed`. Named condition: `text_total` is the true document-budget total for HTML and browser-rendered documents only, because the PDF and non-HTML extraction paths still bound at the caller window (M019 deviation 1) and this plan's non-goals exclude `src/fetch/**` changes. Also corrects M019's plan header from `ready` to `closed`, which its own closure commit left unchanged.
Three verifications that were recorded as gaps now carry measured evidence. - Compatibility gate measured across versions: a throwaway dump test built the same fixed-port fixture at `8bb5c8a` (pre-M020) and in the candidate tree. The single-window payload is deeply identical with no key dropped; the multi-window payload is deeply identical after removing exactly `next_cursor` and `cursor_status` (11 chunks, 11,791 scalars, `text_truncated: true` in both). The temporary test and worktree are removed. - Boundary transcript replaced with one worked example's real observations: `text_total = 38,055` for a 11,791-scalar first page, `offset=5_000` with no `limit` returning the 12,000-scalar default window, `offset=99_999` clamping to an empty window, `limit=0` rejected, focused addressed read carrying `focus_scope: "document"`, `metadata_only` returning `document_unavailable`. - The opt-in live two-window test was run against the real network and passed: RFC 9110, `doc_id=doc_bdacf44c950f8e6d`, `text_total=11981`, windows `[0,4000)` and `[5000,9000)`, 4,000 scalars each. It is also the live evidence for the named condition, because a plain-text document of several hundred kilobytes reports `text_total` as the caller-window bound.
Discharge M020's named condition, with the measurement that showed the condition was half wrong. - The non-HTML document arm in `src/fetch/client.rs` renders its block and chunk set from the document-bounded text (`raw_text`) instead of the window-bounded `pre_framing_text`; `text_chars` stays window-scoped because it describes the legacy `text` field. `text/plain`, CSV, code, notebooks and markdown source now report `text_total` as the extracted document and are addressable past the first window, matching HTML. - Measured on a 20,031-byte `text/plain` body: `text_total` 11,821 -> 19,733, `next_offset` now offered, `offset = 13_000` reaches the tail, and the addressed and unaddressed reads agree on `text_truncated`. On the live RFC 9110 read `text_total` moves 11,981 -> 49,852. - Correct M020's record and close it: PDF was never window-scoped (a six-page fixture reports `text_total` 12,599 against a 12,000 window, because blocks are bounded by `PdfLimits` and the chunk is built from all of them), so the claim that `extract_pdf_text`'s `max_chars` limited addressability was wrong. - Six integration tests that asserted block text is bounded by `max_chars` are retargeted to the corrected contract through one helper; add non-HTML caller-independence and tail-composition tests, and a PDF guard in `tests/pdf_fidelity.rs` (the guard lives there because `src/fetch/pdf.rs` is at a 70,000-byte ratchet ceiling). - Record, for the PDF ingestion subsystem, the findings this exposed: flag conflation, the hardcoded `chunk_0`, the silent `max_pages` cut, and the whole-chunk-per-window payload. Ready PDF ingestion M005 would re-pin the caller window and must reconcile with this record. - Record an unrelated pre-existing counterexample in `property_excerpt_economy::suppression_is_monotone_in_budget`.
… challenge pages M013: honour anubis_base_prefix, solve slow as the fast alias, refuse a spent challenge, project the expected 16^difficulty cost against max_attempts and the remaining wall clock before hashing, and accept a published redir only as a same-origin absolute URL or a safe root-relative path. M016: add the metarefresh time-challenge solver. It echoes only published values after the published meta-refresh wait, performs no hashing, consumes no attempt, and refuses a wait or target the budget cannot honour. M022: classify Anubis, ALTCHA, interactive, and verification challenge pages on the plain HTTP fetch path. A detected page keeps its bounded body but carries manual_interaction_required and one fetch_challenge_page warning; render=auto now escalates a detected wall instead of returning it as an ordinary stub. Also reconcile the stale M010-M012 registry rows and plan headers with their closure records, and record the M013/M016/M022 closures. Gate: fmt, clippy -D warnings, no-default check, 6225 all-features tests (28 ignored), planning-consistency, repo-hygiene, dependency-policy.
M014: derive the pass-cookie reuse window from the upstream-published Max-Age/Expires, clamped by cookie_ttl_seconds, fail closed on malformed or past attributes, evict a rejected cookie, and warn when a routed egress meets cookie reuse. A 21-minute live run re-solved after the published expiry instead of replaying a dead token. M015: add the bounded process-local SolveLedger and always report a provider_status.challenge_solver block (ten-key refusal vocabulary, cookie hits/misses, last candidates and wall clock). M017: add the keyword-query allowlist and keyword_unsupported disposition, measure the shipped default provider set across seven cases twice, and ship [search.routing].intent_mode = "apply"; M006's named condition is discharged. rustsec can no longer count as native coverage for a keyword query. M018: closed by discharge under ADR-0011/M020 (offset windows plus the focus_scope/aggregate_scope labels), with fresh acceptance evidence. M023: when [search.challenge] is enabled, web_fetch solves a detected Anubis challenge once with the existing bounded solver and replays once with spchal-auth. Live Startpage fetch returned the SERP in 2.32 s with no challenge warning; disabled, refused, and still-challenged pages stay marked. Gate: fmt, clippy -D warnings, no-default check, 6291 all-features tests (28 ignored), planning-consistency, repo-hygiene, dependency-policy, docs-check. The agentic retrieval integrity workstream closes with M001-M023.
The plan's live acceptance requires a two-arm solve-count comparison. Startpage publishes a lifetime longer than the shipped 240 s ceiling, so at default the clamp binds and no solve-count reduction is expected; the measured benefit is the avoided dead-token request, and the derived arm was observed re-solving after the published expiry. The comparative arm is the named condition.
Replay the pass-challenge cookie pair set instead of a fixed name, accept both metarefresh presentations, and fail closed on unpublished extensions until css-load and WASM sha256 are authorized by ADR-0012 and ADR-0013. - cookie.rs: parse_issued_pairs()/IssuedCookies join non-deletion pairs, dedupe by name, bound the header at 4 KiB, and derive the reuse window from the longest usable lifetime; parse_issued() stays for ALTCHA. - anubis.rs: opaque pass-cookie replay; metarefresh randomData echo and synthesized pass-challenge URL; bounded extensions with an attributable refusal; one same-origin css-load GET; WASM sha256 with u32 nonce, last-byte endianness, leading-zero-bit difficulty and ceil(bits/4) ceiling mapping. argon2id and hashx remain unsupported. - startpage/mojeek/web_fetch: replay the cached header verbatim; ALTCHA emits chllg=<value> with its Max-Age=0 semantics unchanged. - plans: ADR-0012/ADR-0013 accepted, M024-M028 closed with records, and registry/roadmap/architecture/docs/CHANGELOG updated. Verification: fmt, clippy -D warnings, no-default-features check, dependency-policy, repo-hygiene, planning-consistency, and cargo test --all-features --no-fail-fast (6306 passed, 0 failed, 28 ignored). Live protocol evidence: 2026-11-13 anubis.techaro.lol v1.28.3 accepted challenge=<randomData> and issued techaro.lol-anubis-auth-347ddb4a. css-load and sha256 have no observed deployment and are recorded as synthetic-only.
The shared provider id is also used by the unknown-method tests, which record unsupported scheme concurrently, so an exact +1 delta is racy. Assert monotonic growth instead; the exact ten-key mapping stays pinned by the deterministic ledger unit test.
CI's unlucky proptest draws exposed two test-only assumptions, not product defects. - The www and numeric subdomain properties generated the private TLD `lan`, which validate_url intentionally rejects. Both now use the same private-TLD prop_assume! guard as the deep-subdomain property; the list is shared in one PRIVATE_TLDS constant. - suppression_is_monotone_in_budget asserted admitted passage count monotonicity, but a larger budget can admit a long earlier snippet and leave fewer passages admitted while admitting more characters. Keep the redundancy direction and assert chars_after monotonicity instead. Verified with 10k and 5k proptest cases for the two suites, plus the default runs and clippy.
Make the tool surface callable unchanged by agents trained on the reference fetch server and the common search/fetch MCPs, without adding schema properties or changing behavior. - web_fetch accepts max_length for max_chars, start_index for offset, and question or prompt for focus as serde wire aliases. The canonical field descriptions name the aliases; sending a canonical field and its alias together is a duplicate-field invalid-arguments error, not a silent precedence rule. - All ten tools now advertise readOnlyHint=true, destructiveHint=false, idempotentHint=true, and an accurate openWorldHint, instead of relying on spec defaults for the two omitted hints. - web_search states that it returns source cards, never a synthesized answer, matching the evidence-only contract. - Tests cover alias deserialization and an end-to-end aliased fetch, the duplicate-field rejection, and the four hints; pinned advertisement fingerprints, byte budgets, and the tool-surface baseline are updated with the documented deltas. - Docs: tool matrix (new Annotations and Compatibility aliases sections), architecture/mcp.md, docs/config.md, docs/codegg-integration.md, README, and CHANGELOG.
ADR-0014 selects one default artifact with pdf and browser, a bounded render lifecycle, and a guard-bounded chromiumoxide reqwest edge. Register the browser-artifact-availability workstream with three milestone plans, supersede repository-hardening M006, and keep the planning guards consistent.
…fecycle Default features now include pdf and browser. PDF and browser config defaults resolve from the compiled-in feature state; browser policy defaults to auto with HTTP still the hot path. The shared browser now performs an idle teardown (90 s default, 5 s floor) and enforces max_requests per session, with lifecycle counters in provider_status. Explicit browser requests report browser_not_compiled_in, browser_disabled, or browser_unavailable instead of silently using HTTP, and auto degradation emits a browser_unavailable warning. The dependency-shape guard allows reqwest only through chromiumoxide in both graphs. PDF pre-parse scanning resolves indirect /Length references instead of refusing filtered streams, and auto escalation classifies JavaScript shells from document text rather than framing or source length. The Python wheel is decoupled to pdf only.
added 12 commits
November 14, 2026 07:00
Close render lifecycle bounds and default artifact availability with their requirement-to-evidence matrices, record M003 as conditionally closed with four named conditions, and mark PDF M006 closing because its production change landed with the shared default set. Add the render-gate concurrency test.
…he render gate A browser that does not acknowledge close previously kept the lifecycle write lock and blocked every later render and teardown. The close is now capped at five seconds; the handler is aborted and the lifecycle released regardless.
macOS x86_64 and Windows aarch64 completed qualification with the same ~35% size delta as the other targets. Only aarch64-apple-darwin remains, after the runner cancelled its build job; name that and the remaining independent conditions explicitly.
… test A closed ephemeral port can be reassigned or report a different client error; the test now asserts the endpoint is not Healthy instead of pinning the Refused classification.
Hosted run 43567644057 passed all three jobs after the macOS rerun. Record the bounded browser close and the final candidate run for the bounded-close HEAD.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
git diff --check, andcargo test --locked --all-features --test static_guardspassed.62387208ac8bc20efb545b68f0704f466bab7295(run 43736811317); this PR head still needs its own checks.M003 status
M003 remains blocked. Python wheel smoke fails the typed-request/schema assertion; the realistic-mix browser escalation is unqualified; the hosted macOS arm64 runner configuration is unsupported; and exact-candidate live MCP plus standard installer/release wrapper checks remain outstanding. No production behavior was changed during qualification.