eggsearch provides keyless web, fetch, advisory, registry, and scholarly paths by default. Current source builds also enable PDF extraction and browser rendering by default. The published v0.4.4 binaries predate that change and include neither feature. Other optional capabilities require Cargo features or config changes.
web_fetch extracts PDF text when the binary includes the pdf Cargo feature
and [fetch].pdf_enabled is true. Current source builds enable this feature by
default. Published v0.4.4 binaries do not include it. The config value defaults
to the compiled-in feature state.
No config change is needed in a current source build. To enable extraction in a build that includes the feature but has disabled it in config:
[fetch]
pdf_enabled = trueA source build with --no-default-features leaves PDF out, so [fetch].pdf_enabled
defaults to false; add the feature back explicitly:
cargo build --no-default-features --features pdfThe published Python wheel enables pdf only
(default-features = false, features = ["pdf"]), so it carries PDF extraction and
no browser rendering. See the Python library guide.
- Text-only extraction via
lopdf, with no OCR, rendering, or image extraction - Per-page quality classification: blank, scanned, CID-corrupt, and sparse text pages
- Image presence is always detected for classification, so an image-only page is
reported as scanned even when
include_mediais off;include_mediaonly adds the image count to page metadata - Document metadata (title, author, subject, keywords, creator, producer, dates)
- Bookmark/outline entries where available
The pages field supports 1, 1,3,5, 1-5, and 1,3,7-10 syntax (one-indexed).
- 16 MiB response body (
[fetch].pdf_max_bytes, range 1 byte to 256 MiB); the generic[fetch].max_bytescap still applies to every non-PDF response - 64 MiB decompressed content (
[fetch].pdf_max_decompressed_bytes, range 1 MiB to 1 GiB), measured over the whole document before parsing; a document that would expand further fails withpdf_decompression_bound_exceeded - 25 pages maximum
- 12k chars per page
- 50k chars total
A PDF that exceeds pdf_max_bytes, whether declared in Content-Length or
truncated while streaming, fails with content_too_large naming the limit.
It is never reported as pdf_parse_error.
The decompressed-content budget is enforced before the parser runs, so a
decompression bomb is refused by configuration rather than by the allocator.
Flate streams are measured, ASCII85 streams are decoded under the same cap,
filters lopdf cannot decode are used verbatim (and a stage decoded before
such a filter is still counted), and an LZWDecode stream is refused because
its expansion cannot be measured. A filtered stream whose /Length is an
indirect reference is refused for the same reason: the extent cannot be
measured before parsing.
Encrypted documents are refused with pdf_encrypted whether or not a password
is supplied; pdf_password stays in the schema for compatibility but is
unsupported.
- Extracted text may be incomplete or reordered because PDF layout does not always map to linear text flow
- Scanned or image-based PDFs produce little or no text
metadata_onlymode returns a minimal document with fetch context but no extracted body textpdf_ocr = "auto"extracts text and reports scanned pages as scanned with an OCR-unavailable note per page; no OCR is attempted and no text is recovered for an image-only pagepdf_ocr = "always"fails closed withpdf_ocr_unavailable, because it demands a capability this build does not have- PDF layout reconstruction and OCR are deferred
Some keyless search engines answer automated queries with a published computational proof-of-work challenge instead of results. ADR-0009 permits solving those in-process; it permits nothing else.
Off by default. Opt in per deployment:
[search.challenge]
enabled = true- Solves the methods the engine set actually encounters: Anubis
fast,slow(an alias offast),preact,metarefresh, and WASMsha256, plus ALTCHAPBKDF2/SHA-256(Mojeek);argon2idandhashxstay unsupported - One shared solver module under
src/meta/challenge/; engines call it rather than carrying per-engine copies - At most one solve per provider per search request, and one per
cookie_ttl_seconds - A solved request runs through the normal parser and the normal four-outcome classification
- Four ceilings apply to every solve:
max_difficulty,max_attempts,timeout_ms, and thecookie_ttl_secondsreuse window - A solve that reaches any ceiling fails; it never loops or retries
- Pass cookies are held in process memory only and never appear in logs,
provider_status, diagnostics, or fixtures - The solver sends the operator's configured user agent unchanged and adds no evasion headers
Interactive CAPTCHA or visual puzzle answering, OCR of challenge images, human or third-party solving services, credential or login bypass, session or cookie theft, proxy rotation, user-agent spoofing, and retry widening to conceal an outage. The browser path separately never clicks, simulates input, or uses external services for interactive challenges.
A challenged provider that is disabled, above the difficulty ceiling,
unsupported, or out of budget reports error_class: blocked with a reason
naming the bound, not an empty result. A solved page that fails to parse is
parse_error, not blocked.
Headless Chrome/Chromium rendering supports JavaScript-heavy pages that ordinary HTTP fetching cannot render. Current source builds include the browser Cargo feature by default, but published v0.4.4 binaries do not. Rendering uses a Chrome or Chromium executable already installed on the host.
In current source builds, [fetch.browser].enabled defaults to the compiled-in
feature state, so no config is required. To enable rendering in a build that
includes the feature but has disabled it in config:
[fetch.browser]
enabled = true
policy = "auto"A source build with --no-default-features leaves browser rendering out, and an
explicit render = "browser" or browser_profile request then fails with
browser_not_compiled_in instead of quietly falling back to HTTP. Add the
feature back explicitly:
cargo build --no-default-features --features browserRendering requires a discovered Chrome/Chromium executable. eggsearch never downloads, installs, or manages one.
The render parameter on web_fetch controls transport selection. When the
caller omits it, the configured [fetch.browser].policy applies; a per-call
render overrides the configured policy.
| Value | Behavior |
|---|---|
http_only |
HTTP only, never launches Chrome |
auto (default) |
HTTP first, then at most one browser escalation for JavaScript shells, non-interactive verification, or detected challenge pages |
browser |
Browser directly, no HTTP prefetch. Fails if no Chrome/Chromium executable is available |
auto does not escalate for authentication or rate-limited responses. If browser
rendering finds an interactive challenge, web_fetch returns a manual-interaction
error. It does not solve the challenge.
An explicit render = "browser" or browser_profile request reports a stable
code rather than silently downgrading to HTTP:
| Code | Condition |
|---|---|
browser_not_compiled_in |
The binary was built without the browser feature |
browser_disabled |
The feature is compiled but [fetch.browser].enabled = false |
browser_unavailable |
Enabled, but no Chrome/Chromium executable was discovered |
When auto escalation is unavailable, the HTTP response is returned with the
structured warning browser_unavailable and browser_escalated: false. If the
browser detects an interactive challenge, the call returns a manual-interaction
error instead.
- Public-network-only: Rejects localhost, private IPv4/IPv6, link-local, and cloud metadata addresses regardless of
allow_localhost/allow_private_networksettings - No browser download: Discovers an already-installed system Chrome/Chromium. Never downloads, installs, or manages browser updates
- No challenge solving in the browser: Interactive challenges (CAPTCHAs, Turnstile) are detected and reported as structured error codes. They are never clicked, simulated, or sent to external solving services. Engine-level proof-of-work solving under ADR-0009 is separate, opt-in, and non-interactive
- Deterministic executable path: An explicitly configured invalid browser executable path fails without silently falling back to auto-discovery
- Request interception: All observable requests are intercepted and checked against the network policy
- Bounded extraction: DOM size, request count, navigation time, and post-load wait are all bounded by configuration. The session request bound is enforced: reaching
max_requestscloses the browser session and the next render relaunches, and an idle interval with no in-flight render tears the shared browser down - Existing sanitation pipeline: Rendered DOM flows through the same HTML extraction, text bounding, and prompt-injection sanitation as ordinary HTTP fetches
CAPTCHAs, Turnstile, and login walls return structured error codes through the MCP error response:
browser_manual_interaction_requiredbrowser_profile_requires_attention
These are never automated.
The response includes:
transport:"http"or"browser"browser_escalated: whether auto mode escalatedbrowser_profile: display name when a persistent profile was used
Fresh raw cache entries can satisfy a new extraction mode, link setting, character bound, or PDF page selection without another network request.
| Setting | Default | Description |
|---|---|---|
enabled |
true when compiled |
Enable browser rendering escalation; false in a --no-default-features build |
policy |
"auto" |
Default render policy when render is omitted: http_only, auto, or browser |
executable |
auto-discovered | Path to Chrome/Chromium executable |
startup_timeout_ms |
10000 |
Browser startup timeout |
navigation_timeout_ms |
20000 |
Page navigation timeout |
post_load_wait_ms |
1500 |
Wait after page load |
verification_wait_ms |
10000 |
Wait for non-interactive verification |
max_requests |
100 |
Renders per browser session; reaching it closes the session and the next render relaunches |
idle_timeout_ms |
90000 |
Idle interval with no in-flight render after which the shared browser process is closed (minimum 5000) |
max_dom_bytes |
4000000 |
Maximum DOM size |
global_concurrency |
1 |
Global browser concurrency |
per_origin_concurrency |
1 |
Per-origin browser render cap (combined with origin_browser_concurrency via min) |
block_media |
true |
Block media autoplay |
Persistent browser profiles allow a local operator to establish a dedicated browser session for an origin that requires authentication or interactive human verification.
- Created only through CLI commands. MCP callers cannot create profiles or launch headed browsers
- Each profile requires explicit headed local setup via
browser-login - Restricted to its recorded exact origin
- Disabled by default
# Create a profile and open a headed browser for login
eggsearch browser-login https://example.com --profile my-portal
# List all profiles
eggsearch browser-profiles list
# Inspect a profile
eggsearch browser-profiles inspect my-portal
# Remove a profile
eggsearch browser-profiles remove my-portalOnce established, use it in web_fetch:
{
"url": "https://example.com/dashboard",
"browser_profile": "my-portal"
}[fetch.browser.persistent_profiles]
enabled = true| Setting | Default | Description |
|---|---|---|
enabled |
false |
Enable persistent browser profiles |
profiles_dir |
platform default | Custom profiles directory |
allowed_profiles |
empty (all allowed) | Allowlist of profile names |
profile_process_timeout_ms |
30000 |
Timeout for profile-scoped browser processes |
- Profile metadata:
$XDG_DATA_HOME/eggsearch/browser-profiles/<opaque-id>/profile.toml - Chrome data: sibling
chrome-data/directory - Each profile uses opaque directory IDs for cache partitioning
browser-loginand profile-scoped MCP fetches use the same Eggsearch-ownedchrome-datadirectory- Chrome manages cookies and storage within the profile directory. Eggsearch never exports, logs, or serializes cookies
- Process-local cache is not invalidated when a profile is removed from the CLI (cache is process-scoped)
Optional listener-free HTTP/SOCKS proxy-chain route beneath eggfetch for
fixed operator-owned provider upstreams. Source-build opt-in only: prebuilt
and default binaries do not include it. Build with the egress feature:
cargo build --features egressEggfetch remains the HTTP, pooling, destination-TLS, decompression,
redirect-mechanics, and deadline owner. Eggress owns only physical TCP
proxy-hop establishment. No listener, service lifecycle, admin endpoint,
system-proxy mutation, or full embed runtime is introduced. Direct behavior
is unchanged when no route is configured. Chain failures fail closed and
never fall back to direct. Retry policy stays with OriginController;
request/total timeouts stay with eggfetch/eggsearch with no second outer
Eggress deadline.
Supported base protocols are ordinary TCP http, socks4, and socks5
(httponly accepted as an HTTP variant). Shadowsocks, Trojan, WebSocket,
SSH, QUIC/H3, UDP, legacy crypto, pproxy compatibility, and insecure TLS
remain excluded.
Each hop host accepts a bare DNS hostname, an IPv4 literal, or a bare
IPv6 literal such as ::1 or 2001:db8::1. The host field never takes a
scheme, userinfo, path, bracket wrapping, or an appended port; port
remains the sole port source.
[egress]
enabled = true
[[egress.hops]]
scheme = "http"
host = "127.0.0.1"
port = 8080
[[egress.hops]]
scheme = "socks5"
host = "127.0.0.1"
port = 1080
username = "alice"
password_env = "EGRESS_PROXY_PASS"Proxy passwords are never stored in configuration. Each hop names the
environment variable holding its password via password_env; the value is
resolved at connector-build time and a missing or empty variable fails the
request closed. AppConfig::save never writes raw secrets because only
variable names are persisted. Diagnostics and Debug output use the
redacted route form (****:****@host:port) and never log full proxy URIs
or credentials. Proxy authentication is a hop handshake and is never
forwarded to the destination as a normal header.
When routed, only vendored provider search-engine upstreams use the configured chain. The following remain direct by design and are guarded by static tests:
web_fetch,batch_fetch, andrepo_fetchdynamic targets keep resolved-address pinning on the direct route. Eggfetch 0.2.0 rejects the custom-dialer plusresolved_addressescombination, so these SSRF-sensitive paths cannot silently migrate to proxy DNS.- Forge tree APIs, package-resolver fetches, the self-updater, startup/integration loopback health probes, rmcp Streamable HTTP transport, and browser subprocess traffic remain direct in this phase.
- If the binary reports that the egress feature is missing, rebuild with
--features egressor set[egress].enabled = false. - Only
http,socks4, andsocks5(httponly) schemes are accepted. - If a credential environment variable is missing or empty, export it before starting the server. Unset secrets fail closed.
authentication/connection/timeoutdial errors carry kind/stage/hop/protocol facts without credentials; a failed chain never retries direct.
Enabled by default; there is nothing to turn on.
- Every HTML-ish fetch sends
Accept: text/markdown;q=1.0, text/html;q=0.9, */*;q=0.1. Origins that support Markdown for Agents (Cloudflare-fronted sites, for example) return the smaller representation; origins that ignore the header are unaffected. - PDF, JSON/XML/CSV/YAML/TOML, source/config, binary, and code-host raw targets keep the plain HTML
policy, and
extract_mode = "metadata_only"never negotiates because page metadata lives in the HTML head. - A 406/415, an unsupported content type, or an empty Markdown body falls back to HTML once, so a fetch costs at most two requests.
- The representation is a cache dimension (
md1/html1), so a Markdown body can never be served for an HTML cache key. Responses reportrepresentation, and origin token headers (x-markdown-tokens,x-original-tokens) are surfaced when present. include_linksworks on Markdown bodies: links are extracted from the parsed Markdown AST with the same cap and classification as the HTML path.