fix(app): reach the AI clients the UA gate and empty shell were losing

.github/workflows/bot-serving-check.yml

@@ -28,11 +28,12 @@ permissions:
 jobs:
   bot-serving:
     runs-on: ubuntu-latest
-    # 19 checks x (--retry 2 -> up to 3 attempts x --max-time 30) can reach
-    # ~28.5 min worst-case; 32 leaves room to report a clean failure rather
-    # than dying to the job timeout, which reports nothing useful. Recompute
-    # this when adding checks: the ceiling is checks x 90s, plus margin.
-    timeout-minutes: 32
+    # 22 check() calls x (--retry 2 -> up to 3 attempts x --max-time 30) can
+    # reach ~33 min worst-case, plus one non-retried trailing-slash probe
+    # (30s); 38 leaves room to report a clean failure rather than dying to the
+    # job timeout, which reports nothing useful. Recompute this when adding
+    # checks: the ceiling is check() calls x 90s, plus margin.
+    timeout-minutes: 38
     steps:
       - name: Crawler UAs must get 200 + per-route titles
         run: |
@@ -96,7 +97,10 @@ jobs:
             "Mozilla/5.0 (compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)" \
             "DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)" \
             "Mozilla/5.0 (compatible; Amzn-SearchBot/1.0)" \
-            "Mozilla/5.0 (compatible; Amzn-User/1.0)"
+            "Mozilla/5.0 (compatible; Amzn-User/1.0)" \
+            "Mozilla/5.0 (compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)" \
+            "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" \
+            "Grok/1.0"
           do
             check "$ua" "$ORIGIN/scatter-basic" "<title>Basic Scatter Plot | anyplot.ai</title>"
           done
@@ -107,6 +111,11 @@ jobs:
           check "$GOOGLEBOT"   "$ORIGIN/llms.txt" "# anyplot"
           check "$CHATGPTUSER" "$ORIGIN/llms.txt" "# anyplot"
 
+          # llms-full.txt is proxied to the API for EVERY client (mapped or
+          # not) — before, the SPA catch-all soft-404'd it with the homepage
+          # shell. The catalogue-index line proves the API generated it.
+          check "$HUMAN" "$ORIGIN/llms-full.txt" "# anyplot — full catalogue index"
+
           # A trailing slash must normalise to the canonical URL on THIS host.
           # It used to 307 to http://api.anyplot.ai/seo-proxy/... — internal
           # path, wrong host, plain http, and that host disallows all crawling.

CHANGELOG.md

@@ -53,6 +53,27 @@ aggregate instead: an italic *Catalog* line at the end of the version section an
   project version, injected by Vite from the `[project]` version in `pyproject.toml` — the same
   field the release flow bumps — instead of a literal that nobody would think to update. A test
   asserts the injected value matches `pyproject.toml`, so the two cannot drift (#10485).
+- **The crawler allowlist no longer misses documented crawlers — or Grok** — nginx's UA map
+  lacked `Amazonbot` and `meta-externalagent` (both documented crawler tokens; the existing
+  `amzn-*` / `meta-externalfetcher` entries never matched them) and matched xAI only via
+  `grokbot`/`xai-grok`/`grok-deepsearch`, so a bare `Grok` UA fell through to the empty SPA
+  shell — the exact "insufficient relevant content" a user's Grok session reported. The map now
+  matches `~*grok` (subsuming all three), Amazonbot, meta-externalagent, Diffbot and Firecrawl,
+  and the daily bot-serving monitor covers the new tokens (#10488).
+- **JS-less clients no longer see a completely empty page** — the SPA shell was
+  `<div id="root">` with no `<noscript>`, so any fetcher missing from the UA allowlist rendered
+  literally nothing. A noscript block now points such clients at `llms.txt`, `llms-full.txt`,
+  the JSON API and the GitHub repository (#10488).
+
+### Changed
+
+- **llms.txt now tells agents how to actually fetch things** — the file linked nine human-facing
+  HTML pages and named no machine endpoint. New sections document the REST API (base URL, the
+  retrieval endpoints, OpenAPI), the GCS render URL pattern (themes, responsive widths, WebP),
+  the CORS-open GitHub raw source URLs, and a worked three-step "fetch one plot" recipe — plus an
+  honest note that prerendered page HTML is gated on a crawler user agent while these URLs are
+  not. `anyplot.ai/llms-full.txt` is now proxied to the API's generated catalogue index instead
+  of soft-404ing to the homepage shell, and the daily monitor asserts it (#10488).
 
 ## [3.1.0] — 2026-08-19 — Legible to machines
 

app/index.html

@@ -140,6 +140,22 @@
   </head>
   <body>
     <div id="root"></div>
+    <noscript>
+      <!-- The prerender path is gated on a crawler user-agent allowlist
+           (app/nginx.conf), so any JS-less client NOT on that list lands here.
+           Until now "here" was a completely empty page — Grok's fetcher read
+           exactly that and reported the site as having no content (AI-access
+           audit 2026-08-19). Point such clients at the surfaces that work for
+           everyone. -->
+      <h1>anyplot.ai — the open plot catalogue</h1>
+      <p>
+        This page needs JavaScript. Machine-readable access works without it:
+        <a href="/llms.txt">llms.txt</a> (how to query the catalogue),
+        <a href="/llms-full.txt">llms-full.txt</a> (every spec, one per line),
+        the <a href="https://api.anyplot.ai/openapi.json">JSON API</a>, and the
+        <a href="https://github.com/MarkusNeusinger/anyplot">GitHub repository</a>.
+      </p>
+    </noscript>
     <script type="module" src="/src/main.tsx"></script>
   </body>
 </html>

app/nginx.conf

@@ -66,16 +66,28 @@ map $http_user_agent $is_bot {
     ~*duckassistbot 1;
     ~*amzn-searchbot 1;
     ~*amzn-user 1;
+    # Amazon's documented crawler token is "Amazonbot" — the two amzn-* entries
+    # above never matched it, so the documented crawler got the empty shell
+    # (AI-access audit 2026-08-19). Same for Meta: "meta-externalagent" is the
+    # documented AI-training crawler, distinct from meta-externalfetcher.
+    ~*amazonbot 1;
+    ~*meta-externalagent 1;
     # Community-reported assistants: no vendor documentation exists for these
     # tokens, so they are best-effort. Mapping them is harmless — this map only
     # decides WHAT an agent receives, never whether it may crawl; that is
     # robots.txt plus Cloudflare. Under-mapping is the failure mode that put
     # seven fetchers on the empty shell, so err toward listing.
-    ~*grokbot 1;
-    ~*xai-grok 1;
-    ~*grok-deepsearch 1;
+    #
+    # ~*grok subsumes the previously listed grokbot / xai-grok /
+    # grok-deepsearch AND the bare "Grok" token xAI's fetcher was seen sending
+    # — which fell through to the empty shell and produced Grok's
+    # "insufficient relevant content" (AI-access audit 2026-08-19). No common
+    # browser UA contains the substring.
+    ~*grok 1;
     ~*youbot 1;
     ~*cohere-ai 1;
+    ~*diffbot 1;
+    ~*firecrawl 1;
     # Social Media
     ~*twitterbot 1;
     ~*facebookexternalhit 1;
@@ -225,6 +237,23 @@ server {
         try_files $uri =404;
     }
 
+    # llms-full.txt is generated from the DB (one line per spec), so unlike
+    # llms.txt it cannot be a static file — proxy it to the API's
+    # /llms-full.txt endpoint for EVERY client, mapped crawler UA or not.
+    # Before this, the URL was a soft-404: the SPA catch-all answered it with
+    # 200 + the homepage shell (AI-access audit 2026-08-19).
+    location = /llms-full.txt {
+        set $seo_backend https://api.anyplot.ai;
+        proxy_pass $seo_backend/llms-full.txt;
+        proxy_set_header Host api.anyplot.ai;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_ssl_server_name on;
+        proxy_ssl_verify on;
+        # Same 4-deep chain as @seo_proxy above — default depth 1 breaks it.
+        proxy_ssl_verify_depth 4;
+        proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
+    }
+
     # SPA routing - serve index.html for all routes
     # Social media bots get redirected to backend for proper og:tags
     location / {

app/public/llms.txt

@@ -4,12 +4,43 @@
 > and reviews implementations across fifteen libraries in python, r, julia and javascript.
 > browse, copy, adapt — colorblind-safe by default.
 
-every plot page carries the runnable source code (MIT-licensed). catalog pages are also served
-as prerendered HTML to crawler user agents, so fetching a page URL directly returns the
-per-route title, description and content without executing JavaScript.
+every plot page carries the runnable source code (MIT-licensed). catalog pages are served as
+prerendered HTML to recognised crawler user agents — if your client is not on that list you
+get an empty app shell, so prefer the API and raw URLs below: they work for every client,
+no special user agent required.
+
+## Fetch one plot (worked example)
+
+1. find the spec id: [llms-full.txt](https://anyplot.ai/llms-full.txt) lists every spec on one
+   line (`spec_id | title | hub page | implemented libraries`) — search it for a keyword like
+   "bar", or filter by tag via the API (below).
+2. get the source code: `https://api.anyplot.ai/specs/bar-error/seaborn/code` returns JSON with
+   the runnable code (any library id works — the language is resolved automatically).
+3. get the rendered image:
+   `https://storage.googleapis.com/anyplot-images/plots/bar-error/python/seaborn/plot-light.png`
+   (dark theme: `plot-dark.png`; smaller: `plot-light_400.png`, `_800`, `_1200`; WebP: `plot-light.webp`).
+
+## API
+
+base URL: https://api.anyplot.ai — public, read-only, JSON, no auth.
+
+- `GET /specs` — all specs with metadata
+- `GET /specs/{spec_id}` — one spec with all implementations (preview URLs, quality, review)
+- `GET /specs/{spec_id}/{library}/code` — runnable source for one implementation
+- `GET /plots/filter` — every rendered implementation, filterable and pageable.
+  filter params: `plot=` (plot type, e.g. `plot=bar`), `lib=` (library), `lang=` (language),
+  `spec=` (spec id), `data=` (data type), `dom=` (domain), `feat=` (features);
+  comma-separated values = OR, repeated params = AND; page with `limit=` and `offset=`.
+  example: `https://api.anyplot.ai/plots/filter?plot=bar&lib=seaborn&limit=5`
+- [OpenAPI schema](https://api.anyplot.ai/openapi.json) — the full endpoint reference
+
+raw source is also on GitHub (CORS-open):
+`https://raw.githubusercontent.com/MarkusNeusinger/anyplot/main/plots/{spec_id}/implementations/{language}/{library}.{ext}`
+and the spec itself at `.../plots/{spec_id}/specification.md`.
 
 ## Catalog
 
+- [Full catalogue index](https://anyplot.ai/llms-full.txt): one line per spec — ids, titles, libraries
 - [Plot gallery](https://anyplot.ai/plots): every rendered implementation, filterable by library, language and tags
 - [Specifications](https://anyplot.ai/specs): the library-agnostic plot specs the implementations are generated from
 - [Libraries](https://anyplot.ai/libraries): the fifteen supported plotting libraries across four languages