From 798392b4729c977c06ee99cdeaf523547e62f6dc Mon Sep 17 00:00:00 2001
From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com>
Date: Wed, 19 Aug 2026 20:34:46 +0200
Subject: [PATCH 1/4] fix(app): reach the AI clients the UA gate and empty
 shell were losing
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

App-side half of the 2026-08-19 AI-access audit fixes (api side: #10487):

- nginx UA map: ~*grok (bare 'Grok' token fell through — the reported
  'insufficient relevant content'), Amazonbot, meta-externalagent (both
  documented tokens the amzn-*/meta-externalfetcher entries never matched),
  Diffbot, Firecrawl
- llms.txt rewritten for machine consumers: REST API section, GCS render
  URL pattern, GitHub raw URLs, worked fetch-one-plot recipe, honest note
  that prerendered HTML is UA-gated
- anyplot.ai/llms-full.txt proxied to the API's generated catalogue index
  (was a soft-404: SPA shell with HTTP 200)
- noscript fallback in the SPA shell pointing JS-less clients at llms.txt,
  llms-full.txt, the API and GitHub
- bot-serving-check: new UA tokens + llms-full.txt assertion, timeout
  recomputed for 23 checks

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 .github/workflows/bot-serving-check.yml | 16 ++++++++---
 CHANGELOG.md                            | 21 +++++++++++++++
 app/index.html                          | 16 +++++++++++
 app/nginx.conf                          | 35 ++++++++++++++++++++++---
 app/public/llms.txt                     | 32 +++++++++++++++++++---
 5 files changed, 110 insertions(+), 10 deletions(-)

diff --git a/.github/workflows/bot-serving-check.yml b/.github/workflows/bot-serving-check.yml
index 4eff940ee3..eda7462f53 100644
--- a/.github/workflows/bot-serving-check.yml
+++ b/.github/workflows/bot-serving-check.yml
@@ -28,11 +28,11 @@ permissions:
 jobs:
   bot-serving:
     runs-on: ubuntu-latest
-    # 19 checks x (--retry 2 -> up to 3 attempts x --max-time 30) can reach
-    # ~28.5 min worst-case; 32 leaves room to report a clean failure rather
+    # 23 checks x (--retry 2 -> up to 3 attempts x --max-time 30) can reach
+    # ~34.5 min worst-case; 38 leaves room to report a clean failure rather
     # than dying to the job timeout, which reports nothing useful. Recompute
     # this when adding checks: the ceiling is checks x 90s, plus margin.
-    timeout-minutes: 32
+    timeout-minutes: 38
     steps:
       - name: Crawler UAs must get 200 + per-route titles
         run: |
@@ -96,7 +96,10 @@ jobs:
             "Mozilla/5.0 (compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)" \
             "DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)" \
             "Mozilla/5.0 (compatible; Amzn-SearchBot/1.0)" \
-            "Mozilla/5.0 (compatible; Amzn-User/1.0)"
+            "Mozilla/5.0 (compatible; Amzn-User/1.0)" \
+            "Mozilla/5.0 (compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)" \
+            "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" \
+            "Grok/1.0"
           do
             check "$ua" "$ORIGIN/scatter-basic" "<title>Basic Scatter Plot | anyplot.ai</title>"
           done
@@ -107,6 +110,11 @@ jobs:
           check "$GOOGLEBOT"   "$ORIGIN/llms.txt" "# anyplot"
           check "$CHATGPTUSER" "$ORIGIN/llms.txt" "# anyplot"
 
+          # llms-full.txt is proxied to the API for EVERY client (mapped or
+          # not) — before, the SPA catch-all soft-404'd it with the homepage
+          # shell. The catalogue-index line proves the API generated it.
+          check "$HUMAN" "$ORIGIN/llms-full.txt" "# anyplot — full catalogue index"
+
           # A trailing slash must normalise to the canonical URL on THIS host.
           # It used to 307 to http://api.anyplot.ai/seo-proxy/... — internal
           # path, wrong host, plain http, and that host disallows all crawling.
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5f6f8362de..38ac4bd27a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -53,6 +53,27 @@ aggregate instead: an italic *Catalog* line at the end of the version section an
   project version, injected by Vite from the `[project]` version in `pyproject.toml` — the same
   field the release flow bumps — instead of a literal that nobody would think to update. A test
   asserts the injected value matches `pyproject.toml`, so the two cannot drift (#10485).
+- **The crawler allowlist no longer misses documented crawlers — or Grok** — nginx's UA map
+  lacked `Amazonbot` and `meta-externalagent` (both documented crawler tokens; the existing
+  `amzn-*` / `meta-externalfetcher` entries never matched them) and matched xAI only via
+  `grokbot`/`xai-grok`/`grok-deepsearch`, so a bare `Grok` UA fell through to the empty SPA
+  shell — the exact "insufficient relevant content" a user's Grok session reported. The map now
+  matches `~*grok` (subsuming all three), Amazonbot, meta-externalagent, Diffbot and Firecrawl,
+  and the daily bot-serving monitor covers the new tokens.
+- **JS-less clients no longer see a completely empty page** — the SPA shell was
+  `<div id="root">` with no `<noscript>`, so any fetcher missing from the UA allowlist rendered
+  literally nothing. A noscript block now points such clients at `llms.txt`, `llms-full.txt`,
+  the JSON API and the GitHub repository.
+
+### Changed
+
+- **llms.txt now tells agents how to actually fetch things** — the file linked nine human-facing
+  HTML pages and named no machine endpoint. New sections document the REST API (base URL, the
+  retrieval endpoints, OpenAPI), the GCS render URL pattern (themes, responsive widths, WebP),
+  the CORS-open GitHub raw source URLs, and a worked three-step "fetch one plot" recipe — plus an
+  honest note that prerendered page HTML is gated on a crawler user agent while these URLs are
+  not. `anyplot.ai/llms-full.txt` is now proxied to the API's generated catalogue index instead
+  of soft-404ing to the homepage shell, and the daily monitor asserts it.
 
 ## [3.1.0] — 2026-08-19 — Legible to machines
 
diff --git a/app/index.html b/app/index.html
index cb612de81e..5e457d408a 100644
--- a/app/index.html
+++ b/app/index.html
@@ -140,6 +140,22 @@
   </head>
   <body>
     <div id="root"></div>
+    <noscript>
+      <!-- The prerender path is gated on a crawler user-agent allowlist
+           (app/nginx.conf), so any JS-less client NOT on that list lands here.
+           Until now "here" was a completely empty page — Grok's fetcher read
+           exactly that and reported the site as having no content (AI-access
+           audit 2026-08-19). Point such clients at the surfaces that work for
+           everyone. -->
+      <h1>anyplot.ai — the open plot catalogue</h1>
+      <p>
+        This page needs JavaScript. Machine-readable access works without it:
+        <a href="/llms.txt">llms.txt</a> (how to query the catalogue),
+        <a href="/llms-full.txt">llms-full.txt</a> (every spec, one per line),
+        the <a href="https://api.anyplot.ai/openapi.json">JSON API</a>, and the
+        <a href="https://github.com/MarkusNeusinger/anyplot">GitHub repository</a>.
+      </p>
+    </noscript>
     <script type="module" src="/src/main.tsx"></script>
   </body>
 </html>
diff --git a/app/nginx.conf b/app/nginx.conf
index 1aa88a8f38..467f47f9e3 100644
--- a/app/nginx.conf
+++ b/app/nginx.conf
@@ -66,16 +66,28 @@ map $http_user_agent $is_bot {
     ~*duckassistbot 1;
     ~*amzn-searchbot 1;
     ~*amzn-user 1;
+    # Amazon's documented crawler token is "Amazonbot" — the two amzn-* entries
+    # above never matched it, so the documented crawler got the empty shell
+    # (AI-access audit 2026-08-19). Same for Meta: "meta-externalagent" is the
+    # documented AI-training crawler, distinct from meta-externalfetcher.
+    ~*amazonbot 1;
+    ~*meta-externalagent 1;
     # Community-reported assistants: no vendor documentation exists for these
     # tokens, so they are best-effort. Mapping them is harmless — this map only
     # decides WHAT an agent receives, never whether it may crawl; that is
     # robots.txt plus Cloudflare. Under-mapping is the failure mode that put
     # seven fetchers on the empty shell, so err toward listing.
-    ~*grokbot 1;
-    ~*xai-grok 1;
-    ~*grok-deepsearch 1;
+    #
+    # ~*grok subsumes the previously listed grokbot / xai-grok /
+    # grok-deepsearch AND the bare "Grok" token xAI's fetcher was seen sending
+    # — which fell through to the empty shell and produced Grok's
+    # "insufficient relevant content" (AI-access audit 2026-08-19). No common
+    # browser UA contains the substring.
+    ~*grok 1;
     ~*youbot 1;
     ~*cohere-ai 1;
+    ~*diffbot 1;
+    ~*firecrawl 1;
     # Social Media
     ~*twitterbot 1;
     ~*facebookexternalhit 1;
@@ -225,6 +237,23 @@ server {
         try_files $uri =404;
     }
 
+    # llms-full.txt is generated from the DB (one line per spec), so unlike
+    # llms.txt it cannot be a static file — proxy it to the API's
+    # /llms-full.txt endpoint for EVERY client, mapped crawler UA or not.
+    # Before this, the URL was a soft-404: the SPA catch-all answered it with
+    # 200 + the homepage shell (AI-access audit 2026-08-19).
+    location = /llms-full.txt {
+        set $seo_backend https://api.anyplot.ai;
+        proxy_pass $seo_backend/llms-full.txt;
+        proxy_set_header Host api.anyplot.ai;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_ssl_server_name on;
+        proxy_ssl_verify on;
+        # Same 4-deep chain as @seo_proxy above — default depth 1 breaks it.
+        proxy_ssl_verify_depth 4;
+        proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
+    }
+
     # SPA routing - serve index.html for all routes
     # Social media bots get redirected to backend for proper og:tags
     location / {
diff --git a/app/public/llms.txt b/app/public/llms.txt
index 72f8fcf69e..1c43757cd9 100644
--- a/app/public/llms.txt
+++ b/app/public/llms.txt
@@ -4,12 +4,38 @@
 > and reviews implementations across fifteen libraries in python, r, julia and javascript.
 > browse, copy, adapt — colorblind-safe by default.
 
-every plot page carries the runnable source code (MIT-licensed). catalog pages are also served
-as prerendered HTML to crawler user agents, so fetching a page URL directly returns the
-per-route title, description and content without executing JavaScript.
+every plot page carries the runnable source code (MIT-licensed). catalog pages are served as
+prerendered HTML to recognised crawler user agents — if your client is not on that list you
+get an empty app shell, so prefer the API and raw URLs below: they work for every client,
+no special user agent required.
+
+## Fetch one plot (worked example)
+
+1. find the spec id: [llms-full.txt](https://anyplot.ai/llms-full.txt) lists every spec on one
+   line (`spec_id | title | hub page | implemented libraries`).
+2. get the source code: `https://api.anyplot.ai/specs/bar-error/seaborn/code` returns JSON with
+   the runnable code (any library id works — the language is resolved automatically).
+3. get the rendered image:
+   `https://storage.googleapis.com/anyplot-images/plots/bar-error/python/seaborn/plot-light.png`
+   (dark theme: `plot-dark.png`; smaller: `plot-light_400.png`, `_800`, `_1200`; WebP: `plot-light.webp`).
+
+## API
+
+base URL: https://api.anyplot.ai — public, read-only, JSON, no auth.
+
+- `GET /specs` — all specs with metadata
+- `GET /specs/{spec_id}` — one spec with all implementations (preview URLs, quality, review)
+- `GET /specs/{spec_id}/{library}/code` — runnable source for one implementation
+- `GET /plots/filter` — every rendered implementation, filterable and pageable
+- [OpenAPI schema](https://api.anyplot.ai/openapi.json) — the full endpoint reference
+
+raw source is also on GitHub (CORS-open):
+`https://raw.githubusercontent.com/MarkusNeusinger/anyplot/main/plots/{spec_id}/implementations/{language}/{library}.{ext}`
+and the spec itself at `.../plots/{spec_id}/specification.md`.
 
 ## Catalog
 
+- [Full catalogue index](https://anyplot.ai/llms-full.txt): one line per spec — ids, titles, libraries
 - [Plot gallery](https://anyplot.ai/plots): every rendered implementation, filterable by library, language and tags
 - [Specifications](https://anyplot.ai/specs): the library-agnostic plot specs the implementations are generated from
 - [Libraries](https://anyplot.ai/libraries): the fifteen supported plotting libraries across four languages

From 5e7cd8357bfd96b5f037b65afcd15ac7283a0e44 Mon Sep 17 00:00:00 2001
From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com>
Date: Wed, 19 Aug 2026 20:36:20 +0200
Subject: [PATCH 2/4] docs(llms): document filter params and keyword search in
 llms.txt

An agent asked for 'bar charts' had no documented way to search: the
filter endpoint's params (plot=, lib=, lang=, spec=, data=, dom=, feat=;
comma=OR, repeated=AND) are now in the API section with a live-verified
example, and the fetch-one-plot recipe points at grepping llms-full.txt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 app/public/llms.txt | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/app/public/llms.txt b/app/public/llms.txt
index 1c43757cd9..3f720509e5 100644
--- a/app/public/llms.txt
+++ b/app/public/llms.txt
@@ -12,7 +12,8 @@ no special user agent required.
 ## Fetch one plot (worked example)
 
 1. find the spec id: [llms-full.txt](https://anyplot.ai/llms-full.txt) lists every spec on one
-   line (`spec_id | title | hub page | implemented libraries`).
+   line (`spec_id | title | hub page | implemented libraries`) — search it for a keyword like
+   "bar", or filter by tag via the API (below).
 2. get the source code: `https://api.anyplot.ai/specs/bar-error/seaborn/code` returns JSON with
    the runnable code (any library id works — the language is resolved automatically).
 3. get the rendered image:
@@ -26,7 +27,11 @@ base URL: https://api.anyplot.ai — public, read-only, JSON, no auth.
 - `GET /specs` — all specs with metadata
 - `GET /specs/{spec_id}` — one spec with all implementations (preview URLs, quality, review)
 - `GET /specs/{spec_id}/{library}/code` — runnable source for one implementation
-- `GET /plots/filter` — every rendered implementation, filterable and pageable
+- `GET /plots/filter` — every rendered implementation, filterable and pageable.
+  filter params: `plot=` (plot type, e.g. `plot=bar`), `lib=` (library), `lang=` (language),
+  `spec=` (spec id), `data=` (data type), `dom=` (domain), `feat=` (features);
+  comma-separated values = OR, repeated params = AND; page with `limit=` and `offset=`.
+  example: `https://api.anyplot.ai/plots/filter?plot=bar&lib=seaborn&limit=5`
 - [OpenAPI schema](https://api.anyplot.ai/openapi.json) — the full endpoint reference
 
 raw source is also on GitHub (CORS-open):

From e5aecea21b2bd80edecb464ce09610f946450844 Mon Sep 17 00:00:00 2001
From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com>
Date: Wed, 19 Aug 2026 20:36:47 +0200
Subject: [PATCH 3/4] docs(changelog): reference #10488 in the app-side
 AI-access entries

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 CHANGELOG.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 38ac4bd27a..5151cc5d18 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -59,11 +59,11 @@ aggregate instead: an italic *Catalog* line at the end of the version section an
   `grokbot`/`xai-grok`/`grok-deepsearch`, so a bare `Grok` UA fell through to the empty SPA
   shell — the exact "insufficient relevant content" a user's Grok session reported. The map now
   matches `~*grok` (subsuming all three), Amazonbot, meta-externalagent, Diffbot and Firecrawl,
-  and the daily bot-serving monitor covers the new tokens.
+  and the daily bot-serving monitor covers the new tokens (#10488).
 - **JS-less clients no longer see a completely empty page** — the SPA shell was
   `<div id="root">` with no `<noscript>`, so any fetcher missing from the UA allowlist rendered
   literally nothing. A noscript block now points such clients at `llms.txt`, `llms-full.txt`,
-  the JSON API and the GitHub repository.
+  the JSON API and the GitHub repository (#10488).
 
 ### Changed
 
@@ -73,7 +73,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an
   the CORS-open GitHub raw source URLs, and a worked three-step "fetch one plot" recipe — plus an
   honest note that prerendered page HTML is gated on a crawler user agent while these URLs are
   not. `anyplot.ai/llms-full.txt` is now proxied to the API's generated catalogue index instead
-  of soft-404ing to the homepage shell, and the daily monitor asserts it.
+  of soft-404ing to the homepage shell, and the daily monitor asserts it (#10488).
 
 ## [3.1.0] — 2026-08-19 — Legible to machines
 

From c54359060673414d3e58da5372a77a3910b73786 Mon Sep 17 00:00:00 2001
From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com>
Date: Wed, 19 Aug 2026 20:45:28 +0200
Subject: [PATCH 4/4] docs(monitor): count check() calls and the redirect probe
 separately

Copilot review on #10488: the timeout comment claimed 23 checks; the
script performs 22 retried check() calls plus one non-retried
trailing-slash probe. Ceiling unchanged (38 min).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 .github/workflows/bot-serving-check.yml | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/bot-serving-check.yml b/.github/workflows/bot-serving-check.yml
index eda7462f53..9b1c1d8973 100644
--- a/.github/workflows/bot-serving-check.yml
+++ b/.github/workflows/bot-serving-check.yml
@@ -28,10 +28,11 @@ permissions:
 jobs:
   bot-serving:
     runs-on: ubuntu-latest
-    # 23 checks x (--retry 2 -> up to 3 attempts x --max-time 30) can reach
-    # ~34.5 min worst-case; 38 leaves room to report a clean failure rather
-    # than dying to the job timeout, which reports nothing useful. Recompute
-    # this when adding checks: the ceiling is checks x 90s, plus margin.
+    # 22 check() calls x (--retry 2 -> up to 3 attempts x --max-time 30) can
+    # reach ~33 min worst-case, plus one non-retried trailing-slash probe
+    # (30s); 38 leaves room to report a clean failure rather than dying to the
+    # job timeout, which reports nothing useful. Recompute this when adding
+    # checks: the ceiling is check() calls x 90s, plus margin.
     timeout-minutes: 38
     steps:
       - name: Crawler UAs must get 200 + per-route titles