fix(api): open the machine interface AI agents were locked out of

[MarkusNeusinger]

Summary

• Opens the read API to robots-compliant agents: api.anyplot.ai/robots.txt no longer serves a blanket Disallow: / — only /debug and /proxy stay excluded. The old policy forbade the REST endpoints, openapi.json and the MCP transport on the very host llms.txt advertises as the machine interface (AI-access audit 2026-08-19).
• Fixes the language trap: /specs/{id}/{library}/code resolved language=python by default, 404ing all 7 R/Julia/JS libraries (28% of the catalogue, 1,004 implementations) at the obvious URL. The language now derives from the library registry (core/constants.py, new LIBRARY_LANGUAGES); an explicit ?language= still wins. The equivalent MCP-side fix lands in the upcoming dedicated MCP PR.
• Enriches the machine-facing pages: implementation JSON-LD now carries the real render as an ImageObject (not the 1200×630 og card) plus license/codeRepository/dateModified/keywords/author/isBasedOn; spec-hub bot pages show the best-quality real render with the full asset list instead of the og collage; second-tier meta added (robots max-image-preview:large, og:locale, og:image dims/alt, twitter:site/creator).
• Adds GET /llms-full.txt: one line per spec (id, title, hub URL, libraries) under a header documenting the UA-independent retrieval recipes (code endpoint, GCS render URL pattern, OpenAPI, MCP). The nginx mapping for anyplot.ai/llms-full.txt follows in the app-side PR — merge this one first.
• Answers HEAD like GET via an ASGI middleware (was: 405 on every route, including /health and the seo-proxy pages) without doubling the openapi surface, and unifies version identity: openapi.json (was 1.0.0, "9 libraries") and /health (was 0.2.0) now report the packaged version with a registry-derived description and a servers block.

Plan

Fix order approved from the 2026-08-19 AI-access audit (5-agent live audit of anyplot.ai / api.anyplot.ai / GCS): this PR is steps 2, 3 (REST half), and 6, plus the api-side half of step 4. Bucket CORS (step 1) is already applied and verified live. App-side steps (llms.txt content, nginx UA map, noscript) follow in a separate PR; the MCP overhaul is a dedicated follow-up.

Test plan

• ruff check + ruff format --check clean; mypy api core clean
• 1,677 unit + 67 integration tests pass (new tests: robots policy, language resolution incl. override + unknown-library fallback, HEAD semantics, llms-full builder/endpoint, JSON-LD ImageObject/license, hub render + fallback, robots-meta content)
• Live smoke against a local server on the shared DB: /health reports 3.1.0, HEAD /health 200, new robots.txt served, /llms-full.txt lists all 324 specs, /specs/acf-pacf/{ggplot2,makie}/code 200 without ?language=, hub page shows the seaborn render, impl JSON-LD carries ImageObject + MIT license
• After merge: Cloud Build deploy green, then re-verify the same URLs against api.anyplot.ai

[Copilot]

Pull request overview

This PR improves the “machine interface” of the anyplot API by making the read surface reachable and consistent for bots/agents, and by enriching bot-facing HTML/structured data so assistants can reliably discover code, renders, and metadata.

Changes:

• Opened robots.txt for the read API while keeping /debug and /proxy excluded, and added a bot-friendly /llms-full.txt whole-catalog index endpoint.
• Fixed /specs/{id}/{library}/code to resolve language from the library registry when the query parameter is omitted (while preserving explicit ?language= behavior).
• Unified API version identity across /health and openapi.json, added explicit servers to the OpenAPI surface, and added middleware so HEAD behaves like GET (without adding HEAD ops to OpenAPI). Also expanded SEO/bot meta + JSON-LD to prefer real renders.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/api/test_seo_helpers.py	Adds/updates unit coverage for robots meta, hub render selection/fallback, JSON-LD enrichment, and llms-full builder.
tests/unit/api/test_routers.py	Updates router tests for robots policy, llms-full endpoint behavior, language resolution for code endpoint, and HEAD semantics.
tests/unit/api/test_main.py	Aligns version/title/description/openapi server assertions with the new version source + registry-derived description.
docs/reference/seo.md	Updates SEO reference to reflect new og-image defaults, branded OG endpoint shape, and robots policy text.
docs/reference/api.md	Documents language resolution behavior for code endpoint and adds /llms-full.txt documentation.
core/constants.py	Adds LIBRARY_LANGUAGES registry map to resolve language from a library id.
CHANGELOG.md	Adds unreleased entries covering llms-full, robots opening, language resolution fix, HEAD middleware, structured data upgrades, and version unification.
api/version.py	Introduces a single source of truth (APP_VERSION) derived from installed package metadata.
api/routers/specs.py	Resolves language from LIBRARY_LANGUAGES when absent for /specs/{id}/{library}/code.
api/routers/seo.py	Enhances bot HTML meta tags, hub page render selection, implementation JSON-LD (ImageObject/license/etc.), and adds llms-full builder + endpoint.
api/routers/health.py	Switches / and /health to report APP_VERSION instead of a hardcoded version.
api/main.py	Unifies FastAPI identity (version/description/servers) and adds HeadAsGetMiddleware while preserving MCP trailing-slash behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

api/routers/seo.py:596

• _spec_keywords() currently iterates (spec.tags or {}).values(), but Spec.tags is stored as JSONB (key order is not guaranteed). That makes the JSON-LD keywords list order dependent on DB serialization rather than a deliberate, stable ordering. Consider iterating a defined tag-key order (and optionally appending unknown keys) and deduplicating to keep the structured-data output deterministic.

def _spec_keywords(spec) -> list[str]:
    """Flatten the spec's tag bag ({plot_type, data_type, domain, features}) into a keyword list."""
    keywords: list[str] = []
    for values in (spec.tags or {}).values():
        if isinstance(values, str):

[codecov]

Codecov Report

❌ Patch coverage is 92.13483% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
api/routers/seo.py	91.80%	5 Missing ⚠️
api/version.py	60.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.