Skip to content

feat: llms.txt, llms-full.txt and a Markdown twin of every page - #27

Merged
Snooz82 merged 3 commits into
mainfrom
feat/llms-txt
Nov 2, 2026
Merged

Snooz82 merged 3 commits into
mainfrom
feat/llms-txt

Conversation

@Snooz82

@Snooz82 Snooz82 commented Nov 2, 2026

Copy link
Copy Markdown
Member

Summary

Publishes the site for language models, following llmstxt.org. Every file is prerendered and generated from content/ and the libdoc JSON, so none of it needs maintaining by hand.

 .output/public/
+├── llms.txt          # index: every page + summary, docs in reading order
+├── llms-full.txt     # landing page + all docs + comparisons, one file
+├── keywords.md       # current keyword reference, from libdoc JSON
+└── raw/
+    ├── index.md
+    ├── docs/<chapter>/<page>.md
+    ├── why/<tool>.md
+    └── releases/<version>.md
/llms.txt, /llms-full.txt   nuxt-llms (routes, title, notes in nuxt.config.ts)
  server/plugins/llms.ts      fills the sections, in DOC_SECTIONS order
/raw/**.md                  server/routes/raw/[...slug].md.get.ts
  server/utils/llms.ts        content doc → Markdown, %%tokens%% resolved
    llms-components.ts        MDC components → tables, fences, quotes, lists
/keywords.md                server/routes/keywords.md.get.ts
  lib/libdoc-markdown.ts      libdoc HTML → Markdown (turndown + GFM tables)

Nuxt Content's own llms integration is switched off (content.llms: false). It prints %%browser%% unresolved, lists the docs in database order and serves the styleguide demos.

How agents and crawlers find these files:

 <head>
+  <link rel="alternate" type="text/markdown" href="/raw/docs/concepts/selectors.md">
 content/index.md  (Running in five minutes)
   Next: the getting-started guide, or the keyword reference.
+  Writing your tests with an AI assistant? … llms.txt … llms-full.txt … keywords.md

The docs chapter list moves from the docs page into app/utils/doc-sections.ts, so the sidebar and the exports share one order.

Evidence

  • Before: /llms.txt → 404. Stringified as they are, components come out as tags with JSON in their attributes:
    <terminal-block :sessions="[{"shell":"bash","steps":[{"command":"pip install …
    
    After:
    ```bash
    $ pip install "robotframework-browser[bb]"
    Successfully installed robotframework-browser-21.0.0 …
    $ rfbrowser install
    ✓ chromium   downloaded
    ```
    
  • Keyword reference: every keyword lists its arguments with their documentation, named-only arguments are marked, examples are ```robotframework fences, and header-less libdoc tables become GFM tables. libdoc -f MARKDOWN was considered and not used. It keeps the docs in Robot Framework markup ([url|text], = heading =, %TOC%), and the build would need Python for it.
  • test/llms.spec.ts: unit tests for the libdoc conversion, plus these checks against the built output:
    every llms.txt link            → file exists in .output/public
    every content/docs/**/*.md     → listed in llms.txt
    every exported file            → no %%token%%, no component tag, no placeholder
    every text/markdown <link>     → file exists   (> 40 pages)
    landing page                   → links llms.txt
    no raw/gallery
    
    pnpm generate ✓ · pnpm test: 881 passed · pnpm check:bundle ✓ (client JS unchanged: 418 KB reachable)

Merge Danger

Door: two-way

Adds routes and files and changes no existing URL. A revert removes them cleanly.

Blast Radius: build

The new files are additions and don't change any existing page. The only visible change on the site is one paragraph on the landing page. The risk is in the build: about 50 more prerendered routes, and failOnError stops the deploy if any of them throws. A new MDC component whose content lives in its props needs a case in server/utils/llms-components.ts; otherwise test/llms.spec.ts fails. This is documented in CONTRIBUTING.md.

Publishes the site for language models, following llmstxt.org:

- /llms.txt: every page with a one-line summary, docs in reading order,
  then comparisons, release notes and an Optional section
- /llms-full.txt: landing page, all docs and the comparisons in one file
- /raw/<path>.md: each page on its own, with a Source line to the HTML
- /keywords.md: the current keyword reference, converted from the
  committed libdoc JSON (lib/libdoc-markdown.ts)

nuxt-llms owns the two llms routes; Nuxt Content's own integration is off
because it prints %%browser%% unresolved, ignores the docs order and
serves the styleguide demos. MDC components are rewritten into plain
Markdown (tables, fenced terminal sessions, quotes) so no component tags
or JSON props reach the output.

Every page with a Markdown twin links it with
<link rel="alternate" type="text/markdown">, and the landing page names
llms.txt so crawlers reading only the HTML find it. The docs chapter list
moves to app/utils/doc-sections.ts, shared by the docs shell and the
exports.
Copilot AI balanced review requested due to automatic review settings November 2, 2026 13:52
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Preview

https://rfbrowser-pr-27.surge.sh

Commit 4847bb8
Landing https://rfbrowser-pr-27.surge.sh/
Keywords https://rfbrowser-pr-27.surge.sh/keywords
Releases https://rfbrowser-pr-27.surge.sh/releases

Rebuilt on every push to this branch, and taken down when it closes.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Exported Markdown contains unresolved table links and dead fragment links, while the integration-test skip condition can allow verification to pass when llms.txt is missing.

3 open issues
1 file not reviewed
  • pnpm-lock.yaml: Generated file
What changed in this PR

Adds prerendered Markdown exports to make the site's documentation and keyword reference accessible to language models, using existing content and libdoc data.

Changes:

  • Generates llms.txt, llms-full.txt, per-page Markdown, and the current keyword reference.
  • Converts documentation components to Markdown and shares chapter ordering with the docs sidebar.
  • Adds discovery links, export checks, and contributor guidance.
File Description
test/​llms.spec.ts Tests keyword conversion and generated exports.
server/​utils/​llms.ts Orders pages and converts documents to Markdown.
server/​utils/​llms-components.ts Converts MDC components into Markdown.
server/​routes/​raw/​[...slug].md.get.ts Serves individual Markdown pages.
server/​routes/​keywords.md.get.ts Serves the current keyword reference as Markdown.
server/​plugins/​llms.ts Populates the index and combined documentation.
pnpm-lock.yaml Locks the added dependencies.
package.json Adds export and Markdown conversion dependencies.
nuxt.config.ts Configures exports, server assets, and prerendering.
lib/​libdoc-markdown.ts Converts libdoc HTML and metadata to Markdown.
CONTRIBUTING.md Documents exports and component conversion requirements.
content/​index.md Introduces the exports to readers and crawlers.
app/​utils/​markdown-alternate.ts Builds Markdown alternate-link metadata.
app/​utils/​doc-sections.ts Centralizes documentation chapter ordering.
app/​pages/​why/​[tool].vue Links comparison pages to Markdown twins.
app/​pages/​releases/​[version].vue Links release notes to Markdown twins.
app/​pages/​index.vue Adds the landing page's Markdown alternate.
app/​pages/​docs/​[...slug].vue Uses shared chapter order and adds Markdown alternates.
app/​pages/​community.vue Adds the community page's Markdown alternate.
app/​components/​KeywordReference.vue Advertises Markdown for the current release.

Review effort: Balanced

Comment thread server/utils/llms-components.ts Resolved
Comment thread server/utils/llms.ts Resolved
Comment thread test/llms.spec.ts Resolved
- Links inside doc-table cells bypassed the link rewrite, so
  `/docs/operations/node-process` stayed root-relative in the exports.
- Fragment links (`#start`, `#ways`) pointed at anchors the Markdown does
  not keep, and in llms-full.txt into whichever document had them. Both
  now resolve against the page's HTML URL.
- The export tests skipped when llms.txt was missing; they now skip only
  when the site was not built, and fail on any relative link.
- Links are made absolute in RawBlocks.add, so the notes under a
  comparison and any future component block are covered, not only
  doc-table cells. Code fences and spans are left alone.
- Every href without a scheme resolves against the page URL (`./x`,
  `../x`, `x`, `?q` as well as `/x` and `#x`); titled links keep their
  title. The helpers move to server/utils/llms-links.ts, which imports
  nothing and is unit-tested.
- The export tests read llms.txt only if it exists and fail on it
  explicitly, check keywords.md too, and catch relative Markdown links,
  reference definitions and href/src attributes. One prose() helper
  strips code for every check.
@Snooz82
Snooz82 merged commit e499560 into main Nov 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants