Repository navigation
feat: llms.txt, llms-full.txt and a Markdown twin of every page - #27
Merged
Merged
Conversation
Publishes the site for language models, following llmstxt.org: - /llms.txt: every page with a one-line summary, docs in reading order, then comparisons, release notes and an Optional section - /llms-full.txt: landing page, all docs and the comparisons in one file - /raw/<path>.md: each page on its own, with a Source line to the HTML - /keywords.md: the current keyword reference, converted from the committed libdoc JSON (lib/libdoc-markdown.ts) nuxt-llms owns the two llms routes; Nuxt Content's own integration is off because it prints %%browser%% unresolved, ignores the docs order and serves the styleguide demos. MDC components are rewritten into plain Markdown (tables, fenced terminal sessions, quotes) so no component tags or JSON props reach the output. Every page with a Markdown twin links it with <link rel="alternate" type="text/markdown">, and the landing page names llms.txt so crawlers reading only the HTML find it. The docs chapter list moves to app/utils/doc-sections.ts, shared by the docs shell and the exports.
Contributor
Previewhttps://rfbrowser-pr-27.surge.sh
Rebuilt on every push to this branch, and taken down when it closes. |
There was a problem hiding this comment.
🟡 Changes recommended
Exported Markdown contains unresolved table links and dead fragment links, while the integration-test skip condition can allow verification to pass when llms.txt is missing.
3 open issues
1 file not reviewed
pnpm-lock.yaml: Generated file
What changed in this PR
Adds prerendered Markdown exports to make the site's documentation and keyword reference accessible to language models, using existing content and libdoc data.
Changes:
- Generates
llms.txt,llms-full.txt, per-page Markdown, and the current keyword reference. - Converts documentation components to Markdown and shares chapter ordering with the docs sidebar.
- Adds discovery links, export checks, and contributor guidance.
| File | Description |
|---|---|
test/llms.spec.ts |
Tests keyword conversion and generated exports. |
server/utils/llms.ts |
Orders pages and converts documents to Markdown. |
server/utils/llms-components.ts |
Converts MDC components into Markdown. |
server/routes/raw/[...slug].md.get.ts |
Serves individual Markdown pages. |
server/routes/keywords.md.get.ts |
Serves the current keyword reference as Markdown. |
server/plugins/llms.ts |
Populates the index and combined documentation. |
pnpm-lock.yaml |
Locks the added dependencies. |
package.json |
Adds export and Markdown conversion dependencies. |
nuxt.config.ts |
Configures exports, server assets, and prerendering. |
lib/libdoc-markdown.ts |
Converts libdoc HTML and metadata to Markdown. |
CONTRIBUTING.md |
Documents exports and component conversion requirements. |
content/index.md |
Introduces the exports to readers and crawlers. |
app/utils/markdown-alternate.ts |
Builds Markdown alternate-link metadata. |
app/utils/doc-sections.ts |
Centralizes documentation chapter ordering. |
app/pages/why/[tool].vue |
Links comparison pages to Markdown twins. |
app/pages/releases/[version].vue |
Links release notes to Markdown twins. |
app/pages/index.vue |
Adds the landing page's Markdown alternate. |
app/pages/docs/[...slug].vue |
Uses shared chapter order and adds Markdown alternates. |
app/pages/community.vue |
Adds the community page's Markdown alternate. |
app/components/KeywordReference.vue |
Advertises Markdown for the current release. |
Review effort: Balanced
- Links inside doc-table cells bypassed the link rewrite, so `/docs/operations/node-process` stayed root-relative in the exports. - Fragment links (`#start`, `#ways`) pointed at anchors the Markdown does not keep, and in llms-full.txt into whichever document had them. Both now resolve against the page's HTML URL. - The export tests skipped when llms.txt was missing; they now skip only when the site was not built, and fail on any relative link.
- Links are made absolute in RawBlocks.add, so the notes under a comparison and any future component block are covered, not only doc-table cells. Code fences and spans are left alone. - Every href without a scheme resolves against the page URL (`./x`, `../x`, `x`, `?q` as well as `/x` and `#x`); titled links keep their title. The helpers move to server/utils/llms-links.ts, which imports nothing and is unit-tested. - The export tests read llms.txt only if it exists and fail on it explicitly, check keywords.md too, and catch relative Markdown links, reference definitions and href/src attributes. One prose() helper strips code for every check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Publishes the site for language models, following llmstxt.org. Every file is prerendered and generated from
content/and the libdoc JSON, so none of it needs maintaining by hand.Nuxt Content's own llms integration is switched off (
content.llms: false). It prints%%browser%%unresolved, lists the docs in database order and serves the styleguide demos.How agents and crawlers find these files:
<head> + <link rel="alternate" type="text/markdown" href="/raw/docs/concepts/selectors.md">content/index.md (Running in five minutes) Next: the getting-started guide, or the keyword reference. + Writing your tests with an AI assistant? … llms.txt … llms-full.txt … keywords.mdThe docs chapter list moves from the docs page into
app/utils/doc-sections.ts, so the sidebar and the exports share one order.Evidence
/llms.txt→ 404. Stringified as they are, components come out as tags with JSON in their attributes:```robotframeworkfences, and header-less libdoc tables become GFM tables.libdoc -f MARKDOWNwas considered and not used. It keeps the docs in Robot Framework markup ([url|text],= heading =,%TOC%), and the build would need Python for it.test/llms.spec.ts: unit tests for the libdoc conversion, plus these checks against the built output:pnpm generate✓ ·pnpm test: 881 passed ·pnpm check:bundle✓ (client JS unchanged: 418 KB reachable)Merge Danger
Door: two-way
Adds routes and files and changes no existing URL. A revert removes them cleanly.
Blast Radius: build
The new files are additions and don't change any existing page. The only visible change on the site is one paragraph on the landing page. The risk is in the build: about 50 more prerendered routes, and
failOnErrorstops the deploy if any of them throws. A new MDC component whose content lives in its props needs a case inserver/utils/llms-components.ts; otherwisetest/llms.spec.tsfails. This is documented inCONTRIBUTING.md.