How we built our docs for AI agents

Making docs AI readable means shipping a second surface next to the rendered site, built for a reader that does not run your JavaScript. In practice that is six things: an llms.txt index that maps every page, the full corpus in one file at llms-full.txt, a raw markdown twin of every page advertised with a rel="alternate" link, a one sentence description on every page that retrieval can key off, JSON-LD saying what kind of document each page is, and a robots.txt that welcomes AI crawlers on purpose rather than by accident. All six are live on docs.fixter.dev, and more than one of them took a second pass to get right. Here is what each piece does, and the less obvious half: how you keep that surface honest when nobody owns docs full time.

Who actually reads your docs now

Your docs' fastest growing reader does not render CSS. It fetches a URL, reads what comes back, and answers somebody's question with it. That reader is Claude Code, Cursor, ChatGPT, or a Google AI Overview, and for a developer tool it is increasingly the first reader rather than the last. Our buying moment is somebody typing "what should I use to monitor this" into an assistant and getting three names back.

That reader has three properties worth designing for.

It retrieves pages in isolation. An agent that pulls docs.fixter.dev/using-fixter/alerting/ has not read your getting started page and does not know your vocabulary. Anything load bearing that lives in a sidebar, a previous page, or a diagram is gone.

It does not run your JavaScript. Plenty of docs sites render fine in a browser and return a shell to a plain fetch. If the content only exists after hydration, the agent quotes nothing, or it quotes the shell.

It quotes what it can parse cheaply. Given the choice between reconstructing prose from nested div soup and reading markdown, markdown wins, and the answer shown to your prospective customer is built from whichever one it read.

None of this is exotic. It is the old argument for semantic HTML, with a reader that finally has economic weight behind it.

The machine surface we shipped

Our docs are built on Astro and Starlight, and the site is deliberately small. That is the point: this is not a docs team's budget.

An llms.txt that is a real map. llms.txt is a proposal from Jeremy Howard, published in September 2024: a markdown file at the root of your site with an H1 for the project, a blockquote summary, and H2 sections listing links with a one line description each.¹ It is the one path an agent guesses without being told, so it should be worth guessing.

We started with the starlight-llms-txt plugin, then took the route back. The plugin's llms.txt body is a fixed blurb advertising the other text variants and never lists a single page, which is the opposite of a map. So src/pages/llms.txt.ts is a hand written Astro route that reads the content collection, walks the four sidebar sections in order, and emits one line per page:

Absolute URL, real description, every page. Astro gives src/pages priority over plugin injected routes, so ours wins at /llms.txt.

One fetch for the whole corpus. The plugin still generates llms-full.txt, every page concatenated, about 99 KB. A build hook then does two things to it. It deletes llms-small.txt, which is not abridged at all but the same corpus with collapsed whitespace, so it saves no context and only costs the reader a decision. And it rewrites root relative links to absolute ones, because a file meant to be read detached from its origin turns ](/setup/mcp/) into a dead string.

A markdown twin of every page. src/pages/[...slug].md.ts serves /any/page.md for every page in the collection: the title as an H1, the description as a blockquote, then the raw markdown body. No YAML frontmatter, because frontmatter is metadata for our build, not content for the reader. For MDX pages it strips the import lines and keeps the component tags, since the tags carry meaning: <SlackMessage from="Fixter"> tells the reader who is speaking. Append .md to any page URL to see one, for example /using-fixter/alerting.md.

Two things are easy to get wrong. The twin has to be advertised or nobody finds it, so a Starlight head override emits <link rel="alternate" type="text/markdown" href="..."> with an absolute href on every real page. And the content type has to survive deploy: we serve from S3, which ignores build time response headers, so the deploy syncs *.md a second time with an explicit --content-type "text/markdown; charset=utf-8". Skip that and your markdown arrives as application/octet-stream.

There is no visible "view as markdown" link, deliberately: crawlers do not click. Machines get the twin from rel="alternate" and llms.txt, humans get a "Copy for LLM" button that fetches the same file to the clipboard.

Descriptions as retrieval bait. Every page carries a one sentence description, and it does more work than any other field: it is the line in llms.txt that decides whether an agent fetches the page, the description in the JSON-LD, the blockquote atop the markdown twin, and the meta description. Write it badly once and you have degraded four surfaces. Our linter rejects a missing one with the reason attached: every page needs a one-sentence summary (AI retrieval depends on it).

JSON-LD that names the document type. Each page emits TechArticle JSON-LD² with headline, description, and url, plus dateModified from the page's lastReviewed frontmatter and keywords where present. The 404 page emits neither the JSON-LD nor the markdown alternate, which sounds obvious until you find a site advertising a markdown twin of its own error page.

A robots.txt that is strategy, not oversight. Most robots.txt files are inherited. Ours is a decision and says so at the top: everyone is welcome, including AI crawlers. After User-agent: * and Allow: / it names the major AI crawlers explicitly and allows each one, because some check for their own name: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, CCBot, and meta-externalagent.³⁴ Then the sitemap, and a comment pointing at llms.txt and llms-full.txt, since robots.txt has no field for those. If you sell to developers who ask an assistant what to use, blocking AI crawlers from your product docs is blocking your own top of funnel.

A surface is worthless if it lies

Everything above is the easy half. A machine surface that serves stale docs faster and in more formats is a downgrade. Stale docs used to embarrass you in front of one reader who noticed a flag had been renamed. Now they get read once, compressed into an answer, and repeated to people who never visit your site and have no way to check. Wrong docs scale exactly as well as right ones.

So the harder half is a freshness system. The shape we landed on has three parts, and it is worth describing as a pattern.

Facts live outside the pages. A grounding layer holds product truth and the decisions behind it, and the rule is that the docs must be regenerable from it. That inverts the correction path: a fix lands in the grounding layer first and in the page second, because a fix that only lands in the page is lost the next time that page is regenerated. Same instinct as fixing the migration rather than the row.

Machines own the mechanical facts. A manifest names every repo in the platform and, per service, the machine readable files it publishes: config properties, an OpenAPI document, a tool schema. A sync step pulls those at a pinned commit, records provenance, and renders reference pages deterministically. A check mode regenerates each page from the committed artifact and diffs it against the tree, failing on a missing page, a page somebody hand edited, or an unexpected file in the generated area. It runs in CI on every push, so hand editing a generated page is a build failure rather than a review conversation.

Honest status, since this is a post about receipts: the pipeline and its CI gate are built and tested, but every source in the manifest is still marked internal, so nothing publishes through it yet. The gate is done, the emitters are not.

AI drafts, humans review, and one step is adversarial. Four skills carry the loop. One runs the deterministic sync. One does read only discovery: it collects what changed in each repo since a committed baseline, judges the diffs for doc impact, and writes a report without touching a page. One turns findings into drafts on a branch for a person to review. The fourth checks pages against live code, and it is the step most people skip.

That last step is written to be hostile. It looks for lies, not style. Every claim is classified true, stale, wrong, or unverifiable, and each verdict has to be earned against the repo: commands confirmed by grepping the checkout at the pinned branch, environment variables checked against the extracted config, behavior claims required to cite a file and a line. It reports and does not edit. And it may only recommend bumping a page's lastReviewed date for pages that came out fully true, because a review date on a page you found problems in is worse than no date at all.

The division of labor is the point. Machines own what is mechanically derivable, AI narrows the search space, a human signs off. The same shape applies to any agent you point at your own systems, production telemetry included.

Tests on the docs, style as a build failure

Docs are a build artifact, so they get tests.

npm test runs Vitest, and the suite starts with a real astro build, then asserts against dist/. The suite is over a hundred test cases, about a third of them against built output, and they are not smoke tests. llms.txt has to contain exactly one entry per file in the content collection, a count derived at test time rather than hardcoded. llms-full.txt has to contain zero relative links. llms-small.txt has to not exist. The twins have to be markdown and not HTML, MDX imports stripped while component tags survive, the rel="alternate" href absolute, and TechArticle present on a real page and absent on the 404.

Every one of those is a regression a human review waves through. A plugin upgrade quietly reclaiming the /llms.txt route is invisible in a diff and obvious to an assertion.

Style is enforced the same way. npm run lint:docs checks the frontmatter contract on every page: title, description, owner, and lastReviewed are required, and lastReviewed has to be a strict YYYY-MM-DD that is a real calendar date. It also fails the build on an em dash anywhere in a description or a body, because house style forbids them and a style rule that lives in review comments erodes. CI runs astro check, the doc lint, the drift check, and the test suite, in that order, on every push and before every deploy.

If you are doing this for your own docs

In priority order, because the effort is wildly uneven.

Descriptions first. One sentence per page saying what it covers, written for somebody who has read nothing else on your site. Highest return on the list, no build changes needed, and it feeds retrieval, the index, the twin, and the meta description at once.

Self-contained pages, close behind. Assume every page is retrieved alone. Open by saying what it covers, and do not lean on vocabulary or diagrams the reader met on another page.

llms.txt second, and make it a map. Generate it from your content source so it cannot drift: every page, absolute URLs, the descriptions you already wrote. If your framework's plugin hands you a stub, take the route back.

Markdown twins third. More effort, and worth it if your docs lean on components or your site renders client side. Advertise them with rel="alternate" type="text/markdown" and an absolute href, and check the content type survives your deploy.

robots.txt always, and on purpose. Five minutes. Decide whether you want AI crawlers in your docs, then write a file that says so and point at your sitemap. The failure mode is not a wrong decision, it is an inherited default nobody read.

Then test the built output, not the source. Every piece above is a file a framework upgrade can silently stop producing.

None of this is an SEO trick, and the FAQ below covers why. It is a second reader with different constraints, served properly. Same reason Fixter exposes everything over MCP, and why the Claude Code plugin wires your instrumentation rather than asking you to read a setup page and do it by hand. The reader changed. The interface should follow.

Frequently asked questions

What is llms.txt?

llms.txt is a markdown file at the root of a site that gives AI readers a map of its content. The proposal came from Jeremy Howard in September 2024: an H1 with the project name, a blockquote summarizing it, then H2 sections listing links with a one line description each. It is an index, not a permissions file, so unlike robots.txt it does not tell crawlers what they may read. Many sites also publish llms-full.txt, the whole corpus in a single fetch.

Do AI crawlers respect robots.txt?

Mostly, but it is voluntary and always has been. Google's own documentation is blunt: the instructions in a robots.txt file cannot enforce crawler behavior, and it is up to the crawler to obey them. In August 2025 Cloudflare reported that Perplexity was reaching content on sites that had disallowed it, using undeclared crawlers with rotating user agents, and delisted it as a verified bot over it. So treat robots.txt as a clear signal to well behaved crawlers, and use network level controls if you need enforcement.

Should I block AI crawlers from my docs?

For product documentation, almost certainly not. The strongest case for blocking is economic: Cloudflare measured Anthropic's crawler making roughly 71,000 page requests per referral it sent back during one week in June 2025, a bad trade if your business sells ads against pageviews. Documentation is not that business, and if your buyers ask an assistant what to use, being in that answer is the whole point. Blocking still makes sense for paywalled material and anything you do not want reproduced.

How do I make my docs AI readable?

Start with a one sentence description on every page, and pages that stand alone when retrieved in isolation. Then make sure the content exists in the HTML without JavaScript, publish an llms.txt index generated from your content source, and serve a raw markdown twin of each page advertised with a rel="alternate" link. Add JSON-LD naming the document type and a robots.txt that deliberately allows the crawlers you want. Descriptions and self-contained pages carry most of the value.

Does llms.txt help SEO?

Not for Google. Google's guidance on optimizing for its generative AI features says directly that you do not need to create new machine readable files, AI text files, or markdown to appear in Google Search, and that Google Search ignores them. llms.txt is for a different reader: an agent that fetches URLs directly, where one file listing every page with a description really is cheaper than crawling a site. Publish it for that, not for rankings.