llms.txt Explained: What It Is and Whether It Actually Does Anything

llms.txt is a proposed convention: a markdown file at the root of your domain that gives language models a curated, link-based map of your site's most useful content. It was proposed by Jeremy Howard of Answer.AI in September 2024, it has a clear and sensible specification, and it takes about twenty minutes to implement. It is also, as of now, not read by any major AI search engine that has publicly said so — and at least one of them has publicly said it doesn't. Both of those things are true at once, and an article that tells you only the first half is selling you something.

This piece covers the format precisely, the llms.txt versus llms-full.txt distinction, who is and is not consuming these files, a complete worked example for a SaaS site, and what to do with the rest of your afternoon once you've spent twenty minutes on it.

The problem it was proposed to solve

The stated motivation is context windows. When a language model needs to understand what your product does, the naive path is: fetch your HTML, strip navigation, cookie banners, marketing chrome, script tags and inlined CSS, and hope enough signal survives. That is expensive in tokens and lossy in practice. A modern marketing page can be 200KB of HTML wrapping 800 words of prose.

llms.txt proposes a shortcut. Rather than making the model reverse-engineer your information architecture from a sitemap of 4,000 URLs, you hand it a short curated index: here is what we are, here are the twelve pages that matter, here is what each one covers. The spec also suggests offering clean markdown versions of individual pages by appending .md to their URLs, so a model that follows a link from your index gets prose rather than a DOM.

It is a genuinely good idea. Whether good ideas get adopted is a separate question, and the honest answer for llms.txt today is: mostly not, and not by the parties that would matter most.

The exact format

llms.txt is markdown, served from /llms.txt at the domain root. The spec is deliberately strict about structure so it can be parsed programmatically, which is the part most published examples get wrong.

The required and optional elements, in order:

1. An H1 with the project or site name. This is the only required element, and there must be exactly one H1.
2. A blockquote immediately after, containing a short summary. Optional but strongly conventional.
3. Zero or more markdown sections of free-form detail — paragraphs, lists, whatever — with no headings. This is where you put context that isn't a link: what the product does, key terminology, hard constraints.
4. H2 sections, each containing a markdown bullet list of links in the form - [name](url): optional description. The H2 name groups the links (## Docs, ## API, ## Guides).
5. An optional H2 literally named ## Optional, whose links a consumer may skip when working with a tighter context budget. This is the one piece of semantics in the format: it is a hint about what to drop first, not just another section.

Rules worth internalising: only one H1; the H2 sections must contain link lists rather than prose; the free-form prose belongs above the first H2. If you deviate, a strict parser gets nothing useful, and a lenient consumer (an LLM reading it as plain text) would have been fine either way — which tells you something about how much the strictness currently buys you.

llms.txt versus llms-full.txt

These are different things and the distinction is frequently muddled.

llms-full.txt emerged from documentation platforms — Mintlify and similar tools auto-generate one — because for a docs site, the highest-value thing you can hand a coding assistant is simply all of the documentation in one paste-able file. It works well in that narrow case: a developer drops the URL into their IDE assistant and the whole API reference is in context.

For most sites llms-full.txt is a bad fit. If your content is marketing pages, a blog, and a pricing table, concatenating it all produces a large file nobody asked for. Generate it if you have real documentation. Skip it otherwise.

Who actually reads it — the honest accounting

This is the section that separates useful advice from cargo cult, so here it is bluntly.

Google has said it does not use it. John Mueller has publicly compared llms.txt to the keywords meta tag — a file site owners write about themselves that search systems have no reason to trust — and Google representatives have stated that their systems do not consume it. Google's AI Overviews are built on Google Search infrastructure crawled by Googlebot, which reads your HTML.

No major model provider has committed to crawling for it. OpenAI, Anthropic and Perplexity all document their crawler user agents and their robots.txt tokens. None of them documents llms.txt as an input. Absence of a public commitment isn't proof of non-use, but for a file whose entire value proposition is that consumers know to look for it, silence is close to dispositive.

Publishing one is not the same as consuming one. Several AI companies host an llms.txt for their own documentation. That means their docs platform generates it. It does not mean their crawler reads yours. These get conflated constantly in blog posts arguing adoption is widespread.

Independent log analyses have generally found little to no crawler traffic to /llms.txt from the major AI user agents. Treat these as directional rather than definitive — they are individual site samples, not a census — but multiple people looking at their own logs have reached the same conclusion, and nobody has published logs showing the opposite.

Where it genuinely is used today: humans and agents pointed at it deliberately. A developer pasting https://yoursite.com/llms.txt into Claude or ChatGPT. An IDE assistant configured to index a docs URL. An internal RAG pipeline whose author chose to prefer that file. This is real usage, and for a developer-facing product it is not trivial — but it is pull, not push. Nobody discovers your llms.txt because a crawler went looking for it.

So the accurate one-line summary: llms.txt is a proposal with a good design, meaningful use among developer tools and humans who know to ask for it, and no confirmed adoption by the answer engines most people are hoping to influence. It is not a standard in the sense that robots.txt is a standard. robots.txt has an IETF RFC and universal crawler support; llms.txt has a spec document and optimism.

Why add one anyway

Because the cost is close to zero and the failure mode is benign.

• It takes twenty minutes and it is a static file.
• If a coding agent, an IDE, or a curious developer does fetch it, they get an accurate summary of your product instead of guessing from your homepage hero copy.
• Writing it is a useful exercise in itself. Compressing your site into fifteen links and three sentences of positioning surfaces gaps quickly. If you cannot describe what a page is for in eight words, that is worth knowing.
• If adoption does arrive, you are already there, and the marginal cost of having been early is zero.

What it will not do: rank you, get you cited, or substitute for any of the work in how to get cited by AI search engines. Anyone telling you llms.txt is a ranking factor is describing a mechanism that does not currently exist.

A worked example for a SaaS site

# Acme Analytics

> Acme Analytics is a self-hosted product analytics platform for
> engineering teams. It ingests events over an HTTP API, stores them in
> ClickHouse, and exposes funnels, retention and cohort analysis.

Acme is deployed by the customer, not hosted by us. Pricing is per
ingested event, not per seat. The free tier covers 1M events per month.
Terminology note: we use "actor" where most tools say "user", because
events can be attributed to services as well as people.

## Docs

- [Quickstart](https://acme.example/docs/quickstart.md): Docker Compose setup, first event ingested in under ten minutes
- [Event schema](https://acme.example/docs/events.md): required and optional fields, reserved property names, type coercion rules
- [HTTP ingest API](https://acme.example/docs/api/ingest.md): endpoint, auth headers, batching limits, idempotency keys
- [Query API](https://acme.example/docs/api/query.md): SQL-over-HTTP interface and the funnel/retention helpers
- [Self-hosting guide](https://acme.example/docs/self-hosting.md): ClickHouse sizing, retention policies, backup strategy

## Product

- [Pricing](https://acme.example/pricing.md): per-event tiers, free tier limits, what counts as a billable event
- [Security](https://acme.example/security.md): data residency, SOC 2 status, PII handling and property redaction
- [Changelog](https://acme.example/changelog.md): released versions with breaking-change callouts

## Optional

- [Engineering blog](https://acme.example/blog.md): architecture write-ups, not required to use the product
- [Comparison pages](https://acme.example/compare.md): how Acme differs from hosted analytics tools

Points that make this example work rather than just fill space: the blockquote states what the thing is in one sentence; the prose section carries the two facts an LLM would otherwise get wrong (deployment model, pricing unit) plus a terminology note that prevents a whole class of confused answers; every link description says what is in the page rather than restating the title; and ## Optional genuinely holds the droppable material.

The .md suffixes assume you actually serve markdown at those URLs. If you don't, link to the HTML pages instead — a broken link is worse than an unfashionable one.

Serving it

On any static host, drop the file at the web root. On Next.js, public/llms.txt works and is the right default. If you want it generated from your content collection rather than hand-maintained, a route handler is straightforward — though be aware that hand-curation is most of the value here, so generating it from every page in your sitemap recreates the problem the file was meant to solve.

Two operational checks: make sure it returns 200 with a text content type (text/plain is what a .txt extension usually gets you, and it is fine), and make sure nothing in your robots.txt disallows it. That last one sounds absurd until you find the Disallow: /*.txt rule someone added years ago — the interaction between wildcard disallows and root files is exactly the kind of thing covered in robots.txt vs meta robots.

What actually matters more

If you have an hour for AI visibility work, llms.txt should be the last twenty minutes of it, not the first. Ahead of it, in order:

Crawler access. Verify that GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and friends are permitted in robots.txt and not being blocked at your CDN or WAF. Edge-level bot blocking silently defeats a permissive robots.txt, and several CDNs now block AI crawlers by default.

Server-rendered HTML. Assume AI crawlers do not run your JavaScript. If curl against a page returns an empty root div, no index file will help, because there is nothing to index.

Answer-first content. Retrieval operates on passages pulled out of context. A section whose first sentence states the answer is far more quotable than one that builds to it over four paragraphs.

Clean, honest markup. Correct titles and descriptions, a sane heading hierarchy, accurate structured data that matches the visible page. Every one of these helps classic search too, which is the strongest argument for doing them first.

None of that is glamorous, and all of it is confirmed to matter, which is exactly the inverse of llms.txt's current position. If you'd rather not audit that list by hand, SEO Fix Agent runs about thirty weighted checks across technical, meta, content and GEO categories on a page and turns the findings into a fix prompt you can paste into a coding agent. It won't tell you whether your llms.txt is working, because nobody can tell you that yet.

Add the file. Keep it current. Then go fix the things that demonstrably move the needle, and revisit this article in a year — the honest answer may well have changed by then.