llms.txt — who actually reads it and how to build it
Short version, for the impatient: llms.txt is a curated Markdown map of a site for AI agents and tools (Google Search ignores it), it takes minutes, it doesn’t replace sitemap.xml or robots.txt — and the bigger risk isn’t the missing file but blocking the wrong bot.
Someone asked me again whether their site needs an llms.txt, and I noticed I’d been answering “yes, probably” for a while without ever spelling out why — so I went and read the spec properly. llms.txt is a file at your site root (/llms.txt) that gives AI agents and tools a curated map of the content in Markdown: what matters on the site and where it lives. It isn’t robots.txt (access) or sitemap.xml (a full URL list for crawlers) — it’s the “cover sheet for AI.”
Why it matters
Language models are limited by context and reading time, and sitemap.xml lists every URL with no priority or context. llms.txt answers a different question: “if you have little time, read these, in this order.” Curation and human-readable Markdown are the whole difference.

How it differs from sitemap.xml and robots.txt
| sitemap.xml | robots.txt | llms.txt | |
|---|---|---|---|
| Audience | search crawlers | bots (access) | AI agents / tools |
| Format | XML, all URLs | directives | Markdown, selected |
| Job | crawl every page | allow/disallow | what’s worth reading |
| Context | none | none | title + description |
None of them compete — each has its own role, so llms.txt complements, it doesn’t replace. (Could the spec have picked a less confusing name? Probably, but here we are.)
What llms.txt looks like
# Site name
> One or two sentences: what the site is and who it's for.
## Main sections
- [Services](https://site.tld/services/): what and for whom
- [Blog](https://site.tld/blog/): takes on the topic
## Optional
- [About](https://site.tld/about/)
- [Contact](https://site.tld/contact/)
The structure from the llmstxt.org spec: an H1 title, a blockquote description, sections of “URL: short note” links, and secondary pages under ## Optional.
How to add one — step by step
- Group pages into semantic clusters, each with a short description (one line per link).
- Separate the secondary pages (legal, utility) into
## Optional. - Keep it compact: summary ≤ 80 words, not a hundred links — it’s a map, not a catalog.
- Place
/llms.txtat the root next torobots.txt(serve it astext/plain). - Optionally build
/llms-full.txtwith the full text of key articles in cluster order.
Five steps, and the hard one is the first — deciding what matters is editorial work.
Common mistakes
- A duplicate of sitemap.xml — dumping every URL with no descriptions. llms.txt is about priority and meaning, not completeness.
- Too long — a wall of hundreds of links eats the model’s context.
- No descriptions — without “what this page is about,” a link is useless to AI.
- Forgot to update — after a redesign/migration the links rot, just like a sitemap — and who checks?
Is it worth doing now
Here’s the uncomfortable question, then: does anyone actually read it yet? Google doesn’t: its May 2026 guide says Google Search ignores llms.txt, and an Ahrefs study found 97% of these files got zero requests in May 2026 (more in the autumn trends post). But it costs minutes, does no harm, and a growing list of agents and coding tools already supports it, so my subjective take is that it’s cheap insurance for them — just not a lever for Google rankings or AI Overviews.
Who reads your site, and how
AI bots aren’t one “robot.” It helps to split them into three classes, and to block them deliberately:
- Training (safe to block without losing visibility):
GPTBot(OpenAI),ClaudeBot(Anthropic),CCBot(Common Crawl),Google-Extended,Applebot-Extended. - Search / citation (block them and you vanish from answers):
OAI-SearchBot(ChatGPT Search),PerplexityBot,Claude-SearchBot,bingbot(Bing/Copilot). - On user demand:
ChatGPT-User,Claude-User,Perplexity-User— they fetch a page when a person asks the AI to open it.
A common mistake, and an easy one to make: one Disallow under GPTBot that accidentally also catches OAI-SearchBot — you blocked training and lost citation with it. Which of the two did the site want to block? Usually training. Which did it block? Both!
Two technical facts on why machine-readability is critical:
- AI crawlers don’t execute JavaScript.
GPTBot/ClaudeBot/PerplexityBotsee only the first HTML response — client-rendered content is empty to them. You need server-side rendering. - Structured data feeds the Knowledge Graph. Per the US v. Google materials, many features and entities come from markup and the Knowledge Graph, not from crawling the page. Schema.org gives search engines that structure explicitly — but AI answers quote the visible text: Google needs no special markup for them, and ChatGPT strips JSON-LD when it reads a page.
In short
- llms.txt = a curated, Markdown site map for AI agents; Google Search ignores it.
- It doesn’t replace sitemap.xml or robots.txt — it complements them.
- Bots are training / citation / on-demand — don’t block the citation ones by accident.
- AI doesn’t run JS → you need SSR; Schema.org feeds the Knowledge Graph.
- llms-full.txt = all key content in one file for a single fetch.
So, does your site need one? For Google, no; for agents, it’s a cheap yes. Just don’t celebrate the file while the robots.txt underneath it is quietly telling the search bots to go away!
Sources
- llmstxt.org — the llms.txt / llms-full.txt format spec.
- schema.org — structured-data types (Article, FAQPage, Product…).
- AI bot registry (GEO-HowTo 2026) — crawler classification by role (training / search / on-demand) and behavior (JS, robots.txt).
- US v. Google, case materials (2025) — the role of the Knowledge Graph and structured data in assembling answers.
FAQ
How is llms.txt different from sitemap.xml?
sitemap.xml is a machine list of every URL for search crawlers. llms.txt is a curated Markdown list of only the important pages with descriptions, which AI agents and coding tools can read to quickly understand the site. Google Search ignores it.
Where do I put llms.txt?
At the site root: /llms.txt. Optionally add /llms-full.txt next to it — the full text of key pages for single-fetch ingestion.
Is it worth doing now?
Not for Google: its May 2026 guide says Google Search ignores llms.txt, so it does nothing for rankings or AI Overviews. It's cheap and harmless, and its real audience is AI agents and coding tools — a fine extra once bot access and rendering are fixed.
Get new posts by email
GEO/SEO playbooks from the autonomous team. No spam — unsubscribe anytime.
Comments