How is llms.txt different from sitemap.xml?

sitemap.xml is a machine list of every URL for search crawlers. llms.txt is a curated Markdown list of only the important pages with descriptions, which AI engines read to quickly understand the site.

Where do I put llms.txt?

At the site root: /llms.txt. Optionally add /llms-full.txt next to it — the full text of key pages for single-fetch ingestion.

llms.txt — why your site needs one and how to build it

Q: Is it worth doing now?

The standard is young and doesn't yet guarantee traffic, but it's cheap, harmless and prepares the site for AI ingestion — a reasonable bet on the future.

llms.txt is a file at your site root (/llms.txt) that gives AI engines a curated map of your content in Markdown: what matters on the site and where it lives. It isn’t robots.txt (access) or sitemap.xml (a full URL list for crawlers) — it’s the “cover sheet for AI.”

Why it matters

Language models are limited by context and reading time. sitemap.xml lists every URL with no priority or context. llms.txt answers a different question: “if you have little time, read these, in this order.” Curation and human-readable Markdown are the key difference.

How it differs from sitemap.xml and robots.txt

	sitemap.xml	robots.txt	llms.txt
Audience	search crawlers	bots (access)	AI engines / agents
Format	XML, all URLs	directives	Markdown, selected
Job	crawl every page	allow/disallow	what’s worth reading
Context	none	none	title + description

None of them compete — each has its own role. llms.txt complements, it doesn’t replace.

What llms.txt looks like

# Site name

> One or two sentences: what the site is and who it's for.

## Main sections
- [Services](https://site.tld/services/): what and for whom
- [Blog](https://site.tld/blog/): takes on the topic

## Optional
- [About](https://site.tld/about/)
- [Contact](https://site.tld/contact/)

The structure from the llmstxt.org spec: an H1 title, a blockquote description, sections of “URL: short note” links, and secondary pages under ## Optional.

How to add one — step by step

Group pages into semantic clusters, each with a short description (one line per link).
Separate the secondary pages (legal, utility) into ## Optional.
Keep it compact: summary ≤ 80 words, not a hundred links — it’s a map, not a catalog.
Place /llms.txt at the root next to robots.txt (serve it as text/plain).
Optionally build /llms-full.txt with the full text of key articles in cluster order.

Common mistakes

A duplicate of sitemap.xml — dumping every URL with no descriptions. llms.txt is about priority and meaning, not completeness.
Too long — a wall of hundreds of links eats the model’s context.
No descriptions — without “what this page is about,” a link is useless to AI.
Forgot to update — after a redesign/migration the links rot, just like a sitemap.

Is it worth doing now

It isn’t a mandatory standard yet, and not every engine reads it. But it costs minutes, does no harm, and a growing list of tools already supports it. It’s cheap insurance: when an engine decides to read llms.txt, you’ll already have one — with the right priorities.

Who reads your site, and how

AI bots aren’t one “robot.” It helps to split them into three classes, and to block them deliberately:

Training (safe to block without losing visibility): GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended, Applebot-Extended.
Search / citation (block them and you vanish from answers): OAI-SearchBot (ChatGPT Search), PerplexityBot, Claude-SearchBot, bingbot (Bing/Copilot).
On user demand: ChatGPT-User, Claude-User, Perplexity-User — they fetch a page when a person asks the AI to open it.

A common mistake: one Disallow under GPTBot that accidentally also catches OAI-SearchBot — you blocked training and lost citation with it.

Two technical facts on why machine-readability is critical:

AI crawlers don’t execute JavaScript. GPTBot/ClaudeBot/PerplexityBot see only the first HTML response — client-rendered content is empty to them. You need server-side rendering.
Structured data feeds the Knowledge Graph. Per the US v. Google materials, many features and entities come from markup and the Knowledge Graph, not from crawling the page. Schema.org + llms.txt give the machine structure explicitly, instead of relying on retrieval to “guess.”

In short

llms.txt = a curated, Markdown site map for AI.
It doesn’t replace sitemap.xml or robots.txt — it complements them.
Bots are training / citation / on-demand — don’t block the citation ones by accident.
AI doesn’t run JS → you need SSR; Schema.org feeds the Knowledge Graph.
llms-full.txt = all key content in one file for a single fetch.

Sources

llmstxt.org — the llms.txt / llms-full.txt format spec.
schema.org — structured-data types (Article, FAQPage, Product…).
AI bot registry (GEO-HowTo 2026) — crawler classification by role (training / search / on-demand) and behavior (JS, robots.txt).
US v. Google, case materials (2025) — the role of the Knowledge Graph and structured data in assembling answers.