# llms.txt — why your site needs one and how to build it

> llms.txt is a site map for AI engines: a curated list of your important pages in Markdown. How it differs from sitemap.xml, and how to add llms.txt and…

_Source: https://seomatrix.ai/blog/llms-txt/ · Updated: 2026-06-18_

---

**llms.txt** is a file at your site root (`/llms.txt`) that gives AI engines a curated map of your content in Markdown: what matters on the site and where it lives. It isn't robots.txt (access) or sitemap.xml (a full URL list for crawlers) — it's the "cover sheet for AI."

## Why it matters

Language models are limited by context and reading time. `sitemap.xml` lists *every* URL with no priority or context. `llms.txt` answers a different question: "if you have little time, read these, in this order." Curation and human-readable Markdown are the key difference.

## How it differs from sitemap.xml and robots.txt

| | sitemap.xml | robots.txt | llms.txt |
|---|---|---|---|
| Audience | search crawlers | bots (access) | AI engines / agents |
| Format | XML, all URLs | directives | Markdown, selected |
| Job | crawl every page | allow/disallow | what's worth reading |
| Context | none | none | title + description |

None of them compete — each has its own role. llms.txt complements, it doesn't replace.

## What llms.txt looks like

```markdown
# Site name

> One or two sentences: what the site is and who it's for.

## Main sections
- [Services](https://site.tld/services/): what and for whom
- [Blog](https://site.tld/blog/): takes on the topic

## Optional
- [About](https://site.tld/about/)
- [Contact](https://site.tld/contact/)
```

The structure from the llmstxt.org spec: an H1 title, a blockquote description, sections of "URL: short note" links, and secondary pages under `## Optional`.

> **llms-full.txt** — A companion `llms-full.txt` is often placed alongside — a concatenation of the full text of key pages in one file. It's the "single fetch" convention: an AI agent grabs all the content it needs in one request instead of crawling page by page.

## How to add one — step by step

1. **Group** pages into semantic clusters, each with a short description (one line per link).
2. **Separate the secondary** pages (legal, utility) into `## Optional`.
3. **Keep it compact**: summary ≤ 80 words, not a hundred links — it's a map, not a catalog.
4. **Place** `/llms.txt` at the root next to `robots.txt` (serve it as `text/plain`).
5. **Optionally** build `/llms-full.txt` with the full text of key articles in cluster order.

## Common mistakes

- **A duplicate of sitemap.xml** — dumping every URL with no descriptions. llms.txt is about *priority and meaning*, not completeness.
- **Too long** — a wall of hundreds of links eats the model's context.
- **No descriptions** — without "what this page is about," a link is useless to AI.
- **Forgot to update** — after a redesign/migration the links rot, just like a sitemap.

## Is it worth doing now

It isn't a mandatory standard yet, and not every engine reads it. But it costs minutes, does no harm, and a growing list of tools already supports it. It's cheap insurance: when an engine decides to read llms.txt, you'll already have one — with the right priorities.

## Who reads your site, and how

AI bots aren't one "robot." It helps to split them into three classes, and to block them deliberately:

- **Training** (safe to block without losing visibility): `GPTBot` (OpenAI), `ClaudeBot` (Anthropic), `CCBot` (Common Crawl), `Google-Extended`, `Applebot-Extended`.
- **Search / citation** (block them and you vanish from answers): `OAI-SearchBot` (ChatGPT Search), `PerplexityBot`, `Claude-SearchBot`, `bingbot` (Bing/Copilot).
- **On user demand**: `ChatGPT-User`, `Claude-User`, `Perplexity-User` — they fetch a page when a person asks the AI to open it.

A common mistake: one `Disallow` under `GPTBot` that accidentally also catches `OAI-SearchBot` — you blocked training and lost citation with it.

Two technical facts on why machine-readability is critical:

- **AI crawlers don't execute JavaScript.** `GPTBot`/`ClaudeBot`/`PerplexityBot` [see only the first HTML response](/tools/ai-view/) — client-rendered content is empty to them. You need server-side rendering.
- **Structured data feeds the Knowledge Graph.** Per the *US v. Google* materials, many features and entities come from markup and the Knowledge Graph, not from crawling the page. Schema.org + llms.txt give the machine structure explicitly, instead of relying on retrieval to "guess."

## In short

- llms.txt = a curated, Markdown site map for AI.
- It doesn't replace sitemap.xml or robots.txt — it complements them.
- Bots are training / citation / on-demand — don't block the citation ones by accident.
- AI doesn't run JS → you need SSR; Schema.org feeds the Knowledge Graph.
- llms-full.txt = all key content in one file for a single fetch.

## Sources

- **[llmstxt.org](https://llmstxt.org)** — the llms.txt / llms-full.txt format spec.
- **[schema.org](https://schema.org)** — structured-data types (Article, FAQPage, Product…).
- **AI bot registry (GEO-HowTo 2026)** — crawler classification by role (training / search / on-demand) and behavior (JS, robots.txt).
- **US v. Google, case materials (2025)** — the role of the Knowledge Graph and structured data in assembling answers.

## FAQ

### How is llms.txt different from sitemap.xml?

sitemap.xml is a machine list of every URL for search crawlers. llms.txt is a curated Markdown list of only the important pages with descriptions, which AI engines read to quickly understand the site.

### Where do I put llms.txt?

At the site root: /llms.txt. Optionally add /llms-full.txt next to it — the full text of key pages for single-fetch ingestion.

### Is it worth doing now?

The standard is young and doesn't yet guarantee traffic, but it's cheap, harmless and prepares the site for AI ingestion — a reasonable bet on the future.