# AI-friendly robots.txt

_Source: https://seomatrix.ai/tools/robots-generator/_

---

## What it does

A robots.txt is a plain-text file at the root of your domain that tells each crawler — search engines and AI answer engines alike — which URLs it may fetch. This generator builds one that keeps the crawlers you want (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot) fully allowed so your pages can be indexed and quoted in AI answers, while optionally blocking the AI training bots you don't want. Get the rules wrong and you can silently vanish from ChatGPT, Perplexity, and Google AI Overviews.

## What it checks

- **Search-engine user-agents** — Named rules for Googlebot and Bingbot; allowing them is what lets your pages enter the indexes that both classic search and Google AI Overviews draw from.
- **AI answer-engine crawlers** — Bots like OAI-SearchBot (ChatGPT search), PerplexityBot, and Claude-SearchBot fetch live pages to cite in answers, so blocking them removes you from those citations entirely.
- **AI training crawlers** — GPTBot, CCBot, and Google-Extended collect text to train models, and you can block these independently without losing search or citation visibility.
- **Citation vs. training split** — Answer crawlers and training crawlers are separate user-agents, so a correct robots.txt can allow citation while denying model training — the key move most sites miss.
- **Sitemap directive** — A Sitemap: line pointing to your XML sitemap helps every crawler discover all your URLs in one pass instead of relying on link-following.
- **Disallow paths** — Explicit Disallow rules keep admin, cart, and duplicate URLs out of crawlers' reach so they spend their crawl budget on the pages you actually want cited.

## How to use it

1. Choose which crawlers to allow and whether to block AI training bots, then generate the file.
2. Copy or download the generated robots.txt and upload it to your domain root at /robots.txt.
3. Verify it loads at yourdomain.com/robots.txt and re-check after any CMS or host migration.

## FAQ

### Does blocking GPTBot stop ChatGPT from citing my site?

No. GPTBot is OpenAI's training crawler, while ChatGPT's live citations come from OAI-SearchBot. Block GPTBot and allow OAI-SearchBot to keep your citations while opting out of training.

### Where does robots.txt go?

At the root of your domain, reachable at https://yourdomain.com/robots.txt. It only governs the host it sits on, so each subdomain needs its own file.

### Will robots.txt hide a page from search results?

Not reliably. Disallow stops crawling, but a page can still be indexed if other sites link to it; use a noindex meta tag or header to keep a page out of results.

### Should I block AI crawlers at all?

Only training bots, and only if you don't want your content used for model training. Blocking answer-engine crawlers like PerplexityBot removes you from AI answers, which usually costs more visibility than it protects.