free tool

AI-friendly robots.txt generator

Generate a robots.txt that lets search + AI answer engines crawl you (so you can be ranked AND cited) and optionally blocks AI training crawlers. The action half of the AI access checker.

Block AI training crawlers (keep citation & search bots allowed) Sitemap URL (optional) Disallow paths (one per line, optional) ASCII header style (comments are legal in robots.txt — have some fun) Brand / site name for the header Add a “made with seomatrix.ai” line

robots.txt

Save as /robots.txt at your site root. Citation bots (OAI-SearchBot, PerplexityBot, Googlebot…) stay allowed so AI engines can cite you. Validate this robots.txt →

What this tool checks

A robots.txt is a plain-text file at the root of your domain that tells each crawler — search engines and AI answer engines alike — which URLs it may fetch. This generator builds one that keeps the crawlers you want (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot) fully allowed so your pages can be indexed and quoted in AI answers, while optionally blocking the AI training bots you don't want. Get the rules wrong and you can silently vanish from ChatGPT, Perplexity, and Google AI Overviews.

What it looks at

Search-engine user-agents: Named rules for Googlebot and Bingbot; allowing them is what lets your pages enter the indexes that both classic search and Google AI Overviews draw from.
AI answer-engine crawlers: Bots like OAI-SearchBot (ChatGPT search), PerplexityBot, and Claude-SearchBot fetch live pages to cite in answers, so blocking them removes you from those citations entirely.
AI training crawlers: GPTBot, CCBot, and Google-Extended collect text to train models, and you can block these independently without losing search or citation visibility.
Citation vs. training split: Answer crawlers and training crawlers are separate user-agents, so a correct robots.txt can allow citation while denying model training — the key move most sites miss.
Sitemap directive: A Sitemap: line pointing to your XML sitemap helps every crawler discover all your URLs in one pass instead of relying on link-following.
Disallow paths: Explicit Disallow rules keep admin, cart, and duplicate URLs out of crawlers' reach so they spend their crawl budget on the pages you actually want cited.

How to use it

Choose which crawlers to allow and whether to block AI training bots, then generate the file.
Copy or download the generated robots.txt and upload it to your domain root at /robots.txt.
Verify it loads at yourdomain.com/robots.txt and re-check after any CMS or host migration.

Frequently asked questions

Does blocking GPTBot stop ChatGPT from citing my site?

No. GPTBot is OpenAI's training crawler, while ChatGPT's live citations come from OAI-SearchBot. Block GPTBot and allow OAI-SearchBot to keep your citations while opting out of training.

Where does robots.txt go?

At the root of your domain, reachable at https://yourdomain.com/robots.txt. It only governs the host it sits on, so each subdomain needs its own file.

Will robots.txt hide a page from search results?

Not reliably. Disallow stops crawling, but a page can still be indexed if other sites link to it; use a noindex meta tag or header to keep a page out of results.

Should I block AI crawlers at all?

Only training bots, and only if you don't want your content used for model training. Blocking answer-engine crawlers like PerplexityBot removes you from AI answers, which usually costs more visibility than it protects.

This audit is 1 of 85 operations in seo·matrix

Free tools show you WHAT to fix. The full audit + the autonomous team of 20 AI agents fix it end-to-end — strategy, content, GEO, internal links, publishing and day-2 upkeep.

Get the full audit → Track it weekly — free cabinet →

All free tools →