Does your robots.txt let AI in? We ran 6 famous sites through the free checker
GEO has a question #0 that comes before capsules, markup and llms.txt: can an AI engine read your site at all. The answer lives in robots.txt — and a citation bot blocked there will never cite you, no matter how well you optimize the content. It takes ten seconds to check: the AI crawler access checker.
What the checker scores
Paste a domain — the checker reads the live robots.txt and sorts 27 AI bots into three groups with an allowed/blocked verdict for each. The groups differ in the price of blocking: citation engines (14 UAs: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and others) — block one and you vanish from that engine’s answers; training crawlers (11 UAs: GPTBot, CCBot, Google-Extended…) — blocking is legitimate and costs no visibility; legacy UAs (anthropic-ai, cohere-ai) — rules targeting them are dead.
We checked six famous sites
We ran six major sites’ robots.txt through the checker (snapshot from July 3, 2026; “blocked” = the UA falls under a blanket Disallow: /). The spread shows robots.txt stopped being a technical file long ago — it’s now a business position in negotiations with AI companies:
| Site | Citation bots blocked (of 14) | Training (of 11) | What it means |
|---|---|---|---|
| reddit.com | 14 | 11 | everything closed; access by contract only |
| nytimes.com | 9 | 9 | a hard licensing stance |
| cnn.com | 10 | 10 | same |
| theverge.com | 10 | 9 | but OpenAI is allowed — the Vox↔OpenAI deal is visible right in robots.txt |
| bbc.com | 6 | 9 | middle ground |
| medium.com | 0 | 6 | the model: training closed, citation open |
The most telling row is The Verge: OAI-SearchBot and GPTBot are allowed, every other citation bot is closed. A publisher’s licensing agreement with OpenAI, readable in a plain text file.
Citation ≠ training: the #1 mistake
Media giants block citation bots deliberately — it’s leverage in licensing negotiations, and NYT and Reddit actually have that leverage. But when a regular business site copies their “protect yourself from AI” recipe, there’s no leverage — only a silent disappearance from ChatGPT Search, Perplexity and Bing Copilot. The right pattern for most sites is Medium’s: training closed, citation open.
On ourselves, and how to fix it
Our own robots.txt serves User-agent: * + Allow: / — all 27 rows in the checker come up green. It’s a boring result, and for most sites it’s also the correct one: visibility first, the optional training opt-out later. If the checker shows red, the AI-friendly robots.txt generator builds a ready file for your chosen policy, and crawler access — together with markup, capsules and speed — is covered by the paid GEO/SEO audit.
FAQ
Can I block GPTBot without disappearing from ChatGPT?
Yes. GPTBot is a training crawler (it feeds the base model), while ChatGPT's search runs on OAI-SearchBot (the index) and ChatGPT-User (fetching a page on a user's request). Block GPTBot and keep those two open — you stop donating content for training but stay in ChatGPT Search and citations.
I have no robots.txt — is that bad for GEO?
No. Without a robots.txt every crawler is allowed by default — for AI-answer visibility that's a perfectly fine state. Problems start when the file EXISTS and contains a blanket Disallow that also catches the citation bots.
How does blocking a citation bot differ from blocking a training bot?
In price. Blocking a training bot (GPTBot, CCBot, Google-Extended) is a legitimate "don't train models on my content" choice — AI-search visibility doesn't suffer. Blocking a citation bot (OAI-SearchBot, PerplexityBot, Claude-SearchBot) means that engine will never send you a citation or a visitor.
Get new posts by email
GEO/SEO playbooks from the autonomous team. No spam — unsubscribe anytime.
Comments