# Does your robots.txt let AI in? We ran 6 famous sites through the free checker

> Before optimizing content for AI answers, answer question #0: can AI crawlers read your site at all. A breakdown of the free access checker (27 bots, 3 roles) and the real robots.txt of Reddit, NYT, The Verge and Medium.

_Source: https://seomatrix.ai/blog/ai-access-checker/ · Updated: 2026-07-06_

---

GEO has a question #0 that comes before capsules, markup and llms.txt: can an AI engine read your site at all. The answer lives in robots.txt — and a citation bot blocked there will never cite you, no matter how well you optimize the content. It takes ten seconds to check: the [**AI crawler access checker**](/tools/ai-access/).

## What the checker scores

Paste a domain — the checker reads the live robots.txt and sorts 27 AI bots into three groups with an allowed/blocked verdict for each. The groups differ in the price of blocking: **citation engines** (14 UAs: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and others) — block one and you [vanish from that engine's answers](/tools/geo-check/); **training crawlers** (11 UAs: GPTBot, CCBot, Google-Extended…) — blocking is legitimate and costs no visibility; **legacy UAs** (anthropic-ai, cohere-ai) — rules targeting them are dead.

## We checked six famous sites

We ran six major sites' robots.txt through the checker (snapshot from July 3, 2026; "blocked" = the UA falls under a blanket `Disallow: /`). The spread shows robots.txt stopped being a technical file long ago — it's now a business position in negotiations with AI companies:

| Site | Citation bots blocked (of 14) | Training (of 11) | What it means |
|---|---|---|---|
| reddit.com | 14 | 11 | everything closed; access by contract only |
| nytimes.com | 9 | 9 | a hard licensing stance |
| cnn.com | 10 | 10 | same |
| theverge.com | 10 | 9 | but OpenAI is allowed — the Vox↔OpenAI deal is visible right in robots.txt |
| bbc.com | 6 | 9 | middle ground |
| medium.com | **0** | 6 | the model: training closed, citation open |

The most telling row is The Verge: OAI-SearchBot and GPTBot are allowed, every other citation bot is closed. A publisher's licensing agreement with OpenAI, readable in a plain text file.

## Citation ≠ training: the #1 mistake

Media giants block citation bots deliberately — it's leverage in licensing negotiations, and NYT and Reddit actually have that leverage. But when a regular business site copies their "protect yourself from AI" recipe, there's no leverage — only a silent disappearance from ChatGPT Search, Perplexity and Bing Copilot. The right pattern for most sites is Medium's: training closed, citation open.

> The classic trap is a blanket block "for GPTBot and everything like it" that also catches OAI-SearchBot. The training already happened (the model is trained), and now you've dropped out of AI search too — the worst trade available.

## On ourselves, and how to fix it

Our own robots.txt serves `User-agent: *` + `Allow: /` — all 27 rows in the checker come up green. It's a boring result, and for most sites it's also the correct one: visibility first, the optional training opt-out later. If the checker shows red, the [AI-friendly robots.txt generator](/tools/robots-generator/) builds a ready file for your chosen policy, and crawler access — together with markup, capsules and speed — is covered by the [paid GEO/SEO audit](/audit/).

[**→ Check AI-bot access to your site**](/tools/ai-access/)

## FAQ

### Can I block GPTBot without disappearing from ChatGPT?

Yes. GPTBot is a training crawler (it feeds the base model), while ChatGPT's search runs on OAI-SearchBot (the index) and ChatGPT-User (fetching a page on a user's request). Block GPTBot and keep those two open — you stop donating content for training but stay in ChatGPT Search and citations.

### I have no robots.txt — is that bad for GEO?

No. Without a robots.txt every crawler is allowed by default — for AI-answer visibility that's a perfectly fine state. Problems start when the file EXISTS and contains a blanket Disallow that also catches the citation bots.

### How does blocking a citation bot differ from blocking a training bot?

In price. Blocking a training bot (GPTBot, CCBot, Google-Extended) is a legitimate "don't train models on my content" choice — AI-search visibility doesn't suffer. Blocking a citation bot (OAI-SearchBot, PerplexityBot, Claude-SearchBot) means that engine will never send you a citation or a visitor.