# You 'protected' yourself from AI bots — and silently vanished from ChatGPT Search

> AI crawlers play 4 different roles, and blocking each has its own price. The 2026 registry of 19 bots, the #1 robots.txt anti-pattern, and the correct configuration.

_Source: https://seomatrix.ai/blog/ai-bots-robots/ · Updated: 2026-07-09_

---

## Four roles — four different prices

"Blocking AI bots" isn't one decision — it's four. The 2026 AI crawlers play four functional roles, and blocking each costs something different:

| Role | What they do | Consequence of blocking |
|---|---|---|
| **Training** | train the models | **safe to block** — no visibility impact |
| **Search** | index for AI search | you vanish from ChatGPT Search, Claude Search, Bing Copilot |
| **On-demand** | fetch a page on a user's request | your specific page can't be cited |
| **Rogue** | declare robots.txt, don't honor it | Nginx/WAF only |

The key names (from the 19-bot registry, ISPOLIN methodology):

- **Training**: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Applebot-Extended, YandexAdditional.
- **Search**: OAI-SearchBot (the ChatGPT Search index), Claude-SearchBot, PerplexityBot, bingbot (Bing + Copilot).
- **On-demand**: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User.
- **Rogue**: Bytespider (ByteDance) — more below.

## The anti-pattern: a silent disappearance

```
User-agent: *
Disallow: /
```

The catch-all rule also covers the search AI bots — and the site **silently** vanishes from ChatGPT Search, Claude Search and Bing Copilot. The treachery is in "silently": classic results keep working, Google traffic doesn't drop — but you're already gone from AI answers, and you'll notice only when competitors start collecting your citations.

> **Check this right now** — If your robots.txt has a broad Disallow with a few targeted Allows, [check whether it also covers OAI-SearchBot and Claude-SearchBot](/tools/ai-access/). It's the most common finding of our tech audit: blocking training, sites accidentally block search too.

## The correct configuration

The strategy "don't train other people's models, but stay in AI search" is expressed with explicit rules:

```
# Training — block (no visibility impact)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Search — allow explicitly
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# On-demand — allow (this IS citation)
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: *
Allow: /
```

Business goals mapped to rules:

| Goal | Move |
|---|---|
| Don't train LLMs for free | block GPTBot, ClaudeBot, CCBot, Applebot-Extended |
| Be in ChatGPT Search | allow OAI-SearchBot + ChatGPT-User |
| Be in Claude Search | allow Claude-SearchBot + Claude-User |
| Be in Perplexity | allow PerplexityBot + Perplexity-User |
| Stop ByteDance | WAF, not robots.txt |

## Bytespider: when robots.txt doesn't work

Bytespider (ByteDance) publicly declares robots.txt compliance — yet independent observations have recorded bypasses repeatedly. If you need to stop it, robots.txt is useless: block by User-Agent or ByteDance IP ranges in Nginx or a Cloudflare WAF rule.

## And remember: these bots don't run JavaScript

GPTBot, ClaudeBot and PerplexityBot read **only the first HTTP response**. If less than half your content is visible without JavaScript, your site is empty to AI — whatever rules you write. The full breakdown and the fix: [why AI can't see your SPA](/blog/geo-for-spa/).

## How to check your own site

- **tech-audit** — checks robots.txt for the AI patterns: a citation bot accidentally covered by a broad Disallow; deprecated UAs whose rules are dead.
- **ai-crawler-log** — reads the access log and shows which AI bots **actually came**, to which pages, and what they were served (200/403/404) — including blocked ones that keep knocking.

> **Check it in 2 minutes** — Run your site through the [free check](https://seomatrix.ai/audit) — the AI-crawler block shows who you allow, who you block, and whether a catch-all Disallow swallowed the search bots.

## What to do

1. **Inventory your robots.txt** across the four roles — every rule should be deliberate.
2. **Separate training from search**: blocking GPTBot ≠ blocking OAI-SearchBot.
3. **Bytespider — WAF only.**
4. **Check the logs**: robots.txt is a declaration; the access log is reality.

## Sources

- **GEO-HowTo 2026 — Dmitry Ivanov's talk (ISPOLIN)** — the 19-bot registry with roles (training / search / on-demand / rogue), the catch-all-Disallow anti-pattern, Bytespider bypasses, the no-JS visibility test.
- **Crawler documentation**: [OpenAI bots](https://platform.openai.com/docs/bots) (GPTBot / OAI-SearchBot / ChatGPT-User), [Anthropic crawlers](https://docs.anthropic.com) (ClaudeBot / Claude-SearchBot / Claude-User), [Perplexity crawlers](https://docs.perplexity.ai) (PerplexityBot / Perplexity-User).
- **[robotstxt.org](https://www.robotstxt.org)** — User-agent group and rule semantics.

## FAQ

### Can I block GPTBot without losing visibility in ChatGPT?

Yes. GPTBot is a training bot (it feeds the base model), while ChatGPT's search runs on OAI-SearchBot (the index) and ChatGPT-User (fetching a page on a user's request). Block GPTBot and keep those two open: you stop donating content for training but stay in ChatGPT Search and citations.

### Why is 'User-agent: * Disallow: /' a GEO catastrophe?

Because the catch-all rule also covers the search AI bots. The site silently disappears from ChatGPT Search, Claude Search and Bing Copilot — while classic search results look perfectly normal, so the loss can go unnoticed for months.

### What about bots that ignore robots.txt?

Bytespider (ByteDance) declares robots.txt compliance, but bypasses have been recorded repeatedly. Such bots can only be stopped at the server level: a User-Agent or ByteDance IP-range block in Nginx, or a Cloudflare WAF rule.