You 'protected' yourself from AI bots — and silently vanished from ChatGPT Search

Four roles — four different prices

“Blocking AI bots” isn’t one decision — it’s four. The 2026 AI crawlers play four functional roles, and blocking each costs something different:

Role	What they do	Consequence of blocking
Training	train the models	safe to block — no visibility impact
Search	index for AI search	you vanish from ChatGPT Search, Claude Search, Bing Copilot
On-demand	fetch a page on a user’s request	your specific page can’t be cited
Rogue	declare robots.txt, don’t honor it	Nginx/WAF only

The key names (from the 19-bot registry, ISPOLIN methodology):

• Training: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Applebot-Extended, YandexAdditional.
• Search: OAI-SearchBot (the ChatGPT Search index), Claude-SearchBot, PerplexityBot, bingbot (Bing + Copilot).
• On-demand: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User.
• Rogue: Bytespider (ByteDance) — more below.

The anti-pattern: a silent disappearance

User-agent: *
Disallow: /

The catch-all rule also covers the search AI bots — and the site silently vanishes from ChatGPT Search, Claude Search and Bing Copilot. The treachery is in “silently”: classic results keep working, Google traffic doesn’t drop — but you’re already gone from AI answers, and you’ll notice only when competitors start collecting your citations.

The correct configuration

The strategy “don’t train other people’s models, but stay in AI search” is expressed with explicit rules:

# Training — block (no visibility impact)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Search — allow explicitly
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# On-demand — allow (this IS citation)
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: *
Allow: /

Business goals mapped to rules:

Goal	Move
Don’t train LLMs for free	block GPTBot, ClaudeBot, CCBot, Applebot-Extended
Be in ChatGPT Search	allow OAI-SearchBot + ChatGPT-User
Be in Claude Search	allow Claude-SearchBot + Claude-User
Be in Perplexity	allow PerplexityBot + Perplexity-User
Stop ByteDance	WAF, not robots.txt

Bytespider: when robots.txt doesn’t work

Bytespider (ByteDance) publicly declares robots.txt compliance — yet independent observations have recorded bypasses repeatedly. If you need to stop it, robots.txt is useless: block by User-Agent or ByteDance IP ranges in Nginx or a Cloudflare WAF rule.

And remember: these bots don’t run JavaScript

GPTBot, ClaudeBot and PerplexityBot read only the first HTTP response. If less than half your content is visible without JavaScript, your site is empty to AI — whatever rules you write. The full breakdown and the fix: why AI can’t see your SPA.

How to check your own site

• tech-audit — checks robots.txt for the AI patterns: a citation bot accidentally covered by a broad Disallow; deprecated UAs whose rules are dead.
• ai-crawler-log — reads the access log and shows which AI bots actually came, to which pages, and what they were served (200/403/404) — including blocked ones that keep knocking.

What to do

1. Inventory your robots.txt across the four roles — every rule should be deliberate.
2. Separate training from search: blocking GPTBot ≠ blocking OAI-SearchBot.
3. Bytespider — WAF only.
4. Check the logs: robots.txt is a declaration; the access log is reality.

Sources

• GEO-HowTo 2026 — Dmitry Ivanov’s talk (ISPOLIN) — the 19-bot registry with roles (training / search / on-demand / rogue), the catch-all-Disallow anti-pattern, Bytespider bypasses, the no-JS visibility test.
• Crawler documentation: OpenAI bots (GPTBot / OAI-SearchBot / ChatGPT-User), Anthropic crawlers (ClaudeBot / Claude-SearchBot / Claude-User), Perplexity crawlers (PerplexityBot / Perplexity-User).
• robotstxt.org — User-agent group and rule semantics.