AI crawler access checker

What this tool checks

This checks whether your robots.txt allows or blocks the specific AI crawlers that power answer engines — GPTBot and OAI-SearchBot (OpenAI), PerplexityBot, ClaudeBot, Google-Extended, and others. If you Disallow a bot that gathers or retrieves your content for an AI engine, that engine can't crawl your pages and can never cite you in its answers. It separates model-training bots from the live search bots that actually build cited answers.

What it looks at

user-agent match

The exact bot token in robots.txt (e.g. GPTBot, PerplexityBot); rules are applied by user-agent string, so a typo makes the rule silently do nothing.

Disallow directive

A `Disallow: /` under an AI bot's user-agent blocks it from the entire site, and a blocked citation bot can never fetch or quote your pages.

training vs. retrieval bots

GPTBot and Google-Extended feed model training, while OAI-SearchBot and PerplexityBot fetch live pages for answers — blocking the retrieval bots is what actually costs you citations.

wildcard catch-all rule

A `User-agent: *` block applies to any bot without its own rule, so a broad Disallow can accidentally sweep in AI crawlers you meant to allow.

Allow override

A specific `Allow:` path under an AI user-agent can re-open a section inside a wider Disallow, so rule precedence decides whether the bot really gets in.

missing or unreachable robots.txt

A missing robots.txt or a 4xx response (404 — and per Google even 403) is treated as "crawling allowed," while an unreachable file or a 5xx server error is what Google and RFC 9309 treat as a temporary blanket disallow.

How to use it

1. Paste your site's URL (or its robots.txt URL) and run the check.
2. Review the flagged bots — which AI crawlers are Allowed, which are Disallowed, and which rule matched each one.
3. Fix the robots.txt rules that block citation bots you want visible to, then re-check.

Frequently asked questions