# AI crawler access

_Source: https://seomatrix.ai/tools/ai-access/_

---

## What it does

This checks whether your robots.txt allows or blocks the specific AI crawlers that power answer engines — GPTBot and OAI-SearchBot (OpenAI), PerplexityBot, ClaudeBot, Google-Extended, and others. If you Disallow a bot that gathers or retrieves your content for an AI engine, that engine can't crawl your pages and can never cite you in its answers. It separates model-training bots from the live search bots that actually build cited answers.

## What it checks

- **user-agent match** — The exact bot token in robots.txt (e.g. GPTBot, PerplexityBot); rules are applied by user-agent string, so a typo makes the rule silently do nothing.
- **Disallow directive** — A `Disallow: /` under an AI bot's user-agent blocks it from the entire site, and a blocked citation bot can never fetch or quote your pages.
- **training vs. retrieval bots** — GPTBot and Google-Extended feed model training, while OAI-SearchBot and PerplexityBot fetch live pages for answers — blocking the retrieval bots is what actually costs you citations.
- **wildcard catch-all rule** — A `User-agent: *` block applies to any bot without its own rule, so a broad Disallow can accidentally sweep in AI crawlers you meant to allow.
- **Allow override** — A specific `Allow:` path under an AI user-agent can re-open a section inside a wider Disallow, so rule precedence decides whether the bot really gets in.
- **missing or unreachable robots.txt** — A missing robots.txt or a 4xx response (404 — and per Google even 403) is treated as "crawling allowed," while an unreachable file or a 5xx server error is what Google and RFC 9309 treat as a temporary blanket disallow.

## How to use it

1. Paste your site's URL (or its robots.txt URL) and run the check.
2. Review the flagged bots — which AI crawlers are Allowed, which are Disallowed, and which rule matched each one.
3. Fix the robots.txt rules that block citation bots you want visible to, then re-check.

## FAQ

### Does blocking GPTBot stop ChatGPT from citing me?

Blocking GPTBot only stops OpenAI from crawling you for model training; ChatGPT's live web answers use OAI-SearchBot, so allow that bot if you want to appear in ChatGPT Search citations.

### If I have no robots.txt, are AI bots allowed?

Yes — a missing robots.txt is treated as "everything allowed," so by default all AI crawlers may access your site.

### What's the difference between a training bot and a search bot?

Training bots like GPTBot and Google-Extended collect content to train models, while search/retrieval bots like OAI-SearchBot and PerplexityBot fetch pages in real time to build cited answers — the retrieval bots are what directly drive AI citations.

### Will allowing these bots hurt my regular SEO?

No — AI-crawler rules are separate from Googlebot, so you can allow OAI-SearchBot and PerplexityBot without changing how Google indexes or ranks your site.