AI crawler log analysis: what GPTBot and ClaudeBot actually fetch

In short: your server access logs are the only ground truth about what AI crawlers actually do on your site. Everything else — robots.txt, sitemaps, “I added llms.txt” — is intent. The log line is evidence: it names the bot, the URL, the status code, and the byte count. Filter those lines for GPTBot, ClaudeBot, PerplexityBot and friends, and four questions answer themselves — are they reaching your money pages, are they wasting budget on junk, are they blocked, and are the requests even real?

The user agents worth filtering for

Not every AI bot does the same job. Some crawl in bulk to build a corpus; some fetch a single page the instant a user asks a question. Reading logs correctly starts with knowing which is which.

User agent	Operator	What it’s doing
GPTBot	OpenAI	Bulk crawl for training/data
OAI-SearchBot	OpenAI	Building the ChatGPT search index
ChatGPT-User	OpenAI	On-demand fetch — a user asked something
ClaudeBot	Anthropic	Bulk crawl for Claude
Claude-User	Anthropic	On-demand fetch inside a Claude chat
PerplexityBot	Perplexity	Indexing for Perplexity answers
Googlebot + Google-Extended	Google	Crawl governed by the Extended token (see below)

A spike in ChatGPT-User or Claude-User on one URL is a live signal: someone is asking an AI about that topic and it is fetching your page to answer. That is worth more attention than raw bulk-crawl volume.

What one log line tells you

A single access-log line carries four fields that matter for GEO:

• Path — which URL was fetched. Aggregate these and you get coverage.
• Status — 200 (served), 304 (not modified), 403 (blocked), 404 (missing), 5xx (broken). A bot that only ever gets 403s never sees you.
• Bytes — a 200 that returns 800 bytes on a 4,000-word article means the bot got a shell, not the content.
• Frequency + timestamp — how often, and whether recrawl tracks your publish cadence.

The four gaps to hunt

1. Coverage gaps. Group successful (200) fetches by path and compare against your sitemap. Pages the bots never request are invisible to AI answers no matter how good they are. Deep, poorly-linked pages are the usual casualties — a crawl-budget and internal-linking problem, not a content one.

2. Wasted crawl. Count the requests that hit redirects (301/302 chains), 404s, faceted-filter URLs, or infinite pagination. Every one of those is budget the bot did not spend on a real page. On a large site this is where most AI-crawl waste hides.

3. Blocked or thin responses. 403s mean something — a WAF rule, a rate limiter, or robots.txt — is turning bots away. Small byte counts on big pages mean the bot fetched HTML that renders empty. Since most AI crawlers do not run JavaScript, a client-rendered SPA reads as a blank page in the logs.

4. Verification. Confirm the requests are genuine before you trust any of the above.

Verify the bot is real

The user-agent string is self-reported and trivial to forge — scrapers routinely masquerade as GPTBot to dodge blocks. Before you draw a single conclusion, confirm the source IP:

• OpenAI, Anthropic and Google publish the IP ranges their crawlers use, as machine-readable lists you can match against.
• Forward-confirmed reverse DNS is the fallback: reverse-resolve the IP, check it lands on the operator’s domain, then forward-resolve that hostname back to the same IP.

Skip this and your “GPTBot coverage” report may be counting a scraper wearing GPTBot’s name.

The checklist

• Filtered logs for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot.
• Verified each bot by source IP (published ranges or forward-confirmed rDNS).
• Compared 200-fetched paths against the sitemap — listed the never-crawled pages.
• Counted wasted crawl: redirect chains, 404s, faceted URLs, pagination.
• Flagged 403s and thin responses (small bytes on large pages).
• Checked recrawl frequency tracks your publish cadence.
• Separated on-demand (ChatGPT-User / Claude-User) hits as live-interest signals.

Content pages (200) 45%

Redirects / 404 / facets 40%

Assets & other 15%

источник: internal audits, 2026

The split is the point: when nearly half the crawl lands on redirects and dead ends, the fix is not more content — it is cleaning the paths the bots waste budget on. Our AI crawler log tool does this pass for you: it parses the logs, verifies each bot, and hands back the coverage-gap and wasted-crawl lists.

In short

• Server logs are the only evidence of what AI crawlers actually fetch — the rest is intent.
• Filter for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot; treat on-demand fetchers as live interest.
• Read path, status, bytes and frequency to find coverage gaps, wasted crawl, and blocked or thin pages.
• Google-Extended is a robots token, not a log line — the fetch is still Googlebot.
• Verify every bot by IP before you trust the numbers; user agents are trivially spoofed.

Sources

• OpenAI, “GPTBot” and “OAI-SearchBot” docs — platform.openai.com/docs/bots: official user agents and published crawler IP ranges.
• Google Search Central, “Overview of Google crawlers and fetchers” — developers.google.com/search/docs/crawling-indexing/overview-google-crawlers: confirms Google-Extended is a robots token applied to Googlebot, not a separate crawler.
• RFC 9309, “Robots Exclusion Protocol” — rfc-editor.org/rfc/rfc9309: the standard governing how user-agent tokens and directives are interpreted.
• Related: why SPAs go dark to AI crawlers and managing AI bots in robots.txt.