# AI crawler log analysis: what GPTBot and ClaudeBot actually fetch > Read server logs for GPTBot, ClaudeBot and PerplexityBot: find coverage gaps, wasted crawl and blocked assets, and prove AI bots reach your content. _Source: https://seomatrix.ai/blog/ai-crawler-logs/ · Updated: 2026-07-23_ --- In short: your **server access logs** are the only ground truth about what AI crawlers actually do on your site. Everything else — robots.txt, sitemaps, "I added llms.txt" — is intent. The log line is evidence: it names the bot, the URL, the status code, and the byte count. Filter those lines for GPTBot, ClaudeBot, PerplexityBot and friends, and four questions answer themselves — are they reaching your money pages, are they wasting budget on junk, are they blocked, and are the requests even real? ## The user agents worth filtering for Not every AI bot does the same job. Some crawl in bulk to build a corpus; some fetch a single page the instant a user asks a question. Reading logs correctly starts with knowing which is which. | User agent | Operator | What it's doing | |---|---|---| | `GPTBot` | OpenAI | Bulk crawl for training/data | | `OAI-SearchBot` | OpenAI | Building the ChatGPT search index | | `ChatGPT-User` | OpenAI | On-demand fetch — a user asked something | | `ClaudeBot` | Anthropic | Bulk crawl for Claude | | `Claude-User` | Anthropic | On-demand fetch inside a Claude chat | | `PerplexityBot` | Perplexity | Indexing for Perplexity answers | | `Googlebot` + `Google-Extended` | Google | Crawl governed by the Extended token (see below) | A spike in `ChatGPT-User` or `Claude-User` on one URL is a live signal: someone is asking an AI about that topic and it is fetching your page to answer. That is worth more attention than raw bulk-crawl volume. ## What one log line tells you A single access-log line carries four fields that matter for GEO: - **Path** — which URL was fetched. Aggregate these and you get coverage. - **Status** — 200 (served), 304 (not modified), 403 (blocked), 404 (missing), 5xx (broken). A bot that only ever gets 403s never sees you. - **Bytes** — a 200 that returns 800 bytes on a 4,000-word article means the bot got a shell, not the content. - **Frequency + timestamp** — how often, and whether recrawl tracks your publish cadence. ## The four gaps to hunt **1. Coverage gaps.** Group successful (200) fetches by path and compare against your sitemap. Pages the bots never request are invisible to AI answers no matter how good they are. Deep, poorly-linked pages are the usual casualties — a crawl-budget and internal-linking problem, not a content one. **2. Wasted crawl.** Count the requests that hit redirects (301/302 chains), 404s, faceted-filter URLs, or infinite pagination. Every one of those is budget the bot did not spend on a real page. On a large site this is where most AI-crawl waste hides. **3. Blocked or thin responses.** 403s mean something — a WAF rule, a rate limiter, or robots.txt — is turning bots away. Small byte counts on big pages mean the bot fetched HTML that renders empty. Since most AI crawlers do not run JavaScript, a client-rendered SPA reads as a blank page in the logs. **4. Verification.** Confirm the requests are genuine before you trust any of the above. > **Google-Extended is a token, not a crawler** — You will not find a `Google-Extended` line in your logs. Google-Extended is a robots.txt control token that governs whether content Googlebot already fetched may be used for Gemini and Vertex AI — the fetch is still done by ordinary `Googlebot`. So blocking Google-Extended does not reduce crawl traffic; it only changes permission. Read your logs for `Googlebot`, and manage the AI-use decision separately in robots.txt. ## Verify the bot is real The user-agent string is self-reported and trivial to forge — scrapers routinely masquerade as GPTBot to dodge blocks. Before you draw a single conclusion, confirm the source IP: - **OpenAI, Anthropic and Google publish the IP ranges** their crawlers use, as machine-readable lists you can match against. - **Forward-confirmed reverse DNS** is the fallback: reverse-resolve the IP, check it lands on the operator's domain, then forward-resolve that hostname back to the same IP. Skip this and your "GPTBot coverage" report may be counting a scraper wearing GPTBot's name. > **Don't act on unverified logs** — An unverified user-agent count is worse than no data — it looks authoritative and is wrong. A blocked-bot complaint, a "we're being over-crawled" panic, and a coverage-gap fix can all be traced to a spoofer if you skip IP verification. Verify first, decide second. ## The checklist - [ ] Filtered logs for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot. - [ ] Verified each bot by source IP (published ranges or forward-confirmed rDNS). - [ ] Compared 200-fetched paths against the sitemap — listed the never-crawled pages. - [ ] Counted wasted crawl: redirect chains, 404s, faceted URLs, pagination. - [ ] Flagged 403s and thin responses (small bytes on large pages). - [ ] Checked recrawl frequency tracks your publish cadence. - [ ] Separated on-demand (ChatGPT-User / Claude-User) hits as live-interest signals. The split is the point: when nearly half the crawl lands on redirects and dead ends, the fix is not more content — it is cleaning the paths the bots waste budget on. Our [AI crawler log tool](/tools/ai-crawler-log) does this pass for you: it parses the logs, verifies each bot, and hands back the coverage-gap and wasted-crawl lists. ## In short - Server logs are the only evidence of what AI crawlers actually fetch — the rest is intent. - Filter for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot; treat on-demand fetchers as live interest. - Read path, status, bytes and frequency to find coverage gaps, wasted crawl, and blocked or thin pages. - Google-Extended is a robots token, not a log line — the fetch is still Googlebot. - Verify every bot by IP before you trust the numbers; user agents are trivially spoofed. ## Sources - **OpenAI, "GPTBot" and "OAI-SearchBot" docs** — [platform.openai.com/docs/bots](https://platform.openai.com/docs/bots): official user agents and published crawler IP ranges. - **Google Search Central, "Overview of Google crawlers and fetchers"** — [developers.google.com/search/docs/crawling-indexing/overview-google-crawlers](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers): confirms Google-Extended is a robots token applied to Googlebot, not a separate crawler. - **RFC 9309, "Robots Exclusion Protocol"** — [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309): the standard governing how user-agent tokens and directives are interpreted. - Related: [why SPAs go dark to AI crawlers](/blog/geo-for-spa/) and [managing AI bots in robots.txt](/blog/ai-bots-robots/). ## FAQ ### Which AI crawler user agents should I look for in server logs? The high-traffic ones in 2026 are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), and Googlebot governed by the Google-Extended robots token. ChatGPT-User and Claude-User are the on-demand fetchers triggered when a person asks a question, so they signal live interest, not bulk crawling. ### How do I verify a log line is really GPTBot and not a spoofer? Never trust the user-agent string alone — it is trivial to fake. OpenAI, Anthropic and Google publish the IP ranges their crawlers use. Confirm each request's source IP falls inside the official published range (or passes a forward-confirmed reverse DNS lookup) before you count it as a real bot. ### Do AI crawlers render JavaScript like Googlebot? Mostly no. Most AI training and answer crawlers fetch raw HTML and do not execute client-side JavaScript, so a page that only renders in the browser is often empty to them. Logs make this visible: you will see the HTML fetched but never the JSON or hydration bundle a browser would need.