AI crawler log analyzer

What this tool checks

An AI crawler log analyzer reads your server or CDN access log and shows which AI bots — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — visited your site, how often, and which pages they read. It is the only precise way to confirm whether engines like ChatGPT and Perplexity actually see you: a page an AI crawler never fetched cannot be cited in an AI answer.

What it looks at

AI crawler user-agent

The string in each log line that identifies the bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended); without it you cannot tell which engine visited.

Crawl frequency

How many requests each bot made over the log window; frequency shows how actively an engine is refreshing its picture of your site.

Page coverage

Which of your URLs the bots actually downloaded; a page they never crawled cannot be cited in an AI answer.

Bot purpose (train vs answer)

Separates model-training crawlers (GPTBot, Google-Extended, ClaudeBot) from live-answer crawlers (OAI-SearchBot, PerplexityBot, Claude-User) that fetch a page while building an AI response.

HTTP status per bot

The response code each crawler received: a 200 means it read the page, while a 403 or 404 means it was blocked or the page is missing.

Crawl recency

The date of the last hit; a stale last-seen date means an engine has not re-read your updated content.

How to use it

1. Paste or upload your access log (Combined, JSON, or .gz format) into the field on the page.
2. Review the per-bot breakdown — which AI crawlers hit you, how often, and which pages they read.
3. Open access to the crawlers you want in robots.txt and server config, and add any missing pages to your sitemap and internal links.

Frequently asked questions