robots.txt validator

What this tool checks

A robots.txt validator lints the plain-text file at your domain's root that tells crawlers which paths they may fetch, flagging rules that are dangerous — like an accidental whole-site block — or obsolete, meaning directives search and AI engines no longer honor. Because ChatGPT, Perplexity, and Google's crawlers read robots.txt before touching a page, a single stray `Disallow: /` can make your whole site invisible to the engines that would otherwise cite it. This check catches those errors and missing signals like the Sitemap line before they cost you crawling and citations.

What it looks at

Whole-site block (Disallow: /)

A `Disallow: /` under `User-agent: *` bars every crawler from the entire site; it's the single most damaging line and often a staging config shipped to production by mistake.

Missing Sitemap directive

The `Sitemap:` line points crawlers straight to your URL index; it's optional but recommended, and its absence means engines have to discover pages the slow way.

Syntax and field typos

Misspelled fields like `Dissalow` or malformed lines are silently ignored, so a rule you think is protecting a path may not exist at all.

Crawl-delay directive

`Crawl-delay` is ignored by Google and most AI crawlers — only some engines honor it — so relying on it to throttle bots gives a false sense of control.

Obsolete directives

Directives like `Noindex` or `Nofollow` placed inside robots.txt are not supported; they do nothing and belong in meta tags or HTTP headers instead.

How to use it

1. Paste your robots.txt content, or enter your site URL to fetch the live file.
2. Review the flagged rules — whole-site blocks, syntax errors, obsolete or unsupported directives, and a missing Sitemap line.
3. Fix the risky rules in your robots.txt, redeploy it at /robots.txt, and re-run the validation to confirm it's clean.

Frequently asked questions