# robots.txt validator

_Source: https://seomatrix.ai/tools/robots-validator/_

---

## What it does

A robots.txt validator lints the plain-text file at your domain's root that tells crawlers which paths they may fetch, flagging rules that are dangerous — like an accidental whole-site block — or obsolete, meaning directives search and AI engines no longer honor. Because ChatGPT, Perplexity, and Google's crawlers read robots.txt before touching a page, a single stray `Disallow: /` can make your whole site invisible to the engines that would otherwise cite it. This check catches those errors and missing signals like the Sitemap line before they cost you crawling and citations.

## What it checks

- **Whole-site block (Disallow: /)** — A `Disallow: /` under `User-agent: *` bars every crawler from the entire site; it's the single most damaging line and often a staging config shipped to production by mistake.
- **Missing Sitemap directive** — The `Sitemap:` line points crawlers straight to your URL index; it's optional but recommended, and its absence means engines have to discover pages the slow way.
- **Syntax and field typos** — Misspelled fields like `Dissalow` or malformed lines are silently ignored, so a rule you think is protecting a path may not exist at all.
- **Crawl-delay directive** — `Crawl-delay` is ignored by Google and most AI crawlers — only some engines honor it — so relying on it to throttle bots gives a false sense of control.
- **Obsolete directives** — Directives like `Noindex` or `Nofollow` placed inside robots.txt are not supported; they do nothing and belong in meta tags or HTTP headers instead.

## How to use it

1. Paste your robots.txt content, or enter your site URL to fetch the live file.
2. Review the flagged rules — whole-site blocks, syntax errors, obsolete or unsupported directives, and a missing Sitemap line.
3. Fix the risky rules in your robots.txt, redeploy it at /robots.txt, and re-run the validation to confirm it's clean.

## FAQ

### What's the most dangerous robots.txt mistake?

An accidental `Disallow: /` under `User-agent: *` — it blocks every crawler from your whole site, so nothing gets indexed or cited. It's usually a staging rule that shipped to production.

### Does robots.txt block AI crawlers like GPTBot and PerplexityBot?

Yes. AI crawlers read robots.txt first, so a whole-site `Disallow` or a bot-specific block keeps ChatGPT, Perplexity, and similar engines from reading and citing your pages.

### Is Crawl-delay still supported?

Google and most AI crawlers ignore `Crawl-delay`; only some engines such as Bing still honor it (Yandex dropped the directive in favor of a crawl-rate setting in Yandex.Webmaster). The validator flags it so you don't rely on a directive that does nothing where it matters most.

### Can robots.txt hide a page from Google?

No — robots.txt controls crawling, not indexing. A blocked URL can still appear in results without a snippet; to keep a page out of the index, use a `noindex` meta tag instead.