Free robots.txt and sitemap checker
Two questions, one check: are search engines allowed in, and can they find your pages? Both answers live in files most people never open.
- Reads robots.txt and parses it by user-agent group
- Catches a site-wide Disallow: / block
- Finds your sitemap, declared or at the usual paths
- Flags a sitemap that contradicts robots.txt
- Pages in your sitemap that 404 or redirect — dead entries you can't see from the file
- Pages missing from the sitemap entirely
- Blocked pages that matter, ranked by the traffic they'd earn
- Verification — we re-check the live page to confirm you shipped it
Learn more
Questions
What does robots.txt do?
It tells crawlers which parts of your site they may request. It is a set of instructions, not a security control — well-behaved crawlers follow it and everything else ignores it. Never use it to hide anything you actually need protected.
What happens if robots.txt says Disallow: /?
You've asked every crawler to stay off the entire site. It is the single most expensive one-line mistake in SEO, it usually arrives when a staging file gets deployed to production, and nothing on the page looks wrong. This tool checks for it first.
Do I need a robots.txt file at all?
No. With no robots.txt, everything is crawlable, which is a perfectly good default for most sites. Having one lets you steer crawlers away from low-value paths and point them at your sitemap.
Where should my sitemap live?
Anywhere, as long as you declare it. /sitemap.xml is the convention and worth keeping, but the reliable move is a Sitemap: line in robots.txt — that's the address crawlers look for without being told. This tool checks the declared location first, then the usual paths.
What is a sitemap index?
A sitemap of sitemaps. A single sitemap holds up to 50,000 URLs, so larger sites split them and use an index to point at each one. This tool tells you which kind you have, since an index with no child sitemaps looks fine and does nothing.
Why does my sitemap list pages robots.txt blocks?
Usually because the two files were edited at different times by different people. It's a direct contradiction: the sitemap advertises pages the same site forbids crawling. This tool flags that combination specifically.