Technical

robots.txt

A file at the root of your site that tells search-engine crawlers which paths they may or may not crawl.

robots.txt lives at yourdomain.com/robots.txt and is the first thing most crawlers read. It can allow or disallow whole sections of a site and point to your sitemap. Used well, it keeps crawlers out of admin areas and infinite filter URLs so they spend their time on pages that matter.

Used badly, it's a silent disaster: a single stray "Disallow: /" — often left over from a staging site — blocks your entire site from being crawled. Because the pages still load fine for humans, this can go unnoticed while your rankings quietly disappear.

Note the distinction from noindex: robots.txt controls crawling (whether the bot fetches the page), not indexing (whether it appears in results). A page blocked in robots.txt can still be indexed without its content, which is rarely what you want.

What to do about it

Check that robots.txt doesn't disallow anything you want ranked — especially a blanket "Disallow: /". Reference your sitemap in it, and use it to keep crawlers out of low-value URLs. CommandSEO flags crawl-blocking problems and re-verifies once you fix them.

Common questions

What's the difference between robots.txt and noindex?

robots.txt controls whether a crawler may fetch a page; noindex controls whether a fetched page appears in results. To keep a page out of search, use noindex — not a robots.txt disallow, which can leave the URL indexed without content.

[ GET STARTED ]

Get Search-Ready
for the AI Era.

AI still needs to find you before it can cite you. Start with the foundation.

Start free →

No credit card to start.