Robots.txt: what it blocks and common mistakes

Robots.txt tells search engine crawlers which parts of your site to stay out of, and one wrong line can hide your whole site.

What it is#

Robots.txt is a plain text file at the root of your site, at yoursite.com/robots.txt. It gives instructions to automated visitors called crawlers: which folders or addresses they may fetch and which to skip. Well-behaved crawlers, including the major search engines, follow it. It is a request, not a lock, so it does not secure anything.

Why it matters#

You usually want crawlers to reach every page that can bring leads, and to skip pages that do nothing for search, such as admin screens, cart steps or internal search results. A correct file saves their attention for pages that count. A wrong one can block the pages you most need found, and the damage is silent: nothing looks broken on your site.

How to check#

1. Open yoursite.com/robots.txt in a browser. A missing file is acceptable: crawlers then assume everything is open.
2. Look for a line reading Disallow: / under User-agent: *. That blocks the entire site.
3. Check each Disallow line against your list of pages that should earn leads. None should match.
4. In Google Search Console, use the robots.txt report and the URL inspection tool to see whether an address is blocked.

How to fix#

• Start simple. A common safe file allows everything and points to the sitemap: User-agent: *, a blank Disallow:, then Sitemap: https://yoursite.com/sitemap.xml.
• Block only what has no search value, such as /wp-admin/ or checkout pages.
• Remove an accidental block, then request a recrawl of key pages.
• Edit it where your platform allows: a settings page, an SEO plugin, or the file itself on the server.

Common mistakes#

• Leaving a site-wide block from the build stage. Staging sites are often blocked, and the rule travels with the launch. This is the classic way to vanish from search.
• Blocking CSS or script files. Crawlers then cannot see the page as visitors do.
• Using it to hide a page from results. A blocked address can still be listed if other sites link to it. To keep a page out, allow crawling and add a noindex instruction. A crawler blocked by robots.txt never sees that instruction.
• Using it for privacy. The file is public, and it shows anyone where your private folders are. Protect those with a login.
• Typos and case. Paths are case sensitive, and a stray slash changes the meaning.

Pair this with your sitemap: one says where to go, the other where not to.