Robots.txt
Definition
Robots.txt is a plain text file at your site's root (example.com/robots.txt) that tells crawlers which paths they may request. It uses User-agent to name the crawler, Disallow and Allow to scope paths, and Sitemap to point at your XML sitemap. The critical distinction: robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it — Google simply shows it without a description. To keep a page out of search results entirely, allow crawling and use a noindex meta tag. Common uses are admin areas, staging hosts, faceted-navigation traps, and specific AI bots.
Does robots.txt stop a page from being indexed?
No, and this is the most consequential misunderstanding about the file. Disallow prevents Google from requesting the URL; it does not prevent Google from listing it. A disallowed page that other sites link to can still surface, shown without a snippet. Blocking a page you want removed is actively counterproductive — Google cannot see the noindex tag on a page it is not allowed to fetch.
How do you remove a page from Google properly?
Allow crawling and serve a noindex robots meta tag or X-Robots-Tag header, then wait for a recrawl. Google must fetch the page to learn it should be dropped. For urgent removals, the Removals tool in Search Console hides a URL for roughly six months, which buys time while the noindex propagates. Only add a robots.txt Disallow after the page has actually dropped out.
What are the syntax rules that trip people up?
Matching is by user-agent group and only the single most specific matching group applies — a crawler that matches its own named block ignores the wildcard block entirely. Paths are case-sensitive and match by prefix, so Disallow: /admin also blocks /administration. An empty Disallow permits everything. Google supports * and $ wildcards; crawl-delay is not supported and is silently ignored.
Should you block AI crawlers in robots.txt?
It is a deliberate trade, not a default. Blocking GPTBot, ClaudeBot, or PerplexityBot removes your content from those systems' retrieval and training — which also removes you from answers that would have cited you. Sites wanting citation without training typically allow the search-oriented agents and block the training-only ones. Compliance is voluntary; robots.txt is a request, and abusive scrapers ignore it.
What happens if robots.txt returns an error?
The response code changes crawler behaviour. A 404 is treated as no restrictions, so the whole site is crawlable. A persistent 5xx, however, is treated as a full disallow — Google may stop crawling the site entirely while it lasts. This makes a broken robots.txt one of the fastest ways to deindex a site by accident, and worth monitoring like any other critical endpoint.
Related Tools
Related Terms
Run a free SEO audit to see how your site performs.
Try the SEO Checker