Robots.txt

Definition

Robots.txt is a plain text file placed in your website's root directory (e.g., example.com/robots.txt) that provides instructions to web crawlers about which parts of your site they should or shouldn't access. It uses directives like User-agent (which crawler), Allow (permitted paths), Disallow (blocked paths), and Sitemap (location of your XML sitemap). Important: robots.txt blocks crawling, not indexing. A page blocked by robots.txt can still appear in search results if other pages link to it. For pages you want excluded from search results entirely, use the noindex meta tag instead. Common uses include blocking admin areas, staging environments, duplicate content, and specific AI bots.