Crawl Budget

Definition

Crawl budget is the number of URLs Googlebot will crawl on your site within a given time period. It's determined by two factors: crawl rate limit (how fast Google can crawl without overloading your server) and crawl demand (how much Google wants to crawl based on your site's popularity and freshness). For most small-to-medium sites (under 10,000 pages), crawl budget isn't a concern — Google will crawl all your pages regularly. It becomes important for large sites with millions of pages, sites with significant duplicate content, or sites with many low-quality pages. Optimize crawl budget by eliminating duplicate content, fixing broken pages, using robots.txt to block unimportant pages, and maintaining a clean XML sitemap.