Crawl budget - does it matter for your website
Most small business websites do not need to think about crawl budget. Large ecommerce sites, sites with dynamic URL generation, and sites with significant amounts of thin or duplicate content often do.
Crawl budget is the combination of how often Googlebot wants to crawl your site (crawl demand) and how many pages it can crawl without putting strain on your server (crawl capacity limit). For a 50-page website, this is not something to spend time on. For a site with tens of thousands of pages, crawl budget determines which of those pages Google actually discovers and indexes.
Signs you might have a crawl budget issue
New pages take weeks or months to appear in Google's index. Pages that were indexed have been removed from the index without any obvious reason. Search Console shows large numbers of pages in the Discovered - currently not indexed status. Your site generates many URLs through filtering, sorting, or pagination parameters that create near-duplicate versions of the same content.
The main culprits
URL parameters are the most common crawl budget problem. Ecommerce sites that generate unique URLs for every combination of filter - colour, size, price range, sort order - can create hundreds of thousands of URLs from a catalogue of a few thousand products. Googlebot crawls many of these but cannot keep up with the volume. Use Google Search Console's URL Parameters tool (or robots.txt disallow directives) to prevent crawling of parameter-generated URL variants that do not add indexing value.
Redirect chains waste crawl budget because Googlebot has to follow each redirect in the chain, using crawl resources without reaching unique content. Fix redirect chains to use direct 301 redirects from the original URL to the final destination.
XML sitemaps
A well-maintained XML sitemap helps Googlebot prioritise which pages to crawl. Only include pages in your sitemap that you want indexed and that are accessible without login. Exclude redirected URLs, pages with noindex tags, and parameter-generated duplicates. A clean sitemap that accurately reflects your indexable content helps Googlebot allocate crawl resources to the pages that matter.
When to focus on this
If your site has fewer than 10,000 pages and does not use complex URL parameter structures, crawl budget is not your problem. Focus your SEO effort on content quality, technical fundamentals, and link building. Crawl budget optimisation is a specialist concern for large sites, and spending time on it when you have a 200-page website is misallocated effort.
Found this useful?
Start a conversation - no pitch, no pressure.