Crawl Budget Management

Crawl budget is the number of URLs Googlebot will fetch from your domain in a given period. For most sites with under ~10,000 URLs, it is a non-issue — Googlebot crawls everything important within days. For large sites, faceted commerce, news publishers, and programmatic SEO operations, crawl budget is the difference between fresh content reaching the index in hours and stale URLs sitting unindexed for months.

TL;DR

• Most sites do not need to think about crawl budget. Google has been explicit: sites under ~10,000 URLs with healthy server response times rarely hit the budget ceiling. If you are at this scale and indexation is slow, the problem is quality or discovery, not budget.
• Server logs are the only ground-truth source. GSC > Crawl Stats is sampled and aggregated. Real diagnosis happens in raw access logs filtered by user-agent: Googlebot, Googlebot-Image, bingbot, GPTBot, ClaudeBot, PerplexityBot.
• Infinite crawl spaces are the #1 budget waste pattern. Faceted navigation, calendar widgets, search result pages, session-ID URLs, sort/filter parameter combinations — these can create millions of crawlable URLs from a few thousand canonical pages.

The mental model

Crawl budget is like an editor’s reading queue at a major newspaper. The editor (Googlebot) has 8 hours per day per beat (your domain). They read the most important stories first, then less important ones, until the day is done. If your beat sends 50 stories worth reading, the editor reads them all. If it sends 50,000 stories with 90% noise, the editor never gets to the 5,000 that actually matter.

The editor also adjusts how much time they give your beat based on past experience. A beat that consistently sends well-written, server-fast stories gets more time. A beat that 5xx’s occasionally or sends 30% duplicates gets less. This is the crawl rate dimension of crawl budget.

The editor also has an opinion about what to read first — the crawl demand dimension. Pages with strong external signals (links, traffic, freshness) jump the queue. Pages with no signals sit at the back of the line indefinitely.

Deep dive: the 2026 reality

Google’s documented crawl budget components:

1. Crawl rate limit — how fast Googlebot can hit your server without overloading it. Driven by your server’s response time and 5xx rate. A slow server gets less budget.
2. Crawl demand — how much Googlebot wants to crawl, derived from URL popularity and staleness. A popular fresh URL gets crawled often; a forgotten 5-year-old archive gets crawled rarely.
3. Effective crawl budget = min(rate limit, demand) — the actual ceiling per day.

The diagnostic is server logs. GSC > Settings > Crawl Stats shows aggregate trends and per-purpose breakdowns (refresh, discovery), but it does not let you see which URLs Googlebot fetched today and which it ignored. For that, you need raw access logs. Common log infrastructures in 2026:

Source	Access pattern	Tool
Cloudflare	Logpush to S3/GCS	BigQuery, Athena
Vercel	Log Drains to Datadog/Axiom	Axiom, Datadog APM
Nginx/Apache	/var/log/nginx/access.log	GoAccess, Logflare
AWS CloudFront	S3 access logs	Athena, Logs Insights
Akamai/Fastly	LogPush to object storage	BigQuery, Splunk

Infinite crawl spaces — the patterns that explode URL counts:

• Faceted navigation: ?color=red&size=10&brand=acme produces colors × sizes × brands URLs.
• Calendar widgets: a “next month” link with no terminal condition crawls infinitely into 2099.
• Internal site search: /search?q=* crawled and indexed produces one URL per query.
• Session IDs in URLs: ?sessionid=xyz makes every page unique per visitor.
• Sort/filter parameters that don’t affect content: ?sort=newest and ?sort=oldest are crawled separately.
• Tag clouds: a tag taxonomy with thousands of single-use tags.

For AI crawlers, the crawl-budget concern is different. GPTBot runs nightly bulk scrapes; if blocked at the WAF after some threshold, OpenAI’s logs will show your site as low-yield and de-prioritize you. ClaudeBot is similar. PerplexityBot runs on-demand at user-query time; bandwidth waste is your concern, not theirs.

Visualizing it

flowchart TD
  A[Server logs<br/>14-day window] --> B[Filter user-agent: Googlebot]
  B --> C[Group by URL pattern]
  C --> D{Pattern is canonical content?}
  D -->|Yes| E[Healthy budget use]
  D -->|No| F[Wasted budget]
  F --> G{Why crawled?}
  G -->|Faceted params| H[robots.txt Disallow params]
  G -->|Calendar/infinite| I[Add nofollow + robots block]
  G -->|Session IDs| J[Strip via canonical or 301]
  G -->|Internal search| K[robots.txt Disallow /search]
  H --> L[Re-measure in 30 days]
  I --> L
  J --> L
  K --> L

Bad vs. expert

The bad approach

Faceted ecommerce navigation with no parameter handling. Every filter combination is its own URL, every sort order is its own URL, the calendar widget links to /events?date=2099-12-31:

# Sample of a 14-day Googlebot log for one weak ecommerce site
/products/shoes?color=red&size=10              4 hits
/products/shoes?color=red&size=11              3 hits
/products/shoes?color=red&size=10&sort=price   8 hits
/products/shoes?color=red&size=10&sort=newest  6 hits
/products/shoes?color=red&size=10&sort=rating  5 hits
/events?date=2099-08-15                        2 hits
/events?date=2099-08-16                        2 hits
/events?date=2099-08-17                        2 hits
[... thousands more ...]

Googlebot is spending days each month crawling combinatorial garbage while genuinely new product pages wait. Indexation lag for new products: 3–6 weeks. Sites at this scale typically see GSC’s “Discovered – currently not indexed” balloon into the hundreds of thousands.

The expert approach

Block parameter explosions in robots.txt (parameter handling is the highest leverage):

# robots.txt — block infinite spaces
User-agent: *
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*filter=
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*sessionid=
Disallow: /events?date=20*
Disallow: /events?date=21*
Disallow: /tag/

# Allow critical params if needed
Allow: /*?page=

Sitemap: https://example.com/sitemap.xml

Add nofollow on internal calendar/pagination links that lead to infinite spaces:

<!-- Calendar 'next month' link doesn't waste crawl budget -->
<a href="/events?date=2099-08-16" rel="nofollow">Next month</a>

Server-log analysis pipeline. BigQuery query for Cloudflare Logpush:

-- 14-day crawl-budget waste analysis
WITH crawls AS (
  SELECT
    REGEXP_EXTRACT(ClientRequestURI, r'^/[^/]+') AS first_path_segment,
    ClientRequestURI,
    EdgeStartTimestamp,
    ClientRequestUserAgent,
    EdgeResponseStatus
  FROM `proj.cloudflare.http_requests`
  WHERE EdgeStartTimestamp BETWEEN
        TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
        AND CURRENT_TIMESTAMP()
    AND REGEXP_CONTAINS(ClientRequestUserAgent,
        r'(?i)Googlebot|bingbot|GPTBot|ClaudeBot|PerplexityBot')
)
SELECT
  first_path_segment,
  REGEXP_CONTAINS(ClientRequestURI, r'\?') AS has_params,
  COUNT(*) AS hits,
  COUNT(DISTINCT ClientRequestURI) AS unique_urls,
  COUNTIF(EdgeResponseStatus >= 400) AS error_hits,
  ROUND(COUNTIF(EdgeResponseStatus >= 400) / COUNT(*) * 100, 1) AS error_pct
FROM crawls
GROUP BY first_path_segment, has_params
ORDER BY hits DESC
LIMIT 30;

Parse local nginx logs for a quick assessment without a data warehouse:

# Top 30 URLs Googlebot fetched in the last 14 days
zcat /var/log/nginx/access.log*.gz | \
  grep -E 'Googlebot' | \
  awk '{print $7}' | \
  sort | uniq -c | sort -rn | head -30

# Same, isolating URLs with query params (typical waste)
zcat /var/log/nginx/access.log*.gz | \
  grep 'Googlebot' | \
  awk '{print $7}' | \
  grep '?' | \
  sort | uniq -c | sort -rn | head -30

Strip session IDs and tracking params at the edge with Cloudflare Workers or middleware:

// Cloudflare Worker — canonicalize away tracking params
export default {
  async fetch(request) {
    const url = new URL(request.url);
    const tracking = ['sessionid', 'utm_source', 'utm_medium', 'utm_campaign',
                      'utm_content', 'utm_term', 'fbclid', 'gclid'];
    let stripped = false;
    for (const k of tracking) {
      if (url.searchParams.has(k)) {
        url.searchParams.delete(k);
        stripped = true;
      }
    }
    if (stripped) {
      return Response.redirect(url.toString(), 301);
    }
    return fetch(request);
  },
};

Do this today

1. Open GSC > Settings > Crawl Stats. Note Total crawl requests, Average response time, and the By response breakdown. Any sustained period where 4xx + 5xx > 5% means crawl rate is being throttled.
2. In Crawl Stats > By purpose, compare Discovery vs Refresh percentages. Healthy sites are roughly 20% discovery, 80% refresh once mature. Heavy discovery = Google is finding new URLs, possibly an infinite space.
3. Pull server access logs for the last 30 days. Filter for User-Agent matching Googlebot, bingbot, GPTBot, ClaudeBot, PerplexityBot. Group by URL pattern and count hits.
4. Identify your top 30 most-crawled URLs. Cross-reference with your sitemap. Any URL crawled heavily that is not in the sitemap is likely waste — investigate.
5. Audit for infinite crawl spaces. Use Screaming Frog SEO Spider > Configuration > Spider > Crawl Behaviour with “Crawl all subdomains” off and a depth limit of 8. Watch the URL count; if it climbs without converging, you have an infinite space.
6. Add Disallow rules in robots.txt for parameter patterns that produce no unique content (?sort=, ?filter=, ?sessionid=, ?utm_*= — Google ignores UTM by default but Bing/PerplexityBot do not).
7. Add rel="nofollow" to internal links that lead to combinatorial spaces (calendar “next month”, facet pivot tables, pagination beyond ~page 50). Keep these links present for users but invisible to crawl-equity flow.
8. Use Bing Webmaster Tools > Configure My Site > URL Parameters (still functional in 2026, unlike Google’s deprecated equivalent). Tell Bing which parameters are tracking-only so it consolidates.
9. Build a crawl-budget dashboard. Required charts: hits per day per user-agent, hits by status code, hits by URL pattern, average response time over time. Update weekly.
10. Re-measure after 30 days. Compare wasted-URL hit count before and after blocks. A successful intervention typically frees 30–60% of crawl budget for legitimate URLs and shows up as faster indexation in GSC > Pages.