AI Crawler Blocking in Top Websites

The cohort is exactly the first 1,000 pay-level domains in permanent Tranco list 46W9X, generated on August 26, 2026 and retrieved on August 27. Membership and rank stay fixed across every period. Applying this current cohort backward creates survivorship and selection bias: some members were not prominent in 2023, while historical leaders that later fell out are absent. The chart therefore does not represent each period’s contemporaneous top 1,000.

Robots.txt is a widely used way for publishers to declare whether automated agents may crawl their sites. Changes in full-site blocking indicate how publishers are trying to control access that may support AI training or search and answer products, illuminating negotiations and friction between content owners, AI companies, and web platforms. The rates do not measure actual crawling or content use.

The parser tests the product tokens GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, and Bytespider. They are shown separately because operators publish different crawler identities and may use them for different AI training, indexing, search, or product functions; inclusion does not imply that all six have identical purposes or behavior.

Matching is case-insensitive and follows RFC 9309: the most specific matching user-agent group applies, equally specific repeated groups are combined, and User-agent: * applies only when no more-specific group matches. Empty Disallow values do not block. The longest matching Allow or Disallow path wins, with Allow winning equal-length ties; * wildcards and terminal $ anchors are supported. A site counts as fully blocked only when the applicable result denies the root path /.

For each quarterly target, the collector compares the nearest qualifying HTTP-200 robots.txt captures before and after the target and uses the closer usable snapshot within ±14 days. The timestamp shown is the target date, not the capture time. A 20-domain monthly pilot found partial, nonrandom archive coverage, so quarterly points avoid implying monthly precision and reduce full-cohort replay validation from as many as 37,000 to 13,000 bodies.

Each bot’s rate uses only cohort sites with a successfully fetched, usable robots.txt observation for that period. Missing captures, timeouts, failed replays, HTML error bodies, non-200 responses, and failed live requests are excluded from both numerator and denominator—not counted as allowing a crawler. The separate coverage series reports observed sites as a share of all 1,000 cohort domains, and point notes give the exact count.

Bot rates are not published when fewer than 30% of cohort sites have usable observations. Wayback coverage was 10.8% for May 2026 and 13.4% for August 2026, so those bot-rate points are suppressed because the observed subset may be especially unrepresentative. Their coverage points remain visible as evidence of the archive gap.

GPTBot’s introduction in August 2023 provides a natural starting month for tracking explicit AI-crawler policy. The first target is August 1 and can use a qualifying capture within ±14 days, so it should be read as an August snapshot rather than a precise launch-day measurement. No earlier values are backfilled as zeros. Rules matching User-agent: * may also produce a parsed result even before a named token was widely used.

No. Robots.txt records declared machine-readable policy; it does not enforce access or show crawler compliance. WAF rules, authentication, rate limits, and network blocks can deny access without a robots.txt rule, so this metric can understate total access controls. Conversely, declared rules may not be enforced or honored, so the rate is not a strict numeric floor on effective denial. It also does not establish legal permission or prohibition.

The forward collector requests each frozen-cohort domain’s live robots.txt weekly, following redirects and falling back to HTTP only when HTTPS does not return a usable HTTP-200 robots body. It appends or replaces that collection date without changing cohort membership or the quarterly archive. Because live and Wayback availability differ, compare the transition using the coverage series.

Treat each line as the share of observed sites whose text parses as a full-root block for that named token, not as a ranking of operator compliance or access. Publishers may target operators differently, wildcard rules can affect several tokens at once, crawler purposes differ, and operators may interpret nonstandard syntax differently. Compare direction and gaps alongside coverage, launch timing, and operator documentation.