Web Ecosystem
AI Crawler Traffic Share by Operator
Weekly operator share of requests Cloudflare classifies as AI_CRAWLER, normalized to 100% with six fixed operators and Other. This is Cloudflare-network—not web-wide—traffic; customer mix, products, bot detection, operator attribution, and taxonomy shape the sample and can change over time.
AI Crawler Traffic Share by Operator
- Meta
- Anthropic
- OpenAI
- Amazon
- Huawei
- Other
As of the week beginning 2026-08-31, Meta had the largest Cloudflare-observed AI-crawler request share at 35.6%.
Methodology
The updater calls Cloudflare's authenticated Radar endpoint GET /client/v4/radar/bots/timeseries_groups/BOT_OPERATOR with botCategory=AI_CRAWLER, aggInterval=1w, limitPerGroup=100, and format=JSON. Cloudflare identifies the response normalization as PERCENTAGE and the unit as requests. Values therefore measure each operator's share of HTTP requests attributed to bots in Cloudflare's AI_CRAWLER category, not raw request counts.
Radar covers requests visible on Cloudflare's network, not web-wide traffic. Cloudflare's customer mix and the products through which traffic is observed shape the sample. Cloudflare's bot detection, operator classification, and taxonomy can also be revised over time, so movement may include methodology effects as well as changes in crawler traffic; no revision date is assumed unless Cloudflare supplies one in source metadata.
The API restricts a request window to less than 374 days, so the updater requests consecutive one-year windows and reconciles their overlapping, Cloudflare-aligned Monday timestamps. It loads the complete available history, removes leading all-zero buckets, and retains 105 weekly observations from 2024-08-26 through 2026-08-24. Duplicate boundary observations must agree within one millionth of a percentage point.
Meta, Anthropic, OpenAI, Amazon, Huawei, and Google are fixed because they are the six largest operators in the latest observation at the 2026-08-27 cutoff. For every week, Other is calculated as the direct sum of all other operator shares returned with limitPerGroup=100, including Cloudflare's own lower-cardinality “other” group if present. This makes Other an exact residual at the precision returned by the API rather than an estimate.
Missing named-operator keys within a yearly response are recorded as zero only after the returned operator distribution sums to 100%, meaning the operator had no represented share in that response window. API values are retained to six decimal places, no interpolation or smoothing is applied, and every packaged timestamp must sum to 100% within rounding tolerance.
Cloudflare's returned API metadata specifically marks a BOTS sampling-rate change on August 1, 2025 to address a data-pipeline issue. That sourced annotation is a comparability caution: changes around the date may reflect measurement as well as crawler behavior. More generally, classification or taxonomy revisions need not share that date. The latest 2026-08-24 bucket may also be a partial week because the dataset was retrieved on 2026-08-27; weekly timestamps denote the beginning of Cloudflare's aggregation interval.
This series never uses the Web Crawlers CRAWL_REFER_RATIO dimension. Cloudflare defines crawl-to-refer as crawler requests divided by referral requests for a platform; it is a different metric and is excluded here. Radar data reflects traffic visible on Cloudflare's network and should not be treated as a census of the entire web.
Cloudflare makes Radar API data available under CC BY-NC 4.0. This package attributes Cloudflare Radar, retains official source URLs, and is intended for the repository's noncommercial editorial data product subject to that license.
Frequently asked questions
What does AI crawler traffic share measure?
It is the percentage distribution of HTTP requests visible on Cloudflare's network that Cloudflare Radar attributes to bots in its AI_CRAWLER category, grouped by the organization operating those bots. The denominator is all represented AI_CRAWLER-category requests in the same weekly interval, not AI-crawler requests across the whole web.
Why does AI crawler share matter for the web ecosystem?
The distribution shows which operators are driving automated AI-related requests to sites visible to Cloudflare. Changes can affect publisher bandwidth, access policies, licensing debates, and the balance between content collection and referral value, but share alone does not reveal crawler purpose, content use, or publisher impact.
How should changes in operator share be interpreted?
A rising line means an operator accounted for a larger percentage of Cloudflare-observed AI-crawler requests that week, not necessarily that its raw request volume rose. Shares are compositional: one operator can gain because its traffic increased, because others declined, or both.
How are the weekly figures sourced and calculated?
The updater requests Cloudflare Radar's authenticated BOT_OPERATOR time series with the AI_CRAWLER filter and one-week aggregation. Cloudflare returns request shares normalized to percentages; the chart preserves them to six decimal places, applies no smoothing, and verifies that each weekly distribution sums to 100%.
How are operators classified and selected for display?
Cloudflare supplies the BOT_OPERATOR labels and AI_CRAWLER detection used to attribute requests to operating organizations. Its detection, operator classification, and taxonomy can be revised over time, so movement may reflect methodology as well as traffic; this chart assumes no specific revision date unless source metadata provides one. The latest six leaders remain fixed across history, and every other returned operator is summed into Other.
Is this Cloudflare’s crawl-to-refer ratio?
No. Crawl-to-refer compares a platform's crawler requests with referral traffic sent back to publishers. This chart uses BOT_OPERATOR percentage distributions from the Radar Bots API and contains no referral-request data.
Does this represent AI crawler traffic across the whole web?
No. It represents requests visible on Cloudflare's network that Cloudflare classifies as AI_CRAWLER. Cloudflare's customer mix and products shape which sites and requests enter the sample, so the percentages should not be generalized into web-wide traffic shares.
How is Other calculated?
Other is the sum of every operator outside the six fixed named series for each timestamp. The updater requests up to 100 groups, captures 23 distinct operator labels across the current history, and also includes any lower-volume group that Cloudflare itself labels other. It is not inferred from crawl-to-refer data.
Why do some operators have zero share early in the chart?
The six latest leaders are kept as fixed series for comparability. If one is absent from a yearly response whose returned distribution already totals 100%, the updater records its represented share as zero rather than changing the series lineup or backfilling an estimate.
Why does the chart begin in August 2024?
Requests reaching back to January 2024 returned leading all-zero weekly buckets. August 26, 2024 is the first Cloudflare weekly timestamp with a complete 100% operator distribution. No earlier zero-only period is presented as observed crawler share.
Is the latest week complete and comparable across the full period?
The latest bucket begins August 24, 2026 and may be partial at the August 27 cutoff. Cloudflare's returned metadata annotates an August 1, 2025 BOTS sampling-rate change, and detection, attribution, or taxonomy can change at other times, so shifts should not be attributed solely to crawler behavior. Radar covers Cloudflare-network traffic shaped by its customer mix and products, not the entire web.