Agent Grade study · 24 October 2026
How the top 100,000 websites treat AI bots
Sites block AI training bots about twice as often as AI search bots. But 2,012 sites are open to Google and closed to ChatGPT search, and 515 of them never name it at all: a rule meant for every bot shuts it out.
11.7%
block OpenAI training (GPTBot)
5.3%
block ChatGPT search (OAI-SearchBot)
2,012
open to Google, closed to ChatGPT search
4,189
of 9,313 blocking OpenAI training also block ChatGPT search
3,283
open to Google, closed to at least one AI search bot
11.1%
have an llms.txt file
Every bot we checked
| Bot | What it does | Sites blocking | Share | By a blanket rule only |
|---|---|---|---|---|
| GPTBot | OpenAI training | 9,313 | 11.7% | 2,662 |
| CCBot | Common Crawl | 9,141 | 11.5% | 2,741 |
| Bytespider | ByteDance | 8,904 | 11.2% | 2,752 |
| ClaudeBot | Anthropic training | 8,501 | 10.7% | 2,708 |
| Google-Extended | Gemini training | 7,695 | 9.7% | 2,677 |
| meta-externalagent | Meta AI | 7,660 | 9.6% | 2,734 |
| Applebot-Extended | Apple AI training | 7,509 | 9.4% | 2,744 |
| ChatGPT-User | ChatGPT (user asks) | 5,321 | 6.7% | 2,668 |
| PerplexityBot | Perplexity search | 5,298 | 6.7% | 2,684 |
| OAI-SearchBot | ChatGPT search | 4,232 | 5.3% | 2,718 |
| Claude-SearchBot | Claude search | 3,870 | 4.9% | 2,771 |
| Claude-User | Claude (user asks) | 3,922 | 4.9% | 2,779 |
| Perplexity-User | Perplexity (user asks) | 3,915 | 4.9% | 2,783 |
| Bingbot | Bing / Copilot | 2,824 | 3.5% | 2,436 |
| Googlebot | Google Search | 2,257 | 2.8% | 2,188 |
"By a blanket rule only": the site never names the bot; a rule for every bot (User-agent: *) blocks it.
Well-known sites open to Google but closed to ChatGPT search
From their public robots.txt, highest-ranked first. Many do this on purpose; the point is that the two are separate choices.
facebook.com, instagram.com, twitter.com, amazon.com, x.com, pinterest.com, tiktok.com, baidu.com, msn.com, nytimes.com, cnn.com, bbc.com, meraki.com, bbc.co.uk, t.co, imdb.com, launchpad.net, weibo.com, amazon.co.uk, amazon.de, claude.ai, cnbc.com, kwai.net, quora.com, amazon.co.jp, threads.com, usatoday.com, telegraph.co.uk, dailymail.com, amazon.fr, amazon.ca, amazon.in, yelp.com, fb.com, dailymail.co.uk, amazon.es, pinimg.com, amazon.it, amazon.co.za, amazon.com.br
How we did it
- The list: the Tranco top 1 million, list ID 56Z8N, first 100,000 sites.
- One request per site for robots.txt (with a fallback to www.), and llms.txt only where robots.txt allowed us. Our crawler named itself and gave a contact address.
- Rules read by RFC 9309: a named group beats the catch-all group, the longest matching rule wins, Allow wins a tie, and * and $ patterns are supported. A 4xx robots.txt counts as allow all, a 5xx as block all.
- 79,574 sites answered. 20,426 names never answered (mostly CDN, DNS and app domains) and are left out.
- Limits: robots.txt shows what a site asks for, not what its firewall does. Some sites block AI bots at the network edge too.
Is your site one of them?
Agent Grade checks your robots.txt the same way, loads your page as each AI bot, and checks what attack agents look for first. Free.
Get my gradeBy Kyle Clouthier, Clouthier Simulation Labs. Quote freely with a link to this page.