Agent Grade / AI bots by site / wsj.com
Does wsj.com block ChatGPT?
wsj.com is open to Google but blocks Claude search, Perplexity. ChatGPT search is allowed, and OpenAI's training bot is allowed. From its public robots.txt, checked 23 to 24 October 2026. It ranks #384 on the Tranco list of top sites.
| Bot | What it does | Home page | Deciding rule |
|---|---|---|---|
| Googlebot | Google Search | Allowed | User-agent: Googlebot / Allow: / |
| Bingbot | Bing / Copilot | Allowed | User-agent: Bingbot / Allow: / |
| OAI-SearchBot | ChatGPT search | Allowed | User-agent: OAI-SearchBot / Allow: / |
| ChatGPT-User | ChatGPT (user asks) | Allowed | User-agent: ChatGPT-User / Allow: / |
| GPTBot | OpenAI training | Allowed | User-agent: GPTBot / Allow: / |
| Claude-SearchBot | Claude search | Blocked (blanket rule) | User-agent: * / Disallow: / |
| Claude-User | Claude (user asks) | Blocked (blanket rule) | User-agent: * / Disallow: / |
| ClaudeBot | Anthropic training | Blocked (blanket rule) | User-agent: * / Disallow: / |
| PerplexityBot | Perplexity search | Blocked (blanket rule) | User-agent: * / Disallow: / |
| Perplexity-User | Perplexity (user asks) | Blocked (blanket rule) | User-agent: * / Disallow: / |
| Google-Extended | Gemini training | Blocked (blanket rule) | User-agent: * / Disallow: / |
| Applebot-Extended | Apple AI training | Blocked (blanket rule) | User-agent: * / Disallow: / |
| CCBot | Common Crawl | Blocked (blanket rule) | User-agent: * / Disallow: / |
| meta-externalagent | Meta AI | Blocked (blanket rule) | User-agent: * / Disallow: / |
| Bytespider | ByteDance | Blocked (blanket rule) | User-agent: * / Disallow: / |
"Blanket rule": wsj.com never names that bot; a rule for every bot (User-agent: *) applies. llms.txt: No. robots.txt is what a site asks bots to do; some sites also block bots at the network edge.
Questions
Does wsj.com block ChatGPT search?
No. In its robots.txt as checked 23 to 24 October 2026, wsj.com lets OAI-SearchBot, the crawler behind ChatGPT search, read its home page.
Does wsj.com block GPTBot?
No. wsj.com lets GPTBot, OpenAI's training crawler, read its home page.
Does wsj.com block Claude?
ClaudeBot (Anthropic training): blocked. Claude-SearchBot (Claude search): blocked. Claude-User (fetches for a person): blocked.
Does wsj.com have an llms.txt file?
We did not find an llms.txt file on wsj.com, or its robots.txt did not allow us to fetch it.
What about your site?
Agent Grade checks the same 15 bots against your robots.txt, loads your page as each AI bot, and checks what attack agents look for first. Free, under a minute.
Grade my site