How to Block GPTBot and Other AI Crawlers in robots.txt
On , 64.3% of the sites we measured disallow at least one of the monitored AI bot tokens on the root path in robots.txt (based on 56 sites with a readable policy). The most frequently disallowed token was Applebot-Extended (31 of 56 sites). Methodology · All AI crawler records.
How to block GPTBot
To ask OpenAI not to collect your pages for training, add a group for its training crawler, GPTBot, to the robots.txt file at the root of your site:
User-agent: GPTBot
Disallow: /Disallowing GPTBot is a training opt-out; it does not take the site out of ChatGPT search. OpenAI describes three separate agents. GPTBot crawls content that may be used to train its models. OAI-SearchBot surfaces websites in ChatGPT search (sites that disallow it are not shown in search answers). ChatGPT-User opens a page when a person asks ChatGPT to. OpenAI says robots.txt rules may not apply to those user-initiated visits. A site that wants to stay in ChatGPT search while opting out of training disallows GPTBot and leaves OAI-SearchBot allowed. Source: OpenAI's crawler documentation, read on Oct 11, 2026.
In our sample, robots.txt disallows GPTBot at the root path on 24 of 56 sites with a readable decision.
What robots.txt is and is not
robots.txt is a public text file at the root of a site. It contains groups of rules addressed to crawlers by their product token (for example GPTBot or CCBot), with Allow and Disallow rules for paths. The format is standardised in RFC 9309. robots.txt asks crawlers to stay away; it cannot force them, and this page does not show whether they comply.
To publish rules of your own, you need to be able to put this file at the root of your domain. On shared hosting or a VPS you upload it yourself; site builders differ in what they let you change. Our hosting price tracker shows what the plans we check cost at purchase and at renewal, and our Website budget: domain + hosting + email adds up the domain, hosting and email payments over the years.
How we read the rules
For each site we fetch robots.txt and evaluate it for a fixed panel of 17 bot tokens on the path "/". The evaluation follows RFC 9309: groups whose user-agent matches the token exactly (case-insensitive) are merged; the "*" group applies only when no named group exists; path rules are case-sensitive, the longest matching path wins, and Allow wins when lengths are equal. A token that the file does not mention and that has no "*" restriction is recorded as allowed.
Only the root path counts as blocking. A rule that closes /search/ or /api/ to a bot but leaves "/" open is kept in the record but does not count. The per-site value is the share of the 17 tokens disallowed at "/".
Different tokens, different purposes
The tokens in the panel are not interchangeable. Some identify crawlers that collect training data. Some identify search or answer crawlers that fetch pages to cite them in an AI search product. Some identify fetches made on behalf of a user who asked an assistant to open a page. A few tokens, such as Google-Extended and Applebot-Extended, are not separate crawlers at all: they control whether content fetched by the ordinary crawler may be used for AI purposes. A site that blocks a training token but allows a user-fetch token has made a narrower decision than a site that blocks all of them.
Current counts by token
The count of sites that disallow each token at the root path, from the latest successful reading of each site. The token least often disallowed was OAI-SearchBot (11 of 56 sites).
| Bot token | Sites disallowing it at the root path | Sites with a readable decision |
|---|---|---|
| Applebot-Extended | 31 | 56 |
| Bytespider | 31 | 56 |
| ClaudeBot | 31 | 56 |
| CCBot | 30 | 56 |
| anthropic-ai | 26 | 56 |
| PerplexityBot | 25 | 56 |
| GPTBot | 24 | 56 |
| cohere-ai | 23 | 56 |
| meta-externalagent | 23 | 56 |
| Google-Extended | 22 | 56 |
| Amazonbot | 21 | 56 |
| Claude-User | 19 | 56 |
| Claude-SearchBot | 18 | 56 |
| Perplexity-User | 18 | 56 |
| ChatGPT-User | 16 | 56 |
| DuckAssistBot | 15 | 56 |
| OAI-SearchBot | 11 | 56 |
3 of 56 sites disallow every token in the panel, for example Reddit, USA Today, Yelp. 20 sites disallow none of them, for example Wikipedia, Best Buy, Booking.com, Britannica, Docker Hub, Etsy.
Unknown is not "allowed"
If robots.txt could not be read (an HTTP error, a challenge page, a malformed or truncated file), the policy is recorded as unknown with the reason and an empty value. We do not apply the crawler fallback rules for HTTP errors and present the result as the site's policy. An empty but successfully retrieved robots.txt means no restrictions. The homepage is checked separately: some sites return a challenge page to non-browser clients regardless of what robots.txt says, and that is shown as a separate result on each site page.
Limits
The 60 sites are an editorial sample of news, reference, retail, forum and service sites, not a random sample or a traffic ranking. Subdomains can have their own robots.txt. Whether content was used for training, and what agreements sites have with AI companies, is not part of this record. The history spans 22 observation date(s), from Sep 27, 2026 to Oct 18, 2026; a change in a site's rules appears in its history.