AI Crawler Access: Which Sites Disallow GPTBot and Other AI Bots
We read the robots.txt of 60 selected websites every day and record which of the 17 AI crawler tokens we monitor each one disallows. robots.txt asks crawlers to stay away; it cannot force them, and this page does not show whether they comply. 36 of 56 selected sites disallow at least one of the 17 AI crawler tokens we monitor in robots.txt (path /), as of . The figure counts the 56 sites whose robots.txt could be read at that check; the other 4 gave no readable file. Latest check: .
Caliper Press AI Bot Policy Watch
64.3%
Fraction of measured sites disallowing at least one monitored bot on the root path on the date. based on 56 · range 0% – 100% · . Methodology
Who is in this figure (56)
- Amazon
- Ars Technica
- Associated Press
- BBC
- Best Buy
- Bloomberg
- Booking.com
- Britannica
- Business Insider
- CNN
- Dictionary.com
- Docker Hub
- eBay
- Etsy
- Financial Times
- Fox News
- GitHub
- GitLab
- Hacker News
- Hugging Face
- IKEA
- IMDb
- Los Angeles Times
- MDN Web Docs
- Medium
- Merriam-Webster
- Nature
- NPR
- PyPI
- Quora
- Reuters
- Science
- Shopify
- Substack
- Target
- TechCrunch
- The Atlantic
- The Economist
- The Guardian
- The New York Times
- The Verge
- The Wall Street Journal
- The Washington Post
- Time
- Tripadvisor
- Trustpilot
- USA Today
- Walmart
- WebMD
- Wikipedia
- Wiktionary
- Wired
- Yelp
- Zillow
What changed this week 1 value change across 60 sites checked in the 7 days to Oct 18, 2026
- CNN: 94.1% → 88.2% ·
All websites we check
Latest checks: Oct 17–18, 2026. Rows checked on an earlier day show their date.
| Name | Share of AI bots disallowed | Status |
|---|---|---|
| Amazon | 82.4% | Observed |
| Ars Technica | 82.4% | Observed |
| Associated Press | 58.8% | Observed |
| BBC | 82.4% | Observed |
| Best Buy | 0% | Observed |
| Bloomberg | 70.6% | Observed |
| Booking.com | 0% | Observed |
| Britannica | 0% | Observed |
| Business Insider | 23.5% | Observed |
| CNN | 88.2% | Observed |
Show all 57 sites 47 more
| Name | Share of AI bots disallowed | Status |
|---|---|---|
| Dictionary.com | 17.7% | Observed |
| Docker Hub | 0% | Observed |
| eBay | 52.9% | Observed |
| Etsy | 0% | Observed |
| Financial Times | 58.8% | Observed |
| Forbes (last confirmed Oct 17, 2026) | 58.8% (17 Oct) | Observed |
| Fox News | 0% | Observed |
| GitHub | 5.9% | Observed |
| GitLab | 0% | Observed |
| Hacker News | 0% | Observed |
| Hugging Face | 0% | Observed |
| IKEA | 0% | Observed |
| IMDb | 88.2% | Observed |
| 88.2% | Observed | |
| Los Angeles Times | 17.7% | Observed |
| MDN Web Docs | 0% | Observed |
| Medium | 35.3% | Observed |
| Merriam-Webster | 0% | Observed |
| Nature | 70.6% | Observed |
| NPR | 76.5% | Observed |
| PyPI | 0% | Observed |
| Quora | 52.9% | Observed |
| 100% | Observed | |
| Reuters | 88.2% | Observed |
| Science | 35.3% | Observed |
| Shopify | 0% | Observed |
| Substack | 0% | Observed |
| Target | 0% | Observed |
| TechCrunch | 52.9% | Observed |
| The Atlantic | 76.5% | Observed |
| The Economist | 76.5% | Observed |
| The Guardian | 64.7% | Observed |
| The New York Times | 94.1% | Observed |
| The Verge | 82.4% | Observed |
| The Wall Street Journal | 82.4% | Observed |
| The Washington Post | 47.1% | Observed |
| Time | 0% | Observed |
| Tripadvisor | 52.9% | Observed |
| Trustpilot | 41.2% | Observed |
| USA Today | 100% | Observed |
| Walmart | 0% | Observed |
| WebMD | 17.7% | Observed |
| Wikipedia | 0% | Observed |
| Wiktionary | 0% | Observed |
| Wired | 76.5% | Observed |
| Yelp | 100% | Observed |
| Zillow | 17.7% | Observed |
Not recorded at the latest check on Oct 18, 2026: 3 sites
| Name | Status | Last attempt |
|---|---|---|
| npm | Unknown | Oct 18, 2026 |
| Stack Exchange | Unknown | Oct 18, 2026 |
| Stack Overflow | Unknown | Oct 18, 2026 |
How to cite this CC BY 4.0
Caliper Press, “AI Bot Policy Watch”, based on 56 sites, as of Oct 18, 2026; methodology; licence CC BY 4.0.