Methodology: AI bot access policies
AI bot access policies
- Conditions
What we track. The robots.txt rules of a selected list of websites — news, reference, retail, forums and services — for a fixed panel of AI bot product tokens (GPTBot, ClaudeBot and others) at the root path /. The list is an editorial selection, not a random sample or a traffic ranking.
What counts as a value. For each token, whether the site's robots.txt disallows it at /, by the rules of RFC 9309: a group naming the token applies first, the * group only when no group names it; the longest matching path wins, and Allow wins on equal length. A site's value is the share of the monitored tokens it disallows. Rules for other paths are kept but do not count as disallowing /. We also note whether the homepage opened or showed an interactive check page instead.
What it is not. robots.txt asks crawlers to stay away; it cannot force them, and this record does not show whether they comply or whether data is used for training.
When a value is Unknown. When robots.txt cannot be read or is malformed, the policy is Unknown — never a guessed permission or denial. An empty robots.txt means no restrictions. The homepage is not read where robots.txt disallows / for all bots or the site's terms restrict reading.
How often. Every site is read once a day.
AI Bot Policy Watch. The share of sites with a readable robots.txt on the date that disallow at least one monitored token at /, with the number of sites (N); the share for each token uses the same N.
Version. Version 1 of this method, in effect since 27 September 2026; description revised 15 October 2026.
- Coverage
- On : 60 websites.
- Region
- Switzerland