Check robots.txt for AI Crawlers: GPTBot, ClaudeBot and 15 More

Enter a domain to see what its robots.txt asks of 17 AI bot tokens, from GPTBot and OAI-SearchBot to ClaudeBot, PerplexityBot and Google-Extended: Allowed or Disallowed at the root path /, the line that decides, and what it means for AI search and for training. We read one file, robots.txt, and nothing else on the site, then set the answer next to the selected sites we check every day.

• One file. We request https://<domain>/robots.txt once, from Cloudflare’s network where this tool runs, following up to 5 redirects as crawlers do, within 4 seconds and up to 512 KiB. We never request the site’s pages.
• 17 AI bot tokens. Search and answer bots: OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot; User-requested fetchers: ChatGPT-User, Claude-User, Perplexity-User; Other crawlers and control tokens: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Bytespider, CCBot, Applebot-Extended, meta-externalagent, Amazonbot, cohere-ai.
• The rule that decides. Each token is evaluated under RFC 9309 with the same code as our daily records of selected sites. A group that names the token is used instead of the "*" group. Among the rules that apply, the longest matching path wins and Allow wins a tie. The * and $ wildcards and percent-encoding are handled as the standard says. You see the line that decided.
• One path: "/". The answer is for the root path, as in our records. A rule that closes only part of a site (for example /search/) is shown in the details, but it does not make a token Disallowed.
• "Disallow: /" for everyone. If the "User-agent: *" group disallows the whole site, we say so, and whether Googlebot and Bingbot have a group of their own: such a file asks ordinary search engines to stay away too.
• Comparison. Next to each token, how many of the selected sites we check every day disallow it (table below).

Disallowed means the site’s robots.txt asks that bot not to fetch the root path. Allowed means nothing in the file stops it there. robots.txt is a request: a bot that ignores it can still fetch pages, and this check cannot tell whether any bot obeys it. If the file cannot be read, every token is Unknown — never Allowed.

• Search and answer bots. They fetch pages to show or cite them in an AI search product. If they are disallowed, an assistant that follows robots.txt may not read or cite the site when a user asks about it.
• User-requested fetchers. They open a page when a person asks an assistant to. Disallowing them asks assistants not to open the site even on request; OpenAI, for one, says robots.txt rules may not apply to such visits (see the guide below).
• Other crawlers and control tokens. Crawlers that collect pages for their operators, for example for model training or datasets. Google-Extended and Applebot-Extended are not separate crawlers: they say whether content fetched by the ordinary crawler may be used for AI.

• robots.txt is a request. We do not test whether any bot obeys it, and we do not know what was collected before.
• Blocking at the server, firewall or CDN (for example a rule that refuses AI user agents) is not visible in robots.txt and is not checked; neither are meta robots tags, X-Robots-Tag headers or other files.
• One path, "/". Rules for other paths are listed in the details but do not change the answer.
• Only the host you enter: a subdomain has its own robots.txt. Redirects are followed, as crawlers do.
• The file is fetched by this tool, which runs on Cloudflare’s network. A site can serve a different file, or a challenge page, to different clients. A challenge page, a server error, a refused request (such as HTTP 401, 403 or 429) or a file we cannot read gives Unknown.
• No file at all (HTTP 404 or 410) means no restrictions under RFC 9309, so every token is Allowed. In our daily records any answer other than a readable file stays unknown, because there we compare the same sites over time.
• 17 tokens only; other AI bots are not evaluated. Content-Signal lines are quoted as they are.
• We do not watch your site. After you change robots.txt, run the check again.