Menu

Medium (medium.com): AI Crawler Rules in robots.txt

GPTBot is disallowed by medium.com's robots.txt (checked ). In all, the file disallows 6 of the 17 AI bot tokens we monitor (35.3%). Observed Source · Methodology

What this means for you

robots.txt on this site disallows 6 of the 17 AI agents we monitor: 0 of 4 search and answer bots (none), 0 of 3 user-requested fetchers (none) and 6 of 10 other crawlers.

Search and user-requested agents may still read this site, so assistants that follow robots.txt can cite it; the disallowed agents are crawlers that collect pages for their operators, for example for model training or datasets.

To publish rules like these for your own site, see How to Block GPTBot and Other AI Crawlers in robots.txt. A host or CDN can also block AI crawlers on its own, which robots.txt does not show: our hosting checker names the provider behind any domain, and our hosting price tracker shows what the plans we check cost.

What robots.txt says

6 of 17 bots disallowed on /

robots.txt disallows 6 of 17 monitored bots on path /.

  • Amazonbot · Disallowed · Disallow: /
  • Applebot-Extended · Disallowed · Disallow: /
  • Bytespider · Disallowed · Disallow: /
  • ClaudeBot · Disallowed · Disallow: /
  • GPTBot · Disallowed · Disallow: /
  • meta-externalagent · Disallowed · Disallow: /
  • anthropic-ai · Allowed
  • CCBot · Allowed
  • ChatGPT-User · Allowed
  • Claude-SearchBot · Allowed
  • Claude-User · Allowed
  • cohere-ai · Allowed
  • DuckAssistBot · Allowed
  • Google-Extended · Allowed
  • OAI-SearchBot · Allowed
  • Perplexity-User · Allowed
  • PerplexityBot · Allowed
robots.txt excerpt
User-Agent: Amazonbot User-Agent: Applebot-Extended User-Agent: Bytespider User-Agent: ClaudeBot User-Agent: FacebookBot User-Agent: GoogleOther User-Agent: GPTBot User-Agent: meta-externalagent …

Homepage and robots.txt responses at our latest check

homepage HTTP 403 · not reachable · robots.txt HTTP 200
Homepage
HTTP 403 · not reachable
Challenge page
Detected (Cloudflare): Cloudflare: cf-mitigated=challenge; Cloudflare: challenge marker in body
robots.txt
HTTP 200

History and how we checked

25 checks since Sep 28, 2026 · no change

How we got this number

  • What we read: the robots.txt file at medium.com/robots.txt
  • Checked: ; tracked since Sep 28, 2026
  • Shown in: Switzerland
  • Checks so far: 25 checks on 21 days since Sep 28, 2026
  • What changed: no change since Sep 28, 2026
  • Page updated:

Changes

  1. — First check: 35.3%
Record history 25 records for 1 question
Last record
Shown in
Switzerland
TimeValueStatus
Oct 18, 202635.3%Observed
Oct 17, 202635.3%Observed
Oct 16, 202635.3%Observed
Oct 15, 202635.3%Observed
Oct 14, 202635.3%Observed
Oct 13, 202635.3%Observed
Oct 12, 202635.3%Observed
Oct 11, 202635.3%Observed
Oct 10, 202635.3%Observed
Oct 9, 202635.3%Observed
Oct 8, 202635.3%Observed
Oct 7, 202635.3%Observed
Oct 6, 202635.3%Observed
Oct 5, 202635.3%Observed
Oct 4, 202635.3%Observed
Oct 3, 202635.3%Observed
Oct 2, 202635.3%Observed
Oct 1, 202635.3%Observed
Sep 30, 202635.3%Observed
Sep 29, 202635.3%Observed
Sep 28, 202635.3%Observed
Sep 28, 202635.3%Observed
Sep 28, 202635.3%Observed
Sep 28, 202635.3%Observed
Sep 28, 202635.3%Observed
Download JSON Download CSV