Substack (substack.com): AI Crawler Rules in robots.txt
GPTBot is allowed by substack.com's robots.txt (checked ). In all, the file disallows 0 of the 17 AI bot tokens we monitor (0%). Observed Source · Methodology
What this means for you
robots.txt on this site disallows 0 of the 17 AI agents we monitor: 0 of 4 search and answer bots (none), 0 of 3 user-requested fetchers (none) and 0 of 10 other crawlers.
None of the monitored agents is disallowed, so assistants that follow robots.txt can read and cite this site.
What robots.txt says
0 of 17 bots disallowed on /
robots.txt disallows 0 of 17 monitored bots on path /.
- Amazonbot · Allowed
- anthropic-ai · Allowed
- Applebot-Extended · Allowed
- Bytespider · Allowed
- CCBot · Allowed
- ChatGPT-User · Allowed
- Claude-SearchBot · Allowed
- Claude-User · Allowed
- ClaudeBot · Allowed
- cohere-ai · Allowed
- DuckAssistBot · Allowed
- Google-Extended · Allowed
- GPTBot · Allowed
- meta-externalagent · Allowed
- OAI-SearchBot · Allowed
- Perplexity-User · Allowed
- PerplexityBot · Allowed
robots.txt excerpt
User-agent: * Disallow: /action/ Disallow: /publish Disallow: /sign-in Disallow: /channel-frame Disallow: /session-attribution-frame Disallow: /visited-surface-frame Disallow: /feed/private …
Homepage and robots.txt responses at our latest check
homepage HTTP 200 · not reachable · robots.txt HTTP 200
- Homepage
- HTTP 200 · not reachable
- Challenge page
- Detected (Cloudflare): Cloudflare: challenge marker in body
- robots.txt
- HTTP 200
History and how we checked
24 checks since Sep 28, 2026 · no change
How we got this number
- What we read: the robots.txt file at substack.com/robots.txt
- Checked: ; tracked since Sep 28, 2026
- Shown in: Switzerland
- Checks so far: 24 checks on 21 days since Sep 28, 2026
- What changed: no change since Sep 28, 2026
- Page updated:
Changes
- — First check: 0%
Record history 24 records for 1 question
- Last record
- Shown in
- Switzerland
| Time | Value | Status |
|---|---|---|
| Oct 18, 2026 | 0% | Observed |
| Oct 17, 2026 | 0% | Observed |
| Oct 16, 2026 | 0% | Observed |
| Oct 15, 2026 | 0% | Observed |
| Oct 14, 2026 | 0% | Observed |
| Oct 13, 2026 | 0% | Observed |
| Oct 12, 2026 | 0% | Observed |
| Oct 11, 2026 | 0% | Observed |
| Oct 10, 2026 | 0% | Observed |
| Oct 9, 2026 | 0% | Observed |
| Oct 8, 2026 | 0% | Observed |
| Oct 7, 2026 | 0% | Observed |
| Oct 6, 2026 | 0% | Observed |
| Oct 5, 2026 | 0% | Observed |
| Oct 4, 2026 | 0% | Observed |
| Oct 3, 2026 | 0% | Observed |
| Oct 2, 2026 | 0% | Observed |
| Oct 1, 2026 | 0% | Observed |
| Sep 30, 2026 | 0% | Observed |
| Sep 29, 2026 | 0% | Observed |
| Sep 28, 2026 | 0% | Observed |
| Sep 28, 2026 | 0% | Observed |
| Sep 28, 2026 | 0% | Observed |
| Sep 28, 2026 | 0% | Observed |