GitHub (github.com): AI Crawler Rules in robots.txt
GPTBot is allowed by github.com's robots.txt (checked ). In all, the file disallows 1 of the 17 AI bot tokens we monitor (5.9%). Observed Source · Methodology
What this means for you
robots.txt on this site disallows 1 of the 17 AI agents we monitor: 0 of 4 search and answer bots (none), 0 of 3 user-requested fetchers (none) and 1 of 10 other crawlers.
Search and user-requested agents may still read this site, so assistants that follow robots.txt can cite it; the disallowed agents are crawlers that collect pages for their operators, for example for model training or datasets.
What robots.txt says
1 of 17 bots disallowed on /
robots.txt disallows 1 of 17 monitored bots on path /.
- Bytespider · Disallowed ·
Disallow: / - Amazonbot · Allowed
- anthropic-ai · Allowed
- Applebot-Extended · Allowed
- CCBot · Allowed
- ChatGPT-User · Allowed
- Claude-SearchBot · Allowed
- Claude-User · Allowed
- ClaudeBot · Allowed
- cohere-ai · Allowed
- DuckAssistBot · Allowed
- Google-Extended · Allowed
- GPTBot · Allowed
- meta-externalagent · Allowed
- OAI-SearchBot · Allowed
- Perplexity-User · Allowed
- PerplexityBot · Allowed
robots.txt excerpt
User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: PerplexityBot Allow: /*?tab=achievements&achievement=* Allow: /marketplace/*?tab=readme$ …
Homepage and robots.txt responses at our latest check
homepage HTTP 200 · reachable · robots.txt HTTP 200
- Homepage
- HTTP 200 · reachable
- Challenge page
- Not detected
- robots.txt
- HTTP 200
History and how we checked
25 checks since Sep 28, 2026 · no change
How we got this number
- What we read: the robots.txt file at github.com/robots.txt
- Checked: ; tracked since Sep 28, 2026
- Shown in: Switzerland
- Checks so far: 25 checks on 21 days since Sep 28, 2026
- What changed: no change since Sep 28, 2026
- Page updated:
Changes
- — First check: 5.9%
Record history 25 records for 1 question
- Last record
- Shown in
- Switzerland
| Time | Value | Status |
|---|---|---|
| Oct 18, 2026 | 5.9% | Observed |
| Oct 17, 2026 | 5.9% | Observed |
| Oct 16, 2026 | 5.9% | Observed |
| Oct 15, 2026 | 5.9% | Observed |
| Oct 14, 2026 | 5.9% | Observed |
| Oct 13, 2026 | 5.9% | Observed |
| Oct 12, 2026 | 5.9% | Observed |
| Oct 11, 2026 | 5.9% | Observed |
| Oct 10, 2026 | 5.9% | Observed |
| Oct 9, 2026 | 5.9% | Observed |
| Oct 8, 2026 | 5.9% | Observed |
| Oct 7, 2026 | 5.9% | Observed |
| Oct 6, 2026 | 5.9% | Observed |
| Oct 5, 2026 | 5.9% | Observed |
| Oct 4, 2026 | 5.9% | Observed |
| Oct 3, 2026 | 5.9% | Observed |
| Oct 2, 2026 | 5.9% | Observed |
| Oct 1, 2026 | 5.9% | Observed |
| Sep 30, 2026 | 5.9% | Observed |
| Sep 29, 2026 | 5.9% | Observed |
| Sep 28, 2026 | 5.9% | Observed |
| Sep 28, 2026 | 5.9% | Observed |
| Sep 28, 2026 | 5.9% | Observed |
| Sep 28, 2026 | 5.9% | Observed |
| Sep 28, 2026 | 5.9% | Observed |