Menu

Forbes (forbes.com): AI Crawler Rules in robots.txt

GPTBot is disallowed by forbes.com's robots.txt (checked ). In all, the file disallows 10 of the 17 AI bot tokens we monitor (58.8%). Observed Source · Methodology

Last confirmed Oct 17, 2026; no reading on Oct 18, 2026 (1 check without a reading). We keep showing the last confirmed value for up to 3 days after it was confirmed; after that the value is shown as unknown until a new reading.

What this means for you

robots.txt on this site disallows 10 of the 17 AI agents we monitor: 1 of 4 search and answer bots (PerplexityBot), 0 of 3 user-requested fetchers (none) and 9 of 10 other crawlers.

An assistant that follows robots.txt may not read or cite this site when a user asks about it, because the disallowed agents are the ones that fetch pages for answers.

To publish rules like these for your own site, see How to Block GPTBot and Other AI Crawlers in robots.txt. A host or CDN can also block AI crawlers on its own, which robots.txt does not show: our hosting checker names the provider behind any domain, and our hosting price tracker shows what the plans we check cost.

What robots.txt says

10 of 17 bots disallowed on /

robots.txt disallows 10 of 17 monitored bots on path /.

  • Amazonbot · Disallowed · Disallow: /
  • anthropic-ai · Disallowed · Disallow: /
  • Applebot-Extended · Disallowed · Disallow: /
  • Bytespider · Disallowed · Disallow: /
  • CCBot · Disallowed · Disallow: /
  • ClaudeBot · Disallowed · Disallow: /
  • cohere-ai · Disallowed · Disallow: /
  • GPTBot · Disallowed · Disallow: /
  • meta-externalagent · Disallowed · Disallow: /
  • PerplexityBot · Disallowed · Disallow: /
  • ChatGPT-User · Allowed
  • Claude-SearchBot · Allowed · Allow: /
  • Claude-User · Allowed · Allow: /
  • DuckAssistBot · Allowed
  • Google-Extended · Allowed
  • OAI-SearchBot · Allowed · Allow: /
  • Perplexity-User · Allowed
robots.txt excerpt
User-agent: GPTBot Disallow: / User-Agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Disallow: / User-agent: AhrefsBot User-agent: AhrefsSiteAudit User-Agent: Claude-User …

Homepage and robots.txt responses at our latest check

homepage HTTP 200 · reachable · robots.txt HTTP 200
Homepage
HTTP 200 · reachable
Challenge page
Not detected
robots.txt
HTTP 200

History and how we checked

11 checks since Sep 28, 2026 · no change

How we got this number

  • What we read: the robots.txt file at www.forbes.com/robots.txt
  • Checked: Last confirmed ; no reading on ; tracked since Sep 28, 2026
  • Shown in: Switzerland
  • Checks so far: 11 checks on 10 days since Sep 28, 2026
  • What changed: no change since Sep 28, 2026
  • Page updated:

Changes

  1. — First check: 58.8%
Record history 23 records for 1 question
Last record
Shown in
Switzerland
TimeValueStatus
Oct 17, 202658.8%Observed
Oct 16, 202658.8%Observed
Oct 15, 2026UnknownUnknown
Oct 14, 202658.8%Observed
Oct 13, 202658.8%Observed
Oct 12, 2026UnknownUnknown
Oct 11, 2026UnknownUnknown
Oct 10, 2026UnknownUnknown
Oct 9, 202658.8%Observed
Oct 8, 2026UnknownUnknown
Oct 7, 202658.8%Observed
Oct 6, 202658.8%Observed
Oct 5, 202658.8%Observed
Oct 4, 2026UnknownUnknown
Oct 3, 2026UnknownUnknown
Oct 2, 2026UnknownUnknown
Oct 1, 202658.8%Observed
Sep 30, 2026UnknownUnknown
Sep 29, 2026UnknownUnknown
Sep 28, 2026UnknownUnknown
Sep 28, 202658.8%Observed
Sep 28, 2026UnknownUnknown
Sep 28, 202658.8%Observed
Download JSON Download CSV