AI crawler rules and post-quantum TLS checks
Two sets of technical checks that anyone can repeat: which AI crawlers selected websites ask to stay away in robots.txt, and which hosts support post-quantum key exchange in TLS. Latest check: . How we check
AI crawler rules in robots.txt
36 of 56 selected sites disallow at least one of the 17 AI crawler tokens we monitor in robots.txt (path /), as of .
Which sites disallow which AI crawlers · How to block GPTBot and other AI crawlers
For example: BBC, GitHub, The New York Times, Wikipedia.
Post-quantum TLS
43 of the 52 hosts with a valid result negotiated the hybrid post-quantum key exchange X25519MLKEM768 on ; 7 hosts gave no valid result on that check; 1 more shows the last confirmed result from an earlier day. We keep every result with its date.
Which hosts support hybrid post-quantum TLS
For example: BBC, Cloudflare, GitHub, Wikipedia.
These checks sit next to our price records: what web hosting costs at renewal and what domains cost to register and renew, read from the providers' own pages with the same dated method.