Breach Intelligence Blog

Data-driven insights on cybersecurity breaches, vulnerabilities, and infrastructure signals.

Filtered by tag: AI crawlers · Clear filter

analysis

One in Five Websites Blocks an AI Crawler

We read the robots.txt of 110,744 websites and checked each against 19 known AI crawlers. About one in five blocks at least one. Among those that do, seven in ten block nearly every crawler, and the most-blocked bot is OpenAI's GPTBot.

15 Oct 2026 · 10 min read

Infrastructure Research Directory

Browse observed tech stacks, DNS posture, and open services for organisations. All data sourced from public authority records.

Browse Directory →

Free account

Show me any company's infrastructure

Observed tech stack, DNS posture, hosting geography, and change history. Free, no credit card. Set up in 30 seconds.

Show me the data →

Get GTM Intelligence by Email

Weekly research on breach disclosures, infrastructure shifts, and technical exposure signals matched to your market.