Which companies in United States block Google-Extended in robots.txt?

Name: Organisations in United States blocking Google-Extended in robots.txt
Creator: CipherCue

Google-Extended is Google's AI crawler. CipherCue reads the public robots.txt of tracked organisations and records, per named AI crawler, whether the site declares a rule disallowing it. These are the United States organisations observed blocking Google-Extended. Sign up free, no credit card, to see the full list.

114 United States organisations observed disallowing Google-Extended in their robots.txt

Refine Crawler Country

A sample of the organisations observed

A few of the United States organisations whose robots.txt disallows Google-Extended. Sign up free to filter and view the full list.

Show me Locked Locked Locked Locked Locked

See all 114 United States organisations blocking Google-Extended

Free, no credit card. Filter and view every United States organisation observed disallowing Google-Extended in robots.txt, alongside which other AI crawlers they block and their full observed infrastructure.

Show me the full list See pricing

Companies blocking Google-Extended in other markets

Canada France Germany Ireland Italy Netherlands Poland Spain United Kingdom

Other AI crawlers blocked in United States

GPTBot Amazonbot ClaudeBot Meta-ExternalAgent anthropic-ai Applebot-Extended ChatGPT-User cohere-ai FacebookBot Claude-Web PerplexityBot OAI-SearchBot DuckAssistBot Perplexity-User MistralAI-User Claude-SearchBot CCBot Bytespider

How this is observed: CipherCue reads the public robots.txt each organisation publishes and records the disallow rules it declares per named user-agent. A site is recorded as blocking Google-Extended when its robots.txt disallows the site root for that agent (or the * catch-all). robots.txt is a stated preference the crawler chooses whether to honour; this records what the site declares, not crawler behaviour. See the AI-crawler methodology.