CIPHERCUE

← AI crawler policy

Which companies block Meta-ExternalAgent in robots.txt?

Meta-ExternalAgent is Meta's AI crawler. CipherCue reads the public robots.txt of tracked organisations and records, per named AI crawler, whether the site declares a rule disallowing it. These are the organisations observed blocking Meta-ExternalAgent. Sign up free, no credit card, to see the full list.

15,812 organisations observed disallowing Meta-ExternalAgent in their robots.txt

A sample of the organisations observed

A few of the organisations whose robots.txt disallows Meta-ExternalAgent. Sign up free to filter and view the full list.

Amazon.com, Inc.
US
Consumer Services
www.amazon.com
BidX1
IE
Property Marketplace
bidx1.com
Cvb Financial Corp
US
Financials
financialreports.eu
Facebook
CN
Manufacturing
facebook.com
Proofpoint
US
Security
proofpoint.com
SoundCloud
DE
Technology
soundcloud.com
t-online.de
DE
Telecommunication
t-online.de
Trustpilot
GB
Technology
trustpilot.com
Urząd Miejski w Jaworznie
PL
Public Sector
www.jaworzno.pl
 Show me Locked Locked Locked Locked Locked

See all 15,812 organisations blocking Meta-ExternalAgent

Free, no credit card. Filter and view every organisation observed disallowing Meta-ExternalAgent in robots.txt, alongside which other AI crawlers they block and their full observed infrastructure.

Show me the full list See pricing

Companies blocking other AI crawlers

GPTBot (17,401) Amazonbot (16,355) ClaudeBot (16,305) Google-Extended (15,535) anthropic-ai (15,395) Applebot-Extended (15,222) ChatGPT-User (14,998) cohere-ai (14,896) FacebookBot (14,796) Claude-Web (14,708) PerplexityBot (14,700) OAI-SearchBot (14,139) DuckAssistBot (13,920) Perplexity-User (13,790) Claude-SearchBot (13,766) MistralAI-User (13,742) CCBot (5,503) Bytespider (5,069)
How this is observed: CipherCue reads the public robots.txt each organisation publishes and records the disallow rules it declares per named user-agent. A site is recorded as blocking Meta-ExternalAgent when its robots.txt disallows the site root for that agent (or the * catch-all). robots.txt is a stated preference the crawler chooses whether to honour; this records what the site declares, not crawler behaviour. See the AI-crawler methodology.