Meta-ExternalAgent: what it is and how to control it
Meta-ExternalAgent is Meta's web crawler, which Meta says collects content for uses such as training AI models and improving its products. Its robots.txt token is meta-externalagent, and Meta documents a Disallow rule to block it, taking up to 24 hours to apply. A separate agent, Meta-ExternalFetcher, handles user-requested fetches and may bypass robots.txt.
| Operator | Meta, for Meta AI and Meta products |
|---|---|
| Purpose | AI training |
| robots.txt token | meta-externalagent |
| User agent | meta-externalagent/1.1 |
| Follows robots.txt | Yes. Meta documents robots.txt as the way to control this crawler and says rule changes can take up to 24 hours to apply. |
| Runs JavaScript | Not documented |
| IP ranges | Not published |
| Documentation | Meta |
What Meta-ExternalAgent does
Meta-ExternalAgent crawls the web for uses such as training AI models, and Meta also says it indexes content to improve its products. Meta documents a second agent, Meta-ExternalFetcher, which fetches individual links when a user asks, and which Meta says may bypass robots.txt rules.
Should you block Meta-ExternalAgent?
Treat meta-externalagent as a training crawler and set your policy, knowing that Meta-ExternalFetcher user-initiated fetches may bypass robots.txt. Disallowing meta-externalagent stops this crawler's collection, including its use for training AI models. Meta documents no effect on other Meta crawlers; Meta-ExternalFetcher is a separate agent that may bypass robots.txt.
User-agent: meta-externalagent
Disallow: /User-agent: meta-externalagent
Allow: /Rules that apply
- Rule 02: Treat training bots and search bots as separate robots.txt decisions
- Rule 11: Don't rely on robots.txt to stop ChatGPT-User or Perplexity-User; they fetch on a user's behalf
- Rule 03: Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows
Questions
What is Meta-ExternalAgent?
Meta-ExternalAgent is a Meta crawler that, per Meta's documentation, crawls the web for use cases such as training AI models. Meta also says it indexes content to improve its products.
What is the Meta-ExternalAgent user agent?
Meta documents two strings: meta-externalagent/1.1 and meta-externalagent/1.1 followed by a link to its crawler documentation in parentheses. Meta does not publish an IP range list for this crawler.
How do I block Meta-ExternalAgent?
Add a robots.txt group with User-agent: meta-externalagent and Disallow: /. Meta says crawlers may cache robots.txt, so changes can take up to 24 hours to apply.
Does Meta-ExternalFetcher respect robots.txt?
Meta says Meta-ExternalFetcher may bypass robots.txt rules because it fetches individual links at a user's request. It is a separate agent from Meta-ExternalAgent.
Sources
- Meta Web Crawlers · Meta
Checked against the operators' documentation every Monday. Last reviewed Nov 2, 2026.