Meta-ExternalAgent
Meta-ExternalAgent is Meta's AI crawler (Facebook, Instagram, WhatsApp). According to its official documentation, it goes through the web for use cases such as training foundation AI models, the Llama family, or improving products by indexing content directly. Allowing it helps Meta AI, the assistant built into its apps, learn about your business.
- User-agent
meta-externalagentmeta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)- Does it respect robots.txt?
- Yes
- Official documentation
- https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
How to allow it in your robots.txt
User-agent: meta-externalagent
Allow: /How to block it (not recommended)
User-agent: meta-externalagent
Disallow: /Frequently asked questions
Should I block Meta-ExternalAgent?
For a business, it's not advisable. Meta AI is built into WhatsApp, Instagram and Facebook, apps used by millions of potential customers. Blocking this bot reduces what those assistants know about your business.
Does Meta-ExternalAgent respect robots.txt?
Yes. Meta's documentation states you can control it with standard robots.txt rules using the user-agent "meta-externalagent".
How do I know if Meta-ExternalAgent visits my site?
Search for "meta-externalagent" in your server logs. Its requests are identified by that token followed by the version, e.g. meta-externalagent/1.1.