Meta-ExternalFetcher
Meta-ExternalFetcher is Meta's bot that downloads individual links when a user requests them and helps its "agentic" AIs browse websites to complete tasks. It's not a mass crawler: it acts on request. If you block it, Meta's assistants won't be able to check your site when a user asks for it, and your content will be left out of those answers.
- User-agent
meta-externalfetchermeta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)- Does it respect robots.txt?
- Partially
- Official documentation
- https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
How to allow it in your robots.txt
User-agent: meta-externalfetcher
Allow: /How to block it (not recommended)
User-agent: meta-externalfetcher
Disallow: /Frequently asked questions
Should I block Meta-ExternalFetcher?
It's not advisable. Its visits respond to real user requests within Meta's products. Blocking it only makes those users get answers without your content.
Does Meta-ExternalFetcher respect robots.txt?
Partially. Meta's documentation warns that it can bypass robots.txt rules because its downloads are directly requested by a user, similar to OpenAI's ChatGPT-User.
How do I know if Meta-ExternalFetcher visits my site?
Search for "meta-externalfetcher" in your server logs. Every appearance indicates a user of Meta's products requested content from your site through its AI.