Who runs it
Meta-ExternalAgent is run by Meta, according to Meta for Developers, Meta Web Crawlers.
What Meta says it is for
Meta describes Meta-ExternalAgent this way.
“The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” Meta for Developers, Meta Web Crawlers
Meta lists a separate search crawler for Meta AI, Meta-WebIndexer.
“Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses.” Meta for Developers, Meta Web Crawlers
Meta lists a separate fetcher for user requests, Meta-ExternalFetcher.
“The Meta-ExternalFetcher crawler fetches individual links at a user's request” Meta for Developers, Meta Web Crawlers
Meta says Meta-ExternalFetcher may bypass robots.txt rules.
“Accordingly, this crawler may bypass robots.txt rules.” Meta for Developers, Meta Web Crawlers
User-agent string and robots.txt token
User-agent string, from Meta (the page shows the first part before the link).
meta-externalagent/1.1From Meta for Developers, Meta Web Crawlers.
robots.txt token: meta-externalagent
Meta says blocking its crawlers is done with a robots.txt disallow.
“In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.” Meta for Developers, Meta Web Crawlers
Example rule (standard robots.txt syntax, written by us, not a vendor quote):
User-agent: meta-externalagent
Disallow: /
Does blocking it affect AI search visibility?
Meta does not say in its crawler documentation that blocking Meta-ExternalAgent affects visibility in Meta AI. For a separate crawler, Meta-WebIndexer, it says allowing it helps Meta cite and link to your content in Meta AI's responses.
How to find it in your logs
For request-level inspection, use your host, CDN or web-server logs, as in SearchIntel’s log study. Search the user-agent field for the name. On a server with shell access:
grep -i "meta-externalagent" access.log
Meta's page lists several crawlers with their own names (Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher, Meta-ExternalAds). Search for each name you want to count.
Google warns that user-agent strings can be spoofed, so verify the request before counting it.
What we saw in our own logs
These figures come from our post How to Track AI Crawlers on Your Website: A Week of Logs: one week of logs (19 to 25 October 2026) from two websites, a specialist research site and a smaller services site.
- SearchIntel’s October log study reports for services site, share of crawler requests: “Meta's AI crawler 30%”.
The post reports Meta's AI crawler on the services site only. It does not list Meta separately for the research site.
Read this as a snapshot: the post says to “treat the shares as a snapshot, not a benchmark or a trend”, and says “We used the bot labels Vercel exported”, so the labels are Vercel’s. Full method and limits are in the post.
Frequently asked questions
What is Meta-ExternalAgent?
Meta says its Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.
Does blocking Meta-ExternalAgent affect AI search visibility?
Meta does not say in its crawler documentation that blocking Meta-ExternalAgent affects visibility in Meta AI. For a separate crawler, Meta-WebIndexer, it says allowing it helps Meta cite and link to your content in Meta AI's responses.
How do I find Meta-ExternalAgent in my logs?
Search the user-agent field of your host, CDN or web-server logs for the name, for example with grep -i "meta-externalagent" access.log. Google warns that user-agent strings can be spoofed, so verify the request before counting it.
Changelog
Last updated 17 November 2026.
- 17 November 2026: Pages published.
All bots: what each AI bot does on your website.