All AI bots
AI Bots

What Meta-ExternalAgent Does on Your Website

What Meta says it is for, how to spot it in your logs, and what we saw in ours.

In short: Meta says its Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.

Who runs it

Meta-ExternalAgent is run by Meta, according to Meta for Developers, Meta Web Crawlers.

What Meta says it is for

Meta describes Meta-ExternalAgent this way.

“The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” Meta for Developers, Meta Web Crawlers

Meta lists a separate search crawler for Meta AI, Meta-WebIndexer.

“Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses.” Meta for Developers, Meta Web Crawlers

Meta lists a separate fetcher for user requests, Meta-ExternalFetcher.

“The Meta-ExternalFetcher crawler fetches individual links at a user's request” Meta for Developers, Meta Web Crawlers

Meta says Meta-ExternalFetcher may bypass robots.txt rules.

“Accordingly, this crawler may bypass robots.txt rules.” Meta for Developers, Meta Web Crawlers

User-agent string and robots.txt token

User-agent string, from Meta (the page shows the first part before the link).

meta-externalagent/1.1

From Meta for Developers, Meta Web Crawlers.

robots.txt token: meta-externalagent

Meta says blocking its crawlers is done with a robots.txt disallow.

“In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.” Meta for Developers, Meta Web Crawlers

Example rule (standard robots.txt syntax, written by us, not a vendor quote):

User-agent: meta-externalagent
Disallow: /

Does blocking it affect AI search visibility?

Meta does not say in its crawler documentation that blocking Meta-ExternalAgent affects visibility in Meta AI. For a separate crawler, Meta-WebIndexer, it says allowing it helps Meta cite and link to your content in Meta AI's responses.

How to find it in your logs

For request-level inspection, use your host, CDN or web-server logs, as in SearchIntel’s log study. Search the user-agent field for the name. On a server with shell access:

grep -i "meta-externalagent" access.log

Meta's page lists several crawlers with their own names (Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher, Meta-ExternalAds). Search for each name you want to count.

Google warns that user-agent strings can be spoofed, so verify the request before counting it.

What we saw in our own logs

These figures come from our post How to Track AI Crawlers on Your Website: A Week of Logs: one week of logs (19 to 25 October 2026) from two websites, a specialist research site and a smaller services site.

The post reports Meta's AI crawler on the services site only. It does not list Meta separately for the research site.

Read this as a snapshot: the post says to “treat the shares as a snapshot, not a benchmark or a trend”, and says “We used the bot labels Vercel exported”, so the labels are Vercel’s. Full method and limits are in the post.

Frequently asked questions

What is Meta-ExternalAgent?

Meta says its Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.

Does blocking Meta-ExternalAgent affect AI search visibility?

Meta does not say in its crawler documentation that blocking Meta-ExternalAgent affects visibility in Meta AI. For a separate crawler, Meta-WebIndexer, it says allowing it helps Meta cite and link to your content in Meta AI's responses.

How do I find Meta-ExternalAgent in my logs?

Search the user-agent field of your host, CDN or web-server logs for the name, for example with grep -i "meta-externalagent" access.log. Google warns that user-agent strings can be spoofed, so verify the request before counting it.

Changelog

Last updated 17 November 2026.

All bots: what each AI bot does on your website.

See Whether AI Recommends Your Brand

Run a free check across ChatGPT, Claude, Gemini, Perplexity and Google AI Mode. See your score, your platform breakdown, and the competitors AI recommends instead.

Related bots