Back to the log study
AI Bots

AI Bots: What Each One Does on Your Website

Which bots visit your site, who runs them, and what each company says about them.

In short: One page per AI bot: who runs it, what it is for, how to spot it in your logs, and what blocking it does to your AI search visibility.

Our post How to Track AI Crawlers on Your Website: A Week of Logs covers how to read your logs and what a week of logs from two websites showed. The pages below take it one bot at a time.

The bots

BotRun byRole
GPTBotOpenAITraining crawler
OAI-SearchBotOpenAISearch crawler
ChatGPT-UserOpenAIUser-initiated fetch
ClaudeBotAnthropicTraining crawler
Claude-UserAnthropicUser-initiated fetch
Claude-SearchBotAnthropicSearch crawler
PerplexityBotPerplexitySearch crawler
Perplexity-UserPerplexityUser-initiated fetch
ExaSearchBotExaSearch crawler
Googlebot and Google-ExtendedGoogleSearch crawler and AI training control
Applebot and Applebot-ExtendedAppleSearch crawler and AI training control
Meta-ExternalAgentMetaTraining and indexing crawler
AmazonbotAmazonProduct and AI training crawler
CCBotCommon CrawlOpen web archive crawler

Frequently asked questions

Where do the log figures on these pages come from?

They are the figures already published in our post How to Track AI Crawlers on Your Website: A Week of Logs: one week of logs (19 to 25 October 2026) from two websites.

Can I rely on the user-agent string alone?

No. Google says: “The HTTP user agent string can be spoofed.” (Google Search Central, google-common-crawlers page). Check the request against the vendor's published method before you count it.

Do these pages tell me whether to block a bot?

No. Each page says what the company itself says about blocking and leaves the decision to you.

Changelog

Last updated 17 November 2026.

See Whether AI Recommends Your Brand

Run a free check across ChatGPT, Claude, Gemini, Perplexity and Google AI Mode. See your score, your platform breakdown, and the competitors AI recommends instead.

Start with the bots we hear about most