Our post How to Track AI Crawlers on Your Website: A Week of Logs covers how to read your logs and what a week of logs from two websites showed. The pages below take it one bot at a time.
The bots
| Bot | Run by | Role |
|---|---|---|
GPTBot | OpenAI | Training crawler |
OAI-SearchBot | OpenAI | Search crawler |
ChatGPT-User | OpenAI | User-initiated fetch |
ClaudeBot | Anthropic | Training crawler |
Claude-User | Anthropic | User-initiated fetch |
Claude-SearchBot | Anthropic | Search crawler |
PerplexityBot | Perplexity | Search crawler |
Perplexity-User | Perplexity | User-initiated fetch |
ExaSearchBot | Exa | Search crawler |
Googlebot and Google-Extended | Search crawler and AI training control | |
Applebot and Applebot-Extended | Apple | Search crawler and AI training control |
Meta-ExternalAgent | Meta | Training and indexing crawler |
Amazonbot | Amazon | Product and AI training crawler |
CCBot | Common Crawl | Open web archive crawler |
Frequently asked questions
Where do the log figures on these pages come from?
They are the figures already published in our post How to Track AI Crawlers on Your Website: A Week of Logs: one week of logs (19 to 25 October 2026) from two websites.
Can I rely on the user-agent string alone?
No. Google says: “The HTTP user agent string can be spoofed.” (Google Search Central, google-common-crawlers page). Check the request against the vendor's published method before you count it.
Do these pages tell me whether to block a bot?
No. Each page says what the company itself says about blocking and leaves the decision to you.
Changelog
Last updated 17 November 2026.
- 17 November 2026: Pages published.