All AI bots
AI Bots

What ClaudeBot Does on Your Website

What Anthropic says it is for, how to spot it in your logs, and what we saw in ours.

In short: Anthropic says ClaudeBot collects web content that could potentially contribute to training its generative AI models. Restricting it signals that a site's future materials should be excluded from training datasets.

Who runs it

ClaudeBot is run by Anthropic, according to Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?.

What Anthropic says it is for

Anthropic describes ClaudeBot this way.

“ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?

User-agent string and robots.txt token

The Anthropic page we opened names the robot (ClaudeBot) but does not print a full user-agent string, so match on the name.

robots.txt token: ClaudeBot

Anthropic gives a robots.txt example for ClaudeBot.

“User-agent: ClaudeBot Disallow: /” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?

Vendor's own example, as printed on its page:

User-agent: ClaudeBot
Disallow: /

Does blocking it affect AI search visibility?

Anthropic does not say blocking ClaudeBot affects search visibility. It says restricting ClaudeBot signals that a site's future materials should be excluded from its AI model training datasets. For its search and user-request robots, see Claude-SearchBot and Claude-User.

Anthropic says what restricting ClaudeBot signals.

“When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?

How to find it in your logs

For request-level inspection, use your host, CDN or web-server logs, as in SearchIntel’s log study. Search the user-agent field for the name. On a server with shell access:

grep -i "ClaudeBot" access.log

Google warns that user-agent strings can be spoofed, so verify the request before counting it.

Anthropic says its crawler IP list can confirm a request.

“If a crawler has a source IP address on this list , it indicates that the crawler is coming from Anthropic.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?

What we saw in our own logs

These figures come from our post How to Track AI Crawlers on Your Website: A Week of Logs: one week of logs (19 to 25 October 2026) from two websites, a specialist research site and a smaller services site.

Read this as a snapshot: the post says to “treat the shares as a snapshot, not a benchmark or a trend”, and says “We used the bot labels Vercel exported”, so the labels are Vercel’s. Full method and limits are in the post.

Frequently asked questions

What is ClaudeBot?

Anthropic says ClaudeBot collects web content that could potentially contribute to training its generative AI models. Restricting it signals that a site's future materials should be excluded from training datasets.

Does blocking ClaudeBot affect AI search visibility?

Anthropic does not say blocking ClaudeBot affects search visibility. It says restricting ClaudeBot signals that a site's future materials should be excluded from its AI model training datasets. For its search and user-request robots, see Claude-SearchBot and Claude-User.

How do I find ClaudeBot in my logs?

Search the user-agent field of your host, CDN or web-server logs for the name, for example with grep -i "ClaudeBot" access.log. Google warns that user-agent strings can be spoofed, so verify the request before counting it.

Changelog

Last updated 17 November 2026.

All bots: what each AI bot does on your website.

See Whether AI Recommends Your Brand

Run a free check across ChatGPT, Claude, Gemini, Perplexity and Google AI Mode. See your score, your platform breakdown, and the competitors AI recommends instead.

Related bots