Who runs it
Claude-SearchBot is run by Anthropic, according to Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?.
What Anthropic says it is for
Anthropic describes Claude-SearchBot this way.
“Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
User-agent string and robots.txt token
The Anthropic page we opened names the robot (Claude-SearchBot) but does not print a full user-agent string, so match on the name.
robots.txt token: Claude-SearchBot
Anthropic says site owners can set preferences for each of its three robots.
“Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
Example rule (standard robots.txt syntax, written by us, not a vendor quote):
User-agent: Claude-SearchBot
Disallow: /
Does blocking it affect AI search visibility?
Anthropic says disabling Claude-SearchBot prevents indexing of your content for search optimization, which may reduce your site's visibility and accuracy in user search results.
Anthropic says disabling Claude-SearchBot may reduce visibility.
“Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
How to find it in your logs
For request-level inspection, use your host, CDN or web-server logs, as in SearchIntel’s log study. Search the user-agent field for the name. On a server with shell access:
grep -i "Claude-SearchBot" access.log
Google warns that user-agent strings can be spoofed, so verify the request before counting it.
Anthropic says its crawler IP list can confirm a request.
“If a crawler has a source IP address on this list , it indicates that the crawler is coming from Anthropic.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
What we saw in our own logs
These figures come from our post How to Track AI Crawlers on Your Website: A Week of Logs: one week of logs (19 to 25 October 2026) from two websites, a specialist research site and a smaller services site.
The post does not report Claude-SearchBot separately, so we have no published figure for it.
Read this as a snapshot: the post says to “treat the shares as a snapshot, not a benchmark or a trend”, and says “We used the bot labels Vercel exported”, so the labels are Vercel’s. Full method and limits are in the post.
Frequently asked questions
What is Claude-SearchBot?
Anthropic says Claude-SearchBot navigates the web to improve search result quality for users. It says disabling it may reduce a site's visibility and accuracy in user search results.
Does blocking Claude-SearchBot affect AI search visibility?
Anthropic says disabling Claude-SearchBot prevents indexing of your content for search optimization, which may reduce your site's visibility and accuracy in user search results.
How do I find Claude-SearchBot in my logs?
Search the user-agent field of your host, CDN or web-server logs for the name, for example with grep -i "Claude-SearchBot" access.log. Google warns that user-agent strings can be spoofed, so verify the request before counting it.
Changelog
Last updated 17 November 2026.
- 17 November 2026: Pages published.
All bots: what each AI bot does on your website.