All AI bots
AI Bots

What Claude-User Does on Your Website

What Anthropic says it is for, how to spot it in your logs, and what we saw in ours.

In short: Anthropic says Claude-User may access a website when someone asks Claude a question. It says disabling it may reduce a site's visibility for user-directed web search.

Who runs it

Claude-User is run by Anthropic, according to Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?.

What Anthropic says it is for

Anthropic describes Claude-User this way.

“Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?

User-agent string and robots.txt token

The Anthropic page we opened names the robot (Claude-User) but does not print a full user-agent string, so match on the name.

robots.txt token: Claude-User

Anthropic says site owners can set preferences for each of its three robots.

“Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?

Example rule (standard robots.txt syntax, written by us, not a vendor quote):

User-agent: Claude-User
Disallow: /

Does blocking it affect AI search visibility?

Anthropic says disabling Claude-User prevents retrieval of your content in response to a user query, which may reduce your site's visibility for user-directed web search.

Anthropic says disabling Claude-User may reduce visibility.

“Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?

How to find it in your logs

For request-level inspection, use your host, CDN or web-server logs, as in SearchIntel’s log study. Search the user-agent field for the name. On a server with shell access:

grep -i "Claude-User" access.log

Google warns that user-agent strings can be spoofed, so verify the request before counting it.

Anthropic says its crawler IP list can confirm a request.

“If a crawler has a source IP address on this list , it indicates that the crawler is coming from Anthropic.” Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?

What we saw in our own logs

These figures come from our post How to Track AI Crawlers on Your Website: A Week of Logs: one week of logs (19 to 25 October 2026) from two websites, a specialist research site and a smaller services site.

Read this as a snapshot: the post says to “treat the shares as a snapshot, not a benchmark or a trend”, and says “We used the bot labels Vercel exported”, so the labels are Vercel’s. Full method and limits are in the post.

Frequently asked questions

What is Claude-User?

Anthropic says Claude-User may access a website when someone asks Claude a question. It says disabling it may reduce a site's visibility for user-directed web search.

Does blocking Claude-User affect AI search visibility?

Anthropic says disabling Claude-User prevents retrieval of your content in response to a user query, which may reduce your site's visibility for user-directed web search.

How do I find Claude-User in my logs?

Search the user-agent field of your host, CDN or web-server logs for the name, for example with grep -i "Claude-User" access.log. Google warns that user-agent strings can be spoofed, so verify the request before counting it.

Changelog

Last updated 17 November 2026.

All bots: what each AI bot does on your website.

See Whether AI Recommends Your Brand

Run a free check across ChatGPT, Claude, Gemini, Perplexity and Google AI Mode. See your score, your platform breakdown, and the competitors AI recommends instead.

Related bots