All AI bots
AI Bots

What CCBot Does on Your Website

What Common Crawl says it is for, how to spot it in your logs, and what we saw in ours.

In short: CCBot is the crawler for Common Crawl, a non-profit that maintains an open repository of web crawl data. Common Crawl's page for CCBot does not mention AI products.

Who runs it

CCBot is run by Common Crawl, according to Common Crawl, CCBot.

What Common Crawl says it is for

Common Crawl describes itself this way.

“Common Crawl is a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone.” Common Crawl, CCBot

User-agent string and robots.txt token

CCBot's user-agent string, from Common Crawl.

CCBot/2.0 ( https://commoncrawl.org/faq/ )

From Common Crawl, CCBot.

robots.txt token: CCBot

Common Crawl gives this robots.txt rule.

“User-agent: CCBot Disallow: /” Common Crawl, CCBot

Vendor's own example, as printed on its page:

User-agent: CCBot
Disallow: /

Does blocking it affect AI search visibility?

Common Crawl does not say in its crawler documentation that blocking CCBot affects visibility in any AI product. It says the rule prevents Common Crawl from crawling your website.

Common Crawl says how to stop it crawling.

“To prevent Common Crawl from crawling your website, include the following in your robots.txt” Common Crawl, CCBot

How to find it in your logs

For request-level inspection, use your host, CDN or web-server logs, as in SearchIntel’s log study. Search the user-agent field for the name. On a server with shell access:

grep -i "CCBot" access.log

Google warns that user-agent strings can be spoofed, so verify the request before counting it.

Common Crawl warns that other crawlers use its name.

“Please note that we are aware of crawlers falsely identifying themselves as CCBot.” Common Crawl, CCBot

Common Crawl says CCBot uses dedicated IP ranges, with reverse DNS currently supported for IPv4 but not IPv6.

“CCBot is now run on dedicated IP address ranges with reverse DNS (except over IPv6 where reverse DNS is not yet supported.)” Common Crawl, CCBot

What we saw in our own logs

These figures come from our post How to Track AI Crawlers on Your Website: A Week of Logs: one week of logs (19 to 25 October 2026) from two websites, a specialist research site and a smaller services site.

The post does not report CCBot separately, so we have no published figure for it.

Read this as a snapshot: the post says to “treat the shares as a snapshot, not a benchmark or a trend”, and says “We used the bot labels Vercel exported”, so the labels are Vercel’s. Full method and limits are in the post.

Frequently asked questions

What is CCBot?

CCBot is the crawler for Common Crawl, a non-profit that maintains an open repository of web crawl data. Common Crawl's page for CCBot does not mention AI products.

Does blocking CCBot affect AI search visibility?

Common Crawl does not say in its crawler documentation that blocking CCBot affects visibility in any AI product. It says the rule prevents Common Crawl from crawling your website.

How do I find CCBot in my logs?

Search the user-agent field of your host, CDN or web-server logs for the name, for example with grep -i "CCBot" access.log. Google warns that user-agent strings can be spoofed, so verify the request before counting it.

Changelog

Last updated 17 November 2026.

All bots: what each AI bot does on your website.

See Whether AI Recommends Your Brand

Run a free check across ChatGPT, Claude, Gemini, Perplexity and Google AI Mode. See your score, your platform breakdown, and the competitors AI recommends instead.

Related bots