Who runs it
Googlebot and Google-Extended is run by Google, according to Google Search Central, Google's common crawlers.
What Google says it is for
Google describes its common crawlers this way.
“Google's common crawlers are used to find information for building Google's search indexes, perform other product specific crawls, and for analysis.” Google Search Central, Google's common crawlers
Google says what Googlebot preferences affect.
“Crawling preferences addressed to the Googlebot user agent affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Video mode, Google News, and Discover.” Google Search Central, Google's common crawlers
Google describes Google-Extended this way.
“Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI.” Google Search Central, Google's common crawlers
User-agent string and robots.txt token
Googlebot Desktop user-agent string, from Google (W.X.Y.Z stands for the Chrome version).
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36From Google Search Central, Google's common crawlers.
Google says Google-Extended has no user-agent string of its own.
Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.From Google Search Central, Google's common crawlers.
robots.txt token: Googlebot, Google-Extended
Google gives an example robots.txt group for Googlebot.
“user-agent: Googlebot allow: /archive/1Q84 disallow: /archive” Google Search Central, Google's common crawlers
Example rule (standard robots.txt syntax, written by us, not a vendor quote):
User-agent: Googlebot
Disallow: /
User-agent: Google-Extended
Disallow: /
The first group addresses Googlebot, which Google says affects Google Search. The second addresses Google-Extended, which Google says controls use of crawled content for Gemini training and grounding, not Search inclusion.
Does blocking it affect AI search visibility?
Google says Google-Extended does not affect Google Search inclusion or ranking, but controls content use for Gemini training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. For Googlebot, Google says robots.txt directives for Googlebot are the control for how a site is crawled for Search, and that AI is built into Search.
Google says Google-Extended does not affect Search.
“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.” Google Search Central, Google's common crawlers
Google says Googlebot rules are the control for AI features in Search.
“AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search.” Google Search Central, AI features and your website
How to find it in your logs
For request-level inspection, use your host, CDN or web-server logs, as in SearchIntel’s log study. Search the user-agent field for the name. On a server with shell access:
grep -i "Googlebot" access.log
Google says Google-Extended has no separate HTTP user-agent string; crawling uses existing Google user agents.
Google warns that user-agent strings can be spoofed, so verify the request before counting it.
Google warns that the user-agent string can be spoofed.
“The HTTP user agent string can be spoofed.” Google Search Central, Google's common crawlers
Google gives a reverse DNS check.
“the reverse DNS mask of their hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com .” Google Search Central, Google's common crawlers
What we saw in our own logs
These figures come from our post How to Track AI Crawlers on Your Website: A Week of Logs: one week of logs (19 to 25 October 2026) from two websites, a specialist research site and a smaller services site.
- SearchIntel’s October log study reports for research site, share of crawler requests: “Googlebot 15%”.
- SearchIntel’s October log study reports for services site, share of crawler requests: “Googlebot 10%”.
- SearchIntel’s October log study reports for research site, AI company crawlers against Google and Bing together: “52% of crawler requests, against 45% for Google and Bing together”.
- SearchIntel’s October log study reports for services site, AI company crawlers against Google and Bing together: “73% against 19%”.
Read this as a snapshot: the post says to “treat the shares as a snapshot, not a benchmark or a trend”, and says “We used the bot labels Vercel exported”, so the labels are Vercel’s. Full method and limits are in the post.
Frequently asked questions
What is Googlebot and Google-Extended?
Googlebot is Google's crawler for building its search indexes. Google-Extended is a robots.txt token only. Google says it controls whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI, and does not affect inclusion in Google Search.
Does blocking Googlebot and Google-Extended affect AI search visibility?
Google says Google-Extended does not affect Google Search inclusion or ranking, but controls content use for Gemini training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. For Googlebot, Google says robots.txt directives for Googlebot are the control for how a site is crawled for Search, and that AI is built into Search.
How do I find Googlebot and Google-Extended in my logs?
Search the user-agent field of your host, CDN or web-server logs for the name, for example with grep -i "Googlebot" access.log. Google warns that user-agent strings can be spoofed, so verify the request before counting it.
Changelog
Last updated 17 November 2026.
- 17 November 2026: Pages published.
All bots: what each AI bot does on your website.