Most AI visibility work looks at answers: ask the engines a question, see who they name and which pages they cite. It is the right place to start, and we do it every week. But answers do not show you the traffic itself. Server logs show the requests AI assistants and crawlers make to your website, including activity that citation tracking never captures. They do not show whether a page was used in an answer, or who was asking.
We read one week of logs (19 to 25 October 2026) from two websites: a specialist B2B research site with a few hundred pages, and a smaller services website. Both run on Vercel, and we used the bot labels in its request logs. This piece sets out what came through, what we think it means, and where the data stops.
Two kinds of AI visitor
AI traffic in a log falls into two groups, and mixing them up leads to the wrong conclusions.
Assistants fetching on behalf of a user. When someone uses ChatGPT or Claude and it goes to the web, the request can carry a label such as ChatGPT-User or Claude-User. OpenAI says it uses ChatGPT-User "for certain user actions in ChatGPT and Custom GPTs", and Anthropic says Claude-User "supports Claude AI users" and may access websites when people ask Claude questions. These requests are tied to someone using the product, though a single request tells you nothing about who or why.
Crawlers for search and model development. These visit on their own schedule rather than for one person. In the same documentation, OpenAI describes OAI-SearchBot as being "for search" and GPTBot as making its models "more useful and safe". Anthropic's ClaudeBot collects web content for its models. Meta's external agent crawler works for "use cases such as training foundation AI models". Bingbot and Googlebot sit here too. Some of these crawlers feed search indexes, some feed model training, and a request on its own does not tell you which outcome followed.
What came through
Crawlers, by share of crawler requests.
- Research site: Bingbot 29%, Exa 27%, Googlebot 15%, OpenAI's search crawler 12%, Perplexity 10%, Anthropic's ClaudeBot 3%, Amazon 3%, others under 1% each. OpenAI's training crawler, GPTBot, made a handful of requests, all to the sitemap.
- Services site: Meta's AI crawler 30%, Exa 17%, Perplexity 11%, Googlebot 10%, Bingbot 8%, ClaudeBot 8%, OpenAI's search crawler 7%, Amazon 6%, Apple 2%, GPTBot under 1%.
AI assistants, by share of assistant requests.
- Research site: ChatGPT-related labels 74% (GPT Actions 42.5%, ChatGPT-User 31.5%), Claude 20%, DuckDuckGo's assistant 4%, Mistral 2%.
- Services site: ChatGPT-related labels 78%, Claude 22%, DuckDuckGo's assistant under 1%.
Percentages are rounded separately, so a list may not add to exactly 100%.
What we take from it
Google and Bing made less than half the crawler requests on both sites
On the research site, crawlers from AI companies (Exa, OpenAI, Perplexity, Anthropic and Meta) made 52% of crawler requests, against 45% for Google and Bing together. On the services site the gap was far wider: 73% against 19%. Monitoring that only watched Googlebot and Bingbot would have seen 45% of crawler requests on one site and 19% on the other. Check which bots your robots.txt rules and error alerts actually cover.
Exa is the crawler worth knowing
The most striking result was Exa. It was the second busiest crawler on both sites, and it made more than twice as many requests as OpenAI's search crawler on each (2.2 and 2.3 times). It was ahead of OpenAI's search crawler in the week before as well, on both sites.
Exa is a search engine built for software rather than people. Its homepage describes it as "the world's data for AI agents", and its customer stories include HubSpot, Firefox and monday.com. Developers use it to give their AI products a way to search the web. Exa's crawler page says its index "allows users to find, retrieve, and cite your content". So AI tools and agents built on Exa can reach your pages through its index rather than Google's, and that activity will never show up in Google Search Console.
Three checks follow. Look for an ExaSearchBot group in your robots.txt; if there is none, Exa says it follows the rules you set for major search engines, then the wildcard group, so make sure that is what you intend. Check that your most important pages return cleanly to it. And remember that it can only see a noindex instruction on a page it is allowed to fetch.
ChatGPT dominates assistant requests on both sites
Two very different sites, one a specialist research site and one a services business, had almost the same split: roughly three in four assistant requests under ChatGPT-related labels and about one in five from Claude. That describes the mix of requests on these two sites, not who their readers are, and your own logs may differ. Set your own priorities from your own request mix and answer data.
Assistant requests hit pages that did not exist
Requests under assistant labels reached 13 URLs on the research site that returned a 404. Twelve followed the site's company-profile URL pattern. A 404 like this is a list to investigate, not proof of what the assistant wanted: the URL could come from a guess, an old link or a mistyped address. We checked each one. Ten were companies the site did not yet cover and that fitted its scope, so we wrote those profiles. The other three had a clear existing equivalent, so we redirected them. A weekly report now lists any new ones.
Repeated hits on one page need checking before they count
On the services website, nearly two thirds of assistant-labelled requests went to the homepage. Before reading that as interest in the business, we treat it as a question: repeated requests to a single page can come from monitoring, testing or other automation. Request counts need reading, not just adding up.
What to do with your own logs
- Split assistants from crawlers. Crawler counts show recorded requests, not whether a page was indexed, used for training or included in an answer. Assistant requests are closer to people using AI, but still are not a measure of demand on their own.
- Find out who is crawling you beyond Google and Bing. Rank the crawler labels by volume and look up anything you do not recognise. On both sites we read, that was most of the crawling.
- Check robots.txt bot by bot. Make sure the crawlers you want to reach you are allowed, Exa included. Blocking a crawler can limit discovery and updates, and each provider has its own rules for index removal.
- Investigate the 404s that assistant requests hit. Create a page when it meets a real audience need, redirect when there is a clear equivalent, and leave a 404 or 410 in place when neither applies.
- Look after the pages that get requested most. On the research site, five content paths took 22% of assistant requests. Make sure their dates and figures are current and that the first paragraph answers the question directly.
- Question concentrated, repetitive traffic. Check for monitors, tests and scripts before treating it as audience interest.
Logs show who requested what. They do not show whether you were recommended. For that you still need to ask the engines your buyers' questions and see what they say. Our free AI visibility check does that across the main engines. The two views together are much stronger than either alone. For the citation side, see which pages AI answers pull from and how ChatGPT chooses its sources.
The limits of this data
This is one week from two websites, so treat the shares as a snapshot, not a benchmark or a trend. On the research site they rest on 4,276 assistant requests and 9,705 crawler requests; on the services site, 290 and 1,729. The window ran from 19 October 00:00 UTC to 26 October 00:00 UTC. We used the bot labels Vercel exported; the export does not include the verification behind each label, so we have not authenticated every request ourselves, and Exa notes that user-agent strings can be spoofed. Logs never show the question a person asked or who they were: only which URL was requested, when, and what the server returned.
Frequently asked questions
What is ExaSearchBot?
It is the crawler for Exa, a search engine that developers use to give AI products and agents access to the web. Exa says the crawler is for "search and retrieval" and respects robots.txt. You can give it its own group in robots.txt to allow or restrict it.
Can I see what people asked ChatGPT when it visited my site?
No. The log shows which URL was requested and when, but not the question behind it or who asked it. The subjects of the requested pages can suggest topics worth investigating, nothing more.
What is the difference between ChatGPT-User and OAI-SearchBot?
OpenAI uses ChatGPT-User for certain user actions in ChatGPT and Custom GPTs. OAI-SearchBot crawls the web for OpenAI's search. OpenAI documents both on its bots page.
Should I allow Exa's crawler?
If you want to be found through products that use Exa's index, allowing it makes sense. Exa says noindex keeps a page out of its index once the crawler can fetch it.