AI crawlers
See which AI crawlers read your site.
Every fetch from GPTBot, ClaudeBot, PerplexityBot and the rest, by page, and whether it was really them.
What is AI crawler analytics?
AI crawler analytics shows which AI bots read your site, such as GPTBot, ClaudeBot and PerplexityBot. Traceten logs each fetch by page and by purpose (training, search index or answering a question), and checks it against the vendor's published IP ranges or reverse DNS where they exist. It covers 60 crawlers from 21 providers.
CrawlersAvailable on All plans
60 crawlers from 21 providers.
Training, search and answer crawlers, each named and sorted by what it came for.
Answering questions
Someone asked an AI, and it fetched your page to answer.
Building search index
You are entering an AI search index.
Training data
Your content is collected for model training.
VerificationAvailable on All plans
Real crawler, or just a user agent?
Checked against published IP ranges and reverse DNS wherever the vendor publishes them. Impostors are flagged as spoofed.
A meter of its own
Crawler hits never count against your events.
SetupAvailable on All plans
One middleware, on your server.
Crawlers never run JavaScript, so a middleware reports them from Next.js, Express, Hono or Cloudflare Workers and Pages.
Works with
- Next.js
- Express
- Hono
- Cloudflare
Questions
01What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects pages OpenAI may use to train its models. OAI-SearchBot builds the index behind ChatGPT search, so blocking it can keep you out of ChatGPT's search answers. They are separate robots.txt tokens, so you can block one and allow the other.
02How does Traceten know a crawler is really GPTBot?
It checks the request's IP against the ranges OpenAI publishes, and a request that claims to be GPTBot from outside them is flagged as spoofed. Where a vendor publishes crawler hostnames, reverse DNS is confirmed both ways. A crawler Traceten has no way to check is shown as unverified, not as fake.
03Why can't Google Analytics see AI crawlers?
AI crawlers fetch your pages without running JavaScript, so a browser tag never fires for them, and GA4 excludes known bots as well. Traceten records them from a middleware on your server instead.
04Should I block GPTBot?
Blocking GPTBot asks OpenAI not to train on your pages, and it does not remove you from ChatGPT search, which runs on OAI-SearchBot. Blocking a search or answer crawler is the costly move, because it can drop you from that assistant's answers.
05Do crawler hits count against my event quota?
No. AI crawler hits have their own monthly meter on every plan and never count against your events.
Related reading
AI is already sending you buyers. See the receipts.
No calls, no credit card.