See Which AI Crawlers Read Your Site

Why Browser Analytics Is Blind to AI Crawlers

Every JavaScript analytics tag, including Google Analytics, Plausible, and Fathom, only runs inside a browser. AI crawlers do not use a browser. GPTBot, ClaudeBot, and CCBot request your raw HTML over HTTP and read it as text. They never execute the script that would report the visit, so the crawl is invisible to any client-side tool. The only place to see these requests is on the server, where every request lands before a single line of JavaScript runs.

• AI crawlers fetch raw HTML and never execute JavaScript, so no browser tag can record them
• Googlebot, Bingbot, Applebot, and Google-InspectionTool do render pages, so DataSaaS catches those through the normal events beacon with zero setup
• Every AI answer engine and training crawler needs server-side capture to appear in your reports
• Server-side tracking runs before your JavaScript, so it works even when a crawler ignores robots.txt

IP Verification, Verified vs Unverified

A user agent string is trivial to fake. Anyone can send a request that claims to be GPTBot. DataSaaS never takes the header at face value. When a request matches a known crawler token, DataSaaS checks the request IP against the operator's published address ranges. If the IP falls inside an official range, the hit is marked verified. If it does not, or the operator publishes no ranges, the hit is recorded as unverified so you can filter it out.

• Every hit is matched by user agent token, then checked against the operator's published CIDR ranges
• 4,075 IP ranges across 9 operators are refreshed daily so verification stays current
• A spoofed GPTBot request from a random server is recorded as unverified, never as a real crawl
• Operators that publish no IP list (Meta, xAI, most Chinese crawlers) are matched by user agent only and always shown unverified by design

Which Pages ChatGPT, Claude, and Perplexity Read

Once crawls are captured, you see exactly which content the AI engines pull. Sort by page to find what ChatGPT reads most, by crawler to see who visits, and by discovery file to confirm bots are reading your robots.txt, llms.txt, and sitemaps. DataSaaS groups every crawler into four buckets so you can tell answer engines apart from search indexers and model trainers.

• Bots and Pages views show which crawler hit which URL, and how often, over any date range
• Four categories separate AI answers, search indexing, training, and other traffic
• Discovery filter surfaces hits on robots.txt, llms.txt, and sitemap files so you know your rules are being read
• See at a glance whether a crawler is answering questions, building a search index, or training a model on you

One-Line Install with @datasaas/bot-tracker

Add server-side crawler tracking with a single npm package. @datasaas/bot-tracker is a zero-dependency library that works in any Node runtime and edge environment. Drop one fire-and-forget call into your middleware and every crawler request is reported to DataSaaS without slowing down the response. It never awaits, never throws into your request path, and needs only your website ID.

• Next.js: call trackBotFromRequest(req, { websiteId }) in middleware.ts
• Express, Fastify, Hono, Bun, and Deno: call trackBotRequest with the user agent, IP, and path
• Cloudflare Workers and other edge runtimes are supported out of the box, no Node APIs required
• The call is fire-and-forget, so it adds no latency and can never break a page render

56 Crawlers Across 21 Operators

DataSaaS ships with a maintained registry of 56 crawlers from 21 operators, covering every major AI company. From GPTBot and Claude-User to PerplexityBot, CCBot, Amazonbot, Bytespider, and the crawlers from Meta, xAI, Mistral, Cohere, and DeepSeek. Each entry knows the operator, the category, and how the crawler can be verified, so new bots show up correctly the moment they hit your site.

• 15 AI answer engines, 15 search indexers, 14 training crawlers, and 12 other bots
• 21 operators including OpenAI, Anthropic, Perplexity, Google, Amazon, ByteDance, Meta, and xAI
• Each crawler records its verification method: IP range, reverse DNS, or user agent only
• The registry updates as operators launch new crawlers, so your reports stay complete

Frequently asked questions

Related