Bot & AI Crawler Tracking

The DataSaaS JavaScript tag tracks humans. AI crawlers are different: they fetch your raw HTML and never run JavaScript, so no browser tag can ever see them. That is why bot tracking is a separate, server-side install. It does not replace script.js, it runs alongside it. Add one non-blocking call in your backend, middleware, or edge and DataSaaS identifies and IP-verifies every crawler for you.

What gets tracked

Four search crawlers do render JavaScript, so the JS tag already captures them through the events beacon with no server code at all:

• Googlebot
• Google-InspectionTool
• Bingbot
• Applebot

Every AI crawler needs the server package, because none of them run JavaScript. That includes GPTBot, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, CCBot, Amazonbot, Bytespider, the Meta crawlers, xAI / Grok, Mistral, Cohere, DeepSeek, and more.

DataSaaS recognizes 56 crawlers, grouped into four categories: AI answers (assistants that cite your pages live), Search index (traditional search engines), Training (dataset collection), and Other. See the full crawler directory for every operator, its category, and how it is verified.

Install

Add the package to your server-side project:

npm install @datasaas/bot-tracker

Replace ds_abc123in the snippets below with the website id from Settings → Tracking.

Next.js (middleware)

Drop a single call into your middleware. Match pages, not static assets, since bots crawl pages and not build chunks. On Vercel this is all you need: the package finds the request's waitUntil on the Next request context and keeps the report alive itself.

// middleware.ts
import { NextResponse } from "next/server";
import { trackBotFromRequest } from "@datasaas/bot-tracker";

export function middleware(req: Request) {
  trackBotFromRequest(req, { websiteId: "ds_abc123" }); // fire-and-forget
  return NextResponse.next();
}

// Match pages, not static assets. Bots crawl pages, not chunks.
export const config = { matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"] };

Express / Node

In a raw Node framework, pass the request fields directly with trackBotRequest:

import { trackBotRequest } from "@datasaas/bot-tracker";

app.use((req, res, next) => {
  trackBotRequest(
    {
      userAgent: req.headers["user-agent"] || "",
      ip: (req.headers["x-forwarded-for"]?.split(",")[0] || req.socket.remoteAddress || "").trim(),
      path: req.path,
      host: req.headers.host,
      method: req.method,
    },
    { websiteId: "ds_abc123" }
  );
  next();
});

Hono

Register the same call as middleware on every route:

import { Hono } from "hono";
import { trackBotRequest } from "@datasaas/bot-tracker";

const app = new Hono();
app.use("*", async (c, next) => {
  trackBotRequest(
    {
      userAgent: c.req.header("user-agent") || "",
      ip: (c.req.header("x-forwarded-for")?.split(",")[0] || "").trim(),
      path: new URL(c.req.url).pathname,
      host: c.req.header("host"),
      method: c.req.method,
    },
    { websiteId: "ds_abc123" }
  );
  await next();
});

Cloudflare Workers

trackBotFromRequest accepts the Web Request directly. On Workers, the adapter reads the real client IP from the cf-connecting-ip header for you. Workers hand the execution context to your handler rather than exposing it globally, so pass the returned promise to ctx.waitUntil.

import { trackBotFromRequest } from "@datasaas/bot-tracker";

export default {
  async fetch(request, env, ctx) {
    // The adapter reads the real client IP from cf-connecting-ip.
    ctx.waitUntil(trackBotFromRequest(request, { websiteId: "ds_abc123" }));
    return fetch(request);
  },
};

Any other framework

For Fastify, Bun, Deno, or anything else, build the fields yourself and call trackBotRequest. The endpoint option defaults to https://datasaas.co/api/bot. Override it only when you self-host DataSaaS and want to point at your own instance.

import { trackBotRequest } from "@datasaas/bot-tracker";

trackBotRequest(
  { userAgent, ip, path, host, method },
  { websiteId: "ds_abc123", endpoint: "https://datasaas.co/api/bot" }
);

Serverless and edge runtimes

On a long-running server (a VPS, a container, plain Node) an un-awaited report finishes on its own and there is nothing to do. Serverless and edge platforms are different: they freeze the invocation the moment you return a response, which cancels any request still in flight. Both tracking functions return a promise that settles when the report lands, so you can keep the runtime alive until it does.

• Vercel and Next.js are handled automatically. The package registers the report on the Next request context for you.
• Cloudflare Workers need ctx.waitUntil(trackBotFromRequest(...)), since the context is passed to your handler.
• Other serverless platforms take the same promise in whatever keep-alive primitive they expose.

Verified vs unverified

Verification runs in two steps. First DataSaaS matches the request User-Agent against a known crawler. Then it checks the request IP against that operator's published CIDR ranges, which are refreshed daily. A request whose User-Agent claims to be GPTBot but comes from an IP outside OpenAI's ranges is recorded as unverified, so you can filter spoofed traffic out.

Some operators do not publish an IP list at all, so they can only ever be matched by User-Agent. These are marked ua_only and are always recorded as unverified by design. The dashboard defaults to showing verified crawlers only, with a toggle to include the rest.

Where to find the data

Bot traffic shows up in the Bot traffic card on every site dashboard. It gives you:

• Category tabs to switch between AI answers, Search index, Training, and Other
• Verified only toggle, on by default, to hide spoofed and ua_only hits
• Bots and Pages views to see which crawlers visited and which pages they hit
• Discovery filter to isolate requests for robots.txt, llms.txt, and sitemaps