Bot traffic tracking

See when AI assistants, search engines, and model-training crawlers request pages on your website. Track ChatGPT, Claude, Googlebot, Perplexity, and 40+ other crawlers.

Get started

Install the server-side package, add one tracking call in your backend, then deploy. This is separate from the normal Attribu browser tracking script.

See @attribu/ai-crawl on NPM

1. Install the package

npm install @attribu/ai-crawl

2. Add it to your server middleware

Here is the most common setup for a Next.js app hosted on Vercel (more examples below):

// middleware.ts
import { NextRequest, NextResponse, NextFetchEvent } from "next/server";
import { trackAICrawlerRequest } from "@attribu/ai-crawl";

export function middleware(request: NextRequest, event: NextFetchEvent) {
  trackAICrawlerRequest(request, event, {
    websiteId: "YOUR_SITE_ID",
  });

  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

3. Deploy and check your dashboard

After deployment, open your Attribu dashboard and look for the Bot traffic card. You can filter by AI answers, indexing, training crawlers, and IP verification confidence.

Using Cloudflare, Express, Hono, or another backend? See the platform examples below.

What this tracks

AI tools and search crawlers read your site before many users do. The pages they request show what they are trying to answer, index, or learn from, including missing URLs they expected to find.

Attribu groups bot traffic into three main categories:

Crawler-facing files

Attribu also tracks crawler-facing files when known bots request them:

• /robots.txt
• /llms.txt
• /llms-full.txt
• /sitemap.xml and sitemap XML files
• Content files such as /docs/setup.md or /startup/example.md

These files matter because AI assistants, search engines, and training crawlers often request them before crawling the rest of your site. Seeing those requests helps you know whether bots are discovering your AI/SEO instructions and structured content.

How the package works

The package runs in your backend, middleware, edge function, or worker. For each request, it quickly ignores obvious static assets, API routes, framework internals, and normal human browser traffic. If the request looks like bot traffic, it sends a small event to Attribu.

Attribu then classifies the provider, crawler type, confidence, and IP verification on the server. This keeps crawler lists and IP ranges up to date without asking every customer to upgrade the npm package each time a crawler changes.

Bot traffic tracking runs server-side because AI crawlers often request raw HTML and skip frontend JavaScript.

Platform examples

Next.js / Vercel

Create or update middleware.ts:

// middleware.ts
import { NextRequest, NextResponse, NextFetchEvent } from "next/server";
import { trackAICrawlerRequest } from "@attribu/ai-crawl";

export function middleware(request: NextRequest, event: NextFetchEvent) {
  trackAICrawlerRequest(request, event, {
    websiteId: "YOUR_SITE_ID",
  });

  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

Vercel provides event.waitUntil, so the Attribu request is scheduled in the background. Middleware runs before the final page response, so status code is usually stored as unknown.

Cloudflare Pages

Create functions/_middleware.ts:

import { trackAICrawlerRequest } from "@attribu/ai-crawl";

export async function onRequest(context) {
  trackAICrawlerRequest(context.request, context, {
    websiteId: "YOUR_SITE_ID",
  });

  return context.next();
}

Cloudflare Pages provides context.waitUntil. The package uses it internally, so Cloudflare can continue returning the HTML response while the Attribu request runs in the background.

Cloudflare Workers

Wrap your Worker handler with withAICrawlerTracking:

import { withAICrawlerTracking } from "@attribu/ai-crawl";

export default {
  fetch: withAICrawlerTracking(
    async (request, env, ctx) => {
      return fetch(request);
    },
    {
      websiteId: "YOUR_SITE_ID",
    },
  ),
};

This version can capture status code because the wrapper sees the response your handler created. It still uses ctx.waitUntil when available, so tracking is best-effort and non-blocking.

Express

import express from "express";
import { createExpressAICrawlerMiddleware } from "@attribu/ai-crawl";

const app = express();

app.use(
  createExpressAICrawlerMiddleware({
    websiteId: "YOUR_SITE_ID",
  }),
);

The Express middleware calls next() immediately. It attaches a finish listener and sends the bot traffic event after the response has already been sent, so your app does not wait for Attribu before continuing.

Hono

import { Hono } from "hono";
import { trackAICrawlerResponse } from "@attribu/ai-crawl";

const app = new Hono();

app.use("*", async (c, next) => {
  await next();

  trackAICrawlerResponse(c.req.raw, c.res, c.executionCtx, {
    websiteId: "YOUR_SITE_ID",
  });
});

Use this when your Hono runtime gives you access to both the final response and an execution context.

Generic request / response handler

If your backend gives you a standard Request and Response, track after your app creates the response:

import { trackAICrawlerResponse } from "@attribu/ai-crawl";

export async function handleRequest(request, context) {
  const response = await yourAppHandler(request);

  trackAICrawlerResponse(request, response, context, {
    websiteId: "YOUR_SITE_ID",
  });

  return response;
}

If your backend only gives you the request before the response exists, use request-only tracking:

import { trackAICrawlerRequest } from "@attribu/ai-crawl";

export function middleware(request, context) {
  trackAICrawlerRequest(request, context, {
    websiteId: "YOUR_SITE_ID",
  });

  return next();
}

Request-only tracking is enough to know which page the bot tried to crawl. Response-aware tracking only adds status code when it is easy to get.

Custom Docker, Cloud Run, or reverse proxies

Most backends expose the public request URL automatically. If your runtime instead gives the package an internal hostname such as localhost or 0.0.0.0, set the public origin explicitly:

trackAICrawlerRequest(request, context, {
  websiteId: "YOUR_SITE_ID",
  publicOrigin: "https://yourdomain.com",
});

Attribu preserves the requested path and query string and still validates the resulting hostname against your website configuration.

Optional request authentication

You can add a website-specific Bot traffic token without interrupting an existing integration:

1. Open the Bot traffic card settings and create a token.
2. Add it to your server-side package configuration.
3. After the new configuration is deployed, enable Reject unauthenticated requests.

trackAICrawlerRequest(request, event, {
  websiteId: "YOUR_SITE_ID",
  authToken: process.env.ATTRIBU_BOT_TOKEN,
});

Use without Node.js (PHP or any backend)

You do not need Node.js or the npm package. Any backend that can send an HTTPS POST request can report a crawler request directly.

POST https://attribu.tech/api/bot-traffic
Content-Type: application/json
Authorization: Bearer att_bot_******

The Authorization header is optional unless you enabled Reject unauthenticated requests for this website.

Request body

{
  "websiteId": "YOUR_SITE_ID",
  "domain": "yourdomain.com",
  "href": "https://yourdomain.com/docs/get-started",
  "ai": {
    "userAgent": "Mozilla/5.0 ... ChatGPT-User/1.0",
    "ip": "203.0.113.10",
    "statusCode": 200,
    "source": "server_middleware"
  }
}

The JSON body must be smaller than 16 KB. A tracked or safely ignored request normally returns 200 with {"success":true}. Invalid requests return a 4xx response, including 429 when the caller is sending too quickly. Tracking is best-effort: use a short timeout and do not delay or fail your website response when Attribu is unavailable.

PHP example

Run this from server-side PHP after your application has decided what response to return. The local user-agent check is only a bandwidth pre-filter; Attribu performs the final classification.

<?php

function trackAttribuCrawler(string $websiteId, ?string $authToken = null): void
{
    $method = strtoupper($_SERVER['REQUEST_METHOD'] ?? 'GET');
    if (!in_array($method, ['GET', 'HEAD'], true)) {
        return;
    }

    $userAgent = $_SERVER['HTTP_USER_AGENT'] ?? '';
    $crawlerHints = [
        'bot', 'crawler', 'spider', 'chatgpt', 'gptbot', 'claude',
        'perplexity', 'bing', 'google', 'applebot', 'bytespider', 'ccbot'
    ];

    $normalizedUserAgent = strtolower($userAgent);
    $looksLikeCrawler = false;
    foreach ($crawlerHints as $hint) {
        if (strpos($normalizedUserAgent, $hint) !== false) {
            $looksLikeCrawler = true;
            break;
        }
    }

    if (!$looksLikeCrawler) {
        return;
    }

    $host = strtolower($_SERVER['HTTP_HOST'] ?? '');
    $host = preg_replace('/:\d+$/', '', $host);
    if (!$host) {
        return;
    }

    $isHttps = !empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] !== 'off';
    $scheme = $isHttps ? 'https' : 'http';
    $path = parse_url($_SERVER['REQUEST_URI'] ?? '/', PHP_URL_PATH) ?: '/';

    $payload = [
        'websiteId' => $websiteId,
        'domain' => $host,
        'href' => $scheme . '://' . $host . $path,
        'ai' => [
            'userAgent' => $userAgent,
            'ip' => $_SERVER['REMOTE_ADDR'] ?? null,
            'statusCode' => http_response_code(),
            'source' => 'server_middleware',
        ],
    ];

    $headers = ['Content-Type: application/json'];
    if ($authToken) {
        $headers[] = 'Authorization: Bearer ' . $authToken;
    }

    $request = curl_init('https://attribu.tech/api/bot-traffic');
    curl_setopt_array($request, [
        CURLOPT_POST => true,
        CURLOPT_HTTPHEADER => $headers,
        CURLOPT_POSTFIELDS => json_encode($payload, JSON_UNESCAPED_SLASHES),
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_CONNECTTIMEOUT_MS => 300,
        CURLOPT_TIMEOUT_MS => 1000,
    ]);

    curl_exec($request);
    curl_close($request);
}

trackAttribuCrawler(
    'YOUR_SITE_ID',
    getenv('ATTRIBU_BOT_TOKEN') ?: null
);

Python (Flask / Django)

import re
import requests
from threading import Thread

BOT_RE = re.compile(
    r"bot|crawl|spider|chatgpt|gptbot|claudebot|anthropic|perplexity|googlebot",
    re.IGNORECASE,
)

def track_bot(ua, url, ip, status_code=None):
    try:
        payload = {
            "websiteId": "YOUR_SITE_ID",
            "domain": url.split("//")[-1].split("/")[0],
            "href": url,
            "ai": {
                "userAgent": ua,
                "ip": ip,
                "source": "server_middleware",
            },
        }
        if status_code:
            payload["ai"]["statusCode"] = status_code
        requests.post(
            "https://attribu.tech/api/bot-traffic",
            json=payload,
            timeout=2,
        )
    except Exception:
        pass

# In your middleware or after_request hook:
ua = request.headers.get("User-Agent", "")
if BOT_RE.search(ua):
    Thread(
        target=track_bot,
        args=(ua, request.url, request.remote_addr),
    ).start()

Privacy and safety checklist

Optional category filters

By default, Attribu tracks all relevant bot traffic categories. You can disable categories if you only care about specific crawler types:

trackAICrawlerRequest(request, event, {
  websiteId: "YOUR_SITE_ID",

  disableAnswerFetch: true,
  disableSearchCrawlers: true,
  disableTrainingCrawlers: true,
  disableOtherCrawlers: true,
});

Most websites should keep the defaults. The dashboard lets you filter the data later by AI answers, indexing, training, and verification confidence.

Where to find the data

After installing the package, open your Attribu dashboard and look for the Bot traffic card. You can filter by crawler type, show only IP-verified crawlers, and inspect which pages each provider requested.

If you do not see data immediately, that usually means no known crawler has requested your server-rendered pages yet. Human pageviews do not appear in this card.

Supported crawlers

Attribu detects 40+ crawlers including:

• AI answers: ChatGPT-User, PerplexityBot, YouBot, Phind, Cohere-AI, iAskBot, Meta-ExternalAgent
• Indexing: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot, Slurp, SeznamBot, KagiBot, BraveSearch
• Training: GPTBot, ClaudeBot, Bytespider, CCBot, Applebot, Google-Extended, FacebookBot, Amazonbot, SemrushBot, AhrefsBot, DotBot, PetalBot, Diffbot, DataForSeoBot

What's next

• Connect Stripe to attribute revenue to traffic sources
• Custom events to track signups and conversions
• Proxy the script to bypass ad blockers