AI crawlers

Crawlers do not run JavaScript. The browser snippet on your pages is invisible to every one of them, so a second, server side path exists: a small middleware on your own server reports each crawler request to Peeko, and the AI tab fills in.

Nothing about it touches your visitors. There is no cookie, no browser code and no IP address on this path at all, because a crawler has no visitor to count.

Install the middleware

npm i @usepeeko/bot-track

It is ESM only and needs Node 20 or newer. It has zero runtime dependencies and weighs under 3 KB gzipped, because it runs in edge middleware where every kilobyte is on the request path. pnpm add and yarn add work the same way.

It reports to https://usepeeko.com/b, which is a different path from the browser tracker's /a, with the same site key. Your site settings print the snippet with your real key in it.

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";
import { trackBotRequest } from "@usepeeko/bot-track";

const cfg = {
  siteKey: process.env.PEEKO_SITE_KEY!,
  endpoint: "https://usepeeko.com/b",
};

export function middleware(request: NextRequest) {
  trackBotRequest(request, cfg); // fire and forget, swallows its own errors
  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

What it skips on its own

Not every request is worth a beacon, so the package drops most of them before any network call.

• Anything that is not a GET or a HEAD.
• Sub resource loads, judged by the browser's own Sec-Fetch-Dest header: images, scripts, styles, fonts and the rest.
• App internals and assets: sixteen path prefixes, /api, /_next and /assets among them, and twenty five file extensions.
• Any user agent that is not a known AI or search crawler.

robots.txt, llms.txt, llms-full.txt and any .xml path with "sitemap" in it are always reported despite their extensions, because who asked for them is exactly the thing worth knowing.

What is recorded

For each hit Peeko stores the path, the raw user agent string, the company and agent it classified, what the crawler came for, and the response status when your install can report one.

No address is read or stored on this path, and there is nothing to hash: a crawler is not a person. The raw user agent is kept, up to 512 characters, because it is not personal data and because it is what lets Peeko reclassify old hits as the crawler list grows.

The endpoint is rate limited per site, at 600 reports a minute, rather than per address: the caller is your server, so every report from a busy crawl arrives from one or two addresses anyway. A report over that limit is dropped and answered 204 like everything else, so a very large crawl in a very short minute can lose hits.

The site key travels in the body of the beacon. It is a public value, the same one that ships in your page source, so the endpoint is knowingly open: anybody who reads your HTML could post to it. It answers 204 to everything, valid or not, and never says whether a key is real.

How a crawler is recognized

Peeko matches the user agent against a curated list of 55 agents from 21 companies, and then, for an agent it does not know by name, against a shorter list of company tokens. An agent it cannot place at all is dropped and nothing is stored.

Each recognized hit is filed under one of four purposes.

Because the server does the classifying, you rarely need to upgrade the package. A brand new agent is picked up with no change on your side whenever it comes from one of the fourteen companies whose name Peeko watches for in a user agent, and it is filed under Other AI bots.

That fallback is deliberately narrow, for two reasons worth knowing. A company whose only broad token is already one of the precise agents gets no name watched for, because the precise pass matches it first and the copy would be dead: Apple and Applebot are the clearest case. And a bare company name is unsafe on its own, because every one of these companies now ships a consumer app that puts its product name into a real person's user agent. A "claude" token was once recording people using the Claude desktop app as Anthropic crawlers, which is why a user agent shaped like a browser is now never matched by name. Seven of the twenty one companies therefore have no name watched for at all, and a genuinely new agent from one of them waits for a newer version of the package.

An identity is a claim. A user agent is a string anything can copy, so every name on this tab is what the requester said it was, never something Peeko verified. The product says so on the page, and it is worth repeating here.

Reading the AI tab

The tab opens on every crawler the site has seen in the range, as a list you can select from, with four tiles above the detail: AI hits, Answer fetches, Providers and Clicks from AI. A chart under them shows hits per day by what they came for, on any range longer than a day: Today and Last 24 hours draw no chart, because a crawler's rhythm is a thing you read over days.

The detail pane has up to four views.

• Activity: the most recent hits, each one saying why the bot came and which page it read.
• Pages: the paths crawlers read most.
• 404s: the paths they asked for and did not get. This view is explained below.
• Clicks: people, not bots. When somebody follows a citation from ChatGPT, Claude, Perplexity, Copilot or Gemini, the referrer says so, and those visits are counted here from your ordinary page views. No middleware is involved in this number. It is on the overview only, since a click from an AI product has no crawler to attribute it to.

An answer fetch tells you a page was read while an assistant replied to somebody. The question itself never reaches your server, so nobody, Peeko included, can tell you what was asked. Only which page was read.

Suspicious requests

Some of what arrives is not reading your site at all. Requests for the shapes that hold secrets and configuration are counted separately and shown as a warning you can expand, in three groups: secrets and credentials, version control, and config and backups. You can add your own patterns in the site's settings, or remove ones that are legitimate paths on your site. Nobody links to any of them, so a crawler asking is looking for something rather than reading anything. It is also the clearest evidence that a name is only a claim.

Why your 404 list may be missing

Peeko never probes your site, so it only knows a request came back missing if your own install says so. That takes the second call, reportBotHit, which most frameworks can make after they know the response status.

Next.js is the exception, and it matters. Next renders the not found boundary for some requests that resolve with a 200, so reporting a 404 from app/not-found.tsx would mark live pages as missing. Do not do it. On Next, skip the 404 echo: crawler hits still record and only this one dimension stays dark.

When an install has never reported a status, the 404s view removes itself rather than showing an empty list, and the page says why. An empty list would be a claim that no crawler ever got a 404, which is a different and unknown thing.

The three files crawlers ask for

When a crawler asks for one of these and Peeko can see that it got a 404, the activity feed offers you the fix on that row. On an install that cannot report a status, which includes every Next.js one, the row is there and the offer is not.

• llms.txt, a plain text summary of the site for a language model.
• sitemap.xml, the list of pages you want found.
• robots.txt, who may crawl what, and the Content-Signal line that says whether your pages may be used for search, for answers and for training.

The same numbers elsewhere

get_bot_summary over MCP returns the whole tab as one payload. Check responseStatusAvailable before saying anything about 404s: when it is false, missingHits is null because 404s are unknowable for that site, not because there were none.

The command line has peeko bots, and --provider openai narrows every number and list to one company.