Open source · self-hosted

Bot Observability

See exactly which AI crawlers, search engines, and other bots are reading your sites, which pages they hit, and whether those requests are healthy, broken, or worth blocking.

It's a Next.js + Postgres app you run yourself: point it at your own database, drop one API call into your app, and traffic starts showing up in the dashboard below.

View on GitHub Get started

Already running your own instance? Sign in

Top bots this week

Example data

5,200

3,200

2,800

1,600

1,400

1,100

Bot hits (7d)

18,090

AI share

61.6%

11,140 hits

See it in action

Example data — not a live query

Flip through the same tabs the real dashboard has — the numbers below are illustrative, but the panels, colors, and layout are exactly what you'll see once your own traffic starts flowing in.

Bot hits (7d)

18,090

AI share

61.6%

11,140 hits

Error rate

1.2%

217 4xx/5xx hits

Data health

Healthy

Heartbeat 2m ago · Event 4s ago

Crawler mix

AI training leads

Other

3201.8%

How it works

Send events

Your app POSTs each request's user agent, path, and status code to /api/bot-hit as it's served — from middleware, an edge function, or backend logging code.

Bots get identified

Every event is matched against 130+ known crawler patterns and (where possible) verified via reverse-DNS or IP range, server-side, on the way in.

You inspect the traffic

The dashboard rolls hits up by bot, category, page, and status so you can see who's crawling, what they're reading, and what's breaking.

What you get

Bot identity

Detect crawler families, categories, and verified vs UA-only traffic.

Traffic trends

Compare bot, page, and project movement across daily, weekly, and monthly views.

Response quality

Review status classes, failing paths, API hits, and sensitive route probes.

What you can inspect

Overview

Totals, crawler mix, daily trend, time-of-day distribution, movers, and AI crawls-vs-visits by company.

Bots

Full bot list with categories, project spread, hit share, and last seen time — includes a per-bot detail view and an AI-only filter.

Health

2xx/3xx/4xx/5xx mix, failing paths, and sensitive/API path hits.

Raw Events

Filterable event log for individual bot requests and debugging.

Response status reporting

Status reporting depends on tracked sites sending status_code. Rows without that field are shown as not captured, not as a real HTTP status.

Use the Status tab to find bots reading broken URLs, hitting API routes, probing admin/login paths, or generating server errors.

Useful ingestion fields

projectGroups traffic by product, site, or deployment.

url / pathIdentifies which content bots are requesting.

user_agentDrives bot detection and category assignment.

ipEnables reverse-DNS and CIDR verification when available.

status_codeSeparates successful reads from redirects, errors, and uncaptured rows.

heartbeatTracks whether the logging pipeline is still alive.

Bot categories

AI training

Bulk model data collection

AI search

Citation and retrieval indexes

AI agent

User-triggered page fetches

Indexes pages for search results

SEO

Third-party rank-tracking and SEO tools

Social preview

Link previews and unfurlers

Monitoring

Uptime and performance checks

Generic

Scripts, libraries, archivers, and unclassified bots

Get started

Requires Node.js 20+ and a Postgres database (Aiven, Neon, or any standard Postgres). Clone the repo, then:

npm install
cp .env.example .env        # set DATABASE_URL and BOT_LOG_TOKEN
npm run migrate
npm run dev

Open /dashboard and sign in with your BOT_LOG_TOKEN. Full setup, deployment, and ingestion docs are in the README.