Open source · self-hosted · built for SEO and GEO teams

See who is crawling your websites

Bot Observability is a self-hosted dashboard for tracking AI crawlers, search engines, SEO tools, and automated agents across your web projects.

See which URLs they request, whether those requests succeed, and how crawler activity changes over time. Connect your app with one server-side API call and inspect the results across projects.

View on GitHub Get started

Already running your own instance? Sign in

Top bots this week

Example data

5,200

3,200

2,800

1,600

1,400

1,100

Bot hits (7d)

18,090

AI share

61.6%

11,140 hits

See it in action

Example data — not a live query

Flip through the same tabs the real dashboard has — the numbers below are illustrative, but the panels, colors, and layout are exactly what you'll see once your own traffic starts flowing in.

Bot hits (7d)

18,090

AI share

61.6%

11,140 hits

Error rate

1.2%

217 4xx/5xx hits

Data health

Healthy

Heartbeat 2m ago · Event 4s ago

Crawler mix

AI training leads

Other

3201.8%

How it works

Send events

Your app POSTs each request's user agent, path, and status code to /api/bot-hit as it's served — from middleware, an edge function, or backend logging code.

Bots get identified

Every event is matched against 130+ known crawler patterns and (where possible) verified via reverse-DNS or IP range, server-side, on the way in.

You inspect the traffic

The dashboard rolls hits up by bot, category, page, and status so you can see who's crawling, what they're reading, and what's breaking.

What you get

Bot identity

Detect crawler families, categories, and verified vs UA-only traffic.

Traffic trends

Compare bot, page, and project movement across daily, weekly, and monthly views.

Response quality

Review status classes, failing paths, API hits, and sensitive route probes.

What you can inspect

Overview

Totals, crawler mix, daily trend, time-of-day distribution, movers, and AI crawls-vs-visits by company.

Bots

Full bot list with categories, project spread, hit share, and last seen time — includes a per-bot detail view and an AI-only filter.

Health

2xx/3xx/4xx/5xx mix, failing paths, and sensitive/API path hits.

Raw Events

Filterable event log for individual bot requests and debugging.

Response status reporting

Status reporting depends on tracked sites sending status_code. Rows without that field are shown as not captured, not as a real HTTP status.

Use the Status tab to find bots reading broken URLs, hitting API routes, probing admin/login paths, or generating server errors.

Useful ingestion fields

projectGroups traffic by product, site, or deployment.

url / pathIdentifies which content bots are requesting.

user_agentDrives bot detection and category assignment.

ipEnables reverse-DNS and CIDR verification when available.

status_codeSeparates successful reads from redirects, errors, and uncaptured rows.

heartbeatTracks whether the logging pipeline is still alive.

Bot categories

AI training

Bulk model data collection

AI search

Citation and retrieval indexes

AI agent

User-triggered page fetches

Indexes pages for search results

SEO

Third-party rank-tracking and SEO tools

Social preview

Link previews and unfurlers

Monitoring

Uptime and performance checks

Generic

Scripts, libraries, archivers, and unclassified bots

Get started

Requires Node.js 20+ and a Postgres database (Aiven, Neon, or any standard Postgres). Clone the repo, then:

npm install
cp .env.example .env        # set DATABASE_URL and the role-specific secrets
npm run migrate
npm run dev

Open /dashboard and sign in with your BOT_ADMIN_TOKEN. Send events with project-scoped BOT_INGEST_TOKENS. Full setup, deployment, and ingestion docs are in the README.