Open source · self-hosted · built for SEO and GEO teams

See who is crawling your websites

Bot Observability is a self-hosted dashboard for tracking AI crawlers, search engines, SEO tools, and automated agents across your web projects.

See which URLs they request, whether those requests succeed, and how crawler activity changes over time. Connect your app with one server-side API call and inspect the results across projects.

View on GitHub Get started

Already running your own instance? Sign in

Top bots this week

Example data

5,200

3,200

2,800

1,600

1,400

640

Bot hits (7d)

18,090

AI share

61.6%

11,140 hits

See it in action

Example data — not a live query

Explore illustrative examples of the dashboard views. Your own Overview centers on one traffic chart, request summaries and links to evidence, with detailed analysis available on demand.

Bot hits (7d)

18,090

AI share

61.6%

11,140 hits

Error rate

1.2%

217 4xx/5xx hits

Data health

Healthy

Heartbeat 2m ago · Event 4s ago

Crawler mix

AI training leads

AI training8,40046.4%

Search4,20023.2%

AI search2,10011.6%

SEO1,4508.0%

Social preview9805.4%

AI agent6403.5%

Other3201.8%

How it works

Send events

Your website sends bot request details to /api/bot-hit. The Next.js request proxy reports redirects it returns and 200 for requests it forwards; a backend that observes the final response can report that instead.

Bots get identified

Every event is matched against 130+ known crawler patterns and (where possible) checked using forward-confirmed reverse DNS for supported search crawlers, server-side, on the way in.

You inspect the traffic

Review request counts, observed pages and reported outcomes, with links to retained request evidence.

What you get

Bot identity

Detect crawler families, categories, and verified vs UA-only traffic.

Traffic trends

Compare bot, page, and project movement across daily, weekly, and monthly views.

Response quality

Review status classes, failing paths, API hits, and sensitive route probes.

What you can inspect

Overview

Totals, crawler mix, daily trend, time-of-day distribution, movers, and AI crawls and user-triggered fetches by company.

Bots

Full bot list with categories, project spread, hit share, and last seen time — includes a per-bot detail view and an AI-only filter.

Health

2xx/3xx/4xx/5xx mix, failing paths, and sensitive/API path hits.

Raw Events

Filterable event log for individual bot requests and debugging.

Response status reporting

Status reporting depends on tracked sites sending status_code. Rows without that field are shown as not captured, not as a real HTTP status.

Use the Status tab to find bots reading broken URLs, hitting API routes, probing admin/login paths, or generating server errors.

Useful ingestion fields

project credentialAssigns traffic to the configured website project.

url / pathIdentifies which content bots are requesting.

user_agentDrives bot detection and category assignment.

ipEnables forward-confirmed reverse DNS for Googlebot, Bingbot and Applebot when an original client IP is available.

status_codeSeparates successful reads from redirects, errors, and uncaptured rows.

heartbeatOptional receipt, separate from received bot requests.

Bot categories

AI training

Bulk model data collection

AI search

Citation and retrieval indexes

AI agent

User-triggered page fetches

Indexes pages for search results

SEO

Third-party rank-tracking and SEO tools

Social preview

Link previews and unfurlers

Monitoring

Uptime and performance checks

Generic

Scripts, libraries, archivers, and unclassified bots

Get started

Use Node.js 24 and a Postgres database (Aiven, Neon, or any standard Postgres). Clone the repo, then:

npm install
npm run setup     # creates .env with secrets + runs migrations
npm run dev

Open /dashboard and sign in with your BOT_ADMIN_TOKEN. Open Website connection to install the sender and verify a website probe. Probes do not add bot traffic. Installation and connection steps are in the README.