Bot traffic reporting

Bot Observability

A focused dashboard for seeing which bots crawl your projects, which pages they hit, and whether those requests are healthy, broken, or suspicious.

Operational views

Detect crawler families, categories, and verified vs UA-only traffic.

Compare bot, page, and project movement across daily, weekly, and monthly views.

Review status classes, failing paths, API hits, and sensitive route probes.

Totals, crawler mix, daily trend, time-of-day distribution, and movers.

Full bot list with categories, project spread, hit share, and last seen time.

AI training, AI search, and agent traffic with verification breakdowns.

Stacked bot activity by day, week, and month for pattern changes.

2xx/3xx/4xx/5xx mix, failing paths, API hits, and uncaptured status coverage.

Filterable event log for individual bot requests and debugging.

Status reporting depends on tracked sites sending status_code. Rows without that field are shown as not captured, not as a real HTTP status.

Use the Status tab to find bots reading broken URLs, hitting API routes, probing admin/login paths, or generating server errors.

projectGroups traffic by product, site, or deployment.

url / pathIdentifies which content bots are requesting.

user_agentDrives bot detection and category assignment.

ipEnables reverse-DNS and CIDR verification when available.

status_codeSeparates successful reads from redirects, errors, and uncaptured rows.

heartbeatTracks whether the logging pipeline is still alive.

AI Training	Bulk model data collection
AI Search	Citation and retrieval indexes
AI Agent	User-triggered page fetches
Search Engine	Traditional organic discovery
Social Preview	Link previews and unfurlers
SEO Tool	Audits and competitive scans
Monitoring	Uptime and performance probes
Archival	Historical page preservation
Generic / CLI	Scripts, libraries, unknowns