AI Crawler Monitor and Analytics

Monitor verified AI crawler activity without overstating it

Choose the evidence source before drawing a chart. Origin logs may omit responses served from cache, while CDN, edge, WAF, or supported tracker records may see a broader request stream. Document the production hosts, collection layer, timezone, retention, sampling, exclusions, and proxy-address handling so one period can be compared with another.

Maintain a crawler registry from official provider documentation. Store the exact User-Agent, provider, stated purpose, verification method, source link, and last review date. Keep search, training, and user-triggered agents separate. Report a string-only match as unverified when a source address or trusted verified-bot signal cannot be confirmed.

Normalize every event before aggregation. Resolve host variants and redirected URLs to a canonical path, keep the original status and final outcome, and distinguish content pages from assets, health checks, and internal tests. Show request count, requested-page count, first seen, last seen, status distribution, and verification state rather than presenting a single opaque score.

Compare activity with the site inventory. Use the sitemap and priority public-page list to identify requested, never-observed, redirected, blocked, or failing resources. An unobserved page is not automatically blocked: it may be new, weakly linked, cached at another layer, outside retention, or not selected during the period. Preserve those alternatives in the conclusion.

Use trends for diagnosis. Annotate robots changes, WAF changes, deployments, migrations, major publishing dates, and outages. If activity changes, inspect the affected provider, purpose, path, and response evidence before attributing the movement to content quality or platform preference.

Keep downstream outcomes in separate panels. A crawler request proves a resource was requested at a time. Search indexing, AI answer inclusion, citations, human referrals, and conversions need Search Console, provider reporting or controlled observations, analytics, and business data. They can be compared by date without being collapsed into the same metric.

Turn one verified finding into action. Fix an accidental block, redirect, server failure, missing discovery path, or weak source page; deploy; then repeat the exact collection query. Preserve before and after evidence and avoid broad policy or content changes that cannot be attributed or safely rolled back.

Frequently asked questions

Which AI crawler metrics are useful?

Useful metrics include verified visits, requested URLs, status codes, crawl frequency, blocked requests, and changes after publishing or technical updates.

Which page should this page support?

This page should support the central AI bot tracking tool page.

Related internal links

Methodology and limits

Reviewed on 2026-08-26 against official OpenAI, Anthropic, Perplexity, and Cloudflare crawler documentation. Published access policy, verified request traffic, citations, referrals, and conversions are reported as separate evidence layers.

Sources

• OpenAI: crawler purposes, controls, and published IP ranges
• OpenAI: publisher controls and ChatGPT referral attribution
• Anthropic: ClaudeBot, Claude-SearchBot, and Claude-User
• Perplexity: crawler identities and verification guidance
• Cloudflare: AI crawler directory and categories