Permission, visits, and outcomes are different.
robots.txt can express crawler policy, infrastructure logs can record requests, and citations or referrals require their own evidence. One layer cannot prove the next.
Track the Claude Crawler You Actually Mean.
Name the identity, verify the request, and keep policy, traffic, and citation claims separate.
Turn a verified access or traffic finding into one bounded website task.
Open the AI Workspace
Quick Answer
To track ClaudeBot correctly, define whether you mean ClaudeBot, Claude-SearchBot, or Claude-User; inspect the deployed robots.txt rule for the named identity; collect server, CDN, edge, or WAF requests; match the exact User-Agent; verify against Anthropic guidance and the current source IP list where available; record requested URLs, status codes, first and last seen times, and uncertainty; and never present a crawler hit as proof that Claude trained on, cited, or recommended the page.
AI Summary
This provider-specific tracker covers Anthropic crawler families, access intent, verification, traffic evidence, and safe interpretation in one operational workspace.
Identity needs verification.
A User-Agent string can be copied. Use provider guidance and available IP or verified-bot evidence before labeling a request as confirmed.
One focused workspace
Stop jumping between tools.
Stop jumping between tools.
Keep the workflow in one place.
Audit, AI visibility, research, content optimization, and reporting-organized around the website you are improving.
ReplacesWhat you getOther tools
TruboRankAI
TruboRankAIEverything above vs TruboRankAI - one subscription$755+/mo$19.99/moSave ~97%
Third-party names and logos are trademarks of their owners and are shown for comparison. Other-tool prices are illustrative monthly benchmarks; exact plans and costs may vary.
Crawler evidence workflow
Track ClaudeBot by identity, path, and evidence layer
Start with the question you need to answer, then use the evidence layer that can actually answer it.
A defensible ClaudeBot tracking workflow
Anthropic documents separate crawler identities. ClaudeBot is associated with model development, Claude-SearchBot supports search-related discovery, and Claude-User is associated with user-directed retrieval. Build filters and policy notes around those published names rather than grouping every request containing “Claude” into one chart.
Test the deployed policy for representative paths. A global rule can differ from a documentation, media, parameter, or private directory rule. Anthropic documents robots support and a Crawl-delay behavior, but edge security, rate limits, authentication, and origin health can still determine the actual response.
For observed traffic, retain the raw request and mark verification status. Compare the source address with Anthropic’s current guidance when feasible and investigate forwarded-address handling behind a proxy. A verified request plus a successful response is meaningful crawl evidence, but it still stops short of proving later use.
Review coverage after identity verification. Compare requested Claude crawler paths with the canonical sitemap and priority content inventory. A page with no observed request may be new, weakly linked, outside the retention window, cached at another layer, or simply not selected for crawling; report these possibilities instead of calling absence a block.
Keep a small evidence record for every important observation: canonical URL, requested path, timestamp and timezone, status code, response bytes, User-Agent, verification state, and the edge or origin source. Aggregate by provider purpose only after preserving raw evidence. Sampling, caching, proxying, log retention, and privacy controls can change totals, so document those boundaries beside the chart.
Separate crawler families by purpose. OpenAI documents GPTBot, OAI-SearchBot, and ChatGPT-User separately; Anthropic distinguishes ClaudeBot, Claude-SearchBot, and Claude-User; Perplexity distinguishes PerplexityBot and Perplexity-User. A policy intended for training collection may not be the policy intended for search discovery or a user-triggered fetch.
Use the result as a diagnosis, not a visibility score. A verified request proves a resource was requested at a particular time. It does not prove indexing, model training, answer inclusion, citation, recommendation, referral traffic, or conversion. Measure those outcomes with their own named reports and dates.
Frequently asked questions
Are ClaudeBot and Claude-SearchBot the same?
No. Anthropic documents them as separate identities with different purposes.
Does robots.txt prove ClaudeBot visited?
No. It describes policy; a visit requires server-side request evidence.
Does a ClaudeBot request prove a Claude citation?
No. Crawling and answer citation are separate outcomes.