Track the exact Anthropic crawler identity and preserve enough infrastructure evidence to verify the request.
To track ClaudeBot traffic, first decide whether the target is ClaudeBot, Claude-SearchBot, or Claude-User. Check the deployed robots policy for that identity, collect edge or server requests, match the full User-Agent, verify source evidence using Anthropic’s current guidance where possible, normalize canonical URLs and status codes, and report verified and unverified activity separately. Do not infer training, answer use, citations, or human traffic from a request.
This implementation guide turns Anthropic’s published crawler distinctions into a safe logging, verification, and reporting workflow.
Main Explanation
Create one registry entry per documented Anthropic identity. Store its purpose, exact matching rule, current documentation link, verification procedure, and review date. This makes later policy decisions explicit and prevents a search crawler, training crawler, and user-directed fetch from being reported as one behavior.
Inspect the complete delivery path. A robots rule may allow a crawler while a CDN challenge, WAF rule, rate limit, authentication layer, redirect loop, or origin timeout prevents access. Record edge and origin outcomes where possible and test representative public pages rather than only the homepage.
Build a reproducible query over logs. Select the production host and time range, preserve UTC alongside local time, match the exact identity, apply IP or verified-bot evidence, exclude internal tests, group by canonical path and status, and retain a sample of raw events for audit. Label coverage and retention gaps.
Use the report to answer bounded questions: which public pages were requested, whether the response succeeded, whether failures cluster on a route, and whether activity changed after a deliberate policy or deployment change. Citation monitoring and Claude-driven human referrals belong in separate reports.
Do not collapse access, activity, and visibility into one metric. robots.txt answers what a cooperative crawler is permitted to request. A verified infrastructure record answers what requested a resource. A provider answer or citation study answers whether a source appeared in a defined observation set. Referral analytics answers whether a human session arrived. Each report needs its own numerator, denominator, scope, and date.
Preserve implementation safety. Logs can contain IP addresses, URLs, query strings, and identifiers. Minimize collection, avoid exposing private routes in dashboards or exports, restrict access, define retention, and redact sensitive parameters. robots.txt is not authorization; account, billing, API, webhook, admin, and user-data routes need real authentication and access control.
After establishing a baseline, fix one confirmed dependency and repeat the same test. Examples include an accidental disallow, a WAF rule blocking a desired verified bot, a server error on a priority page, or missing logging at the correct layer. Keep the before-and-after evidence and monitor separately for citations, referrals, and business outcomes.
Practical Steps
- List the three documented Claude identities.
- Choose the intended policy for each.
- Test representative paths and delivery layers.
- Query infrastructure logs.
- Verify source evidence.
- Normalize URLs, statuses, and time.
- Report confirmed and uncertain events separately.
- Retest after one change.
FAQ
Can GA4 track ClaudeBot?
It is not reliable for crawler requests; use infrastructure logs.
Does Anthropic respect robots.txt?
Anthropic documents robots support for its crawlers, but actual delivery can also be affected by security and server controls.
What should a ClaudeBot report include?
Identity, verification state, path, status, timestamp, count, collection scope, and stated limitations.
Sources and methodology
Reviewed on 2026-08-26 using official provider crawler documentation. Recommendations distinguish policy tests, verified server-side traffic, provider visibility observations, web referrals, and conversions; User-Agent-only matches are not treated as conclusive identity.
These references support the changeable facts and study findings discussed above. Results depend on each source's sample, date, market, query set, and measurement method.