Cloudflare now distinguishes AI bot behavior by Search, Agent, and Training use, so a single “block AI” decision is no longer precise enough for many publishers.
Open Cloudflare Security Settings and review AI bot policies, Bot Fight Mode, custom WAF rules, managed robots.txt, and AI Crawl Control together. Cloudflare documents separate Search, Agent, and Training classifications with Allow, Block on pages with ads, or Block on all pages options, plus updated defaults scheduled for new domains on September 15, 2026. Choose a policy by business purpose, test representative public URLs, and verify the result in Security Analytics and origin logs.
This guide maps Cloudflare's AI bot controls, execution order, behavior classifications, upcoming defaults, managed robots.txt, Bot Fight Mode, custom rules, and AI Crawl Control. It shows how to preserve security while preventing accidental blocks of wanted search or agent traffic and avoids assuming that an allowed crawler will index, cite, or recommend the site.
Cloudflare AI access decision table
| Traffic purpose | Typical public-page decision | Verification |
|---|---|---|
| Search | Allow when AI search discovery supports the content strategy | Security Analytics plus verified crawler logs |
| Agent | Allow or constrain based on real-time user value and cost | Test paths, rate limits, and origin behavior |
| Training | Choose according to content licensing and policy | Review behavior setting and managed robots.txt |
| Private application routes | Block or require strong authentication | Confirm denial without relying on robots.txt |
| APIs and webhooks | Use explicit authentication and signatures | Test valid and invalid requests separately |
Main Explanation
Start with policy, not a toggle. Search crawlers can collect content for later answers, Agent traffic can act in real time for a user, and Training crawlers can collect material for model development. A publisher may want different outcomes for each use. Document which public sections should be discoverable, which agent actions are acceptable, which training uses are allowed, and which private or expensive routes must remain unavailable.
Cloudflare's current AI bot policy documentation offers behavior-level controls and three actions: allow, block on pages detected as displaying ads, or block on all pages. It also announces new defaults for new domains on September 15, 2026, with Search remaining allowed while some Agent, Training, and mixed-purpose behavior is blocked depending on the page and classification. Verify the live dashboard because defaults, classifications, and rollout dates can change.
Review all enforcement layers. A permissive robots.txt file does not override a WAF rule, Bot Fight Mode challenge, AI Crawl Control block, rate limit, or origin denial. Cloudflare documents that AI Crawl Control crawler-blocking custom rules run before bot solutions, while pay-per-crawl processing occurs afterward. Conflicting layers can make a dashboard setting appear ineffective until the earlier rule is identified.
Protect private and state-changing routes regardless of crawler preference. Account, dashboard, checkout, API mutation, webhook, admin, staging, tracking, and user-data endpoints should rely on authentication, authorization, CSRF or signatures, and appropriate WAF controls—not on robots.txt. Public documentation, product explanations, evidence pages, and canonical articles can use a different policy from sensitive application surfaces.
Test representative URLs after every change. Request the home page, an article, a product page, a static asset, an API route, and a private route with the intended policy context. Review Cloudflare Security Analytics for the action and service, then confirm origin logs where the request should pass. A user-agent string alone can be spoofed; use the verification method published by the crawler provider when identity matters.
Watch for false conclusions. Seeing zero crawler requests after allowing a class does not mean the configuration failed; the crawler may not visit during the observation window. Seeing a successful request does not prove indexing, model training, citation, referral traffic, or conversion. Preserve timestamps, paths, status codes, Cloudflare actions, and verified network evidence so each conclusion stays bounded.
Revisit the configuration after Cloudflare platform changes, migrations, advertising changes, or new public sections. TurboRankAI can inspect the public robots.txt and supported crawler behavior from the website side, while Cloudflare remains the authority for edge actions. The goal is an intentional access matrix: wanted public discovery works, expensive or private routes remain protected, and the owner can explain why.
Practical Steps
- Write the desired Search, Agent, and Training policy.
- Inventory public, private, advertising, API, and expensive routes.
- Review Cloudflare AI bot policies and scheduled defaults.
- Inspect Bot Fight Mode, custom rules, managed robots.txt, and AI Crawl Control.
- Resolve conflicts according to Cloudflare's execution order.
- Test representative paths and review Security Analytics.
- Verify wanted traffic at the origin and preserve sensitive-route controls.
- Monitor after rollout and document platform or policy changes.
FAQ
Does turning off Block AI Bots allow every bot?
No. Other Cloudflare layers, custom WAF rules, Bot Fight Mode, AI Crawl Control, rate limits, origin controls, and robots.txt can still affect requests.
Should search and training bots have the same policy?
Not necessarily. Cloudflare now classifies Search, Agent, and Training behavior separately so publishers can choose according to discovery, licensing, cost, and security needs.
Can I protect private routes with robots.txt?
No. robots.txt communicates crawler preferences and is not access control. Protect private routes with authentication, authorization, and edge or origin security.
Sources and methodology
The workflow follows Cloudflare's current product documentation and OpenAI's published crawler distinctions, reviewed on 2026-08-17. The September 15, 2026 defaults are presented as a scheduled Cloudflare change and should be rechecked at deployment. No allow rule is described as guaranteeing crawling, indexing, training, citations, or traffic.
These references support the changeable facts and study findings discussed above. Results depend on each source's sample, date, market, query set, and measurement method.