Cloudflare syncs AI bot rules with robots.txt, closing its own policy gap

Why it matters

Cloudflare is turning its position in front of websites into an AI traffic policy layer, linking crawler identification, published consent and edge enforcement in one control.

Cloudflare product manager Jin-Hee Lee will roll out Bot Preference Sync in an upcoming launch, giving website operators one control panel for the AI crawler rules they enforce at Cloudflare's edge and the preferences they publish in robots.txt.

Lee joined Cloudflare full time as a product manager after interning in its product organization, according to her 2024 announcement. Her work carries forward the operating idea that Matthew Prince, Michelle Zatlyn and Lee Holloway used to build Cloudflare: observing abusive Internet traffic was useful, but customers wanted infrastructure that could act on what it saw. Bot Preference Sync applies that instinct to a less tidy problem, where a site's public instructions and its security rules can say different things.

Cloudflare introduced separate controls for Search, Agent and Training crawlers on July 1st. Lee's new feature takes the choices already made in that dashboard and writes the corresponding directives into robots.txt. If a customer has an existing file, Cloudflare says its generated section will be added at the top while preserving the site's existing Disallow directives.

The setting will be available from Cloudflare's Free tier through Enterprise and can be disabled. Cloudflare says it will be switched on by default for new customers. Existing users of Cloudflare's older managed robots.txt feature will be asked to review their settings before moving across.

One decision, two layers

Robots.txt tells a cooperative crawler where it should and should not go. Cloudflare's bot controls can enforce a block at the network edge. Website operators have had to maintain both layers, creating room for a crawler to encounter a public Disallow instruction without facing a corresponding technical block, or to be blocked despite seeing a permissive file.

Bot Preference Sync removes that configuration drift for Cloudflare's category-level controls. Search and Agent traffic can be allowed, blocked on pages serving ads or blocked everywhere. Selecting Disallow for Training causes Cloudflare to publish a no-training preference while retaining separate enforcement against crawlers that fail Cloudflare's transparency requirements.

Cloudflare will build the generated section from bots tracked in BotBase and periodically update it as classifications change. Operators can inspect the classifications in Cloudflare Radar's public bots directory.

That maintenance work matters because bot identities and purposes are moving targets. A crawler may build a search index, retrieve a page for an AI agent and collect material for model training under one user agent. Cloudflare's July 1st taxonomy treats those as separate behaviors, even when one operator combines them.

Lee has worked on that classification problem across several Cloudflare releases. Her Cloudflare author page credits her on the July traffic controls, BotBase-related behavior systems, managed robots.txt controls and cryptographic bot recognition. Bot Preference Sync turns that body of detection work into a simpler product decision for a site owner: choose what a class of crawler may do, then let Cloudflare maintain the matching public instructions.

Transparency becomes an access condition

Cloudflare is also using the feature to pressure mixed-use crawler operators to disclose more about what happens after they fetch a page. A bot combining Search and Training must respect a no-training preference, offer site owners a way to opt out of AI summaries, provide URL-level visibility into pages made available for training and publish evidence that declining training does not reduce conventional search visibility.

Cloudflare says crawlers meeting those conditions may retain search access when a customer disallows training. Crawlers that do not qualify remain subject to the block. Cloudflare's verification framework also depends on bots identifying themselves and honoring the relevant robots.txt preference.

The distinction is central to the product. Syncing a file cannot compel an unidentified crawler to behave. It gives cooperative operators a consistent instruction and lets Cloudflare enforce the customer's decision against traffic it can identify. Cloudflare documented the limit last year when it accused Perplexity of using undeclared crawlers after encountering blocks and no-crawl directives. Perplexity rejected Cloudflare's characterization at the time.

Bot Preference Sync therefore reduces an administrative failure mode without pretending robots.txt has become an access-control system. Its practical value comes from combining the published preference with Cloudflare's edge enforcement and bot directory.

Cloudflare wants to own the policy layer

Cloudflare's founders originally built a service between websites and unwanted traffic. AI crawlers have expanded that position from security infrastructure into a policy layer for the content economy. Publishers, retailers and software businesses now have different incentives around search discovery, agent access and model training, and those incentives can change page by page.

An online store may want its product pages indexed and available to shopping agents. An ad-supported publisher may want search referrals while keeping training crawlers away from articles whose value depends on a human page view. Bot Preference Sync packages those choices into broad categories rather than requiring each operator to maintain an expanding list of user agents.

The trade-off is precision. Cloudflare says the sync does not read individual custom rules or reproduce special arrangements with particular crawler operators. Customers with exceptions or more complex logic must disable the category-wide sync and manage their own file.

The same distinction applies to the agentic web: the relevant question is who may read a page, for what purpose and under whose stated identity.

For Lee, the release is a product-management answer to a problem created by the growing sophistication of Cloudflare's own controls. Once Search, Agent and Training became separate policy choices, asking customers to manually restate those choices in a static file left the product half-finished. The sync closes that loop. Whether crawlers honor the published answer remains a question of identity, enforcement and the conduct of the bot operator.