Cloudflare syncs AI bot rules with robots.txt, closing its own policy gap

Product manager Jin-Hee Lee's feature is headed for an upcoming rollout, publishing Search, Agent and Training choices while preserving existing Disallow rules.

By · Published

Charcoal and chalk sketch showing robots.txt rules like Disallow and User-agent on a central network gate, directing various AI bots.

Product manager Jin-Hee Lee's feature is headed for an upcoming rollout, publishing Search, Agent and Training choices while preserving existing Disallow rules.