Cloudflare Launches New /crawl API for AI and RAG Data Collection

Website operators have struggled to prevent their content from being used to train artificial intelligence models without also sacrificing traffic from traditional search. The conflict is most acute with mixed-use crawlers such as Googlebot, Applebot and Bingbot, which can serve both search indexing and AI training. The issue matters for publishers and other sites funded by advertising, subscriptions or referrals because AI-generated answers can replace direct visits, weakening the business models that finance online content.

Cloudflare launched its Disallow AI Training setting on September 15, 2026, separating controls for Search, Training and Agent traffic. Apple, Google and Microsoft have met or committed to its “Accountable” requirements, allowing their mixed-use crawlers to index participating sites while honoring no-training preferences; other training crawlers can be blocked. New ad-supported domains are offered presets that allow search, disallow training and block AI agents on pages carrying ads. Cloudflare said 17% of sites restrict training, while fewer than 1% block search; mixed-use crawlers account for 36.6% of verified crawler traffic.