Cloudflare Launches New /crawl API for AI and RAG Data Collection

Cloudflare has long helped websites block malicious crawlers and is now offering a compliant content-extraction tool. The /crawl API, introduced in July 2025, streamlines the creation of AI training datasets and retrieval-augmented generation knowledge bases, reducing the cost of handling site navigation, content cleaning and format conversion in-house.

The new /crawl API is currently in open beta. Developers can crawl an entire website with a single API call and convert its pages into Markdown, or use Workers AI to produce JSON in a specified structure. The service also supports incremental updates, allowing RAG pipelines to retrieve only new or modified content and improving data-refresh efficiency.