Cloudflare Lets Websites Block AI Training While Keeping Search Visibility

WebMCP is an emerging browser standard that lets websites expose structured tools to AI agents, replacing brittle page interpretation, simulated clicks and scraping with direct, typed actions. Experimental support is shipping in Chrome 146. The approach could make tasks such as search, form completion and transactions more reliable while keeping activity inside the visitor’s browser session, preserving the publisher’s interface, traffic and control.

Cloudflare launched a developer preview on Aug. 6, 2026, allowing domain owners to activate WebMCP from Agent Readiness > Labs in the Cloudflare Dashboard. The company uses HTMLRewriter at the edge to inject a single reference to a same-origin bridge script, leaving origin code unchanged. The preview includes two tool packs — Content Credentials and Site MCP Server — enabled by default and executed in the visitor’s browser. The latter discovers tools from an existing MCP endpoint and registers them for compatible agents, including Cloudflare’s BrowserRun.

Cloudflare Workers is a serverless platform that runs applications across the company’s global edge network. As AI agents increasingly generate and execute code autonomously, developers face a sharper trade-off between rapid deployment and control over untrusted software. Dynamic Worker Loader addresses that risk by placing generated code in separate V8 isolates, helping prevent an agent’s output from affecting the host application or reaching systems beyond its authorized scope.

As of Aug. 4, 2026, Cloudflare has made Dynamic Worker Loader available to customers on paid Workers plans. Developers can load a new Worker at runtime and execute AI-generated code in an isolated environment that starts within milliseconds. The service can block outbound network access, intercept requests and inject selected bindings. Cloudflare has also integrated the capability with Code Mode, bringing code generation, execution and governance onto one architecture, without disclosing a separate price.

Cloudflare has long helped websites block malicious crawlers and is now offering a compliant content-extraction tool. The /crawl API, introduced in July 2025, streamlines the creation of AI training datasets and retrieval-augmented generation knowledge bases, reducing the cost of handling site navigation, content cleaning and format conversion in-house.

The new /crawl API is currently in open beta. Developers can crawl an entire website with a single API call and convert its pages into Markdown, or use Workers AI to produce JSON in a specified structure. The service also supports incremental updates, allowing RAG pipelines to retrieve only new or modified content and improving data-refresh efficiency.