Meta AI Crawlers Hit 9.1 Billion Quarterly Requests

Meta is accelerating efforts to build its own web-search index, a move aimed at reducing its AI products’ reliance on external services such as Google. A proprietary index could give Meta’s models broader and more timely access to online information. The push also raises concerns for publishers and developers, who bear the bandwidth, computing and maintenance costs of automated crawling without necessarily receiving the referral traffic traditionally generated by search engines.

Meta’s AI crawlers made 9.1 billion requests in the latest quarter covered by a DataDome report, driven largely by rapid growth in Meta-WebIndexer, the security company said. Some developers reported that the volume pushed VPS loads high enough to trigger alerts. Despite the heavy crawling, the activity generated almost no return traffic for affected websites, sharpening scrutiny of Meta’s indexing strategy and its impact on online content providers.