Cloudflare Starts Blocking AI Training Bots Today

A change that has been coming for months lands today. Cloudflare, which sits in front of a large share of the web, now blocks two kinds of AI crawler by default on pages that carry ads. The shift applies to new domains, new customers, and existing sites on the free tier, and it reflects how much the traffic mix has moved. Cloudflare says 52 percent of the crawler requests on its network are now collecting data to train models, up from about 22 percent in the spring of 2025.

The company sorts crawlers into three buckets. Search means bots that index a page so it can answer a question later. Agent means software acting in real time on a person's behalf. Training means a crawler taking content to train or fine-tune a model. Under the new defaults, Search stays welcome. Agent and Training do not.

The part that trips people up

Here is the complication, flagged by Help Net Security and others. A lot of the big crawlers do more than one job. Googlebot, Applebot, and Bingbot gather pages for search and for model training with the same visit. Cloudflare's system cannot split one request in two, so if a site blocks Training, those multi-purpose bots get blocked outright, even though Search was meant to stay open. Publishers who want AI companies kept out may find they have quietly dropped out of Google's index as well.

That trade sits at the centre of a fight over who owns the value in web content. For two years, site owners have watched their pages feed models that then answer readers directly, sending nothing back. Cloudflare is offering a blunt instrument to push back, packaged as a candidate industry standard through its Content Signals format. Whether the major labs honour those signals, or route around them, is the open question.

The timing is pointed. It arrives in the same week that lab leaders are publicly arguing about how fast to build and whether the industry should regulate itself. Control over training data is one of the few levers content owners actually hold, and today they get to pull it by default rather than by choice.