On July 1st, Cloudflare, Inc., the leading connectivity cloud company, recently announced it is now the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation.
What is Cloudflare?
Cloudflare is a top internet infrastructure provider that helps improve websites' and applications' speed, security, and reliability. A key feature is its content delivery network (CDN), which delivers web content through a global network of servers to ensure faster load times and enhanced protection. Many hosting companies that support Mediavine publishers—such as BigScoots—or WordPress support services like NerdPress, offer integrated access to Cloudflare. Publishers can also connect to Cloudflare directly or by using a WordPress plugin.
What is Mediavine's stance on using Cloudflare to restrict access for AI bots?
Mediavine supports using Cloudflare to block AI crawlers that do not provide value in return through meaningful traffic or direct compensation. We believe strongly in a fair value exchange: if AI companies want to use publisher content to train their models or power their platforms, they should drive traffic back to the source or pay for access. While that balance doesn’t yet exist across the industry, Cloudflare’s tools give content creators greater control and are a step toward a more equitable ecosystem.
Could there be negative consequences from preventing AI crawlers via Cloudflare?
Mediavine does not feel there are immediate negative impacts from preventing specific AI crawlers via Cloudflare. Mediavine is closely monitoring the evolving landscape around AI crawlers and their effect on publishers. While blocking specific AI bots through Cloudflare can help protect your content, it may also limit future visibility or potential monetization opportunities. Our stance may adapt as the industry and regulatory environment continue to develop, and we will keep our content creators aware of our point of view.
Might restricting AI bots affect my site's visibility or organic search traffic?
According to Cloudflare, blocking AI bots through their tools will not impact your site’s visibility in traditional search results. Their approach distinguishes between AI crawlers used for model training or generative tools and the bots used for standard SEO indexing (like Googlebot). This means your site’s organic search rankings remain unaffected, while allowing you to control how your content is used by AI companies that do not provide direct value in return.
Can you explain the concept of "pay per crawl" and how it works?
Pay Per Crawl is a private beta program from Cloudflare that allows creators to charge AI crawlers a fee to access their content.
How It Works:
- Detection: Cloudflare identifies AI crawlers accessing your site.
- Response Mechanism: If a crawler attempts to access content without prior payment, Cloudflare responds with an HTTP 402 "Payment Required" status, providing pricing details.
- Payment Processing: AI crawlers that agree to the terms can include payment credentials in their requests. Upon successful payment, they receive standard HTTP 200 responses, granting access to the content.
- Publisher Control: Website owners can configure settings to allow, charge, or block specific AI crawlers, tailoring access based on their preferences.
Creators interested in participating can reach out to Cloudflare for more information.
Will these settings be enabled by default?
Cloudflare has stated that AI bot blocking will be enabled by default for users on the free tier. However, paid users will need to activate this feature manually. You can manage and customize these settings directly through your Cloudflare dashboard or contact the provider from whom you receive Cloudflare services.