Infrastructure providers like Cloudflare are redefining who controls access to AI for web content, and this shift is affecting AI companies, publishers, retailers, merchants and consumers.
When Cloudflare announced it would start blocking Perplexity AI’s stealth crawling, it became a formidable gatekeeper of the open web.
This new control raises profound debates about openness, fairness, advancement and the economics of content on the internet.
In fact, Vercel CEO Guillermo Rauch takes the opposite stance from Cloudflare, suggesting, The fastest path to irrelevance: blocking progress. Blocking what consumers actually want: low-friction interfaces. The internet is changing. The answer to AI is: more AI. Not to block and stagnate. Our metrics show Perplexity and ChatGPT have had an extremely positive effect on our business.”
The Perplexity Incident: Enforcement and Accountability in Action
Serving nearly 20% of web traffic and boasting over 60% market share in content delivery networks, Cloudflare’s decisive action to delist and block Perplexity AI’s crawlers is a big deal.
The company’s investigation revealed Perplexity employed stealth crawling tactics, bypassing robots.txt restrictions by spoofing user agents, rotating IP addresses and scraping content from private domains without permission.
While Perplexity framed the issue as an essential service blockade, the underlying problem is clear: Publishers and infrastructure providers demand respect for established web protocols and content ownership.
Cloudflare’s Pay-Per-Crawl Model: Empowerment or Control?
Cloudflare’s initiative to block AI crawlers by default with an option for pay-per-crawl access gives publishers and merchants unprecedented control and monetization power over AI data extraction. This tactic transforms bots from invisible freeloaders into regulated, consent-based participants.
However, the pay-per-crawl method consolidates significant power in Cloudflare’s hands. It allows them to control access for millions of sites on a global scale, effectively positioning it as an internet tollbooth for AI crawling and discovery. Critics compare this to the gatekeeper roles Apple and Google assumed in app stores, now unprecedentedly broadened to core web infrastructure.
Impact on Publishers: Protection, Monetization and Visibility Challenges
Publishers have been among the loudest advocates for Cloudflare’s new policies. Years of unregulated AI crawling has skyrocketed server costs, drained advertising revenue and caused the free-riding of content without referral traffic.
For example, AI bots from companies like OpenAI and Anthropic exhibit crawl-to-referral ratios as high as 1,700-to-1 and 73,000-to-1, severely impacting publisher monetization. Comparatively, Google’s search crawlers have a crawl-to-referral ratio of roughly 14-to-1.
Cloudflare’s tools let publishers flip a red button to block AI crawlers network-wide, ending endless Whac-A-Mole battles with unauthorized bots. The new pay-per-crawl model offers the potential to monetize content and regain economic value, with major media companies like Condé Nast, Dotdash Meredith and Gannett.
Yet, this comes with trade-offs. Blocking AI crawlers may reduce publishers’ visibility in AI-generated summaries and answers, possibly shrinking indirect discovery channels.
Impact on Retailers and Merchants: Visibility Risks and Strategic Choices
As more brands and retailers are becoming media companies and more publishers are evolving into retailers, they face a nuanced dilemma. AI-powered shopping tools and zero-click commerce rely on crawling for product data aggregation and discovery.
Unintentional blocking of AI crawlers risks invisibility in these emerging sales channels, potentially costing brand awareness and sales. Conversely, allowing crawlers grants visibility but risks data commodification and loss of direct traffic control.
Cloudflare’s model offers retailers and transactional first publishers (e.g., Tripadvisor) the option to monetize AI access to their product content. But AI companies’ willingness to pay remains limited, particularly outside major publishers.
Retailers, especially small and midsize companies, could become a major casualty. They must adopt a strategic approach, either auditing Cloudflare settings or maybe even moving away from Cloudflare. Additionally, they must selectively permit trustworthy AI bots and ensure structured data markup to maximize discoverability without sacrificing control.
Broader Implications and the Governance Debate
Cloudflare’s shift from infrastructure enabler to content access gatekeeper raises pressing questions about the future of the open web.
Is this a needed step to protect creators and support a fair AI ecosystem? Supporters say curbing AI free-riding and enabling monetization is key to sustaining journalism and creative work.
Or does it risk centralizing too much power in one provider? Critics warn that handing crawling control to Cloudflare could lead to opaque gatekeeping and stifle competition and innovation.
Can AI progress coexist with content ownership? Limiting access may slow AI or raise costs, but open scraping threatens content value and long-term quality.
This shift demands new governance frameworks, including:
- Standards to distinguish ethical crawling from exploitation
- Transparency in crawler behavior and blocking
- Infrastructure diversity to prevent monopolies
- Legal frameworks balancing IP, innovation and public interest
A New Era of Content Control and Internet Governance
Cloudflare’s AI tollbooth addresses publisher grievances and empowers merchants with new leverage. However, the new method centralizes an unprecedented degree of power over AI data access in one company’s hands.
The Perplexity incident signals the end of the AI Wild West and the start of a tightly policed, monetized internet—a digital web where who controls the bots influences what content thrives and which businesses succeed.
All stakeholders—publishers, retailers, AI companies, governments and users—must engage in this debate, shaping how openness, fairness and innovation coexist with AI.
Is Cloudflare’s AI tollbooth the necessary evolution for a sustainable internet or the first step toward a gatekept web ruled by monopolistic infrastructure?