🎉 #Gate xStocks Trading Share# Posting Event Is Ongoing!
📝 Share your trading experience on Gate Square to unlock $1,000 rewards!
🎁 5 top Square creators * $100 Futures Voucher
🎉 Share your post on X – Top 10 posts by views * extra $50
How to Participate:
1️⃣ Follow Gate_Square
2️⃣ Make an original post (at least 20 words) with #Gate xStocks Trading Share#
3️⃣ If you share on Twitter, submit post link here: https://www.gate.com/questionnaire/6854
Note: You may submit the form multiple times. More posts, higher chances to win!
📅 End at: July 9, 16:00 UTC
Show off your trading on Gate Squ
"Cyber Bodhisattva" shows its power, the era of "free lunch" for AI giants has come to an end.
Elon Musk and Donald Trump, this pair of White House duo has recently entered the "verbal battle" 2.0 phase. Similar to this pair of frenemies are the love-hate relationships between foreign publishing groups and AI giants—on one hand, major publishers want to collaborate with AI companies, while on the other hand, some publishers vow to bankrupt the AI giants.
According to the data, after the emergence of AI search and ChatGPT, global website traffic has been declining; on the other hand, AI giants' "AI crawlers" are continuously eroding data from all websites by disregarding crawling protocols and making tens of thousands of requests.
At this moment, a construction company finally stepped forward, holding the hands of content creators and saying, "We can say no to the AI giants!"
Cloudflare, the internet infrastructure giant that controls approximately 20% of global web traffic, has been dubbed the "Cyber Bodhisattva" by netizens. In July 2025, it launched an experimental product and trading market: "Pay Per Crawl" - establishing new rules for AI crawlers:
Either obtain permission or pay.
In simple terms, the essence of this feature is to give website content creators an option "switch": they can choose to allow AI crawlers free access, charge per crawl, or directly block access.
According to the founder of Cloudflare, "Content is the fuel that drives the AI engine, so it is fair for content creators to be compensated directly."
For AI companies, if they want to continue scraping content from the entire internet to train models, they can no longer "eat a big meal for free" as they did before. However, there are benefits as well, because paying according to the clear pricing can help avoid copyright disputes.
Can Cloudflare's recent "anti-bot" measures alleviate the rampant attacks from AI crawlers? More importantly, can this company leverage its unique position to establish a brand new content distribution and monetization model for the AI era?
01
The "free lunch" of AI giants
For the past few decades, most web pages have been publicly "crawlable" by default. Search engines like Google and Bing bring traffic to websites, and with traffic, websites monetize through advertising or subscription sales - this is the unspoken contract of the search era.
In the era of AI, traditional search traffic has plummeted, and the more you calculate, the more losses you incur.
AI companies treat all online content as training fuel, yet they rarely compensate most creators. When users ask questions directly in AI chatbots, the answers often come from summarized content rather than dozens of blue links, which does not drive more traffic to the websites.
Even search giants like Google are changing; in the past, they provided lists of website links, but now they have launched "AI Overview" on their search page. According to their report, 75% of query users get answers without clicking any links.
Cloudflare's latest data from July 2025 shows that Google's crawler brings back 1 click for about every 6 to 7 crawls to a website, while OpenAI only achieves 1 referral for every 1500 times, and Anthropic's ratio is even more exaggerated, reaching 1 for every 73300.
The click-through rate brought to websites by AI crawlers from major companies each time | Image source: Cloudflare
This means that the traditional "content for traffic" model has failed. Compared to traditional search engines, AI giants consume massive amounts of website content without providing "traffic", and this imbalance makes it increasingly difficult for some content creators to survive.
"With OpenAI, the difficulty of acquiring website traffic is 750 times higher than in the Google era, and with Anthropic, the difficulty is as high as 30,000 times. The reason is simple: we are increasingly consuming derivatives of original content rather than the original content itself." Cloudflare CEO Matthew Prince said in a blog post, "This is not a fair trade."
AI companies also pay a price for scraping data. Over the past two years, AI giants have been accused of "stealing content" to train large models, leading to a wave of copyright lawsuits globally, particularly with ongoing lawsuits involving news organizations like The New York Times and OpenAI.
Active large factory AI crawler robots | Image source: Cloudflare
Therefore, Cloudflare has launched "Pay Per Crawl" to establish a market for "pay-per-crawl" that aims to solve this problem.
The company has designed a permissions and payment system where the website can choose to "allow, block, or charge" AI crawlers in the backend. If an AI crawler wants to scrape content from the website, it must register, verify its identity, and make a payment each time it accesses the site.
If all goes well, this model can allow online content to shift from "advertising monetization" to "content licensing monetization," opening up new sources of income. Whether it's large media or niche blogs, they can have bargaining power in the AI era and be paid for their use by AI.
To emphasize its significance, the Cloudflare CEO referred to the day of the launch of "Pay Per Crawl" as:
"Independence Day of Content".
02
How is the AI "toll" collected?
Of course, the vision is beautiful, but how does the technology come to fruition?
Cloudflare started as a company providing CDN, DDoS protection, DNS, zero trust security, and other services. It has deployed nodes in over 300 cities worldwide, handling about 20% of web traffic, which facilitates its role as an "intermediary."
"Pay Per Crawl" is built on the intermediate layer of its global CDN network: it can identify and process AI crawlers before access requests reach the origin site. Webmasters can set three modes in the Cloudflare backend: Allow, Charge, Block.
The site administrator can set permissions, charges, or block in the backend | Image source: Cloudflare
All websites newly added to Cloudflare block AI crawlers by default, unless the webmaster actively allows it. Only AI companies that have established a partnership with Cloudflare can participate in the payment mechanism; otherwise, they will be blocked.
If an AI crawler makes a request to a paid URL without having paid, Cloudflare will return an HTTP 402 Payment Required status code—an almost unused status code in the past, specifically reserved for "online payments." The AI crawler can include payment information in the request to indicate agreement to pay the configured price, and once the price matches, it will be released with a 200 OK response and automatically settle the payment.
Cloudflare itself is the "cash register" for this transaction, responsible for aggregating bills and distributing profits.
Cloudflare will return an HTTP 402 Payment Required status code | Image source: Cloudflare
Crawlers can include payment information in the request | Image source: Cloudflare
HTTP 200 OK response confirms charge | Image source: Cloudflare
More importantly, this cannot be bypassed simply by using User-Agent spoofing. Cloudflare requires AI companies to register keys and guarantee identity with digital signatures. This is also to prevent "counterfeit crawlers" from posing as compliant entities to evade payment.
In the past, robots.txt was a plain text file placed in the root directory of a website to inform search engine crawlers which pages could be crawled and which could not. However, it was merely a "polite suggestion" from the website, and many AI crawlers simply ignored it. Cloudflare's solution changed this by transforming the existing "soft constraints" of robots.txt into "hard gates."
However, according to Cloudflare, only about 37% of the top 10,000 domains currently have a robots.txt file.
Set levels for AI crawlers | Image source: Cloudflare
To participate in Cloudflare's paid crawling market, both crawlers and the crawled must set up Cloudflare accounts. As of now, "Pay Per Crawl" is still in the beta phase, with only a few large publishers involved, such as BuzzFeed, The Atlantic, and Fortune. Cloudflare is also continuously soliciting interested content creators and crawlers.
"We expect significant growth in the pay-per-use model," said Cloudflare officials.
Although it is still in the early stages, the company has many ideas for the future. For example, publishers or other organizations could charge different fees for different types of content, or implement dynamic pricing based on the number of users of the AI application, or introduce more granular pricing strategies based on different areas such as training, inference, and search.
They also believe that the true potential of pay-per-use crawlers may be revealed in the world of Agent intelligent agents.
"What if smart agents could operate entirely programmatically behind a paywall? Imagine being able to ask your deep research assistant to compile the latest cancer research, legal briefs, or help you find the best restaurants—then giving that smart agent a budget to acquire the most useful and relevant content."
"The first solution based on the HTTP 402 response code will open up a future where a smart agent can programmatically negotiate access to digital resources," said Cloudflare.
03
Crossroads of the Internet
From an economic perspective, this could be the beginning of AI and a wide range of content creators "renegotiating revenue shares."
Now, only major media outlets can negotiate licensing with AI companies (for example, the New York Times only reached a settlement after reporting on OpenAI), while the vast majority of small and medium-sized websites, forums, and even individual authors have been "silently crawled away" with no ability or awareness to resist. Cloudflare's solution can actually extend this bargaining power to a wider range of websites.
According to the Cloudflare team, they have had hundreds of conversations with news organizations, publishers, and major social media platforms, and they all "hope to allow AI crawlers to access their content, but expect to be compensated."
For supporters, the "Pay Per Crawl" model is conceptually very "fair": creators earn income, AI companies avoid legal risks, and in the long run, it can drive the entire industry towards more compliant content licensing.
Image source: Cloudflare
Of course, AI companies may not be happy that internet data is no longer free. To capture new content, they have to spend money, which means there are cost factors beyond computing power.
On the other hand, this may also suppress rampant scraping and force AI model developers to be more selective with their data—such as selectively purchasing high-value content instead of indiscriminately feeding all kinds of website content into their models.
Matthew Prince said, "The AI engine is like a Swiss cheese; the truly original content that can fill the holes in this cheese is more valuable than the repetitive, low-value content that currently occupies most of the web."
In his view, traffic has always been unable to accurately measure the value of content. "If we could start scoring and evaluating content not based on how much traffic it generates, but rather on how it promotes knowledge (measured by how many existing holes it fills in the AI engine's 'Swiss cheese') — we could not only help AI engines progress faster but also potentially foster a new golden age of high-value content creation."
However, digital rights advocates may argue: can small AI startups, researchers, and open-source communities bear such data costs? Will academic research and public archiving, these "benign crawlers," struggle to move forward and only be able to access limited, low-value data sources?
In a reality where advertising revenue is declining and traffic costs are rising, how many websites would be willing to open themselves up for AI crawlers to take advantage of for free? Will this mark the beginning of "closure," causing the internet to lose its spirit of freedom and sharing?
If the entire network defaults to blocking fees, will this inadvertently exacerbate "big company monopoly"? After all, big companies have more money.
The "Pay Per Crawl" model attempts to address the issue of AI sucking content without giving back, while also potentially unintentionally raising the threshold for AI innovation, returning to the old topic of copyright protection and knowledge openness.
Of course, Cloudflare just gives websites more autonomy. Website owners can completely choose to continue providing free access to public welfare and non-profit projects. The power is still in the hands of the creators. In any case, they deserve to receive "compensation."
In the words of the Cloudflare CEO, the goal of this transformation is to "build a better internet." "We do not have all the answers yet, but we are working with some top economists and computer scientists to find the answers."
At present, other CDN and security providers (such as Akamai, Fastly, Amazon CloudFront) have not announced similar features.
Keep AI crawlers outside the door | Image source: Cloudflare
Although Cloudflare's "Pay Per Crawl" may seem like just a new feature of a CDN product, in a sense:
It could be a signal that the internet has reached a crossroads.
In the era of search, the value of content is converted into advertising revenue through user visits. But in the AI era, users may not even click into websites anymore—all answers are summarized and generated by chatbots. Should we continue to let AI large models freely mine web content, or should we return to the principle of "reciprocity" in data acquisition, allowing creators to receive their due compensation? How much compensation can there be?
This early experiment may be paving the way for a new data economy in the AI era. Regardless of success or failure, its stance is clear: AI cannot endlessly exploit the patience of creators and turn human labor into free fuel in the name of "openness."
"The internet is undergoing a transformation, and its business model will change accordingly. In this process, we have the opportunity to learn from the good aspects of the past 30 years to make it better in the future."
As for whether things can really get better, as Cloudflare itself admits:
This is just the beginning.