(Credit: SOPA Images / Contributor / LightRocket via Getty Images)
Cloudflare has introduced a new way for websites to stop AI crawlers from stealing their content without permission—and to potentially get them to pay up.
Cloudflare is an internet infrastructure service that helps websites manage and secure their traffic, such as fending off Distributed Denial of Service (DDoS) attacks. Starting today, Cloudflare is "changing the default to block AI crawlers unless they pay creators for their content," says CEO Matthew Prince. "That content is the fuel that powers AI engines, and so it's only fair that content creators are compensated directly for it."
Some AI companies have inked deals with publishers for access to their content; others have sued for copyright infringement. But not everyone has lawsuit money. Cloudflare says it wants to help its customers get paid with "Pay per crawl," a new service that creates "authentication mechanisms to create a framework for paid content access."
"Even if a crawler doesn’t have a billing relationship with Cloudflare, and thus couldn’t be charged for access, a publisher can still choose to ‘charge’ them—the functional equivalent of a network level block," Cloudflare says.
Sites determine pricing, and they have the option to allow specific crawlers. Pay per crawl is currently in private beta; interested sites and crawlers can reach out to Cloudflare.
A Sustainable Digital Ecosystem?
The feature could replace an old way websites typically block bots—updating their robots.txt file—which is no longer sufficient in the AI era, as many data scrapers ignore the instructions in the robots.txt file. That's why PCMag's parent company Ziff Davis is suing OpenAI.
"We applaud Cloudflare for advocating for a sustainable digital ecosystem that benefits all stakeholders—the consumers who rely on credible information, the publishers who invest in its creation, and the advertisers who support its dissemination," says Vivek Shah, CEO of Ziff Davis, in a statement provided to Cloudflare.
Reddit CEO Steve Huffman, who has expressed frustration with AI crawlers, praised Cloudflare's new solution as a "more transparent" approach and "a step in the right direction for everyone." And Cloudflare's press release lists several top publishing companies who "embrace a permission-based model for AI crawling," including Condé Nast, The Atlantic, the Associated Press, Buzzfeed, ADWEEK, Fortune, Dotdash Meredith, and Gannett Media.
Cloudflare says it manages traffic for 20% of the web and "trillions of requests daily," which may be enough to take a meaningful bite out of AI companies' crawlers. However, the impact on the overall AI ecosystem depends on how many publishers choose to adopt the new setting.
Prince tells The New York Times he is "100% confident we can block them from accessing the content. And if they don't get to access the content, then their product will be worse."
If widely adopted, this could encourage a new business model for AI companies and publishers, requiring payment for content and a more consent-based approach to data scraping.