The nonprofit Creative Commons (CC), renowned for championing open licensing for creators, has announced its cautious support for “pay-to-crawl” technology. This system aims to automate compensation for website content accessed by machines, such as AI webcrawlers, addressing a significant shift in the digital landscape.

Earlier this year, CC introduced a framework for an open AI ecosystem, intended to provide legal and technical guidelines for data sharing between content controllers and AI providers. Their tentative backing of pay-to-crawl stems from the belief that, if implemented responsibly, it could offer websites a sustainable model for content creation and sharing. This mechanism could manage substitutive uses of content, ensuring its public accessibility rather than it disappearing behind more restrictive paywalls.

The concept of pay-to-crawl, spearheaded by companies like Cloudflare, proposes charging AI bots each time they scrape a site for content to train and update their models. Historically, websites permitted webcrawlers to index their content for search engines like Google, benefiting from increased traffic and clicks. However, the rise of AI chatbots has altered this dynamic; consumers often receive direct answers, eliminating the need to click through to original sources. This trend has already significantly impacted publishers by reducing search traffic, threatening their financial viability.

A pay-to-crawl system could provide a much-needed revenue stream for publishers to offset the losses incurred due to AI. It is particularly beneficial for smaller web publishers who lack the leverage to negotiate individual content deals with major AI providers, unlike larger entities such as OpenAI with Condé Nast or Perplexity with Gannett.

Despite its support, Creative Commons emphasized several critical caveats. They warn that such systems could concentrate power on the web and potentially restrict access for public interest actors, including researchers, nonprofits, cultural heritage institutions, and educators. To mitigate these risks, CC proposed a series of principles for responsible pay-to-crawl implementation. These include not making pay-to-crawl a default setting, avoiding blanket rules, allowing for content throttling instead of outright blocking, preserving public interest access, and ensuring the systems are open, interoperable, and built with standardized components.

The pay-to-crawl space is seeing increased activity beyond Cloudflare. Microsoft is developing an AI marketplace for publishers, while startups like ProRata.ai and TollBit are also innovating in this area. Furthermore, the RSL Collective introduced the Really Simple Licensing (RSL) standard, a specification that dictates which parts of a website crawlers can access without completely blocking them. RSL has gained support from major players including Cloudflare, Akamai, Fastly, Yahoo, Ziff Davis, and O’Reilly Media. Creative Commons has also endorsed RSL, aligning it with its broader CC signals project aimed at developing tools for the AI era.