There’s a new AI licensing standard for scraping web content, which is now official. The question is: will it protect their content?
According to The Verge (A pay-to-scrape AI licensing standard is now official, written by Emma Roth and available here), an open licensing standard that aims to make AI companies pay for the content they vacuum up across the web is now an official specification. Really Simple Licensing 1.0 — or RSL for short — gives publishers the ability to dictate licensing and compensation rules to the web crawlers that visit their sites.
The RSL Collective announced the standard in September with backing from Yahoo, Ziff Davis, and O’Reilly Media. It’s an expansion of the robots.txt file, which outlines the parts of a website a web crawler can access. Though RSL alone can’t block AI scrapers that don’t pay for a license, the web infrastructure providers that support the standard can — a list that now includes Cloudflare and Akamai, in addition to Fastly.
RSL’s 1.0 release lets publishers block their content from AI-powered search features, like Google’s AI Mode, while maintaining a presence in traditional search results. Currently, Google doesn’t give websites an individual option to opt out of AI-powered features without booting them out of traditional search, too.
“RSL provides exactly that missing layer,” RSL Collective cofounders Doug Leeds and Eckart Walther say in an emailed statement to The Verge. “Using RSL, Google can respect a publisher’s preferences at the use case level, which means a publisher can stay fully available in traditional search, while opting out of AI training, grounding, or generative answers.”
As I discussed yesterday, Google is currently facing an investigation from the European Commission, which is looking into whether the company has violated antitrust policies by using web publishers’ content in AI search features “without offering them the possibility to refuse such use of their content.”
“With this release, and the support for it across the internet ecosystem, RSL 1.0 becomes the expected and trusted way to communicate how content may be used in AI systems, giving those signals real weight in both practice and legal interpretation,” Leeds says.
The RSL Collective also worked with the Creative Commons to add a new “contribution” payment option for nonprofit organizations and individuals behind the webpages, code repositories, and datasets that make up “the shared pool of freely available knowledge and creative work on the internet.”
I love the idea! The question is: can they make it work and force the AI companies to “buy the cow” when they’ve been “getting the milk for free”? We’ll see.
So, what do you think? Do you think this AI licensing standard will ensure publishers get paid for their content? Please share any comments you might have or if you’d like to know more about a particular topic.
Image created using Microsoft Designer, using the term “robot hand reaching through a computer screen grabbing money.”
Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.