Cloudflare offers free detection and blocking of AI bots
Cloudflare helps website owners control how AI models use their content, and even try to monetize it.
This comes after the company earlier this year announced a simple tool for website owners to prevent AI developers from scraping content.
Cloudflare said its AI Audit product provides a suite of tools to help customers understand how AI models are used for content.
Once they know what the content is being used for, they can then decide whether or not they are willing to allow AI developers to access it.
Furthermore, they can also set what they consider a fair price for allowing AI bots to extract content for use in model training and other purposes.
The practice of scraping content from websites has become very common in the AI industry, as the internet provides a treasure trove of seemingly free data that can be used to train AI models.
This mass extraction of website data is controversial, with a group of content creators and publishers claiming it is unfair, especially since they are not aware that it is happening.
Cloudflare said it helps every web developer leverage data for a fair price, and AI Audit is designed to give control back to content creators, allowing for a transparent exchange between the two parties.
AI Audit includes a simple tool that automatically blocks access to content for all types of AI bots, along with a suite of analytics tools that help site owners understand what AI bots are doing on their sites.
According to Cloudflare, this helps website owners understand why AI models are accessing web pages, when they are, how frequently, and even distinguish between AI bots that attribute their data source and those that don't.
Additionally, Cloudflare's AI Audit product provides a tool for site owners to determine a fair price for allowing AI bots to access their content, based on benchmark prices negotiated by major publishers.
Cloudflare says this is necessary because small site owners lack the resources and expertise to understand the value of their content and negotiate deals with AI companies.
Moreover, the AI companies themselves do not have the ability to strike a deal with every website they scrape data from, because there are millions of such sites.