<p>As the demand for accurate data to fuel AI models continues to rise, an escalating battle is unfolding on the internet. Companies are taking measures to protect their data from AI web crawlers, with OpenAI’s GPTBot and Common Crawl’s CCBot facing increased opposition. Recent data reveals that over 250 leading websites are now blocking GPTBot, while nearly 14% of the most popular websites have also barred CCBot, impacting the availability of vital information for AI training.</p> 
<h2 class="wp-block-heading" id="the-growing-battle-against-ai-web-crawlers">The growing battle against AI web crawlers</h2> 
<p>In an era dominated by data-driven decision-making and artificial intelligence, web crawling bots have become both essential tools and significant sources of controversy. Last month, OpenAI introduced GPTBot, which was designed to respect the decades-old robots.txt protocol, allowing websites to signal their desire to remain unscraped. Initially, approximately 70 out of the top 1,000 websites implemented blocks against GPTBot, including internet giants like Amazon and Tumblr.</p> 
<p>Recent findings from Originality.ai reveal a significant shift in the landscape. Over a span of just three weeks, the number of prominent websites blocking GPTBot has surged to more than 250. This comprehensive list includes well-known platforms like Pinterest, Vimeo, GrubHub, Indeed, Apartments.com, The Guardian, Live Science, USA Today, NPR, CBS News, NBC News, CNBC, The New Yorker, People, and seemingly all titles published by Hearst and Conde Nast. Even weather.com has joined the ranks of those shielding their content from the AI crawler.</p> 
<h2 class="wp-block-heading" id="the-challenge-to-data-accessibility">The challenge to data accessibility</h2> 
<p>The impetus behind this wave of web crawler blocks lies in the critical role that accurate data plays in training powerful generative AI models, such as OpenAI’s GPT-4. These models rely heavily on vast amounts of text data, much of which is sourced from the internet, despite the majority of it being subject to copyright or owned by specific entities. The awareness of this practice has surged in recent times, leading to numerous legal disputes and the possibility of new government regulations.</p> 
<p>Simultaneously, many companies are taking measures to secure their user-generated content and online activities. Through updates to their terms of service and user policies, tech companies are asserting their rights to access and utilize user data for AI projects and training purposes. This shift in approach represents a broader trend towards data protection and sovereignty, with companies increasingly seeking to control how their data is used, particularly by AI-driven entities.</p> 
<h2 class="wp-block-heading" id="ccbot-is-another-target-of-the-blockade">CCBot is another target of the blockade</h2> 
<p>While GPTBot has drawn significant attention due to its association with OpenAI and the GPT-4 model, another web crawler, CCBot, employed by Common Crawl, is facing its share of resistance. Common Crawl, a European-based organization, has spent years amassing vast amounts of web data, including copyrighted content, and organizing it for use as training data for large language models like Meta’s Llama.</p> 
<p>As of late September, data from Originality.ai indicates that almost 14% of the 1,000 most popular websites have implemented blocks against CCBot. This list includes Amazon, Vimeo, Masterclass, Kelly Blue Book, The New York Times, The New Yorker, and The Atlantic, among others. It’s noteworthy that many of the websites blocking CCBot also extend their restrictions to GPTBot, illustrating the growing trend of safeguarding data from AI web crawlers, regardless of their affiliations.</p> 
<p>The battle between companies seeking to protect their data and AI web crawlers in pursuit of training material continues to intensify. The growing number of websites blocking GPTBot and CCBot underscores the increasing importance of data accessibility and control in an <a href="https://www.cryptopolitan.com/meta-unveils-generative-ai-enhancements/" data-type="post" data-id="374978" rel="noopener noreferrer nofollow">AI-driven</a> world. As legal and regulatory scrutiny mounts, the balance between data utilization and data protection remains a critical challenge for both businesses and AI developers.</p>

Companies Block AI Web Crawlers including GPTBot, Impacting Data Availability

Table of Contents

Cryptopolitan_News

Other articles published on Sep 28, 2023

Are Meme Coins Losing Steam? A Decline in DOGE and SHIB Suggests Waning Interest from Traders

Uniswap Foundation hopes to raise $62 million more.

Shanghai Court Recognizes Unique Traits of Bitcoin

Now that ChatGPT can use the internet, it is not only confined to information from 2021.

PARSIQ Will Release Their Reactive Network

Coinbase Secures Regulatory Approval for Derivatives Trading in International Markets

Join our free newsletter for daily crypto updates!