Gemini

Future

Standard

Notable

<p>analysis conducted by Press Gazette, it has been revealed that a significant proportion of the top 100 news websites in the English language employ measures to block<a href="https://www.cryptopolitan.com/researchers-create-ai-web-agent/"> AI web </a>crawlers from accessing their content. Out of the 106 sites examined, 45 were found to have no AI crawlers blocked whatsoever, while the remaining sites exhibited varying degrees of restriction.</p> 
<h2 class="wp-block-heading" id="insights-into-ai-crawler-blocking-trends-among-top-news-websites">Insights into AI crawler blocking trends among top news websites</h2> 
<p>Among the surveyed news websites, more than four in ten allow all AI web crawlers to scrape their content without any hindrance. However, a considerable portion, comprising 61 sites, impose restrictions by blocking at least one AI bot. Notably, 32 sites go a step further by blocking two or more AI crawlers, with some sites even barring up to five.</p> 
<p>Leading the list of blocked AI crawlers is GPTBot, the web crawler associated with ChatGPT, developed by OpenAI. A striking 56.6% of the surveyed websites disallow access to GPTBot. Following closely behind is Google-Extended, another frequently blocked crawler utilized by Google’s AI chatbot Gemini (previously named Bard).</p> 
<p>&nbsp;Additionally, crawlers such as Claude-Web, Claudebot, anthropic-ai, Cohere-ai, Perplexity-ai, Seekr, and Meltwater face varying degrees of restriction across the surveyed websites.</p> 
<h2 class="wp-block-heading" id="notable-exclusions-and-inclusions">Notable exclusions and inclusions</h2> 
<p>While some major publishers opt to block certain AI bots, others choose not to impose any restrictions. For instance, Mirror, Express, Manchester Evening News, Ladbible, Unilad, and publications under the Lebedev-owned Independent and Evening Standard umbrella allow unrestricted access to AI crawlers.</p> 
<p>&nbsp;Similarly, Politico, Axel Springer’s subsidiary, permits access to AI crawlers due to a content-sharing agreement with OpenAI.</p> 
<p>In a surprising move, the Daily Beast, owned by IAC, refrains from blocking any AI bots despite the company’s chairman advocating for compensation to publishers by AI companies. Conversely, some politically conservative websites, including GB News, Newsmax, Zero Hedge, Breitbart, and Fox News, choose not to block AI crawlers, diverging from other publications under the Murdoch-owned umbrella.</p> 
<h2 class="wp-block-heading" id="implications-and-future-outlook">Implications and Future Outlook</h2> 
<p>The varying approaches adopted by news publishers regarding AI crawler access reflect the ongoing debate surrounding content usage and intellectual property rights in the digital era. While some publishers opt for strict control over their content to safeguard against unauthorized usage and maintain control over distribution, others prioritize accessibility and collaboration with AI companies for content dissemination and innovation.</p> 
<p>As the landscape continues to evolve, it remains to be seen how publishers, AI companies, and regulatory bodies will navigate the complex intersection of technology, content ownership, and user privacy.&nbsp;</p> 
<p>The decisions made by news publishers regarding AI crawler access not only impact the dissemination of news but also shape the broader conversation surrounding digital content usage and intellectual property rights.</p>

Are Top News Websites Blocking AI Crawlers? Unveiling the Latest Findings

Table of Contents

Cryptopolitan_News

Other articles published on Feb 28, 2024

Crypto Tsunami: Pepe Token Price Surges over 60% Amid Massive Whale Movements

Braintrust (BTRST) Price Prediction 2025, 2026, 2027, 2028, 2029 and 2030

Morgan Stanley Considers Integrating Bitcoin ETFs into Investment Portfolio

G20 bullish on global economy: But why should it even matter?

Ethereum whales moves $64M worth of ETH from major exchanges

Famous Analyst Predicts BTC Price in 2025 as Bitcoin Hits Over $64,000

Join our free newsletter for daily crypto updates!