Gemini

Future

Standard

Notable

<p>analysis conducted by Press Gazette, it has been revealed that a significant proportion of the top 100 news websites in the English language employ measures to block<a href="https://www.cryptopolitan.com/researchers-create-ai-web-agent/"> AI web </a>crawlers from accessing their content. Out of the 106 sites examined, 45 were found to have no AI crawlers blocked whatsoever, while the remaining sites exhibited varying degrees of restriction.</p> 
<h2 class="wp-block-heading" id="insights-into-ai-crawler-blocking-trends-among-top-news-websites">Insights into AI crawler blocking trends among top news websites</h2> 
<p>Among the surveyed news websites, more than four in ten allow all AI web crawlers to scrape their content without any hindrance. However, a considerable portion, comprising 61 sites, impose restrictions by blocking at least one AI bot. Notably, 32 sites go a step further by blocking two or more AI crawlers, with some sites even barring up to five.</p> 
<p>Leading the list of blocked AI crawlers is GPTBot, the web crawler associated with ChatGPT, developed by OpenAI. A striking 56.6% of the surveyed websites disallow access to GPTBot. Following closely behind is Google-Extended, another frequently blocked crawler utilized by Google’s AI chatbot Gemini (previously named Bard).</p> 
<p>&nbsp;Additionally, crawlers such as Claude-Web, Claudebot, anthropic-ai, Cohere-ai, Perplexity-ai, Seekr, and Meltwater face varying degrees of restriction across the surveyed websites.</p> 
<h2 class="wp-block-heading" id="notable-exclusions-and-inclusions">Notable exclusions and inclusions</h2> 
<p>While some major publishers opt to block certain AI bots, others choose not to impose any restrictions. For instance, Mirror, Express, Manchester Evening News, Ladbible, Unilad, and publications under the Lebedev-owned Independent and Evening Standard umbrella allow unrestricted access to AI crawlers.</p> 
<p>&nbsp;Similarly, Politico, Axel Springer’s subsidiary, permits access to AI crawlers due to a content-sharing agreement with OpenAI.</p> 
<p>In a surprising move, the Daily Beast, owned by IAC, refrains from blocking any AI bots despite the company’s chairman advocating for compensation to publishers by AI companies. Conversely, some politically conservative websites, including GB News, Newsmax, Zero Hedge, Breitbart, and Fox News, choose not to block AI crawlers, diverging from other publications under the Murdoch-owned umbrella.</p> 
<h2 class="wp-block-heading" id="implications-and-future-outlook">Implications and Future Outlook</h2> 
<p>The varying approaches adopted by news publishers regarding AI crawler access reflect the ongoing debate surrounding content usage and intellectual property rights in the digital era. While some publishers opt for strict control over their content to safeguard against unauthorized usage and maintain control over distribution, others prioritize accessibility and collaboration with AI companies for content dissemination and innovation.</p> 
<p>As the landscape continues to evolve, it remains to be seen how publishers, AI companies, and regulatory bodies will navigate the complex intersection of technology, content ownership, and user privacy.&nbsp;</p> 
<p>The decisions made by news publishers regarding AI crawler access not only impact the dissemination of news but also shape the broader conversation surrounding digital content usage and intellectual property rights.</p>

Are Top News Websites Blocking AI Crawlers? Unveiling the Latest Findings

目次

Cryptopolitan_News

2024年2月28日公開の他の記事

New Cryptocurrency Move from Russia! "The First Approval Has Arrived!"

Seedify.fund (SFUND) Price Prediction 2025, 2026, 2027, 2028, 2029 and 2030

Trading Veteran Explains Bitcoin Price Prediction: $150,000, $275,000 & $400,000

Shiba Inu (SHIB) Holders Move To Catch the Next Wave of Memecoin With Rebel Satoshi ($RBLZ) Presale

Ethereum whales moves $64M worth of ETH from major exchanges

6 Best Altcoins to Invest in Right Now February 28 – Arweave, Injective, Dogecoin

ニュースレターを購読して暗号資産の最新情報を毎日入手！