when

Security

A pair of researchers from Salus Security, a blockchain security company with offices in North America, Europe and Asia, recently published research showcasing GPT-4’s talents when it comes to parsing and auditing smart contracts.
As it turns out, artificial intelligence (AI) is pretty good at generating and parsing code, but you wouldn’t want to use it as a security auditor.
Per the paper:
<blockquote data-ct-non-breakable="true">
 “GPT-4 can be a useful tool in assisting with smart contract auditing, especially in code parsing and providing vulnerability hints. However, given its limitations in vulnerability detection, it cannot fully replace professional auditing tools and experienced auditors at this time.”
</blockquote>
The Salus researchers <a href="https://arxiv.org/pdf/2402.12023.pdf" target="_blank" rel="noopener nofollow">used</a> a data set of 35 smart contracts (called the SolidiFI-benchmark vulnerability library), which contained a total of 732 vulnerabilities, to judge the AI’s ability to detect potential security weaknesses across seven common types of vulnerabilities. 
Related: <a href="https://cointelegraph.com/news/bnb-chain-losses-decline-crypto-report-2023-avengerdao">Crypto lost in BNB Chain heists down by 85% in 2023: Report</a>
According to their findings, ChatGPT is good at detecting true positives — actual vulnerabilities that, outside of a testing environment, would be worth investigating. It reached greater than 80% precision in testing.
However, it has an apparent problem with generating false negatives. This is expressed through a statistic called “recall rate,” and in the Salus team’s experiments, GPT-4’s recall rate was as low as only 11% (higher is better). 
This indicates, as the researchers concluded, “that GPT-4’s vulnerability detection capabilities are lacking, with the highest accuracy being only 33%.” As such, the researchers recommend using dedicated auditing tools and good old-fashioned human know-how for auditing smart contracts until AI systems such as GPT-4 can be brought up to speed. 
<blockquote data-ct-non-breakable="true">
 “In summary, GPT-4 can be a useful tool in assisting with smart contract auditing, especially in code parsing and providing vulnerability hints. ... When using GPT-4, it should be combined with other auditing methods and tools to enhance the overall accuracy and efficiency of the audit.”
</blockquote>

ChatGPT can write smart contracts; just don’t use it as a security auditor

Cointelegraph

Other articles published on Feb 20, 2024

Ripple’s XRP Rides the Waves of Whale Transfers and Market Resilience

UK crypto firms can now rely on Englebert for FCA-compliant ads

Dogecoin Price Prediction – DOGE Signals Bullish Break, Bulls Aims For $0.10

How Much Will You Be Worth in 2025 if You Invest $1000 in Ethereum (ETH), Solana (SOL) and Retik ...

XRP Community Endorses Crypto Lawyer John Deaton For US Senate

Bitcoin Teeters on the Edge: Will $52,000 Mark Usher in a Bull Run?

Join our free newsletter for daily crypto updates!