OpenAI’s ChatGPT Struggles: Researchers Stumped by Deteriorating Performance

OpenAI’s advanced chatbot, ChatGPT, has recently been under scrutiny as researchers from Stanford and UC Berkeley discovered a perplexing decline in its performance over just a few months. In a study conducted on July 18, the researchers found that ChatGPT’s latest models were becoming increasingly incapable of providing accurate answers to identical questions.

Despite extensive analysis, the study’s authors were unable to pinpoint the exact reasons behind the AI chatbot’s deteriorating capabilities. To assess the reliability of different ChatGPT models, researchers Lingjiao Chen, Matei Zaharia, and James Zou tested ChatGPT-3.5 and ChatGPT-4 on tasks such as solving math problems, answering sensitive questions, coding, and spatial reasoning.

The research revealed a significant drop in accuracy for ChatGPT-4. In March, this model achieved a remarkable 97.6% accuracy in identifying prime numbers. However, when the same test was conducted in June, the accuracy plummeted to a mere 2.4%. In contrast, the earlier GPT-3.5 model showed improvement in prime number identification during the same timeframe.

The decline was not limited to prime number identification alone. Both ChatGPT-3.5 and ChatGPT-4 exhibited a substantial deterioration in generating lines of new code between March and June. Moreover, ChatGPT’s responses to sensitive questions underwent a noticeable change. Previous iterations provided extensive reasoning for not answering such questions, but in June, the models simply apologized and refused to answer, with some examples even showing a focus on ethnicity and gender.

The study’s authors highlighted the fact that the behavior of large language models like ChatGPT can change significantly within a relatively short period. They emphasized the need for continuous monitoring of AI model quality. Users and companies relying on LLM services in their workflows were advised to implement some form of monitoring analysis to ensure the chatbot’s performance remains reliable and up to par.

In a separate development, OpenAI announced plans on June 6 to establish a dedicated team to manage the potential risks associated with superintelligent AI systems, which they anticipate could emerge within the next decade. This proactive step reflects OpenAI’s commitment to addressing the challenges posed by AI advancement.

As the future of AI unfolds, it is crucial to closely monitor and address the fluctuations in performance observed in ChatGPT and other similar models. By doing so, we can ensure that AI chatbots continue to serve as valuable tools while maintaining accuracy, reliability, and ethical standards.