Why Language Models Hallucinate
OpenAI has published a detailed explanation about why language models, like ChatGPT, sometimes "hallucinate" — that is, confidently generate information that is false or misleading. These hallucinations are not just bugs, but are partly a result of how models are trained and evaluated.
The core issue is that most models are trained to predict the next word in a sentence, rather than to distinguish between true and false statements. This approach rewards models for producing plausible-sounding answers, even if they are incorrect. During evaluation, models are typically scored based on whether their answer is right or wrong, which leads them to always attempt an answer, even when unsure. This creates a system that encourages guessing instead of admitting uncertainty.
OpenAI argues that to reduce hallucinations, both training and evaluation methods need to change. Specifically, models should be trained and evaluated in ways that reward admitting uncertainty or abstaining from an answer when appropriate. Evaluation systems should penalize confident wrong answers more heavily and should give some credit when a model says, "I don’t know."
They also point out that hallucinations are more likely to happen when the model hasn’t seen enough data on a particular fact. Some questions don’t have clear or available answers, and forcing the model to respond regardless can produce hallucinated content.
Contrary to some beliefs, OpenAI suggests that hallucinations won’t go away simply by making models more accurate or larger. In fact, smaller or less capable models may sometimes perform better at avoiding hallucinations by being more cautious when they don’t know something.
To tackle the problem, OpenAI is working on newer versions of its models (like GPT-5) that are less likely to hallucinate, especially in complex reasoning tasks. They’re also pushing for better evaluation practices across the field that focus on reducing incentives for false confidence.
Image: Sora
Comments
Post a Comment