Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

 

Researchers from Arizona State University argue that many Chain-of-Thought (CoT) “reasoning” gains are largely pattern-matching within the training distribution. Using DataAlchemy, a controlled synthetic framework, they probe three axes—task, length, and format—and show performance collapses under even moderate distribution shifts, often yielding fluent but inconsistent chains (e.g., correct rules with contradictory conclusions). The paper formalizes a data-discrepancy view, provides bounds connecting test error to train-test divergence, and finds that small amounts of targeted SFT can “patch” gaps but do not confer genuine out-of-distribution reasoning. The authors advise OOD-focused evaluation and caution against treating CoT as plug-and-play in high-stakes settings.

Source: https://arxiv.org/abs/2508.01191v3

Image: Imagen 3 (Gemini)