Apple Study: Reasoning Models Collapse on Complex Problems

A group of Apple researchers — Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio and Mehrdad Farajtabar — published a paper this past weekend that calls into question one of the central promises of recent generative AI: that so-called reasoning models actually reason. The study, titled "The Illusion of Thinking," was posted to arXiv on June 7 and is already making the rounds among researchers.

Reasoning models, technically known as LRMs (Large Reasoning Models), are the generation of systems that generate an explicit, step-by-step thinking process the user can read before producing an answer. That capability has been the big selling point of the past several months: models that don't just answer, but show how they arrive at the answer — and that, in theory, should be better at solving complex math and logic problems.

A puzzle lab for measuring thought

Until now, most evaluations of these models have relied on math and coding benchmarks that only measure whether the final answer is correct. The problem, the authors note, is that these benchmarks tend to be contaminated — meaning the model has likely seen very similar exercises during training — and they say nothing about how the model actually thinks internally.

To get around this, Apple's team designed controllable puzzle environments, including variants of the Tower of Hanoi, where difficulty can be precisely dialed up or down while the underlying logical structure stays identical. That makes something possible that traditional benchmarks don't offer: isolating the effect of complexity and, in the process, examining not just whether the model gets the right answer, but the reasoning trace it produces before responding.

Three regimes, and a total collapse

Comparing LRMs against their equivalent standard versions (models without that explicit "think before answering" step) under the same compute budget, the authors identify three distinct behaviors depending on problem difficulty:

• Low complexity: standard models, without explicit reasoning, outperform LRMs.
• Medium complexity: this is where the reasoning models' edge shows up, outperforming their standard counterparts.
• High complexity: both types of model collapse equally. The paper describes it literally as a "complete accuracy collapse" past a certain threshold, with results approaching 0% accuracy.

The most striking part, according to the authors, isn't just that the models fail, but how they fail. As a problem gets harder, reasoning effort — measured by the length of the thinking process the model generates — increases, as one would expect. But only up to a point: past that threshold, effort actually decreases, even though the model still has token budget left to keep thinking. In other words, the model doesn't run out of room to reason further; it simply stops trying once the problem gets too hard.

Failures in exact computation

The study also documents that these models have concrete limitations when it comes to exact computation: they fail to consistently apply explicit algorithms and reason inconsistently depending on the scale of the problem, even when the underlying logic doesn't change. The authors also dig deeper into the reasoning traces themselves, studying which solutions the models explore and how they behave computationally, in an effort to better understand their real strengths and limits.

Why this debate matters

The paper's title is no accident. "The Illusion of Thinking" goes straight to the heart of the debate that has dominated the AI industry over the past year: whether the "thinking out loud" process these models display reflects genuine reasoning, or is, in practice, a learned pattern that falls apart the moment a problem stops resembling something seen during training.

The question isn't academic. Companies and developers have spent months adopting these reasoning models precisely for tasks that demand reliability on complex problems: planning, code debugging, financial or scientific analysis. If performance itself collapses beyond a certain complexity — and the model even scales back its effort right when it would be needed most — it's worth calibrating expectations before handing critical decisions to these systems without supervision.

For now, the work is a preprint: it hasn't gone through peer review or been published at a conference. That doesn't take away from the methodology, but it does mean replications and counterarguments from other labs should be expected in the coming weeks — especially from the companies that build these reasoning models and have a lot riding on this debate.