AI makes us less likely to say “I don’t know,” even when it’s wrong

An AI assistant can influence more than the answer we give: it also appears to lower the threshold at which we decide whether we know enough to respond. A paper published on arXiv finds that, when faced with difficult questions, access to an AI recommendation sharply reduces people’s willingness to say “I don’t know,” even when that recommendation is incorrect.

The effect matters because recognizing uncertainty is a core skill in sound decision-making. In areas such as education, work or information seeking, confidently answering a misunderstood question can be worse than suspending judgment and checking the facts.

Five experiments with deliberately wrong advice

The study, by Chiara Marcoccia, Walter Quattrociocchi and Valerio Capraro, brings together five experiments involving 3,132 participants. Four were preregistered—a mechanism through which researchers set their design and analysis before seeing the results—and the fifth directly replicated one of the trials.

Participants had to answer difficult questions, with the option to decline to respond at all times. The key to the design was that the authors constructed the questions so that the AI advice offered would be wrong. This allowed them to separate two phenomena that are often conflated: the potential effect of using AI and the effect of receiving a correct answer.

The point, then, was not to measure whether the model helped people find the right solution. The test examined what happens when someone encounters a fluent, seemingly useful answer that in fact leads them in the wrong direction.

More answers, but fewer correct ones and greater confidence

Simply having AI available nearly eliminated participants’ inclination to abstain. The result held both when they could actively request the advice and when it appeared directly on screen.

Participants answered more questions, but were correct only about one-third as often as they were when they had no access to AI. At the same time, their confidence nearly doubled.

That combination is especially significant. The problem is not just that a false suggestion can prompt an isolated mistake, something to be expected in any imperfect system. The study points to a shift in metacognition: the ability to assess what we know, what we do not know and when it is best to withhold judgment.

Conversational assistants are designed to sustain useful interactions and provide complete answers. That fluency can make a recommendation seem more reliable than it deserves, especially when the user lacks sufficient prior knowledge to spot the error.

Incentives correct part of the effect

The researchers also tested what happened when correct answers were rewarded and mistakes penalized. With those incentives, participants sought and followed AI advice less often, answered more accurately and chose not to respond more frequently.

However, they still withheld judgment far less often than participants without access to AI. Incentives improved behavior, but did not eliminate the effect of having an automated answer readily available.

The findings do not support simply applying these percentages to every chatbot, task or professional setting. The questions were expressly designed to make the advice false, and a controlled experiment does not replicate the full complexity of using AI at work or at home. But it does identify a risk that standard model evaluations often leave out: it is not enough to measure how many answers are correct; it is also necessary to observe how AI changes people’s judgment about when to ask for help, verify it or acknowledge uncertainty.

As AI-generated suggestions become integrated into search engines, documents, emails and customer-service systems, that judgment may become a design issue. Showing verifiable sources, communicating the system’s uncertainty and making it easier for users to check an answer may be more useful measures than presenting every recommendation with the same appearance of confidence.