A chatbot that produces “hallucinations” and elaborates on them eloquently is seen as just as credible as one that gives concise, error-free answers, according to the journal Computers in Human Behavior Artificial Humans.
Most people are familiar with tools such as ChatGPT, Gemini, Claude and other chatbots. Every day, we ask them millions of questions and receive answers. Language models, however, tend to fall into so-called “hallucinations”: They generate implausible, dubious content and present it with great confidence as truth.
Research conducted by media scholar Marianna de Sa Siqueira and colleagues from Radboud University in Nijmegen, the Netherlands, shows that users are more likely to see a chatbot that uses natural language as intelligent and useful, even if it is actually spouting nonsense.
In the experiment, participants were shown one of four versions of a conversation with a chatbot about “The Lord of the Rings.” Two versions featured fluent, natural-sounding sentences. In one, the chatbot gave correct answers, and in the other, incorrect ones. The other two versions relied on short, matter-of-fact messages: One contained only true information, while the other presented (subtly) fabricated facts.
As expected, the chatbot that used elegant language and gave correct answers was rated highest. Significantly, however, the chatbot that wrote fluently but gave false information was rated by participants as just as intelligent and pleasant to deal with as the chatbot that communicated in a stiff, sparing manner but conveyed correct data.
It also turned out that as many as 80 percent of participants did not notice the errors at all, even though these people had been screened beforehand for basic knowledge of “The Lord of the Rings.”
“Above all, this shows how easily people can be misled and how quickly we begin to trust something that is actually spouting nonsense,” said Barbara Müller, an associate professor and co-author of the study. “If a text sounds elegant and natural, people automatically attribute intelligence to it. Subconsciously, we then assume: ‘Everything is fine, I don’t need to check this any further.'”
The study draws attention to the danger of blindly trusting artificial intelligence. Professor Tibor Bosse, also a co-author, explained: “So-called language models, such as GPT or Claude, are trained primarily to create sentences that are statistically probable and easy to take in. These systems do not always attach great importance to truth. They were designed to sound like a human and to be convincing, but whether the information they contain is actually correct is of secondary importance to them.”
“People used to complain that chatbots were annoying and didn’t know anything. Now they sound so intelligent that we believe them too easily,” Bosse noted.
In his view, this problem cannot be solved with a simple disclaimer at the outset. “Most chatbots display a brief notice at the start saying that generated content may not be fully accurate, but these companies certainly are not going to note on their own with every answer that the system is unsure of its words. If we really want to protect users from chatbots spouting nonsense, we must, for example, require technology companies to make this uncertainty visible. If the chatbot itself expresses doubt, we can prevent people from taking nonsense for truth,” the researcher pointed out.
Author: Paweł Wernicki (PAP)
