How Large Language Models are Designed to Hallucinate

I feel a real sense of relief that my new paper has finally been posted on arxiv.org: http://arxiv.org/abs/2509.16297. Getting it accepted was not easy, and perhaps that is fitting because the argument it makes does not come easily either. This is not a paper that confirms the prevailing assumptions of the AI industry. It asks the reader to take a step back and to consider whether the very foundations of the current paradigm are as solid as they appear.

The field has become locked in, not only to the transformer as the vehicle for artificial general intelligence, but also to a mindset that is, in many ways, Cartesian. The progress of AI is measured statistically, by climbing leaderboards and comparing benchmarks, without a moment’s pause to ask what exactly is being measured. When we look closely we see that the models are not scaling intelligence. They are scaling coherence, the ability to keep words and phrases flowing in plausible succession. And then, when coherence is not enough, scaffolding is added in order to extend performance. But neither coherence nor scaffolding, as impressive as they are, constitute intelligence.

The paper therefore asks for a paradigm shift in how we think about both AI and intelligence. It is not a matter of adding more data or stacking more parameters. It is a matter of understanding the structural conditions of meaning itself. The diagnosis I offer is that hallucination, which everyone acknowledges as a serious problem, is not a bug in the system or an accidental side effect. It is designed in, a predictable outcome of the way the transformer operates.

Hallucination arises because the transformer is compelled to keep pattern matching where pattern alone is not enough. It is pressed by its human masters to continue producing words even when the conditions for truth are absent. The model can relate words to other words endlessly, but it has no way of touching world. It has no way of grounding language in the ontological structures that allow meaning to be tested, corrected, and secured. What emerges are outputs that can seem coherent, sometimes dazzlingly so, but that fail at the threshold where language must answer to reality.

To see this requires stepping outside the Cartesian inheritance that has shaped so much of modern thought. In that view intelligence is an inner faculty that can be measured by its results. In the view I am asking the reader to consider, intelligence is inseparable from being-in-the-world, from the way words and things hang together in significance. That significance does not come from defining an isolated object. It comes from knowing how things relate to one another and to the purposes they serve. It is precisely here that the transformer shows its limit, for it has no access to this web of relations, only to the statistical shadows cast by words.

I know this paper asks a lot. It challenges deeply entrenched habits of thinking and entrenched technical investments. Yet if we want AI systems that can be more than engines of coherence, we must begin with a clear diagnosis of why they hallucinate. That diagnosis, in my view, has to be ontological, because the problem is not one of more clever engineering alone but of the very conditions under which language and truth arise.

So I am relieved that the paper is out at last, and I hope it can open a conversation that has so far been postponed. Hallucination is not simply an obstacle to be managed with patches and workarounds. It is a symptom of a deeper misalignment between statistical patterning and the structures of world and truth. To recognize that is to begin the search for AI that does not merely sound intelligent, but that might, in some way that we can now intuit, become intelligent.