> ## Content Index
> Fetch the complete content index at: https://www.ffpurpose.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# The Word "Hallucination" Is Doing Too Much Work
- URL: https://www.ffpurpose.com/the-word-hallucination-is-doing-too-much-work/
- Published: 2026-09-09T15:28:54.000Z
- Updated: 2026-09-09T15:28:54.000Z
- Author: tanner stahl
- Tags: Essay

I spent the first stretch of my career in bioengineering, then in quality engineering on FDA-regulated medical devices. In that world, "biologically inspired" is not a finished thought, but more so the start of one. Before a biological analogy can justify anything about how a device works, you have to say exactly what it predicts, then test whether the prediction holds. Nobody signs off on "it mimics the body" as a design rationale. It has to become a spec, and the spec has to survive contact with a test.

The everyday vocabulary around AI products is looser than that. We call systems attentive, reasoning, remembering, understanding, hallucinating. Sometimes those words describe something technically specific, and researchers do the work to define what they mean. Just as often, in the way teams actually talk about what they've built, the words describe what the output looks like from the outside, not what the system is doing underneath. A metaphor is useful shorthand for thinking. It becomes dangerous the moment it's allowed to substitute for a specification. This essay is about one word where that substitution has real consequences: hallucination.

## A word borrowed from psychiatry

"Hallucination" comes from clinical language describing a break in perception, seeing or hearing something that isn't there while a person's underlying senses are otherwise intact. Applied to a language model, the word carries an assumption most people never examine: that something is malfunctioning, that a normally reliable system has come unglued from reality in this one instance.

OpenAI's own research on the subject makes the case that this framing is a poor description of the mechanism.[1](#user-content-fn-1) Generative errors during pretraining aren't a mysterious breakdown. They parallel ordinary misclassification errors in supervised learning, arising from the same statistical pressures that produce classification mistakes anywhere else, as a natural consequence of minimizing cross-entropy loss. Some baseline rate of error isn't a bug that crept in. It falls naturally out of how these models are trained to predict text in the first place.

What happens after pretraining is the more useful half of the argument. The persistence of hallucinations, despite substantial effort to fix them, traces back to the incentives built into how these models get evaluated: most of the primary evaluations used to grade language models reward confident guessing over an honest expression of uncertainty.[1](#user-content-fn-1) Under a common scoring scheme, guessing when unsure maximizes the expected score, so guessing becomes the strategy that wins. As models and their post-training get optimized against evaluations shaped like that, the incentive tilts toward confident answers instead of calibrated ones.

That's two different claims stacked inside one word. Some amount of error is close to unavoidable given how these models learn. The part that persists, that gets reinforced instead of corrected, is a consequence of what we chose to reward. "Hallucination" flattens the distinction into one story, something a better model will eventually outgrow. The more honest read: in a meaningful share of cases, the model isn't malfunctioning. It's doing exactly what our grading scheme rewarded it for doing.

## Where I made a smaller version of the same mistake

I caught myself doing this while building an LLM-based clinical trial eligibility matching pipeline. Working through the extraction and evaluation logic, I kept describing what the model was doing as "understanding" free-text eligibility criteria, matching patients against trial requirements written in dense, inconsistent clinical language.

The evaluation eventually exposed a much less human failure. One confirmed failure mode was a drug-class matching gap: the system handled surface-level term matching reliably but missed cases requiring semantic equivalence, recognizing that two differently-named drugs belonged to the same therapeutic class, something a human reviewer does automatically. "Understanding" had been a convenient word for the output I was seeing when things went well. It was never a tested claim about what the system could actually do. The word didn't cause the failure. But it let me skip a more useful question: what would this system have to do consistently before I was justified calling its behavior understanding? The eval ended up answering part of that question for me, later than it should have.

## Not all of this borrowing is equally bad

Mechanistic interpretability is the counterexample worth taking seriously. Anthropic's interpretability research deliberately imports neuroscience-style methodology: identifying features and tracing circuits to explain how particular outputs are produced.[2](#user-content-fn-2)[3](#user-content-fn-3) It's also honestly incomplete, and Anthropic says so directly. Its own research on circuit tracing in Claude 3.5 Haiku reports that the resulting attribution graphs give satisfying insight into the model's computation for only about a quarter of the prompts tested, with the published case studies drawn from the successes.[3](#user-content-fn-3) That's what disciplined borrowing looks like. It doesn't claim more than it's earned, and it says plainly where it runs out.

## The actual rule

Every cognitive verb attached to an AI system implies a mechanism, and that mechanism should be testable before the word gets used to justify a design decision. If a system "remembers," that implies persistence, so define what information survives, for how long, and under what conditions it can be retrieved. If a system "understands" a document, define the behavior that would distinguish understanding from extraction, then check for it. If a system "hallucinates," ask what you're actually measuring: a factual error, a calibration failure, or unsupported generation dressed up as an answer. Each of those has a different fix, and "hallucination" alone tells you none of them. That's not a vocabulary problem. It changes whether a team reaches for a better model, a confidence threshold, retrieval, a human review step, or a different evaluation altogether.

AI doesn't need to abandon its borrowed vocabulary. Attention, memory, understanding, hallucination are useful shorthand. But shorthand is only useful once we know what it's shorthand for. Otherwise, the metaphor isn't explaining the system. It's hiding the specification we never wrote.

---

### Sources

1. Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). *Why Language Models Hallucinate*. OpenAI. [https://cdn.openai.com/pdf/d04913be-3f6f-4d2b-b283-ff432ef4aaa5/why-language-models-hallucinate.pdf](https://cdn.openai.com/pdf/d04913be-3f6f-4d2b-b283-ff432ef4aaa5/why-language-models-hallucinate.pdf?ref=ffpurpose.com) (also available at [https://arxiv.org/abs/2509.04664](https://arxiv.org/abs/2509.04664?ref=ffpurpose.com)) [↩](#user-content-fnref-1) [↩2](#user-content-fnref-1-2)
2. Ameisen, E., Lindsey, J., Pearce, A., Gurnee, W., Turner, N. L., Chen, B., Citro, C., et al. (2025). *Circuit Tracing: Revealing Computational Graphs in Language Models*. Transformer Circuits Thread, Anthropic. [https://transformer-circuits.pub/2025/attribution-graphs/methods.html](https://transformer-circuits.pub/2025/attribution-graphs/methods.html?ref=ffpurpose.com) [↩](#user-content-fnref-2)
3. Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N. L., Citro, C., et al. (2025). *On the Biology of a Large Language Model*. Transformer Circuits Thread, Anthropic. [https://transformer-circuits.pub/2025/attribution-graphs/biology.html](https://transformer-circuits.pub/2025/attribution-graphs/biology.html?ref=ffpurpose.com) [↩](#user-content-fnref-3) [↩2](#user-content-fnref-3-2)