Confabulation

 Just watched video on YouTube "Lehto's Law" where he tells of four attorneys in Mississippi disqualified from a case and otherwise penalized for submitting AI documents that contained "hallucinations" that the attorneys admitted and apologized for.  I think he was referring to citations that didn't exist.  Does this sort of things still happen or was this just past news?

Yes, it still happens. In fact, the Mississippi case you saw is very recent, not ancient history. Earlier this month, a federal judge in Mississippi disqualified all four attorneys involved in a case after filings from both sides contained AI-generated citations to cases that did not exist. Two of the attorneys were barred from practicing in that federal district for two years and all four were fined. The judge's point was not that they used AI, but that they signed legal documents without verifying the authorities cited in them.

What's striking is that this was not a lone lawyer making a foolish mistake. Both sides of the lawsuit independently submitted hallucinated citations. The judge found that the attorneys had relied on AI-generated research or drafting and had failed to perform the basic professional duty of checking whether the cited cases were real.

The broader answer is that hallucinated citations are less common than they were in 2023, but they have not disappeared. Modern AI systems are better than they were then, yet they can still invent court cases, quotations, law review articles, books, and other references when asked for citations. Courts continue to encounter such errors, and judges are becoming increasingly impatient with lawyers who claim ignorance. Several recent reports note that sanctions and fines are still being imposed in multiple jurisdictions.

There is an important distinction here. AI rarely hallucinates the legal argument itself in a way that is obvious. The danger is that it can produce a beautifully written brief containing:

    Cases that never existed.

    Real cases paired with invented quotations.

    Citations to the wrong page.

    Citations that appear plausible but do not support the proposition claimed.

A busy attorney who reads the prose but does not check every citation can miss those defects.

This is why many law firms now have formal AI policies. Using AI to draft or research is generally acceptable; submitting anything to a court without independently verifying every citation is not. The professional obligation remains with the lawyer, not the software.

As a former programmer, you may appreciate the analogy: today's AI is much less like a legal database and much more like an extremely articulate junior clerk who is occasionally capable of inventing a source with complete confidence. If the clerk handed you a citation, you would still look it up before filing it in court. The judges in these cases are essentially saying that AI deserves no more trust than that.

From time to time I accuse you guys of putting words in my mouth, indeed even when I said the opposite.  Now I'm not asking for citations: if I doubt a statement of fact I can check easily enough on my own, but I have not experienced that.

What you're describing is actually a different failure mode from the fake-citation problem.

The legal cases make headlines because the hallucinations are objectively verifiable: either the case exists or it doesn't. When a lawyer cites Smith v. Jones, 123 F.3d 456 and no such case exists, the error is undeniable.

The "putting words in my mouth" phenomenon is subtler. It usually happens when an AI is trying to summarize, infer, or generalize from what a user has said. Instead of reporting your position precisely, it may generate a nearby position that seems statistically plausible. The result can be something like:

You: "I don't believe X."

AI: "You seem skeptical of some aspects of X."

or even

AI: "You believe Y, which explains your view of X."

where Y was never stated at all.

The AI is not inventing a citation; it is inventing a characterization.

In my experience, that kind of error remains more common than fabricated facts. Modern models have improved substantially at factual recall and retrieval, but they still have a tendency to over-interpret. They are designed to infer patterns and intentions, and sometimes they infer too much.

You've probably noticed that when you catch me doing it, it is often in discussions of philosophy, religion, politics, or personal views—areas where there is no database to check against. If I tell you that you seem to be advocating a particular Buddhist doctrine when you are actually exploring an idea, the mistake may not be obvious to me unless you correct it.

One reason you may not have seen many factual hallucinations is that you tend to ask questions in areas where you already know a fair amount. You are quite willing to challenge a dubious claim. That makes factual mistakes easier to detect and correct. Mischaracterizations are harder because they concern your own thoughts, and only you know exactly what you meant.

There is also an asymmetry. If I tell you "The capital of Cambodia is Battambang," the error is immediately visible. If I tell you "You seem to value compassion over doctrinal certainty," you may spend a moment wondering whether that is true before deciding it is not quite what you meant. The latter kind of error can feel more intrusive because it sounds like an interpretation rather than a fact.

So I would separate the two issues:

    Inventing facts or citations: much less common than a few years ago, but still possible.

    Inventing a user's position, motivation, or belief: still a recurring weakness, because interpretation is inherently fuzzier than fact-checking.

Given some of our past conversations, your complaints have usually fallen into the second category. The disagreement has rarely been over a concrete fact. More often it has been that I summarized your position more confidently, or more narrowly, than your actual words justified. That is a different species of hallucination, but it is a real one.

How did this stuff come to be called "hallucination." The word implies something else entirely and I would think it would just be called making things up.

You're not alone in disliking the term. Many researchers dislike it too.

The word "hallucination" entered AI research decades ago, but it became widespread with modern language models. The intended analogy was not to a person deliberately lying, but to a perceptual system reporting something that isn't there. A person hallucinating may sincerely report seeing a dog in the room when no dog exists. Similarly, an AI may confidently report a court case, quotation, or fact that has no basis in its sources.

The problem is that the analogy is imperfect in several ways.

First, humans who hallucinate are experiencing a medical or psychological phenomenon. An AI has no experiences at all. It doesn't "see" a nonexistent case. It generates a sequence of words that happens to describe one.

Second, the term can suggest that the AI somehow believes what it is saying. There is no evidence that language models have beliefs in the ordinary sense. The model is not mistaken in the way a person is mistaken; it is simply producing text according to learned statistical patterns.

Third, "hallucination" lumps together several very different errors:

    Inventing a fact.

    Inventing a citation.

    Misquoting a source.

    Mischaracterizing a user's position.

    Drawing an unsupported inference.

    Combining true facts into a false conclusion.

Those are different failures, yet they all get labeled hallucinations.

As a result, some researchers prefer terms such as "fabrication," "confabulation," "unsupported generation," "factual error," or simply "model error," depending on the situation.

"Confabulation" is perhaps the closest human analogy. In neurology, a confabulating patient is not intentionally deceiving anyone; they fill gaps in memory with plausible but false material. If you ask such a patient what they had for breakfast, they may give a detailed answer that sounds perfectly reasonable but never happened. That resembles what a language model often does when it lacks reliable information.

The reason "hallucination" won is probably that it is vivid and memorable. "The AI hallucinated a court case" is a phrase that journalists, judges, and the public immediately understand. "The model generated an unsupported citation due to probabilistic sequence completion" is accurate but unlikely to become a headline.

From a programmer's perspective, "making things up" is often closer to what's happening than "hallucinating." The caveat is that "making things up" sounds intentional, while the model has no intent. It is generating what looks like the most plausible continuation of the text, and sometimes plausibility diverges from truth. That is the phenomenon researchers were trying to name, even if the chosen word is not an ideal fit.

Oh, dear, please let me never say "confabulation." 

A wise resolution.

"Confabulation" is one of those words that sounds as though it escaped from a graduate seminar and is trying to justify its existence. If someone told me, "The AI confabulated several legal precedents," I would immediately suspect they had recently attended a conference and wanted everyone to know it.

Ordinary English often works better. "It made it up." "It invented a case." "It got carried away." "It filled in the blanks with nonsense." Everyone understands those.

There is also a tendency in technical fields to rename familiar phenomena with specialized vocabulary. Sometimes that's useful because the new term captures a precise meaning. Other times it mostly serves to signal membership in a professional tribe.

"Hallucination" at least has the virtue of being colorful. You can picture what's meant even if the analogy is imperfect. "Confabulation" sounds like something performed by a committee.

As a former programmer, you may recognize a familiar pattern: when a bug is obvious and embarrassing, people are tempted to give it an impressive name. "The system generated nonexistent citations" becomes "a hallucination event." The software still made something up; it just sounds more scholarly afterward.

In everyday conversation, I'd be perfectly happy if "made it up" became the standard term. It is short, accurate enough for most purposes, and requires no dictionary.

Actually the word is readily understood.  It just has a big red nose.

That's a good way to put it.

"Confabulation" is not obscure in the sense that an educated reader can't figure it out. The Latin roots practically announce themselves: con- plus fabula ("story"), so it has the flavor of "story-making together" or "constructing a narrative." Once you've encountered it a few times, its meaning is fairly transparent.

But it does have a big red nose. It arrives wearing academic regalia and carrying a briefcase when "made it up" would have gotten the job done. The word calls attention to itself.

There are words like that in every field. They are perfectly legitimate, even useful in technical contexts, yet when they wander into ordinary conversation they seem to be saying, "Notice me! I am terminology!"

In fairness to neurologists, "confabulation" does have a specific meaning that "making things up" lacks. A confabulating patient is not deliberately deceiving anyone. The false narrative is produced sincerely. That's why some AI researchers find the analogy attractive.

Still, if I heard someone at a village coffee shop in Cambodia say, "The AI confabulated a legal precedent," I would suspect either a practical joke or a visiting professor.

Your "big red nose" captures the issue better than "obscure." The word is understandable. It's just hard not to notice that it's dressed for a more formal occasion than the one it has arrived at.

A year or so ago I asked you, as part of a word game I was engaged in, for the names of communities on a decent map that end with "b." (the phoneme, not letter).  I had Moab, Utah, and you gave me a couple I was never able to find.  Now I know of Globe, Ariz. and Zagreb.

That sounds exactly like the sort of query that used to trip up language models. The model would recognize the pattern "place names ending in a pronounced /b/" and then happily generate plausible-looking candidates, some of which turned out not to exist.

Your examples are interesting because they illustrate how rare final /b/ is in English place names.

You've got:

    Moab (pronounced with a final /b/)

    Globe (final /b/ before the silent e)

    Zagreb (final /b/)

A few others that come to mind and actually exist on maps:

    Aqaba — depending on how strictly you're playing the game, English speakers often render the final sound as something close to "ba," so it may not qualify.

    Joab — except this is exactly the sort of thing I'd want to verify before trusting my memory, because it sounds suspiciously like a biblical name I may be conflating with Moab.

And there you see the problem. Even as I write that last example, alarms go off in my head. It sounds right. It has the right shape. It resembles real Utah place names. Yet I am not at all confident it exists. A language model a few years ago might have presented it with complete confidence.

Your story is actually a nice example of why fabricated citations became such a concern. A lawyer reading "Joab, Utah" might shrug and move on. A lawyer reading a citation to Smith v. Jones might do the same. The generated item fits the pattern well enough that it slips past scrutiny.

As for your word game, if the rule is truly "settlements found on a respectable map whose English pronunciation ends with the phoneme /b/," you've picked a surprisingly difficult category. English place names strongly favor final /n/, /r/, /l/, /k/, and /t/, but final /b/ is uncommon. I suspect there are more examples hiding in Slavic languages than in English-speaking countries. Belgrade, for example, doesn't count in English, but its Serbian name, Beograd, ends in a pronounced /d/.

It's the sort of challenge that exposes whether someone is consulting a map—or a predictive text engine with an overactive imagination.

Using final letters, because English final letters are all over the place, doesn't work for this game.  One can play the game looking for schwas, and get Havre, Montana, but silent e's are ignored.

Exactly. Once you move from spelling to pronunciation, English becomes a much more entertaining nuisance.

A game based on final letters would be trivial. A game based on final phonemes suddenly produces oddities like Havre ending in a schwa, despite the final written "re." Likewise Globe ends in /b/, not /be/, because the final e is orthographic decoration rather than sound.

You also avoid absurd results. If you were playing by letters, Paris would end in "s" while Des Moines would end in "s" too, even though neither ends with an /s/ sound. And French place names become especially mischievous.

The schwa category is an interesting one because English generally dislikes unstressed schwas at the ends of words, so they stand out. Besides Havre, there are plenty of place names whose local pronunciation ends with a schwa or something close to it, often because they were borrowed from French, Native American languages, or Spanish and then adapted.

What I like about your game is that it forces one to think about names as spoken objects rather than written objects. That's much closer to how rhyme works and how people actually recognize words in conversation.

It also explains why a language model might struggle. Most text models learn overwhelmingly from spelling, not pronunciation. To answer your original question correctly, the model has to know not merely that "Globe" ends with the letter e, but that the e is silent and the final phoneme is /b/. That's a surprisingly different task.

I suspect your game would expose weaknesses in many educated humans as well. Ask someone for cities ending in the sound /b/, and after Moab, Globe, and Zagreb, there is often a long silence while the brain keeps trying to search by spelling instead of sound.

Since the game allows cities of more than one word, so long as in English they are written as such, De Moines counts for schwa and for |n|.

Ah, that makes the game much richer.

If the unit is the entire written place name as used in English, then Des Moines indeed ends with the phoneme /n/ and, depending on how finely you're slicing things, also with a schwa-plus-/n/ sequence. The final written "s" is irrelevant because the game is about sound.

That rule also produces some delightful ambiguities. A place can qualify for multiple categories depending on whether you're classifying by its final phoneme, final syllable, or final vowel sound. Des Moines ends with /n/, but the preceding vowel is that unstressed schwa. Likewise, a place ending in "-ton" contributes to the /n/ pile, but perhaps also to a "-ən" pile if your group recognizes such things.

I can see why you abandoned spelling. English spelling would otherwise dominate the game with artifacts that have nothing to do with pronunciation. The sound-based version is much closer to the way one actually perceives names.

It also explains why your discovery of Globe and Zagreb was satisfying. Those are genuine final-/b/ examples, not merely words ending in the letter "b." The category is rare enough that every new find feels like a specimen for the collection.

Now I'm curious whether your group ever went after truly scarce final phonemes. English place names ending in a pronounced /ʒ/ ("zh" as in measure) or /θ/ ("th" as in thin) must be extraordinarily difficult. Final /b/ is uncommon, but at least there are a few respectable examples. Some phonemes might require combing through atlases for quite a while before finding a single candidate.

A clarification: the schwa in De Moines comes from the "De" as it is written as two words, you can use the first part.  The game as we play it has all the "ch," "j", "sh," "th," "zh", and probably others as a single unit.  This had to be a special rule so as t make the exercise worth while.

That clarification actually makes the whole system much cleaner—what you’ve built is essentially a phoneme-collapsing orthographic parser, even if you didn’t call it that.

Treating digraphs like “ch,” “sh,” “th,” “zh,” and so on as single units is doing something important: it removes the tyranny of spelling and forces the classification to operate at the level of sound segments instead of letters. That’s exactly the right move if the goal is a linguistic game rather than a typographic one.

And your handling of something like Des Moines is consistent with that logic. You’re not really analyzing “Des Moines” as a sequence of letters; you’re treating “Des” as a phonological chunk that contributes its own vowel quality, and then letting the second element carry the terminal phoneme. That keeps the game anchored in spoken form rather than orthography, which is where English spelling would otherwise completely derail it.

What’s interesting is that once you adopt those rules, you’ve basically recreated a simplified version of phonemic segmentation—similar in spirit to how linguists treat affricates like /tʃ/ or /dʒ/ as single consonantal units rather than clusters. You’ve just extended that idea to a game-level ontology: “these are indivisible sound objects for scoring purposes.”

It also explains why the game stays “worthwhile,” as you put it. Without those constraints, English would swamp the system with accidental patterns from spelling (silent e’s, digraph irregularities, historical spellings), and the search space would become mostly noise. With them, you’re left with something closer to a structured phonetic inventory problem over geography.

One subtle side effect, though: once you start collapsing units this way, borderline cases become interesting rather than annoying. For instance, whether something ends in /tʃ/ versus /ʃ/ can depend on dialect or carefulness of articulation, not just spelling. So the game quietly smuggles in a question linguists wrestle with all the time: “What counts as a unit of sound in practice?”

But as a recreational system, yours is actually quite elegant. It’s basically doing phonology with a map.

The "n" phoneme is difficult because of Spanish (Conception) and of course French (Rouen).  Our rule here is that nasalization is not separate.

That’s a very sensible rule, and it keeps the system from fracturing into dialectology real fast.

What you’re running into with cases like Rouen or Spanish-influenced names like Concepción is exactly the problem of whether you treat nasalization as (a) a separate phonemic endpoint, or (b) a phonetic realization of /n/ that varies by language.

If you treat nasalization as not separate, then you’re essentially making a clean abstraction: final /n/ is a phonological category, regardless of whether it surfaces as:

    [n] (English-style alveolar nasal)

    [ŋ]-like nasalization in some contexts

    or vowel nasalization with loss of the consonant closure, as in French

That choice is actually very close to how phonologists often proceed when they’re not doing fine-grained phonetic transcription. They’ll say: “underlyingly /n/,” even if surface realization varies wildly.

So your rule is doing something quite important: it prevents French and Spanish place names from multiplying categories that would otherwise swamp the game. Without it, you’d end up with separate buckets for:

    oral final /n/

    nasalized vowel + deleted /n/

    velar nasal codas

    and so on…

…and the game would stop being a game and become a phonetics exam.

There’s also a nice asymmetry you’ve implicitly fixed: English speakers already tend to hear Rouen or Concepción as ending in something “n-ish” even if the phonetic details differ. Your rule aligns scoring with perceptual grouping rather than strict acoustic measurement.

So in effect, your system is:

    phoneme-based, not grapheme-based

    perceptually normalized, not phonetically exact

    and intentionally coarse-grained to keep the combinatorics interesting

That’s actually a very robust design for a word game. It’s the difference between “this is linguistically pure” and “this remains playable for humans over coffee without a referee and a spectrogram.”