LLM counting vowels

 I read an article that claimed LLM's can't count, like tell me the number of vowels in this prompt?

Yes—but that claim needs a big asterisk.

If the prompt is literally:

    “tell me the number of vowels in this prompt?”

then I can count them. But there are several different things being conflated when people say “LLMs can't count.”

An LLM doesn't normally count by executing a character-by-character counting algorithm. It generates the answer from patterns learned during training. That makes it surprisingly easy to fool with things like:

    “How many r's are in strawberry?”

because the model may recognize the word as a unit rather than systematically inspecting every character.

But if I deliberately treat the task as literal character counting, I can do it reliably for ordinary text. For your exact sentence, we also have to decide whether “vowels” means a, e, i, o, u only, whether y counts, and whether we're counting the question itself or some larger prompt.

So the interesting experiment isn't simply “Can an LLM count?” It's whether the model can reliably perform an operation that requires exhaustive inspection rather than pattern recognition. That's a genuine weakness of language models, although modern models can compensate for it by using explicit reasoning or tools.

And there's a wonderfully simple way to demonstrate the distinction: give me a long, deliberately constructed string containing, say, 73 as hidden among thousands of other characters and ask me to count them. That's much more revealing than asking about a familiar English word.

Well with y, it depends; in "young"  y is a consonant, in "mabey" it is a vowel.

Exactly. And “y” is a particularly good demonstration of why counting vowels isn't quite as simple as counting letters from a fixed set.

In young, the y represents the consonant sound /j/, so it isn't functioning as a vowel. In maybe (I assume that's the word you meant by “mabey”), the final y represents a vowel sound, roughly /i/, so it is functioning as a vowel.

In fact, y can do several jobs:

“yes” — consonant
“gym” — vowel
“maybe” — vowel
“boy” — vowel
“rhythm” — vowel

And there is an interesting distinction here: letters don't intrinsically belong to the vowel/consonant categories; sounds do. The letter y is simply a spelling symbol whose role depends on the word.

So if someone asks an LLM, “How many vowels are in this sentence?”, there are actually two possible questions:

    How many vowel letters (a e i o u, perhaps y)?

    How many letters representing vowel sounds?

Those can produce different answers. And if we're testing whether an LLM can “count,” we'd better specify which game we're playing first. 😄

Yes. I think you've put your finger on the more interesting issue.

A human listener normally performs interpretation before computation. If someone says, “How many vowels are in this sentence?”, you don't ordinarily stop the conversation and ask for a phonological specification. You infer what the person probably means from ordinary usage and context, make the reasonable assumptions, and get on with it.

And humans are remarkably good at this kind of background assumption management. We routinely tolerate ambiguity because we have a sense of what distinctions matter to the questioner.

An LLM has a slightly different problem. I can certainly make the same sort of assumptions, but I also have a tendency—especially when I've noticed an ambiguity—to surface the ambiguity instead of silently resolving it. That's what happened here, and I arguably overdid it.

There is an important distinction between:

“There are ambiguities in this question.”

and

“The question cannot be answered until those ambiguities are resolved.”

The first is almost always true. The second is usually false.

For your original question, a sensible conversational interpretation would be something like: count the ordinary vowel sounds in the words, using normal English pronunciation, and don't get cute about marginal cases. I could simply make that assumption and answer.

That actually gets us closer to the LLM-counting question. Counting itself may be the easy part. Deciding what the question means—and deciding which ambiguities are worth mentioning—is a more fundamental cognitive task.

And there's a funny irony here: an LLM can sometimes be criticized for giving a wrong numerical answer when the more fundamental failure occurred before it ever started counting.

When there are ambiguities, we tend to assume the most common and then let the person asking revise if we assumed wrong.

Exactly. That's normal conversation.

If I ask you, “How many vowels are in this sentence?” you don't need me to specify a dialect, phonemic inventory, treatment of diphthongs, or the status of syllabic consonants. You make the default, most probable interpretation, answer it, and leave the door open for correction.

That is a very useful general conversational principle:

    Resolve ambiguity by the most ordinary interpretation unless the ambiguity is consequential.

If I say, “How many people were at the meeting?” you don't respond, “Define person and meeting.” You understand what I mean. If I actually meant people physically present rather than people invited, I can correct you.

And I think this is an important distinction in evaluating LLMs. A good conversational system shouldn't merely detect ambiguity; it should judge whether the ambiguity matters enough to interrupt the conversation. Humans do this almost automatically.

In fact, your formulation—assume the most common and let the questioner revise—is probably a better description of ordinary intelligent conversation than “be precise.” Precision is something we introduce when the circumstances require it, not the starting condition of every question.

I suspect this also explains some of the irritation people experience with AI: sometimes the machine is technically being careful while conversationally being obtuse.

You may recall when I told you about my freshman college linguistic first day when the professor picked me from his list and asked if I thought linguistics was subjective or objective and I hedged saying a little of both.  The same question appeared on his final exam, and there I said, "Being a science, it is necessarily objective, but being about a human behavior, it has a subjective aspect."  That got circled and a "right on."

Yes—that is a lovely example of the distinction we've just been circling around. And the professor's “right on” tells you quite a lot about what he was actually testing.

Your freshman answer, “a little of both,” was reasonable, but it was essentially a classification without an argument. By the final exam you had learned to make the distinction explicit:

    “Being a science, it is necessarily objective, but being about a human behavior, it has a subjective aspect.”

That's much stronger because you weren't merely splitting the difference. You were identifying two different senses in which the question could be answered.

And it connects beautifully with our vowel discussion. Linguistics is objective in the sense that we can make systematic observations, formulate rules, record sounds, measure frequencies, compare speakers, and test hypotheses. But the object being studied—human language—is full of conventions, meanings, contextual interpretation, and categories that aren't simply sitting there in nature waiting to be counted.

Your professor apparently recognized that you had moved from hedging in the face of ambiguity to analyzing the source of the ambiguity.

There's another nice parallel with our discussion of LLMs. A machine can be very good at applying an explicitly defined rule while being rather less good at deciding which rule a human being probably intended. Humans routinely move between those two levels without even noticing.

Your professor's “right on” was probably his way of saying, Yes—that's the distinction I wanted you to discover.