How can a model that writes working code, passes law exams and explains quantum physics fail to count the letters in a word a five year old can spell? If you’ve spent any time with LLMs, you have probably seen the famous example: ask “How many r’s are in the word strawberry?” and, for a long time, many models confidently answered two.

It became one of the most shared AI fails on the internet. Modern models are much better at it now, but it’s still a great question, because the answer tells you something important about how LLMs actually read text, and why character level tasks can still trip them up.

In this article, we’ll go over why counting letters was so hard for LLMs, what the model actually “sees” when you type a word, why newer models get it right, and what this means for you when you build with LLMs. We’ll also run a real tokenizer in Python to see the problem with our own eyes.

LLMs Don’t Read Letters

When you read “strawberry”, you see ten letters. An LLM never sees those letters. Before your text reaches the model, it goes through a tokenizer, which chops the text into chunks called tokens and replaces each chunk with a number from a fixed vocabulary.

Tokens are usually common pieces of words. Very frequent words get a token of their own, and rarer words are built from a few pieces. GPT-4o’s tokenizer has a vocabulary of about 200,000 tokens, and most of them are several characters long. This is done for efficiency: if every letter were its own token, every prompt would be four or five times longer, and as we saw in the KV cache article, longer sequences cost a lot more compute and memory.

Here is what that looks like for our word:

In a normal sentence like “How many r’s are in the word strawberry?”, the word comes with a space in front of it, and ” strawberry” is a single token: number 101830. That’s all the model gets. Not ten letters, not three r’s, just one number. Let that sink in for a second. You are asking the model to count letters inside something it has never seen spelled out.

Let’s See It In Python

OpenAI’s tiktoken library lets us run the same tokenizer GPT-4o uses. Install it with pip install tiktoken and run:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")  # the tokenizer used by GPT-4o

for text in ["strawberry", " strawberry", "Strawberry", "STRAWBERRY", "s t r a w b e r r y"]:
    ids = enc.encode(text)
    pieces = [enc.decode([i]) for i in ids]
    print(f"{text!r:24} -> {len(ids)} tokens: {pieces}  ids={ids}")

Running it gives:

'strawberry'             -> 3 tokens: ['st', 'raw', 'berry']  ids=[302, 1618, 19772]
' strawberry'            -> 1 tokens: [' strawberry']  ids=[101830]
'Strawberry'             -> 3 tokens: ['Str', 'aw', 'berry']  ids=[3504, 1134, 19772]
'STRAWBERRY'             -> 4 tokens: ['ST', 'RAW', 'B', 'ERRY']  ids=[1117, 46176, 33, 132354]
's t r a w b e r r y'    -> 10 tokens: ['s', ' t', ' r', ' a', ' w', ' b', ' e', ' r', ' r', ' y']  ids=[82, 260, 428, 261, 286, 287, 319, 428, 428, 342]

A few interesting things jump out:

• The same word gets very different tokens depending on a leading space or capital letters. To the model, “strawberry”, ” strawberry” and “STRAWBERRY” are three unrelated sequences of numbers. It has to learn from data that they mean the same thing.
• Even when the word is split, the r’s are hidden. In ['st', 'raw', 'berry'], one r is buried inside “raw” and the other two inside “berry”. The token ids 1618 and 19772 say nothing about which letters they contain.
• Spelled out with spaces, each letter becomes its own token. Look at the last line: token 428, which is ” r”, appears exactly three times. Now the r’s are actually visible to the model, and counting them becomes a much easier job.

So How Do Models Know Spelling At All?

If the model only sees token 101830, how can it spell “strawberry” when you ask it to? Because it has picked up spelling indirectly from its training data. Somewhere on the internet there are spelling lists, crossword clues, children’s books and people writing “s-t-r-a-w-b-e-r-r-y”. From millions of examples like these, the model learns a rough association between a token and the letters it is made of.

The key word is rough. This knowledge is memorised, not looked up, so it can be fuzzy, especially for the exact number of repeated letters. And “berry” with its double r is exactly the kind of detail that gets blurred. One popular theory for the answer “two” is that people online often ask whether “berry” is spelled with one r or two, so the model has seen “two r’s” next to this word many times, even though the whole word has three.

Counting Is Hard Too

Tokenization is the main culprit, but it isn’t the only one. Counting is also an awkward task for a transformer. To count, you need to keep a running tally as you go: one, two, three. A model answering in a single token, “2” or “3”, has to do all of that inside one forward pass, with a fixed number of layers and no notepad to write on.

That’s why LLMs also tend to struggle with related tasks, even on words that are split into individual characters:

• counting the words in a paragraph, or writing exactly 50 words
• reversing a string, or spelling a word backwards
• staying under a strict character limit, like a 160 character meta description
• anagrams, rhymes and other word games that depend on letters

Why Newer Models Get It Right

If you try the strawberry question on a recent model, it will most likely answer three. The tokenizer hasn’t changed much, so what did?

1. Reasoning. Reasoning models think before they answer. They write out something like “s, t, r (1), a, w, b, e, r (2), r (3), y” in their hidden thinking. Just like our Python example, spelling the word out turns each letter into its own token, and writing the tally down gives the model the notepad it was missing. Fittingly, OpenAI’s first reasoning model, o1, was reportedly developed under the codename “Strawberry”.
2. Training on the problem. Once a failure becomes famous, it ends up in training data and evaluation sets, and models get taught to handle it. This fixes the famous example, but not always the general skill, so slightly different questions, like counting the s’s in “Mississippi” or the e’s in a long sentence, can still trip models up.
3. Tools. Many assistants can now run code. A model that writes "strawberry".count("r") and runs it will never get this wrong.

What This Means For You

Whenever your task depends on individual characters, keep tokenization in mind:

1. Ask the model to spell it out first. A prompt like “Spell the word letter by letter, numbering each r, then give the total” works far better than asking for the number directly.
2. Use code for exact counting. Character counts, word counts, string lengths and validation belong in regular code, not in the model. If you are building an app, count in Python and pass the result to the model if it needs it.
3. Don’t trust hard limits. If you ask for “under 160 characters”, check the length yourself and ask for a shorter version if needed.
4. Don’t judge a model by this one test. Failing at strawberry says a lot about tokenization and very little about how well a model can reason, write or code.

Key Takeaway

LLMs process tokens, not individual letters, by default. Modern models have become much better at reasoning about the characters inside those tokens, but the underlying limitation hasn’t disappeared entirely.

Older models trying to count the r’s in “strawberry” were a bit like someone counting the letters in a word they had only ever heard, never seen written down. Today’s models usually get it right, but they can still slip on unfamiliar strings, long sequences or more complex character counting. The strawberry problem was never a sign that LLMs are stupid. It’s a window into how they work: everything you send them is turned into numbers first, and anything that gets lost in that step, like the letters inside a word, has to be learned the hard way.

That’s all I wanted us to go over in this article. If you found it useful, share it with the next person who sends you a screenshot of an AI failing at strawberry, and see you in the next one.

Cheers!