It is Stupid to Ask How Many R’s ‘Strawberry’ Has

Whenever a new LLM is released, users tend to quiz it first with basic questions like: “How many R’s does 'Strawberry' have?” or “Which one is bigger – 9.9 or 9.11?”.  Most models, like GPT-3.5, Claude, and Llama, get the answer wrong. The problem starts when users try to benchmark the reasoning capabilities of a model based on these questions.  Steve Wilson, the CEO of Exabeam, explained that when an LLM processes a word, it doesn’t see it as individual letters but rather as tokens. These tokens may represent entire words or subword units, depending on the model’s design.  “For example, “strawberry” might be broken into one or more tokens that don’t directly correspond to individual letters. This can lead to errors when trying to analyse or