What an LLM is: large language models without the mystique

The explanation you hear most often is: a large language model predicts the next word. That is true and also unhelpful. Your phone's autocomplete predicts the next word too, and nobody expects it to refactor their code.

The difference is in everything the model had to learn in order to get that word right.

Prediction as a side effect

Imagine your only job is finishing the last word of a sentence. For "The coffee is hot, so I'll wait for it to ___" a feel for language is enough. For "The sum of 47 and 58 is ___" you now have to do arithmetic. For "This function crashes because the variable is ___" you have to understand code.

That is the whole trick. Train a model to complete text across the entire internet — books, documentation, forums, source code — and it cannot do the job without picking up grammar, arithmetic, a chunk of world knowledge and the structure of programming languages along the way. The abilities are not the goal. They are a by-product of one dumb-sounding task, repeated at enormous scale.

The "large" in LLM does not mean a big file. It means billions of parameters — numbers tuned during training. Everything the model learned lives in them.

Tokens: the model does not see words

Text is chopped into tokens before processing. A token is roughly a piece of a word — common words tend to be one token, longer or unusual ones break into several.

It sounds like trivia, but it explains oddities you will hit. Why does the model sometimes miscount letters in a word? Because it never sees letters, it sees tokens. Why do APIs bill per token rather than per character? Because tokens are the unit the model actually works in. And why does Slovak or Czech burn more tokens than English for the same sentence? Because the tokeniser was trained mostly on English.

The context window is a desk, not a memory

The context window is how many tokens the model can hold at once — your question, the conversation so far, attached documents, system instructions. All of it together.

The important part: the model remembers nothing between calls. When it seems to recall what you said five minutes ago, that is because the application resent the entire conversation. Every single time. The model is a colleague with perfect amnesia: you hand over the whole folder each time and they read it from scratch.

Two consequences follow. Long conversations are expensive, because the whole thing gets resent. And when the window overflows, something falls out — usually the oldest part. That is why long chats occasionally act as though they forgot what you agreed. They did not forget. They never saw it.

Temperature, and why you get a different answer twice

The model does not pick one next word. It computes a probability across the whole vocabulary and samples from it. The temperature setting controls how far it may stray from the most likely choice.

Low temperature means predictable, dull, consistent. High means creative, surprising, unreliable. For pulling fields off an invoice you want it low. For brainstorming campaign names, higher. There is no correct value, only a correct value for the task.

This also explains why testing AI features is tedious. The same input gives you a slightly different answer every time.

Hallucination is not a bug

The model never distinguished between "I know this" and "this sounds plausible". It is doing the same thing throughout: finding the most natural continuation of the text. Ask about a fact that was not in the training data and it will not say "I don't know" — because the natural continuation of a question is an answer, not an admission of ignorance.

So it will confidently invent a standard that does not exist, cite the wrong clause, or recommend a book nobody wrote. This is not lying. It is statistics doing exactly what it was built to do.

You can reduce it — give the model real source material and have it answer from that (the idea behind RAG), demand citations, verify output with code. You cannot remove it.

Why this matters

Most disappointment with LLMs comes from expecting a database and getting a language model. A database returns a fact or an error. A language model always returns text that reads well.

Once that lands, the rest follows: why context matters, why blind trust is a mistake, and why the best AI features look boring — the model proposes, a human confirms.