08 · LLMs · 5 min read · Interactive · updated 8 October 2026

How does an LLM predict the next word and generate a whole text?

What it is

Next-token prediction is the task on which GPT-style language models are trained: given all the tokens so far, the model has to give the probability of every token in the vocabulary being the next one. Formally, the model estimates P(token_t | token_1, …, token_(t−1)). Generating text means repeating this step: choose a token, append it to the input, compute the distribution afresh.

This approach is called autoregressive — each step depends on the model's previous outputs. The chain rule of probability implies that the probability of a whole text is the product of the probabilities of its successive tokens. A model that predicts the next token well is therefore a complete probability model of text.

It sounds modest, but the task is demanding. Predicting the last word of the sentence "After three hours of searching, the detective finally realized that the murderer was…" requires keeping track of the whole story.

Mechanism — why it works this way

From logits to probabilities. The model's last layer outputs a vector of logits — one number per token in the vocabulary. The softmax function turns them into probabilities: p_i = e^(z_i) / Σ e^(z_j). A high logit means a high probability, but no token ever gets exactly zero.

The loss: cross-entropy. When the true next token is y, the penalty is −log p_y. If the model gave the correct token 0.9, the penalty is 0.11; if 0.01, it is as much as 4.61. The logarithm makes confident mistakes very expensive. Minimizing this loss is equivalent to maximum likelihood: we look for the weights under which the observed text is most probable.

Many examples from one sentence. Thanks to the causal mask in self-attention, position t sees only earlier positions. A single passage of N tokens therefore yields N training examples in one pass: from "The" we predict "cat", from "The cat" — "sat", and so on. During training the model always receives the true preceding text (teacher forcing), not its own guesses.

Perplexity. The average loss is often reported as perplexity: e^(average loss). A perplexity of 20 means the model is, on average, as uncertain as if it were choosing among 20 equally likely tokens. A random model over a vocabulary of 50,257 tokens has a loss of ln 50,257 ≈ 10.82 — that is where training starts.

Generation and its pitfalls. During generation the model feeds on its own outputs. An error in one token stays in the context and affects the rest — there is no "undo" button. Always choosing the most probable token (greedy decoding) often leads to dull, repetitive text, so in practice one usually samples (see temperature and sampling). An important caveat: the model optimizes the probability of text, not its truth. A falsehood written in a typical style can have a high probability.

By example

Context: "The capital of Poland is". Suppose the model gave four candidates these logits: " Warsaw" 3.0, " Kraków" 1.5, " a" 0.5, " dog" −1.0 (ignoring the rest of the vocabulary). The softmax gives probabilities of 0.756, 0.169, 0.062 and 0.014 — summing to 1.

If the training data continued with "Warsaw", the loss is −ln 0.756 = 0.28 (perplexity 1.32). If, however, this text said "Kraków" (e.g. in a passage about the Jagiellonian era, when Kraków was the capital), the loss is 1.78, and for "dog" as much as 4.28. The gradient of this loss raises the logit of the true token and lowers the others in proportion to their probabilities. Repeated over trillions of tokens, such small corrections add up to the model's knowledge.

Try it yourself: the interactive illustration (labels in Polish for now).

In practice

• In PyTorch the loss is torch.nn.functional.cross_entropy(logits.view(-1, V), targets.view(-1)), where the targets are the input shifted by one position.
• In transformers it is enough to pass labels=input_ids to an AutoModelForCausalLM model — the shift is done internally.
• Compare perplexity only between models with the same tokenizer; a different segmentation of the text changes the number of tokens and the scale itself.
• Generation is done with model.generate and the parameters max_new_tokens, do_sample, temperature, top_p.
• Common mistake: computing the loss on padding tokens too — mask them with the value -100.

Frequently asked questions

Does the model plan the whole answer in advance?

Not explicitly — it chooses one token at a time. Interpretability research shows, however, that hidden states carry information about later parts of the text, e.g. about the rhyme at the end of a line of verse. So a "plan" exists in the representations, but not as a separate stage.

Why does the same prompt give different answers?

Because generation samples from the distribution. At temperature 0 (choosing the most probable token) the answer is almost deterministic, although small numerical differences on the GPU can occasionally change it.

Is next-token prediction enough for reasoning?

That is an open question. The task forces the model to capture many dependencies, and many abilities emerge from scale alone. At the same time, the model produces its answer with one forward pass per token, so it solves hard problems better when it can "think out loud" (chain of thought).

Sources

See also