Probability you actually need

1

Concept

A probability is a number between zero and one expressing uncertainty under a model. Zero rules an event out; one makes it certain within that model. For mutually exclusive possibilities, the probabilities add to one. A language model’s next-token output is a categorical distribution: one probability for each vocabulary item. The distribution is the prediction. Selecting one token from it is a separate decoding operation.

Probability is always attached to assumptions and information. We write $P(A)$ for the probability of event $A$, and $P(A\mid B)$ for the probability of $A$ given that $B$ is known. Language modelling is conditional: $P(x_t\mid x_1,\ldots,x_{t-1})$. The same token can be likely after one prefix and unlikely after another. “Mercury” after “the closest planet to the Sun is” differs from “the liquid metal is.”

A random variable maps possible outcomes to values. A distribution says how probability is allocated among them. For a discrete variable $X$, its expected value is a probability-weighted average, $E[X]=\sum_x p(x)x$. Expectation is not necessarily an outcome you will observe. The expected roll of a fair six-sided die is 3.5, although no face shows 3.5. In machine learning, expected loss summarizes performance across examples or possible outputs.

The chain rule decomposes the probability of a sequence. For tokens $x_1,\ldots,x_T$,

This identity explains why repeated next-token prediction defines a probability for an entire passage. It also explains why long sequence probabilities become tiny: many numbers below one are multiplied. Tiny values are awkward for finite-precision computers, so implementations work with logarithms.

Because $\log(ab)=\log a+\log b$, a product of probabilities becomes a sum of log probabilities. The log of a probability is at most zero. A more likely event has a log probability closer to zero; an impossible event tends toward negative infinity. Training often minimizes negative log-likelihood, which simply changes the sign so better predictions yield a smaller objective. Averaging it over tokens makes sequences comparable.

Two probability rules prevent common reasoning errors. First, $P(A\mid B)$ is not generally $P(B\mid A)$. A symptom may be common among people with a condition while the condition remains rare among people with that symptom. Second, high probability is not truth. A language model estimates regularities in its data and training process; a fluent misconception can receive high probability. Calibration asks whether events assigned, say, 70 percent probability occur about 70 percent of the time across comparable cases.

Softmax bridges arbitrary model scores and a distribution. Given logits $z_i$, it computes $p_i=e^{z_i}/\sum_j e^{z_j}$. Adding the same constant to every logit leaves probabilities unchanged, because only relative differences matter. Temperature divides logits before softmax and changes how concentrated the distribution is. These transformations manipulate uncertainty; they do not supply new evidence.

The practical toolkit is small: normalize distributions, condition on context, use the chain rule for sequences, use expectations for averages, and use logs for stable products. Keep the distinction between a model’s distribution and the world it describes. With those pieces, loss, entropy, perplexity, sampling, and cross-entropy become variations on familiar ideas rather than isolated formulas.