Cross-entropy Loss

Training is a loop: the model predicts, something measures how wrong the prediction was, and backpropagation nudges every weight downhill on that measurement. The measurement has to be a single differentiable number — that is the entire job description of a loss function. For a language model, the prediction is a probability distribution over the whole vocabulary (the output of the final softmax), and the “right answer” is simply whichever token actually came next in the training text.

Cross-entropy is the standard loss for this, and the formula is shorter than its name:

Read that again, because it is stranger than it looks: the loss reads one entry of the prediction vector. GPT-2’s output distribution has 50,257 entries, and cross-entropy looks at exactly one of them — the probability on the token that actually occurred. The other 50,256 numbers never appear in the formula. (They still matter, through the gradient — section 2.)

And what supervises this? The training text itself. The “label” at every position is just the next token of the document, so raw text is its own labeled dataset — this is what self-supervised means, and it is why LLMs can train on trillions of tokens without a single human annotation. A 1,000-token document is 1,000 classification problems, each one “predict entry number $t{+}1$,” and the reported training loss is the average $-\log(p_c)$ over every position.

Two facts carry this whole topic.

Fact 1: only $p_c$ enters the formula. Shuffle probability mass among the wrong answers however you like — the loss does not move. A prediction of [cat 0.7, dog 0.29, bird 0.01] and [cat 0.7, dog 0.15, bird 0.15] cost exactly the same when the truth is “cat.”

Fact 2: the −log curve is savagely asymmetric. Here is the ladder, worth memorizing:

The pattern in the ladder: equal ratios in probability become equal steps in loss. Every halving of $p_c$ adds the same 0.69; every ÷10 adds the same 2.3. That means the curve is nearly flat at the top and a cliff at the bottom — losing 0.09 of probability mass from 0.99 → 0.9 costs +0.095 of loss, while losing the same 0.09 of mass from 0.1 → 0.01 costs +2.303. Same mass, 24× the price, purely because of where on the curve it happens. High confidence on the right answer is rewarded cheaply; high confidence on a wrong answer is punished without limit.

Feel both facts directly:

So the loss value ignores the wrong answers — but the gradient does not. In every real model, cross-entropy sits directly on top of softmax, and that pairing produces one of the cleanest results in deep learning: the gradient of the loss with respect to logit $i$ is

$\frac{\partial L}{\partial z_i} = p_i - y_i$

— predicted probability minus one-hot target. The true token’s logit gets pushed up with force $1 - p_c$; every wrong token’s logit gets pushed down with force equal to its own current probability. The log exactly cancels the softmax’s exp, which is why the two are always implemented as a single fused operation and why this pairing, rather than some other loss on top of softmax, became the default.

The true next token is “cat”. Three models make three predictions over the tiny vocabulary {cat, dog, bird}:

Notice what the third row does not say: it doesn’t matter that dog got 0.50 — [cat 0.01, dog 0.98, bird 0.01] would cost exactly the same 4.605. The loss only asks one question: how much probability did you put on what actually happened?

The core computation is one line, and the ladder from section 2 is three more:

import numpy as np

p = np.array([0.70, 0.20, 0.10])       # cat, dog, bird
-np.log(p[0])                           # 0.357 — the loss, truth = index 0

-np.log(0.99)                           # 0.010
-np.log(0.01)                           # 4.605 — 460× the loss of 0.99
-np.log(0.001)                          # 6.908

What actually runs in training never materializes probabilities at all — it goes straight from logits to loss with the fused log-softmax (section 4 is about why):

import torch.nn.functional as F

# logits: (seq_len, vocab_size) — raw scores, pre-softmax
# targets: (seq_len,)           — the actual next-token ids
loss = F.cross_entropy(logits, targets)   # mean of -log p_correct over positions

Two numbers worth carrying into any interview:

• The step-0 sanity check. A freshly initialized model knows nothing, so its softmax is roughly uniform: $p_c \approx 1/50{,}257$ for GPT-2’s vocabulary, giving loss $\approx \ln(50{,}257) = 10.82$ (about 12.2 for a ~200k vocabulary). If your loss at step 0 isn’t close to $\ln(V)$, your wiring is broken before training even starts.
• Perplexity is just $e^{\text{loss}}$. A mean loss of 2.303 nats is perplexity 10: on average the model is as uncertain as a uniform choice among 10 tokens. Papers report perplexity; the optimizer minimizes cross-entropy; they are the same measurement on different scales.

• One confidently-wrong token dominates the batch. A token predicted at 0.001 contributes 6.9 to the sum — as much as ~690 tokens predicted at 0.99. And its gradient is the strongest the loss can emit: $p_i - y_i \approx -1$ on the true token’s logit. This asymmetric pressure is the point — it is what forces models to hedge realistically instead of gambling on favorites — but it also means loss spikes during training often trace back to a handful of rare tokens the model was sure couldn’t come next.
• A hard zero is an infinite loss. If anything in the pipeline puts an exact 0 on a token that then actually occurs, the loss is $-\log(0) = \infty$ and the gradients are NaN — one poisoned position kills the whole batch, and a NaN loss step can corrupt the weights for good. Softmax with finite logits can never emit an exact 0 (the Softmax topic covers why), which is one more reason the pairing is safe — but code that post-processes probabilities (banning tokens by zeroing them, aggressive rounding, hand-rolled masking applied in the wrong place) rediscovers this failure the hard way. This is also why you never hard-zero a vocabulary entry you might still need to score.
• log(softmax(x)) as two separate ops underflows. A logit 104 nats below the max has a true log-probability of about −104 — perfectly representable. But computed naively, softmax first produces $e^{-104} \approx 7 \times 10^{-46}$, which rounds to exactly 0 in float32 (the smallest positive value is about $1.4 \times 10^{-45}$), and then the log returns −inf. The fix is algebra, not clamping: $\log(\mathrm{softmax}(z)_c) = z_c - \mathrm{logsumexp}(z)$, computed with the same subtract-the-max trick softmax itself uses. This is exactly what F.cross_entropy fuses — which is why it takes logits, and why feeding it probabilities is a silent correctness bug.
• The loss can never reach zero — and shouldn’t. Zero loss requires $p_c = 1$ at every position, which softmax cannot produce with finite logits — and which language itself forbids: after “My favorite color is” there is genuine uncertainty about the next token, and the best possible model matches that uncertainty rather than eliminating it. Formally, $H(P,Q) = H(P) + \mathrm{KL}(P\|Q)$: training can only drive the KL term toward zero, and the entropy of real text remains as an irreducible floor. A flattening loss curve near that floor is not a model that stopped learning — chasing it to zero anyway is called overfitting.