Top-k vs Top-p

The Temperature lesson ended on a warning: temperature reshapes the next-token distribution but never truncates it. That matters because softmax outputs are strictly positive — every one of a ~200,000-token vocabulary gets nonzero probability at every step, including tokens that would derail the sentence outright. Sampling draws from that entire distribution, so each generated token is a small lottery in which every garbage token holds a ticket.

The tail is individually negligible and collectively real. Suppose the plausible top 50 tokens carry 97% of the mass and the remaining ~199,950 tokens split the last 3%. One draw is 97% safe. But a 500-token response makes 500 draws, and the chance of never hitting the tail is $0.97^{500} \approx 2 \times 10^{-7}$ — a derailment is essentially guaranteed, and one absurd token early in a generation poisons everything conditioned on it. The fix is not to reshape the distribution — it is to cut the tail off before sampling. That is truncation, and top-k and top-p are the two standard knives.

The two rules sound interchangeable. They are not, and the difference is the entire point of this lesson: top-k commits to a candidate count before seeing how confident the model is; top-p reads the confidence off the distribution and sizes the candidate pool to match.

Top-k is one line: sort descending, keep ranks 1 through $k$, zero out the rest, renormalize. The candidate pool is $k$ tokens whether the model is certain or clueless. Predictable, cheap, and blind to the one signal that matters — the shape of the distribution it is cutting.

Top-p walks down the same sorted list accumulating probability and stops at the first token that pushes the running total to $p$. That prefix is the nucleus — the smallest set of tokens that collectively account for probability $p$ — and everything after it is deleted. The pool size is now an output of the distribution, not an input: when one token hoards the mass the nucleus closes after a token or two, and when the mass is spread thin the nucleus stretches to cover dozens of legitimate candidates.

Feel the asymmetry below. The peaked preset is a model that knows the answer; the flat preset is a model with genuine options. Top-p resizes between them. Top-k cannot.

distributionpeaked (confident)flat (uncertain)

The capital of France is

top-kk = 5

top-pp = 0.90

sort ↓ · cut · renormalizesurvivors: Σ = 1.00

Paris

Σ 87%87.0% → 95.6%

the

Σ 91%4.0% → 4.4%

located

Σ 94%2.5%×p

a

Σ 95%1.5%×p

one

Σ 96%1.0%×p

home

Σ 97%0.8%×k ×p

known

Σ 98%0.7%×k ×p

France

Σ 98%0.6%×k ×p

Lyon

Σ 99%0.5%×k ×p

beautiful

Σ 99%0.5%×k ×p

situated

Σ 100%0.5%×k ×p

Marseille

Σ 100%0.4%×k ×p

kept 2 of 12 — nucleus wanted 2, top-k allowed 5 · deleted tail mass 9.0%, inherited by the survivors (the teal sliver) · same k and p on the flat (uncertain) preset keep 5 — top-p adapts, top-k doesn't.

Tokens cut by either rule fade out (×k / ×p tags show which rule cut them), survivors renormalize, and the teal sliver on each surviving bar is the mass it inherits from the deleted tail. The Σ column is the running cumulative sum top-p walks down. To see the adaptivity: slide top-k to 12 (off), then flip between presets — at p = 0.90 the nucleus keeps 2 tokens on the peaked distribution and 10 on the flat one. Now set k = 5 and flip again: exactly 5 survive on both, junk included or good candidates excluded — the readout shows the collision (nucleus wanted 10, top-k allowed 5).

Take the two distributions from the widget. After The capital of France is, the model puts 87% on Paris and scraps on eleven alternatives. Walk the sorted list with both rules — top-k with $k = 5$, top-p with $p = 0.9$:

rank	token	$p_i$	cumulative	top-k = 5	top-p = 0.9
1	Paris	0.870	0.870	keep	keep
2	the	0.040	0.910	keep	keep — crosses 0.9, nucleus closes
3	located	0.025	0.935	keep	cut
4	a	0.015	0.950	keep	cut
5	one	0.010	0.960	keep	cut
6–12	home … Marseille	0.040 total	1.000	cut	cut

Top-p keeps 2 tokens and renormalizes over their 0.91 of mass: Paris $0.87 / 0.91 = 95.6\%$, the $4.4\%$. Top-k keeps 5 — dragging located, a, and one into the lottery as 2.6%, 1.6%, and 1.0% shares after renormalizing over 0.96, alternatives the nucleus correctly judged noise.

Now the flat distribution, For dinner I'm making: pasta 0.14, chicken 0.12, a 0.11, some 0.10, rice 0.09, soup 0.08, tacos 0.08, pizza 0.07, curry 0.07, salmon 0.06, stir-fry 0.05, lasagna 0.03. The cumulative sum crawls — it reaches 0.92 only at rank 10. Same $p = 0.9$, five times the survivors: the nucleus read the model’s uncertainty and widened. Top-k = 5 still keeps exactly 5, amputating soup through lasagna — 44% of the probability mass, all of it legitimate dinner.

import numpy as np

peaked = np.array([0.870, 0.040, 0.025, 0.015, 0.010, 0.008,
                   0.007, 0.006, 0.005, 0.005, 0.005, 0.004])  # "The capital of France is"
flat   = np.array([0.140, 0.120, 0.110, 0.100, 0.090, 0.080,
                   0.080, 0.070, 0.070, 0.060, 0.050, 0.030])  # "For dinner I'm making"

def top_k(probs, k):                        # fixed COUNT
    keep = np.argsort(probs)[::-1][:k]
    out = np.zeros_like(probs)
    out[keep] = probs[keep]
    return out / out.sum()                  # renormalize the survivors

def top_p(probs, p):                        # fixed MASS (nucleus)
    order = np.argsort(probs)[::-1]
    cum = np.cumsum(probs[order])
    n = np.searchsorted(cum, p) + 1         # smallest prefix with cum >= p
    out = np.zeros_like(probs)
    out[order[:n]] = probs[order[:n]]
    return out / out.sum()

(top_p(peaked, 0.9) > 0).sum()    # 2  — confident: the nucleus collapses
(top_p(flat,   0.9) > 0).sum()    # 10 — uncertain: the nucleus expands
(top_k(peaked, 5)   > 0).sum()    # 5  — top-k keeps 5 either way
(top_k(flat,   5)   > 0).sum()    # 5

top_p(peaked, 0.9)[:2]            # [0.956, 0.044] — survivors inherit the deleted mass
np.random.choice(12, p=top_p(flat, 0.9))    # sampling happens AFTER truncation

• Top-k on a peaked distribution keeps junk alive. With $k = 40$ after The capital of France is, one candidate is right and 39 are noise — and renormalization guarantees the noise gets real tickets. In the worked example, $k = 5$ hands 5.2% of the final distribution to located, a, and one. This is the exact failure top-p exists to fix: the nucleus sees 87% on Paris and closes at 2.
• Top-k on a flat distribution clips good candidates. The same $k = 5$ on the dinner distribution deletes 44% of legitimate mass — tokens the model rated nearly as good as the survivors. Fixed count means one $k$ cannot be right for both regimes, and every generation visits both: a model is peaked mid-word and flat at the start of a sentence.
• Tiny p collapses to near-greedy. At $p = 0.5$ the nucleus on the peaked distribution is just Paris — sampling with one ticket is argmax. Push $p$ low everywhere and you inherit greedy decoding’s repetition loops from the Beam Search vs Greedy lesson, with extra steps. At the other end, $p = 1.0$ keeps everything: no truncation at all, lottery tickets for the whole tail.
• Order of operations silently changes the result. Temperature and truncation do not commute. Take logits 5, 2, 1 from the Softmax lesson with $T = 2$ and $p = 0.9$. Truncate first ($T = 1$ probabilities 0.936, 0.047, 0.017): the nucleus is one token — greedy, and temperature has nothing left to do. Temperature first (0.736, 0.164, 0.100): the cumulative hits 0.900 at rank 2, so two tokens survive. Identical settings, deterministic output vs a live coin-flip — which is why “same parameters, different library, different behavior” is usually an order-of-warpers bug, not a model difference.
• Truncation can’t fix a wrong distribution. Like temperature, both rules operate on the model’s existing ranking — they only ever delete from the bottom. If the right token is ranked 60th, top-k = 50 and any sane top-p delete it; no sampling knob resurrects an answer the model didn’t rate.

Top-k and top-p both truncate the tail. What is the fundamental difference, and when does it show?

Top-k fixes the candidate count; top-p fixes the candidate mass and lets the count float. The difference shows whenever distribution shape varies — which is every step of every generation. On a peaked distribution (87% on one token), p = 0.9 keeps 2 tokens while k = 40 hauls 39 junk alternatives into the renormalized lottery; on a flat distribution the same p = 0.9 expands to dozens of survivors while k = 5 amputates nearly half the legitimate mass. Top-p adapts to the model’s confidence; top-k ignores it. That adaptivity is why nucleus sampling generally wins on quality.

Walk through exactly how nucleus sampling builds its keep-set.

Sort tokens by probability descending. Walk down the list accumulating a running sum. Stop at the first token where the cumulative reaches the threshold p — that prefix (the nucleus) is the smallest set whose total probability is ≥ p. Zero out everything after it, renormalize the survivors to sum to 1 (each survivor scales by 1/kept-mass), then sample from it — temperature, if you use it, was already applied to the logits upstream of the cut. One subtlety: the token that crosses the threshold is included, which is why p = 0.9 on a distribution with 87% + 4% at the top keeps two tokens, not one.

Production APIs expose temperature, top-k, and top-p simultaneously. Isn't that redundant?

No — the knobs do disjoint jobs. Temperature reshapes the distribution (moves every pairwise ratio, deletes nothing); truncation deletes the tail (moves no ratio among survivors — renormalization is a uniform scale-up). High temperature without truncation hands real mass to garbage tokens; truncation without temperature gives no control over diversity inside the kept set. The standard stack layers them — Hugging Face transformers applies temperature, then top-k, then top-p — using top-p for adaptive quality and top-k as a hard ceiling on candidates behind it: top-p quality with a top-k safety cap.

Same model, same temperature, same top-p — two libraries produce visibly different behavior. What's your first suspect?

The order of the sampling warpers. Temperature and truncation don’t commute: with logits 5, 2, 1, T = 2, p = 0.9 — truncating on the raw distribution (0.936, 0.047, 0.017) closes the nucleus at one token, making generation greedy regardless of temperature; applying T = 2 first (0.736, 0.164, 0.100) leaves two tokens in the nucleus and real randomness. Identical parameters, qualitatively different sampling. Check where each library inserts temperature relative to top-k/top-p before blaming the model.

When does top-k actively hurt quality? Give both failure directions.

Peaked distributions: k is too big — after “The capital of France is” the model puts 87% on Paris, and k = 40 keeps 39 noise tokens that renormalization then funds with real probability; every sample is a fresh chance to say something absurd. Flat distributions: k is too small — at the start of a sentence with twenty comparable continuations, k = 5 deletes legitimate candidates carrying ~44% of the mass in the worked example, narrowing the model artificially. Since one generation passes through both regimes, no single k is right, which is the argument for the adaptive rule.

Does truncation change the model's ranking or the ratios between surviving tokens?

Neither. Both rules cut strictly from the bottom of the sorted list, so whatever survives keeps its rank order — like temperature, truncation can never promote a token the model ranked low. And renormalization divides every survivor by the same kept-mass constant, so the ratios among survivors are untouched: with p = 0.9 on the peaked example, Paris goes 87% → 95.6% and “the” goes 4% → 4.4%, still exactly 21.75× apart. Contrast temperature, which exists precisely to change those ratios (softmax turns a logit gap Δ into a ratio e^Δ, and T rescales it to e^(Δ/T)). Truncation picks the entrants; temperature sets the odds.