Modelsllm

The Statistics of Token Selection: Logits, Temperature, and Top-P Explained

P&C

Prompt & Circuit Editorial

May 31, 2026 · 12 Min Read

The Statistics of Token Selection: Logits, Temperature, and Top-P Explained

TL;DR

  • Logits are the raw, unnormalised scores a language model produces for every token in its vocabulary.
  • Softmax turns logits into a probability distribution.
  • Temperature divides the logits before softmax: below 1 sharpens the distribution, above 1 flattens it.
  • Top-k keeps only the k highest-scoring tokens.
  • Top-p (nucleus sampling) keeps the smallest set of tokens whose cumulative probability reaches p.
  • Lower temperature and tighter filtering make output more predictable; higher values make it more varied.

What are logits?

In a decoder-only transformer, the final hidden state for the last position is projected by a linear layer (the "LM head") onto a vector as long as the vocabulary. Each entry is a logit: an unbounded real number that says how strongly the model favours that token as the next one.

Logits are not probabilities. They can be negative or large, and they do not sum to 1. Softmax converts them:

Pi=ezi∑jezjP_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

where ziz_i is the logit for token ii. The result is a valid probability distribution over the vocabulary.

A worked example with a toy five-token vocabulary and logits [4, 3, 2, 1, 0]:

TokenLogitProbability (softmax)
A40.636
B30.234
C20.086
D10.032
E00.012

A one-point gap in logits becomes roughly a 2.7x gap in probability, because softmax exponentiates.

Temperature

Temperature tt rescales the logits before softmax:

zi′=zitz'_i = \frac{z_i}{t}
  • t<1t < 1 magnifies the gaps between logits, so the distribution sharpens and the top token dominates.
  • t>1t > 1 shrinks the gaps, so the distribution flattens and less likely tokens gain probability.
  • t=1t = 1 leaves the distribution unchanged.

The same toy logits at three temperatures:

TemperatureABCDE
0.50.8650.1170.0160.0020.000
1.00.6360.2340.0860.0320.012
2.00.4290.2600.1580.0960.058

As tt approaches 0, all the probability mass moves to the highest logit, which is the same as greedy decoding. Dividing by exactly zero is undefined, so implementations treat it as a special case (the code below maps it to greedy). In Hugging Face Transformers you get greedy decoding by setting do_sample=False.

Because temperature changes every logit, it also changes the softmax denominator, so every probability moves, not just the top one.

Top-k sampling

Top-k restricts the candidates to the k tokens with the highest scores:

  1. Find the k largest logits.
  2. Set every other logit to negative infinity (probability 0).
  3. Apply softmax to what remains.
  4. Sample.

With k = 3 on the toy example, tokens D and E are removed and A, B and C are renormalised. The weakness of top-k is that k is fixed: when the model is confident, k = 50 still admits 49 unlikely tokens; when the distribution is genuinely flat, k = 50 may cut off reasonable options.

Top-k does not make the model itself faster. The forward pass that produces the logits costs the same regardless of k; filtering only changes which token is drawn.

Top-p (nucleus) sampling

Nucleus sampling was introduced by Holtzman et al. in "The Curious Case of Neural Text Degeneration" (ICLR 2020). The authors observed that likelihood-maximising decoding tends to produce bland, repetitive text, and proposed sampling from the "nucleus" of the distribution, which trims the unreliable low-probability tail while keeping diversity.

The procedure:

  1. Sort tokens by probability, highest first.
  2. Accumulate probabilities until the running total reaches p.
  3. Keep that set and discard the rest.
  4. Renormalise and sample.

On the toy example the cumulative probabilities are 0.636, 0.871, 0.957, 0.988 and 1.0. With p = 0.9, the nucleus is A, B and C, because 0.871 is still below 0.9 and adding C takes the total to 0.957.

Unlike top-k, top-p adapts to the shape of the distribution: a peaked distribution yields a small nucleus, a flat one a large nucleus. With p = 1.0 nothing is removed and only temperature affects randomness.

How the steps combine

The order matters, because temperature changes the probabilities that top-p measures. In Hugging Face Transformers, when sampling is enabled, the generation code applies temperature first, then top-k, then top-p (and min-p if set).

Click to expand and zoom

Combining top-k and top-p is common. Top-k sets a hard upper bound on candidates and top-p trims further when the model is confident.

A from-scratch implementation

This NumPy function implements the pipeline above. It runs as-is and produces the numbers in the tables.

import numpy as np

def softmax(z):
    z = z - z.max()            # subtract the max for numerical stability
    e = np.exp(z)
    return e / e.sum()

def sample_next(logits, temperature=1.0, top_k=0, top_p=1.0, rng=None):
    rng = rng or np.random.default_rng()
    logits = np.asarray(logits, dtype=np.float64)
    if temperature <= 0:                       # treat 0 as greedy decoding
        return int(np.argmax(logits))
    z = logits / temperature                   # 1. temperature
    if top_k > 0:                              # 2. top-k: keep the k largest logits
        kth = np.sort(z)[-min(top_k, z.size)]
        z = np.where(z < kth, -np.inf, z)
    p = softmax(z)                             # 3. softmax
    if top_p < 1.0:                            # 4. top-p: smallest set with mass >= top_p
        order = np.argsort(p)[::-1]
        cum = np.cumsum(p[order])
        keep = order[: np.searchsorted(cum, top_p) + 1]
        mask = np.zeros_like(p, dtype=bool)
        mask[keep] = True
        p = np.where(mask, p, 0.0)
        p = p / p.sum()                        # renormalise
    return int(rng.choice(p.size, p=p))        # 5. sample

logits = np.array([4.0, 3.0, 2.0, 1.0, 0.0])
for t in (0.5, 1.0, 2.0):
    print(f"t={t}:", np.round(softmax(logits / t), 3))

rng = np.random.default_rng(0)
print(sample_next(logits, temperature=0.7, top_k=3, top_p=0.9, rng=rng))

Using it with Hugging Face Transformers

In practice you pass the parameters to generate() rather than writing the loop yourself. Sampling must be switched on with do_sample=True; otherwise Transformers uses greedy decoding and ignores temperature, top-k and top-p. If a model's generation_config.json does not set them, the defaults are temperature=1.0, top_k=50 and top_p=1.0.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen2.5-0.5B-Instruct"   # small, ungated model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

def generate(prompt, temperature=0.7, top_p=0.9, top_k=50, max_new_tokens=100):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    output = model.generate(
        **inputs,
        do_sample=True,
        temperature=temperature,
        top_k=top_k,
        top_p=top_p,
        max_new_tokens=max_new_tokens,
    )
    return tokenizer.decode(output[0], skip_special_tokens=True)

print(generate("Write a short haiku about sunrise."))

This needs transformers, torch and accelerate installed (device_map="auto" relies on Accelerate).

Practical tuning guidance

There are no universally correct values; the right settings depend on the model, the task and how much variation you can tolerate. Some guidance that follows directly from the mechanics:

  • Deterministic or factual tasks (extraction, classification, code): use greedy decoding or a low temperature. Low temperature does not make a model more accurate; it makes it pick its most likely answer more consistently.
  • Open-ended writing and brainstorming: start at temperature 1.0 with a top-p around 0.9, then adjust.
  • Change one knob at a time. Temperature and top-p both control diversity, so moving both at once makes it hard to tell which change helped.
  • Min-p is an alternative. Transformers also supports min_p, which drops tokens whose probability falls below a fraction of the top token's probability. Its documentation suggests values of 0.01 to 0.2, which it describes as comparably selective to top-p values of 0.99 to 0.8.
  • Beam search is a different strategy. In Transformers, beam search keeps several candidate sequences and picks the one with the highest overall probability. It suits input-grounded tasks such as captioning or speech recognition more than creative generation.

FAQ

Does temperature affect the softmax denominator? Yes. It rescales every logit, so the sum of exponentials changes and every probability shifts.

Can I use top-k and top-p together? Yes. In Transformers both are applied when sampling, top-k first and then top-p.

What happens if p = 1.0 or k equals the vocabulary size? That filter becomes a no-op, and only temperature shapes the distribution.

Is there a best temperature? No. It depends on the task, the model and how much variation you want. Measure on your own prompts.

Sources

Tags:llmdecodingsamplingtemperaturetop-p
Share Article

Responses (0)

Sign in to join the conversation and share your thoughts.

P&C

Prompt & Circuit Editorial

Contributing Editor

Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.