The Statistics of Token Selection: Logits, Temperature, and Top-P Explained
Prompt & Circuit Editorial
May 31, 2026 · 12 Min Read

TL;DR
- Logits are the raw, unnormalised scores a language model produces for every token in its vocabulary.
- Softmax turns logits into a probability distribution.
- Temperature divides the logits before softmax: below 1 sharpens the distribution, above 1 flattens it.
- Top-k keeps only the k highest-scoring tokens.
- Top-p (nucleus sampling) keeps the smallest set of tokens whose cumulative probability reaches p.
- Lower temperature and tighter filtering make output more predictable; higher values make it more varied.
What are logits?
In a decoder-only transformer, the final hidden state for the last position is projected by a linear layer (the "LM head") onto a vector as long as the vocabulary. Each entry is a logit: an unbounded real number that says how strongly the model favours that token as the next one.
Logits are not probabilities. They can be negative or large, and they do not sum to 1. Softmax converts them:
where is the logit for token . The result is a valid probability distribution over the vocabulary.
A worked example with a toy five-token vocabulary and logits [4, 3, 2, 1, 0]:
| Token | Logit | Probability (softmax) |
|---|---|---|
| A | 4 | 0.636 |
| B | 3 | 0.234 |
| C | 2 | 0.086 |
| D | 1 | 0.032 |
| E | 0 | 0.012 |
A one-point gap in logits becomes roughly a 2.7x gap in probability, because softmax exponentiates.
Temperature
Temperature rescales the logits before softmax:
- magnifies the gaps between logits, so the distribution sharpens and the top token dominates.
- shrinks the gaps, so the distribution flattens and less likely tokens gain probability.
- leaves the distribution unchanged.
The same toy logits at three temperatures:
| Temperature | A | B | C | D | E |
|---|---|---|---|---|---|
| 0.5 | 0.865 | 0.117 | 0.016 | 0.002 | 0.000 |
| 1.0 | 0.636 | 0.234 | 0.086 | 0.032 | 0.012 |
| 2.0 | 0.429 | 0.260 | 0.158 | 0.096 | 0.058 |
As approaches 0, all the probability mass moves to the highest logit, which is the same as greedy decoding. Dividing by exactly zero is undefined, so implementations treat it as a special case (the code below maps it to greedy). In Hugging Face Transformers you get greedy decoding by setting do_sample=False.
Because temperature changes every logit, it also changes the softmax denominator, so every probability moves, not just the top one.
Top-k sampling
Top-k restricts the candidates to the k tokens with the highest scores:
- Find the k largest logits.
- Set every other logit to negative infinity (probability 0).
- Apply softmax to what remains.
- Sample.
With k = 3 on the toy example, tokens D and E are removed and A, B and C are renormalised. The weakness of top-k is that k is fixed: when the model is confident, k = 50 still admits 49 unlikely tokens; when the distribution is genuinely flat, k = 50 may cut off reasonable options.
Top-k does not make the model itself faster. The forward pass that produces the logits costs the same regardless of k; filtering only changes which token is drawn.
Top-p (nucleus) sampling
Nucleus sampling was introduced by Holtzman et al. in "The Curious Case of Neural Text Degeneration" (ICLR 2020). The authors observed that likelihood-maximising decoding tends to produce bland, repetitive text, and proposed sampling from the "nucleus" of the distribution, which trims the unreliable low-probability tail while keeping diversity.
The procedure:
- Sort tokens by probability, highest first.
- Accumulate probabilities until the running total reaches p.
- Keep that set and discard the rest.
- Renormalise and sample.
On the toy example the cumulative probabilities are 0.636, 0.871, 0.957, 0.988 and 1.0. With p = 0.9, the nucleus is A, B and C, because 0.871 is still below 0.9 and adding C takes the total to 0.957.
Unlike top-k, top-p adapts to the shape of the distribution: a peaked distribution yields a small nucleus, a flat one a large nucleus. With p = 1.0 nothing is removed and only temperature affects randomness.
How the steps combine
The order matters, because temperature changes the probabilities that top-p measures. In Hugging Face Transformers, when sampling is enabled, the generation code applies temperature first, then top-k, then top-p (and min-p if set).
Click to expand and zoom
Combining top-k and top-p is common. Top-k sets a hard upper bound on candidates and top-p trims further when the model is confident.
A from-scratch implementation
This NumPy function implements the pipeline above. It runs as-is and produces the numbers in the tables.
import numpy as np
def softmax(z):
z = z - z.max() # subtract the max for numerical stability
e = np.exp(z)
return e / e.sum()
def sample_next(logits, temperature=1.0, top_k=0, top_p=1.0, rng=None):
rng = rng or np.random.default_rng()
logits = np.asarray(logits, dtype=np.float64)
if temperature <= 0: # treat 0 as greedy decoding
return int(np.argmax(logits))
z = logits / temperature # 1. temperature
if top_k > 0: # 2. top-k: keep the k largest logits
kth = np.sort(z)[-min(top_k, z.size)]
z = np.where(z < kth, -np.inf, z)
p = softmax(z) # 3. softmax
if top_p < 1.0: # 4. top-p: smallest set with mass >= top_p
order = np.argsort(p)[::-1]
cum = np.cumsum(p[order])
keep = order[: np.searchsorted(cum, top_p) + 1]
mask = np.zeros_like(p, dtype=bool)
mask[keep] = True
p = np.where(mask, p, 0.0)
p = p / p.sum() # renormalise
return int(rng.choice(p.size, p=p)) # 5. sample
logits = np.array([4.0, 3.0, 2.0, 1.0, 0.0])
for t in (0.5, 1.0, 2.0):
print(f"t={t}:", np.round(softmax(logits / t), 3))
rng = np.random.default_rng(0)
print(sample_next(logits, temperature=0.7, top_k=3, top_p=0.9, rng=rng))
Using it with Hugging Face Transformers
In practice you pass the parameters to generate() rather than writing the loop yourself. Sampling must be switched on with do_sample=True; otherwise Transformers uses greedy decoding and ignores temperature, top-k and top-p. If a model's generation_config.json does not set them, the defaults are temperature=1.0, top_k=50 and top_p=1.0.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "Qwen/Qwen2.5-0.5B-Instruct" # small, ungated model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
def generate(prompt, temperature=0.7, top_p=0.9, top_k=50, max_new_tokens=100):
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
**inputs,
do_sample=True,
temperature=temperature,
top_k=top_k,
top_p=top_p,
max_new_tokens=max_new_tokens,
)
return tokenizer.decode(output[0], skip_special_tokens=True)
print(generate("Write a short haiku about sunrise."))
This needs transformers, torch and accelerate installed (device_map="auto" relies on Accelerate).
Practical tuning guidance
There are no universally correct values; the right settings depend on the model, the task and how much variation you can tolerate. Some guidance that follows directly from the mechanics:
- Deterministic or factual tasks (extraction, classification, code): use greedy decoding or a low temperature. Low temperature does not make a model more accurate; it makes it pick its most likely answer more consistently.
- Open-ended writing and brainstorming: start at temperature 1.0 with a top-p around 0.9, then adjust.
- Change one knob at a time. Temperature and top-p both control diversity, so moving both at once makes it hard to tell which change helped.
- Min-p is an alternative. Transformers also supports
min_p, which drops tokens whose probability falls below a fraction of the top token's probability. Its documentation suggests values of 0.01 to 0.2, which it describes as comparably selective to top-p values of 0.99 to 0.8. - Beam search is a different strategy. In Transformers, beam search keeps several candidate sequences and picks the one with the highest overall probability. It suits input-grounded tasks such as captioning or speech recognition more than creative generation.
FAQ
Does temperature affect the softmax denominator? Yes. It rescales every logit, so the sum of exponentials changes and every probability shifts.
Can I use top-k and top-p together? Yes. In Transformers both are applied when sampling, top-k first and then top-p.
What happens if p = 1.0 or k equals the vocabulary size? That filter becomes a no-op, and only temperature shapes the distribution.
Is there a best temperature? No. It depends on the task, the model and how much variation you want. Measure on your own prompts.
Sources
Responses (0)
Sign in to join the conversation and share your thoughts.

Prompt & Circuit Editorial
Contributing Editor
Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.
Continue Reading

Demystifying LangChain Models: From LLMs to Embeddings in Plain English
A plain-English guide to LangChain's model layer: chat models, legacy text-completion LLMs, embedding models, invoke(), key parameters, and how embeddings power semantic search, with code for the current LangChain 1.x APIs.

Context Engineering: The Hidden Moat Behind AI Product Success
Models are converging; what you put in their context window is not. Five disciplines of context engineering, grounded in published guidance from Anthropic, Manus and the long-context research literature.

From "Just a Chatbot" to a Full AI System: The Layers Explained
Modern AI products are not a single model behind a text box. A layer-by-layer guide to the stack: LLMs, retrieval-augmented generation, tools and agents, the Model Context Protocol, and the runtime that keeps it all reliable.