Top-P Sampling (Nucleus Sampling)

Definition

Top-p sampling, formally known as **nucleus sampling**, is a token selection strategy used during AI language model inference to control the diversity and quality of generated text. It was introduced as an improvement over both greedy decoding (always pick the top token) and top-k sampling (always sample from the k most likely tokens). **How it works:** At each generation step, an LLM produces a probability distribution over its entire vocabulary. Rather than sampling from all possible tokens or a fixed-size top-k subset, top-p sampling: 1. Sorts all tokens by probability in descending order. 2. Walks down the sorted list, accumulating probability mass until the cumulative total reaches or exceeds the threshold **p** (a value between 0 and 1). 3. Discards all tokens outside this 'nucleus.' 4. Samples the next token from only the tokens within the nucleus. For example, with **p = 0.9**, the model identifies the smallest set of tokens whose combined probability is at least 90%, then samples exclusively from that set. **Why this matters:** The size of the nucleus adapts dynamically to the model's confidence: - When the model is **highly confident** (one token has very high probability), the nucleus is small — perhaps just 1–3 tokens. Output is focused. - When the model is **uncertain** (probability spread across many tokens), the nucleus is larger, allowing more diverse output. This adaptivity is top-p's key advantage over top-k, where the nucleus size is always fixed regardless of the probability landscape. **Common values:** - **p = 1.0**: No filtering — sample from the full distribution (equivalent to pure temperature sampling). - **p = 0.9**: A common production default that balances diversity with quality. - **p = 0.1–0.3**: Very restrictive nucleus; highly focused, near-deterministic output. **Used with temperature:** Top-p and temperature are typically applied together. Temperature reshapes the probability distribution first; top-p then determines the candidate pool to sample from. Both parameters interact to define the final output behavior. **Relevance to video AI:** In platforms like OpenClip, top-p settings in underlying language models affect how the AI analyzes video transcripts, detects viral moments, and generates any text-based output. Conservative top-p values help ensure that clip scoring and analysis remains coherent and grounded in the source transcript rather than wandering into low-probability interpretations.