Nucleus Sampling

The Curious Case of Neural Text Degeneration

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi · ICLR 2020

Read paper · arXiv:1904.09751

Draw the next vocabulary token from the renormalized distribution over a dynamic nucleus.

What you will implement

A nucleus-probability helper is provided; practice inverse-CDF draws, vocabulary IDs and caller-owned RNG with 8192 actual synthetic samples.

• 93. Top-p: Draw Tokens from the Nucleus

  Draw vocabulary token IDs from nucleus probabilities using an inverse CDF.

  The nucleus probability helper is provided; implement per-row sampling, random-stream ownership and vocabulary indexing.