The Typicality Bias: Mitigating Mode Collapse via Verbalized Sampling

The research identifies typicality bias—the human tendency to prefer familiar or stereotypical content—as a primary driver of mode collapse in large language models.

This phenomenon occurs when aligned models lose the creative diversity of their base versions, instead repeatedly generating a narrow set of predictable responses.

To resolve this, the authors introduce Verbalized Sampling (VS), a training-free prompting technique that directs models to explicitly describe a distribution of multiple possibilities and their probabilities.

Experiments demonstrate that this method significantly restores generative variety in tasks such as creative writing, social simulations, and data generation. Crucially, this improvement in diversity does not undermine the model's factual accuracy or safety.

The study suggests that while post-training alignment often suppresses variety, the underlying models retain a vast range of behaviors that can be unlocked through principled prompting.