The Diffusion Duality

Subham Sekhar Sahoo1, Justin Deschenaux2, Aaron Gokaslan1, Guanghan Wang1, Justin Chiu3, Volodymyr Kuleshov1
1Cornell Tech, NY     2EPFL, Lausanne     3Cohere, NY
ICML 2025
arXiv Code Colab Logo Colab

🤗

 HuggingFace

We unlock few-step generation in discrete diffusion language models via the underlying Gaussian diffusion.

MY ALT TEXT

An illustration of Uniform-state discrete diffusion and the underlying Gaussian diffusion.

A descriptive text for the GIF A descriptive text for the GIF A descriptive text for the GIF

Key Innovations

  1. We show that uniform-state discrete diffusion emerges from Gaussian diffusion, enabling the transfer of techniques from continuous to discrete domains.
  2. Building on this insight, we propose the DUO framework, which improves training through a low-variance curriculum.
  3. We further introduce Discrete Consistency Distillation, adapting consistency distillation to the discrete setting and accelerating DUO sampling by two orders of magnitude.

An eternal theme in mathematics is that discreteness emerges from underlying continuity. From quantum mechanics, where the quantized energy states of electrons arise as solutions to continuous wave equations, to the binary logic of digital circuits, fundamentally driven by smooth analog currents, discreteness has repeatedly and naturally emerged from an underlying continuum. Our work continues this tradition by demonstrating that a discrete diffusion process is, in fact, an emergent phenomenon of an underlying continuous Gaussian diffusion process. This perspective enables the design of faster training and sampling algorithms for discrete diffusion models.

BibTeX

@inproceedings{
  sahoo2025the,
  title={The Diffusion Duality},
  author={Subham Sekhar Sahoo and Justin Deschenaux and Aaron Gokaslan and Guanghan Wang and Justin T Chiu and Volodymyr Kuleshov},
  booktitle={Forty-second International Conference on Machine Learning},
  year={2025},
  url={https://openreview.net/forum?id=9P9Y8FOSOk}
}