The Diffusion Duality

Subham Sekhar Sahoo¹, Justin Deschenaux², Aaron Gokaslan¹, Guanghan Wang¹, Justin Chiu³, Volodymyr Kuleshov¹

¹Cornell Tech, NY ²EPFL, Lausanne ³Cohere, NY
ICML 2025

arXiv Code

Colab

🤗

HuggingFace

We unlock few-step generation in discrete diffusion language models via the underlying Gaussian diffusion.

An illustration of uniform state discrete diffusion (top) and the underlying Gaussian diffusion (bottom). While both are separate Markov processes, applying \(\texttt{arg max}\) maps Gaussian latents \(\mathbf{w}_t \in \mathbb{R}^n\) to discrete latents \(\mathbf{z}_t \in \mathcal{V}\), transforming their marginals from \(\tilde{q}_t(.|\mathbf{x}; \tilde{\alpha}_t)\) to \(q_t(.|\mathbf{x}; \mathcal{T}(\tilde{\alpha}_t))\) and adjusting diffusion parameters from \(\tilde{\alpha}_t\) to \(\alpha_t = \mathcal{T}(\tilde{\alpha}_t)\) .

Key Innovations

We show that uniform-state discrete diffusion emerges from Gaussian diffusion, enabling the transfer of techniques from continuous to discrete domains.
Building on this insight, we propose the DUO framework, which improves training through a low-variance curriculum.
We further introduce Discrete Consistency Distillation, adapting consistency distillation to the discrete setting and accelerating DUO sampling by two orders of magnitude.

Introduction

An eternal theme in mathematics is that discreteness emerges from underlying continuity. From quantum mechanics, where the quantized energy states of electrons arise as solutions to continuous wave equations, to the binary logic of digital circuits, fundamentally driven by smooth analog currents, discreteness has repeatedly and naturally emerged from an underlying continuum. Our work continues this tradition by demonstrating that a discrete diffusion process is, in fact, an emergent phenomenon of an underlying continuous Gaussian diffusion process. This perspective enables the design of faster training and sampling algorithms for discrete diffusion models.