How Do Diffusion Models Write Text Faster Than Autoregressive Models?

Stefano Ermon says a diffusion model generated text 10x faster than an autoregressive model with the same number of parameters, because it outputs many tokens at the same time.

With PhD student Yang Song, he started score-based generative models in 2019 by training a neural network to denoise images. A network that can denoise an image understands enough structure to support a generative procedure, and that approach became the technology behind modern diffusion models. Generation starts from pure noise and gradually refines it into a clean picture, rather than painting one pixel at a time from left to right. The best models for image generation and video generation are based on diffusion, music is based on diffusion to some extent, and a lot of protein work is based on diffusion too. His group built ways to produce samples very quickly and to raise quality, then turned to text, code, and other discrete objects, looking for something more parallel than autoregressive models and with built-in error correction. Research at Stanford led to a 2024 paper: a transformer trained as a diffusion model on the same data, with less than a billion parameters, matched an autoregressive model at the GPT-2 scale on perplexity and fit the data just as well.

He started Inception to scale the technology into commercial diffusion-based language models.