How Does a Diffusion Language Model Trade Compute for Quality?

Stefano Ermon says a diffusion-based model gives Inception many ways to trade compute for quality at inference time. Inception is about two years old, with around 50 people, and still spends much of its time on how to train the models and how to accelerate inference.

An autoregressive model generates one token at a time, with little room to change the pattern. Image and video diffusion already use distillation and differential-equation solvers to sample very quickly. Research covers training, inference, and engineering: data mixes, evals, RL, and post-training infrastructure. Recipes for a diffusion language model still have to be built. The model stays transformer-based, so existing architecture work can stay. Attention stays, along with public data sets, evals, and benchmarks. Inception stays scrappy, reuses what already exists, and focuses where impact and differentiation are highest. Right now the bet is speed. Nobody knows whether a diffusion-based language model will be significantly more intelligent than an autoregressive one.

Ermon doubts the best way of building powerful AI systems has been discovered. Other ways of building the models remain open, and efficiency will matter. Nobody really knows how the AI factory will work, which is why alternative ways of creating intelligence are worth chasing.