How Do Diffusion Models Learn Structure by Compressing Data?

Stefano Ermon says training a generative model finds structure in data by building a compression scheme. A diffusion model does the same work as an autoregressive model: both try to learn an efficient way to compress the data. The more the data can be compressed, the more patterns have been identified. Predicting the next word is already a way of learning the structure of the data.

In the 2024 paper, parity with autoregressive models was measured by perplexity, a score for how much structure the model has identified. At GPT-2 scale, the diffusion model identified the same amount of structure as an autoregressive model.