Stefano Ermon

No Priors: Artificial Intelligence | Technology | Startups · Why Diffusion Will Win AI Inference with Inception Co-Founder and…

• Are Inception's Mercury Models Faster Than Haiku?

  Stefano Ermon says Inception's Mercury models are on par with the Haiku models, the Flash models, and OpenAI's mini and nano models on benchmarks, while being significantly faster. Inception is serving those diffusion-based language models in production.

  Getting diffusion to write text took new research. Image diffusion denoises continuous data, and two pixel colors can still be interpolated. Two words do not have a point between them, so the same idea had to be extended to discrete objects. Once that worked, Mercury moved from a research prototype to a model customers use.

  Diffusion language models cannot run on vLLM or SGLang. Inception built its own serving engine so it can handle produ

  ▶ at 11:53 Share Transcript

• Why Didn't Inception Open Source Its Diffusion Models?

  Stefano Ermon says Inception did not open source everything in part to keep the IP a little closer to the company. Diffusion is a different stack, and a lot of it had to be built in-house because there is no mature ecosystem for the serving engine or the kernels.

  Much of the work had to be developed inside the company. Some open source models exist, but they are not particularly good, so the models are a little harder to deploy and customers are not used to trying them. Inception has to build its own stack for SFT, for RLHF, and for RL, and the stack becomes IP to some extent. One reason for keeping it closed was to hold that IP closer and not open it.

  The closed choice has downsides. The

  ▶ at 30:13 Share Transcript

• Are Diffusion Models Easier to Control Than Autoregressive Models?

  Stefano Ermon says diffusion models are typically easier to control than autoregressive models. Generation can be steered while it is still underway, using an external reward function or a set of constraints.

  An autoregressive model has to finish the whole object before anyone can tell whether it meets a constraint, stays aligned, or stays on brand. Generating a molecule when solubility matters means waiting for the full object, then scoring it with a reward function. A diffusion model generates from coarse to fine, so the check can happen progressively. From the very beginning it is possible to ask whether the object is the kind of thing wanted, then steer generation in the direction an ex

  ▶ at 26:01 Share Transcript

• How Do Diffusion Models Learn Structure by Compressing Data?

  Stefano Ermon says training a generative model finds structure in data by building a compression scheme. A diffusion model does the same work as an autoregressive model: both try to learn an efficient way to compress the data. The more the data can be compressed, the more patterns have been identified. Predicting the next word is already a way of learning the structure of the data.

  In the 2024 paper, parity with autoregressive models was measured by perplexity, a score for how much structure the model has identified. At GPT-2 scale, the diffusion model identified the same amount of structure as an autoregressive model.

  ▶ at 22:35 Share Transcript

• Can Diffusion Models Match Cerebras Speed on NVIDIA GPUs?

  Stefano Ermon says a diffusion language model can match the speed of an autoregressive model on custom hardware while running on NVIDIA GPUs. OpenCall, a company building voice agents, is the example he names.

  The voice pipeline uses an ASR model, then a reasoning LLM that makes the tool calls and decides what to say next, then a text-to-speech component. The reasoning LLM is what delivers the highest quality, and speed matters for the experience. OpenCall had been serving those LLMs on Cerebras, using custom chips to reach that speed. A diffusion-based LLM built to run in parallel accelerates at the software level, so the same speed is available on NVIDIA GPUs.

  GPUs are scarce, but there

  ▶ at 17:40 Share Transcript

• How Does a Diffusion Language Model Trade Compute for Quality?

  Stefano Ermon says a diffusion-based model gives Inception many ways to trade compute for quality at inference time. Inception is about two years old, with around 50 people, and still spends much of its time on how to train the models and how to accelerate inference.

  An autoregressive model generates one token at a time, with little room to change the pattern. Image and video diffusion already use distillation and differential-equation solvers to sample very quickly. Research covers training, inference, and engineering: data mixes, evals, RL, and post-training infrastructure. Recipes for a diffusion language model still have to be built. The model stays transformer-based, so existing archit

  ▶ at 13:30 Share Transcript

• How Do Diffusion Models Write Text Faster Than Autoregressive Models?

  Stefano Ermon says a diffusion model generated text 10x faster than an autoregressive model with the same number of parameters, because it outputs many tokens at the same time.

  With PhD student Yang Song, he started score-based generative models in 2019 by training a neural network to denoise images. A network that can denoise an image understands enough structure to support a generative procedure, and that approach became the technology behind modern diffusion models. Generation starts from pure noise and gradually refines it into a clean picture, rather than painting one pixel at a time from left to right. The best models for image generation and video generation are based on diffusion, m

  ▶ at 3:24 Share Transcript