Who Will Win the AI Inference Market?
Erik Bernhardsson says the AI inference market will be enormous, like a new cloud, and inference alone will not be enough to win it. Selling tokens for dollars is a commodity. There is no switching cost, and customers can move to another provider. That is a race to the bottom.
What lasts, in his view, is a suite. Inference has to come with post-training, sandboxes, reinforcement learning, low-latency inference, and batch inference. Buyers want a set of tools for building AI applications, not one trick. Modal has been building that platform for five years. He does not call the breadth a moat. He calls the infrastructure underneath a very hard product to copy.
From the transcript
Can I ask you, we've spoken about three different providers there. Inference as a market has many different speculators commenting on it. When you look at the next five years, how does the inference market play out? First of all, I think inference is going to be an enormously large market. It's like the new cloud, basically. But I don't think inference in itself is maybe the only thing you need.
One of the things that I feel very strongly about is it has to come with a lot of other products. It has to come with post-training, sandboxes, agentic stuff. All kinds of other types of inference, like low latency inference, batch inference. So building that platform, by the way, we've been working on this for the last five years. I think that to me is not a moat, but it's a very hard product to build, that infrastructure.