Has Open-source LLM Inference Peaked?
Erik Bernhardsson says open-source LLM inference has not peaked. Both open-source inference and frontier providers will grow 5x, 10x, or 20x over the next 12 months and beyond, and both will grow by orders of magnitude over the next few years. Which side ends up larger is hard to call.
More resources should have gone to language-model inference earlier. The work sat on the sidelines because the team did not realize how big open-source inference would be and expected a race to the bottom in a commodity space. Staffing then jumped from one or two people to 30 people 12 months later, and Modal is now one of the top three providers.
He calls the peak argument a fixed-pie view: frontier providers already sell very cheap products, US enterprises do not want Chinese models, frontier demand comes back, and open-source inference shrinks or plateaus.
From the transcript
Again, I think it's about building products that you anticipate will be useful in the future. And from your own experience are things that you always wanted to see. I think you have to kind of trust your gut in that. You said that about resource allocation and that being very, very important as you think about this breadth of products. Where did you allocate resources where with the benefit of hindsight you wish you hadn't? And how do you reflect on that? I think we should have allocated more resources towards LLM inference like earlier.
I think we're a little bit on the sidelines for a while. We didn't realize how big open source inference would be. And we thought it would be kind of a race to the bottom, just a commodity space. And then DeepSea came out and we're like, fuck, like we got to move very fast. And so we staffed up a team immediately, you know, went from like, you know, one or two people working on this to like 30 people 12 months later.