Smash Notes

Back to 20VC: Who Wins the Inference Market | Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute | The Real Limits of Post-Training and RL with Erik Bernhardsson, Modal CEO, with Erik Bernhardsson.

Will GPU Prices Come Back Down?

Erik Bernhardsson says GPU prices will come back down, though he cannot say when. Hardware prices are like gravity and fall in the very long run. A supply shortage has pushed prices up, so the same number of GPUs costs more, and paying more than other buyers is what gets a large allocation. NVIDIA is shipping an enormous number of GPUs. He does not know how many hundred thousand Blackwells are being installed, and he does not call the market efficient. Prices may keep rising for a year, two years, or five years, until supply catches demand.

Modal buys on a three-to-four-month lead time. He draws a straight line or an exponential line through demand from a diversified set of customers. Growth has run about 30 to 40 percent month to month for the last 12 months, so three to four months out is about two to three times current demand, and Modal buys that capacity. He says Modal is more long than short GPUs, because customers pay more when the underlying price rises.

He does not think data centers can be overbuilt in the next five years. Interest rates are rising and data centers cannot be built fast enough, which he treats as a check on overbuild, while demand is still too large for a glut in that window.

Transcript

20VC: Who Wins the Inference Market | Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute | The Real Limits of Post-Training and RL with Erik Bernhardsson, Modal CEO ·

The Twenty Minute VC: Venture Capital | Startup Funding | The Pitch

More from this episode

  1. Why Did Erik Bernhardsson Wait Until His Mid-30s to Start Modal?Erik Bernhardsson started Modal, his first company, at 36 or 37 because he never had quite enough confidence in himself, and a little of the money was missing…
  2. Can You Raise a Venture Round Without a Pitch Deck?Erik Bernhardsson says he has never made a pitch deck, and the rounds still closed. He kind of hates fundraising.
  3. How Much Does Modal Burn Each Month?Erik Bernhardsson says Modal burned only a few million a month up until last year, and monthly burn is now a lot.
  4. Who Will Win the AI Inference Market?Erik Bernhardsson says the AI inference market will be enormous, like a new cloud, and inference alone will not be enough to win it.
  5. Do Companies Still Have Software Margins?Erik Bernhardsson says almost no company today has software margins anymore. The 90% margins plus that Silicon Valley showed 10 years ago are basically gone.
  6. Is Buying GPUs Like an Open-air Drug Market?Erik Bernhardsson says buying GPUs at NVIDIA's GTC conference in San Jose felt like an open-air drug market, with GPUs instead of drugs.
  7. Is CUDA Still a Moat for NVIDIA?Erik Bernhardsson says CUDA is a little overstated as NVIDIA's moat. He does not think CUDA itself is the moat. He calls it an annoying programming model.
  8. Is Nebius a Better Business Than CoreWeave?Erik Bernhardsson says he does not know whether Nebius is a better business than CoreWeave. Both, in his view, have executed extremely well.
  9. How Did Harry Stebbings Thank New Followers?Harry Stebbings sent a personal thank-you to every follower who followed him before 50,000. He typed each note himself. Every night from 1 till 2.30 a.m.
  10. Has Open-source LLM Inference Peaked?Erik Bernhardsson says open-source LLM inference has not peaked.
  11. Are Software Engineers Going to Be Fine?Erik Bernhardsson says software engineers are going to be fine for the next 10 to 20 years.
  12. How Much Do AI Labs Pay New Graduates?Erik Bernhardsson has lost new graduates to some of the big labs for 20-plus million.
  13. What Does Erik Bernhardsson Think Makes a Good Dad?Erik Bernhardsson treats being a good dad as setting his two kids up for success in life, and he always thinks about the right amount to push them.

People

More answers like this

More with Harry Stebbings

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.