Smash Notes

Back to Muse Wonder: Zuckerberg and Wang's $100B AI, with Mark Zuckerberg & Alexandr Wang.

How Much Does Training a Frontier AI Model Cost?

Mark Zuckerberg puts the figure you would see in public, to operate a gigawatt cluster for a year, on the order of $15 billion. John Collison calls that roughly a $15 billion effort to train a frontier model. Zuckerberg separates that operating cost from construction. Building the capital expenditure is probably closer to $50 or $60 billion, paid up front, and that is one cluster for one part of training.

The final run is not the whole research bill. Each change is tested up a ladder to full scale, and that testing is probably an equal amount again, before anyone counts serving the model. Zuckerberg agrees that tens of billions of dollars is a fair estimate for training a frontier model now.

Transcript

Muse Wonder: Zuckerberg and Wang's $100B AI ·

Cheeky Pint

More from this episode

  1. How Does Mark Zuckerberg Use Muse?Mark Zuckerberg uses Muse to coach his mixed martial arts.
  2. Why Do Winning AI Products Look So Simple?John Collison says winning AI products look simple because the field collapses quickly onto a few designs you could describe on the back of a postcard.
  3. What Is Muse Wonder?Alexandr Wang says Muse Wonder is a new class of model at the frontier, and the lab has been working toward it since the start.
  4. What Is Active Memory in an AI Model?Alexandr Wang says Muse Wonder is state of the art, by Meta's own measurement, on a slate of personal and professional agent tasks.
  5. How Big Should a Pre-training Team Be?Mark Zuckerberg says he does not know how big a pre-training team is, and it does not need to be that big. Other labs are probably quite big.
  6. Why Does Muse Wonder Lead on Legal and Finance?Alexandr Wang says Muse Wonder is the state of the art model on a lot of the finance evaluations and a lot of the legal evaluations, and the domain skill…
  7. Why Doesn't Scaling One Part of an AI Model Work?Alexandr Wang's broad rule for AI is to scale everything, and to scale it together. Scaling one thing in isolation does not work.
  8. How Is Meta Scaling AI Past Five Gigawatts?Mark Zuckerberg says you can measure a lab's compute strength by how big its largest single supercomputer is.
  9. Why Is Meta Spending $100 Billion Before the Model Exists?Alexandr Wang says frontier AI is the opposite of a lean startup. A lean startup gets something working and only then doubles down. You cannot do that here.
  10. Why Is Meta's AI Spend Less Risky Than the Price Tag?Mark Zuckerberg says Meta's huge AI spend is not as risky as the size of the numbers suggests, because Meta has a backstop for how it can use compute.
  11. How Does Muse Tell Mark Zuckerberg From His Daughter?Mark Zuckerberg says Muse voice mode segments the voices so it knows which one is him and which one is not.
  12. Why Are AI Agents Different From Chatbots?Alexandr Wang says an agent is a big step change because the mental model is different from a chatbot that only shares information.
  13. How Does Meta Plan to Pay for Its AI Buildout?John Collison asks how the AI buildout gets paid for.
  14. Why Hasn't One AI Lab Run Away With the Lead?John Collison recalls the 2023 story that one AI lab would get ahead, start recursive self-improvement, and pull away from everyone else.
  15. Why Did Researchers Join Meta Superintelligence Labs?Alexandr Wang says a reason people were excited to join Meta and Meta Superintelligence Labs is that they wanted to work somewhere focused on giving this…
  16. Why Did Mark Zuckerberg Choose Alexandr Wang?Mark Zuckerberg says he and Alexandr Wang had talked about once every few months for years, and he was always impressed by how Wang thought about strategy.
  17. Why Did Mark Zuckerberg Reboot Meta's AI Lab?John Collison names four moves he thinks even an opinionated founder would not make. Mark Zuckerberg treated the recent prior AI effort as not good enough.
  18. Why Did a Meta Lead Reject a Reasoning Model?Alexandr Wang heard a key lead on Meta's existing effort reject a reasoning model because the lead did not think consumers needed one and said a normal model…

People

More answers like this

More with John Collison

More with Mark Zuckerberg

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.