Smash Notes

Back to Dylan Patel – Anthropic & OpenAI will have most of the world’s compute by 2028, with Dylan Patel.

How Much Power Does Anthropic's Mythos Pre-train Use?

Dylan Patel says Anthropic's Mythos pre-train uses under 200 megawatts for about two months, even while Anthropic is nearly five gigawatts by the end of the year, or more.

Leading Chinese labs have 100 to 200 megawatts of compute in total. ByteDance Seed is the outlier with significantly more, and the others are not running a gigawatt or anywhere close. A lab budget has so far been about sixty percent training and forty percent inference. Broken down further, about fifty percent of the compute is research, 10 percent is development, and 40 percent is inference. Research is researchers generating ideas and testing new architectures, data mixes, hyperparameters, and attention techniques. The most Anthropic ever used at one moment was maybe 200 megawatts. They had multiple gigawatts, so most of the compute went to research, not to developing the model. Clusters are hard to coordinate and co-locate, multi-site training is hard, and generating more rollouts during RL does not necessarily make it better. Single-sided RL training used even less than the pre-train. Total RL compute was probably higher, but it ran in sequence. Of about two gigawatts on hand, only 200 megawatts could be leveraged for training.

He expects automated coding, an automated researcher, and continual learning to push a larger share of the compute budget into training the model. Right now the gap matters less, which is why Chinese companies are not that far behind in AI models, at least as the public sees them, relative to how little compute they have.

Grokipedia·Anthropic·Dylan Patel

Transcript

Dylan Patel – Anthropic & OpenAI will have most of the world’s compute by 2028 ·

From the transcript

The other aspect of this that I would think is noteworthy is Chinese companies today are not that far behind in AI models, at least perceivably by the public, relative to the amount of compute they have, right? The leading Chinese labs have 100%. 200 megawatts total of compute at most. ByteDance seed being the one outlier where they have significantly more than that.

But, you know, Kimmy is not running, you know, a gigawatt or anywhere close to it. Yeah. Whereas Anthropic is, you know, nearly five gigawatts by the end of the year, right? Or more, sorry. And so, you know, the question is sort of, well does it matter and I think right now it doesn't matter that much that this difference in compute because you know when we break down the compute ratio or budget of a lab historically it's been you know let's say six or so far it's been like sixty percent training forty percent inference but that training gets broken down further and it's actually like fifty percent of the compute is research 10 percent of the compute is development and

See in transcript

Dwarkesh Podcast

More from this episode

  1. How Much Revenue Does Anthropic Make Per Megawatt?Dylan Patel says Anthropic's revenue has gone as high as $50 million per megawatt, while the base cost of compute runs around $10, $13, or $15 million per…
  2. Will Anthropic and OpenAI Control Most AI Compute by 2028?Dylan Patel says Anthropic and OpenAI are on track to control most of the usable compute in the world by the end of 2028, and he sees nothing that stops the…
  3. Why Can't AI Labs Fund Next Year's Buildout?Dylan Patel says the labs can spend hundreds of billions of dollars and will generate hundreds of billions of revenue next year, but capital spending next…
  4. What Must AI Labs Pay Per Megawatt in 2028?Dylan Patel says the labs will keep buying a larger share of the world's compute, but they cannot do it at today's prices or anywhere close.
  5. Who Captures the Value That AI Models Create?Dylan Patel says most of the value these models create does not go to OpenAI or Anthropic. So far it goes mostly to the users.
  6. Will AI Labs Shift More Compute Into Training?Dylan Patel believes the labs will allocate less and less compute to inference over time and ratchet up the share dedicated to training.
  7. How Much AI Compute Will the World Have by 2028?Dylan Patel agrees that adding his yearly figures puts world AI compute over 200 gigawatts by the end of 2028, and he confirms that total is global.
  8. How Much AI Compute Will China Have by 2028?Dylan Patel says China will have about 30 gigawatts of AI compute or less in 2028, and China's share of new data-center compute stays under 10 percent.
  9. Why Can't an AI Release Ban Stop a Singularity?Dylan Patel's main worry is a singularity that a public release ban fails to stop, because outside deployment is what is helping.
  10. How Fast Is Frontier AI Labor Growing?Dwarkesh Patel says the effective AI workforce at frontier labs is growing about 10 times a year, and one lab could hold more labor than the number of people…

People

More answers like this

The other side

More with Dylan Patel

More with Dwarkesh Patel

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.