How Much Power Does Anthropic's Mythos Pre-train Use?
Dylan Patel says Anthropic's Mythos pre-train uses under 200 megawatts for about two months, even while Anthropic is nearly five gigawatts by the end of the year, or more.
Leading Chinese labs have 100 to 200 megawatts of compute in total. ByteDance Seed is the outlier with significantly more, and the others are not running a gigawatt or anywhere close. A lab budget has so far been about sixty percent training and forty percent inference. Broken down further, about fifty percent of the compute is research, 10 percent is development, and 40 percent is inference. Research is researchers generating ideas and testing new architectures, data mixes, hyperparameters, and attention techniques. The most Anthropic ever used at one moment was maybe 200 megawatts. They had multiple gigawatts, so most of the compute went to research, not to developing the model. Clusters are hard to coordinate and co-locate, multi-site training is hard, and generating more rollouts during RL does not necessarily make it better. Single-sided RL training used even less than the pre-train. Total RL compute was probably higher, but it ran in sequence. Of about two gigawatts on hand, only 200 megawatts could be leveraged for training.
He expects automated coding, an automated researcher, and continual learning to push a larger share of the compute budget into training the model. Right now the gap matters less, which is why Chinese companies are not that far behind in AI models, at least as the public sees them, relative to how little compute they have.
Dylan Patel – Anthropic & OpenAI will have most of the world’s compute by 2028 ·
From the transcript
The other aspect of this that I would think is noteworthy is Chinese companies today are not that far behind in AI models, at least perceivably by the public, relative to the amount of compute they have, right? The leading Chinese labs have 100%. 200 megawatts total of compute at most. ByteDance seed being the one outlier where they have significantly more than that.
But, you know, Kimmy is not running, you know, a gigawatt or anywhere close to it. Yeah. Whereas Anthropic is, you know, nearly five gigawatts by the end of the year, right? Or more, sorry. And so, you know, the question is sort of, well does it matter and I think right now it doesn't matter that much that this difference in compute because you know when we break down the compute ratio or budget of a lab historically it's been you know let's say six or so far it's been like sixty percent training forty percent inference but that training gets broken down further and it's actually like fifty percent of the compute is research 10 percent of the compute is development and