Back to Ryan Greenblatt – What happens once AI can automate AI research?, with Ryan Greenblatt.
How Much Algorithmic Progress Buys Five Years of AI?
Ryan Greenblatt says five years of AI progress would take around eight years of algorithmic progress, very roughly. Automated AI research in 2022, using only the compute labs had then, would have produced Mythos by the end of 2022.
GPT-3 was released in 2020 and trained about six and a half to seven years ago. A first guess put training compute near 1E23, and a lookup put it near 3E23. Mythos is probably a little over three orders of magnitude higher, a thousand-times compute gap. Closing the gap means discovering all the algorithmic progress since GPT-3 and still more, because Mythos used far more compute than labs had in 2022. A model trained now on GPT-3 level compute would match the best model from about three years ago and come out a moderate amount better than GPT-4.
Most of the gains have come from a mix of algorithms and data. Huge further improvements on both can keep training strong models with less compute.
Ryan Greenblatt – What happens once AI can automate AI research? ·
From the transcript
Yeah. Suppose we were back and when GPT-3 is developed. The idea is, basically with the level of compute they've had back in 2022, you could have trained, if we had automated AI R&D back then,
at the end of that year have Mythos. That would be the idea, yes. Mythos took way more compute than they had back then, but even with the level of compute they had back then, not only do they do all the breakthroughs, but they also train Mythos with their level of compute. What would be required is, obviously, discovering all the algorithmic progress since then, discovering even more actually, because you had to make up for the fact that Mythos uses I don't know, what was GPT-3 trained on, like 1E23? We can look it up.