Back to Ryan Greenblatt – What happens once AI can automate AI research?, with Ryan Greenblatt.
How Fast Can an AI Learn a New Codebase?
Ryan Greenblatt says an AI can understand a new codebase much faster than a human, and more shallowly. A model such as Mythos 5 or Fable 5 gets some understanding in significantly less than an hour, sometimes much less. An AI in an hour can match a human with a few weeks, maybe, depending on how complicated the codebase is, and it still will not match someone who has worked on that codebase for two years. Older models, around 3.5 Sonnet or 3.7 Sonnet, might only have matched about a day of human familiarity. The amount of understanding models can match has gone up.
Asked to really understand a codebase and then implement a feature, a current model can spawn many sub-agents, pull context back, and investigate a few things. It is not amazing at that, but it is fast. Implementing a complicated feature in a very large codebase is extremely verifiable, so it is a task models can be trained to do better.
Ryan Greenblatt – What happens once AI can automate AI research? ·
From the transcript
And my sense is that like, that's not true for literally every domain. And my sense is that the AIs will develop increasingly good mechanisms for quickly acquiring understanding and expertise in a given domain. So consider for example, how fast AIs can like understand a new code base. AIs can understand a new code base much faster than humans can, but to a degree that's shallower than humans could currently understand, but is getting better over time, right? So let me spell that argument out a bit more.
So let's say you take, you know, Fable 5 or Mythos 5 or whatever, and you like wanted to make some kind of complicated change to a really massive code base, the model will get some understanding of the code base very fast, like in the course of maybe like, you know, significantly less than an hour, potentially much less than an hour. And then its understanding of the code base will like plateau a little bit where it won't get as deep of an understanding as a human would have gotten over a much longer period.