Back to The Bumpy Road To AGI — With Kevin Roose, with Kevin Roose.
What Happens If AI Starts Improving Itself?
Kevin Roose says that if AI starts improving itself, new models do not have to wait on human researchers, and he thinks the labs should stop before that loop closes. He believes recursive self-improvement is possible, the labs are racing to do it, and it is dangerous.
Alex Kantrowitz cites an Anthropic figure that Claude is now the lead on 26 percent of Anthropic's AI research and development projects, not a team member, a pair programmer, or a coding assistant. OpenAI has put out versions of the same kind of disclosure. Roose sees no reason the share would not rise to 50 percent, 75 percent, or someday 100 percent. OpenAI's goal is to build an automated AI researcher. If the labs close the loop and take the humans all the way out, new models might be trained every two weeks or every couple of hours. He does not know the exact speed, or whether every human can be removed, but that is what the labs are trying to do.
The story that the labs are losing control of their models looks at the race the wrong way. They are giving control to their models. They want to automate their researchers out of jobs and speed up development. Roose calls losing control too passive a description of that choice.
The Bumpy Road To AGI — With Kevin Roose ·
From the transcript
I believe in recursive self-improvement. I believe that it is possible. I believe that the labs are racing to do it. And I think it's dangerous. I think they should stop.
But there was a statistic that came out the other day that I would love to know what you think of this. Anthropic is trying to give, and OpenAI has done versions of this too. They're trying to give people a sense of how much of their own research is being automated by their own tools right now. According to Anthropic, Claude is now the lead on 26 percent of their AI research and development projects.
Not a team member or a pair programmer or a coding assistant. It is leading the project.
I don't see any reason why that number couldn't go up to 50 or 75% or maybe someday 100%. I mean, OpenAI has also said like their goal is to build an automated AI researcher. If they do that, if they can kind of close the loop and take the humans all the way out, then we don't have to wait six months between new models. Then new models might start being trained and developed like at machine speed.