What Happens If AI Starts Improving Itself?

Kevin Roose says that if AI starts improving itself, new models do not have to wait on human researchers, and he thinks the labs should stop before that loop closes. He believes recursive self-improvement is possible, the labs are racing to do it, and it is dangerous.

Alex Kantrowitz cites an Anthropic figure that Claude is now the lead on 26 percent of Anthropic's AI research and development projects, not a team member, a pair programmer, or a coding assistant. OpenAI has put out versions of the same kind of disclosure. Roose sees no reason the share would not rise to 50 percent, 75 percent, or someday 100 percent. OpenAI's goal is to build an automated AI researcher. If the labs close the loop and take the humans all the way out, new models might be trained every two weeks or every couple of hours. He does not know the exact speed, or whether every human can be removed, but that is what the labs are trying to do.

The story that the labs are losing control of their models looks at the race the wrong way. They are giving control to their models. They want to automate their researchers out of jobs and speed up development. Roose calls losing control too passive a description of that choice.