What Is the Key Question for AI Progress?

Leopold Aschenbrenner says the key question for AI progress in the next few years is how hard it is to unlock the test-time compute overhang. GPT-4 now answers with a few hundred tokens of chain of thought. At a hundred tokens a minute, that is about three minutes of thinking. Millions of tokens on one problem would be four orders of magnitude more test-time compute, about a few months of working time, and a few months of work can do far more than a few minutes.

Models cannot hold that yet. A little iterative debugging on code works, then the model gets stuck and cannot correct its errors. An AlphaGo paper shows train-time compute and test-time compute can trade off, and four orders of magnitude more test-time compute is almost like a three and a half orders of magnitude bigger model. Aschenbrenner says the short-timelines view is that unlocking the overhang might not be that hard, because models only need a few new kinds of tokens: error-correction tokens, as in noticing a mistake and thinking again, and planning tokens, as in making a plan, writing a draft, and critiquing the draft.

He describes two paths to agents. One is scaling until systems pick up more nines of reliability, the path another guest laid out on the show. The other is an unhobbling path: learn a system-two process, spend millions of tokens, and stay cohesive and coherent.