Back to Why Sam Altman is against a single superintelligence, with Sam Altman.
What Is Test-time Compute?
Sam Altman says test-time compute is how a model spends more work on one problem and returns a better result. A few years ago, more than a dollar of compute on a problem did not help. GPT-4 could grind out 100,000 tokens and still not write software, because there was no way yet to turn extra compute into a better answer.
Chain of thought lets the model think hard. A whole team of agents can work one problem, and that team can run 10,000 times in parallel. Those teams need chain of thought and test-time compute. A swarm made of GPT-3 was not a useful swarm. The compute that improves a single problem keeps climbing, and the gap between ordinary use and the most productive use keeps spreading.
Why Sam Altman is against a single superintelligence ·
From the transcript
One way to look at this problem or this question is how much compute can we productively throw at a single problem?
And a few years ago, we forget how fast this is all gone, but a few years ago it was impossible to throw more than a dollar of compute at a problem and have it help you at all. Why? You could have GPT -4 grind out for you 100 ,000 tokens or something and it was going to answer your question. And it certainly was not going to be able to go productively write you a piece of software.
However, we had not figured out how to use a lot of test time compute to get you a better result. And then we figured out chain of thought and the models could think hard about a problem. And now, and then like people really started using a lot of compute. And now you can throw a whole team at something and you can do that 10 ,000 times in parallel.