What Is Test-time Compute?

Sam Altman says test-time compute is how a model spends more work on one problem and returns a better result. A few years ago, more than a dollar of compute on a problem did not help. GPT-4 could grind out 100,000 tokens and still not write software, because there was no way yet to turn extra compute into a better answer.

Chain of thought lets the model think hard. A whole team of agents can work one problem, and that team can run 10,000 times in parallel. Those teams need chain of thought and test-time compute. A swarm made of GPT-3 was not a useful swarm. The compute that improves a single problem keeps climbing, and the gap between ordinary use and the most productive use keeps spreading.