Why Does More Compute Keep Improving Word Prediction?

Emmanuel Ameisen says pouring more compute and more data into the task of predicting the next word makes the model's skill at predicting the next word keep improving in a predictable way. The trend just keeps going and keeps going and keeps going as an empirical law.

A lot of the early team at Anthropic made the observation. People speculated as to why at the time, and Emmanuel Ameisen says the result is still mostly an empirical finding. The finding means you can just keep pouring more compute in and you keep improving on the metric. A field where more compute keeps improving the metric is rare. It is also rare for the metric to then correlate with a bunch of tasks people actually care about.