Where Did the Original AI Scaling Laws Come From?

Dario Amodei says the original scaling laws grew out of brain science. As monkeys evolved into humans, larger brains tracked new abilities, so the number of neurons is tied to what a mind can do and scale is important.

He formulated the scaling laws while looking at the evolution of animals and the size of animal brains. In human evolutionary history, primate brains grew very rapidly at about the time humans developed language and could use opposable thumbs. He takes the pattern as a neuroscientific inspiration: brain size and the number of neurons are intimately tied to capabilities. The second influence is mechanistic interpretability, which means doing neuroscience for artificial neural networks. Phenomena in artificial neural networks were easier to see than in animal brains, because animal work means cutting up the brain and the experiments are messy and difficult to replicate. Looking inside artificial neural networks yields much more, and some of what was learned there has helped guide discoveries in neuroscience.

Brain evolution supplied a reason to treat scale as important, and work on artificial neural networks now helps guide discoveries in real brains.