Where Did the Original AI Scaling Laws Come From?
Dario Amodei says the original scaling laws grew out of brain science. As monkeys evolved into humans, larger brains tracked new abilities, so the number of neurons is tied to what a mind can do and scale is important.
He formulated the scaling laws while looking at the evolution of animals and the size of animal brains. In human evolutionary history, primate brains grew very rapidly at about the time humans developed language and could use opposable thumbs. He takes the pattern as a neuroscientific inspiration: brain size and the number of neurons are intimately tied to capabilities. The second influence is mechanistic interpretability, which means doing neuroscience for artificial neural networks. Phenomena in artificial neural networks were easier to see than in animal brains, because animal work means cutting up the brain and the experiments are messy and difficult to replicate. Looking inside artificial neural networks yields much more, and some of what was learned there has helped guide discoveries in neuroscience.
Brain evolution supplied a reason to treat scale as important, and work on artificial neural networks now helps guide discoveries in real brains.
Dario & Daniela Amodei: "This Is Gonna Determine the Fate of Nations and the Future of Humanity” ·
From the transcript
Dario, your studies, your scientific studies as a researcher often involved probes into human neurons. To what extent when you're doing artificial neural networks, do you see that understanding of the brain and does that affect how we're going to build AI? Yeah, I think for me it's affected it in two ways. One is when I was formulating the original scaling laws, like just looking at the evolution of animals and the size of animal brains as monkeys evolved into humans.
There was a period in human evolutionary history where the brains of primates grew very rapidly. And that was about the time that humans developed language and were able to use opposable thumbs. And so the idea that brain size, number of neurons is somehow intimately tied to capabilities, that scale is important. I feel that that was a neuroscientific inspiration.