Have Physical Efficiency Gains in Chips Stopped?
Naveen Rao says physical efficiency gains in chips have largely stopped. The last few generations of GPUs and other chips show little new efficiency at the physical layer. Moore's law and Dennard scaling made devices smaller, denser, and more efficient, and the innovations have largely stopped.
He splits how a computer is built into three levels. The physical level is the energy required to move a bit between two states, the way a phone's weight changes the energy needed to flip between zero and one. Most efficiency gains over the last several years, and over several generations, have come from the next layer, computing architecture, by moving to smaller data types: 32-bit, then 16-bit, then 8-bit, then 4-bit. Fewer bits means less data moved and less computation. The level above that is still not well exploited. Mosaic started work on enforcing locality and cutting data movement at the model level.
Five or six years ago the model was not the primary driver of compute. Compute was a general-purpose tool for websites and databases. A computer treated as a physical device built to deliver intelligence can be co-designed from the model to the architecture to the physical layer. The remaining advantage sits in the top layers, not only in the physical layer.
From the transcript
more for that electricity. When we go slightly further down from energy, which is the core foundational power, to the compute that uses it, what do we know now about compute that you wish you had known two years ago? I think one thing I wish I'd known is the fact that we don't really see physical efficiency gains anymore. So you can separate out the way we build a computer and let's call it, let's call it three levels.
So the first level is the physical. So if I can send a bit from this point to that point and do it cheaper, you know, let's say I represent zero as being here and one is being here. My phone has a certain weight. So there's energy required to move between these two states. If I make a lighter phone, it changes the amount of energy required.