Have Physical Efficiency Gains in Chips Stopped?
Naveen Rao says physical efficiency gains in chips have largely stopped. The last few generations of GPUs and other chips show little new efficiency at the physical layer. Moore's law and Dennard scaling made devices smaller, denser, and more efficient, and the innovations have largely stopped.
He splits how a computer is built into three levels. The physical level is the energy required to move a bit between two states, the way a phone's weight changes the energy needed to flip between zero and one. Most efficiency gains over the last several years, and over several generations, have come from the next layer, computing architecture, by moving to smaller data types: 32-bit, then 16-bit, then 8-bit, then 4-bit. Fewer bits means less data moved and less computation. The level above that is still not well exploited. Mosaic started work on enforcing locality and cutting data movement at the model level.
Five or six years ago the model was not the primary driver of compute. Compute was a general-purpose tool for websites and databases. A computer treated as a physical device built to deliver intelligence can be co-designed from the model to the architecture to the physical layer. The remaining advantage sits in the top layers, not only in the physical layer.
