How Do Google and DeepMind Change a Chip Before Tape-out?

Amin Vahdat says Google and DeepMind co-design hardware and models in the same rooms, and they will change a chip already in progress when the gain is large enough. DeepMind has brought model optimizations, including changes to transformers or to a particular kind of math, while a chip was still unfinished. If hardware support would make training or serving significantly faster, engineers and researchers sit together for a few days, a week, or two weeks. Often the chip cannot do exactly what the model team wanted. The model then shifts so most of the gain remains, and the hardware is intercepted. Sometimes tape-out waits a week or two because the benefit is worth the delay.

Several stages run at once: chips in production, chips back from the manufacturer and being debugged, chips about to tape out, chips in design, and chips still in concept. A late change is much harder across company boundaries. Planning still starts two, three, four, or five years ahead. A tweak worth half a percent or 1 percent on a chip about to tape out is usually not worth stopping. A large opportunity is. An idea from two years earlier can already be in production, lifting workloads across Google.