Smash Notes

Back to Google's AI Infrastructure Chief, Amin Vahdat, on the Physics & Economics of Frontier AI, with Amin Vahdat.

What Is an AI Data Center?

Amin Vahdat says an AI data center shares the basics of a non-AI data center and is not radically different. Both use concrete and an enclosure, electrical yards, mechanical yards, cooling, row after row of power, a huge network to connect large amounts of compute, and a fair amount of storage. The big difference is specialization.

A traditional data center is a 20, 25, or 30 year building investment, planned for how the site evolves over that full span. Hardware inside might last six years, so many generations have to be planned for, including servers, networking, storage, and accelerators such as GPUs and TPUs. An AI data center is oftentimes more purpose-built, with the building co-designed around the hardware inside, and it might skip a lot of storage. A storage rack might draw 10, 20, 30, or 40 kilowatts. Next to a TPU rack or a GPU rack, power easily hits hundreds of kilowatts today and might hit megawatts in the next few years.

Designing a row for 30 storage racks is a very different job from designing a row for one or two AI racks, because each AI rack uses so much more power.

Transcript

Google's AI Infrastructure Chief, Amin Vahdat, on the Physics & Economics of Frontier AI ·

Training Data

More from this episode

  1. What Is Goodput in AI Infrastructure?Amin Vahdat says goodput is the useful work a real workload delivers, not the theoretical peak a chip can claim.
  2. Why Must Google Double Token Capacity Every Six Months?Amin Vahdat says the hardware has to double how many tokens it can generate every six months, and software will deliver as much of that capacity as the chips…
  3. Why Did Google Split Its AI Chips Into 8i and 8t?Amin Vahdat says Google split the line because, by 2026, inference and serving had grown large enough to justify a chip specialized for each job.
  4. Why Co-design an AI Chip With the Software?Amin Vahdat says co-design is how a stack stops leaving performance unused between the chip, the network, and the software.
  5. How Do Google and DeepMind Change a Chip Before Tape-out?Amin Vahdat says Google and DeepMind co-design hardware and models in the same rooms, and they will change a chip already in progress when the gain is large…
  6. What Has Stayed the Same Since Google's First Tensor Processing Unit?Amin Vahdat says the tensor processing unit, at a medium level of detail, has not really changed since the first version.
  7. How Do Long-horizon Agents Change a Data Center?Amin Vahdat says long-horizon agents change the data center because the human pause between prompts disappears, and the CPUs, network, and storage around the…
  8. How Does Google Replace a Failed Tensor Processing Unit Rack?Amin Vahdat says a failed tensor processing unit rack can be replaced with another rack without moving any fiber, because the light is redirected to a spare…
  9. How Does Google Get a Gigawatt of Power?Amin Vahdat says Google's preferred model for a new data center is to stay connected to a utility, even when the site needs a gigawatt of power.
  10. Why Does Google Build Separate Inference Clusters?Amin Vahdat says serving clusters have to be built because inference demand is too large to cover with older training clusters that are no longer fully used…
  11. Are Google's Old AI Chips Still Fully Used?Amin Vahdat says Google's seven and eight year old tensor processing units are still at 100% utilization.
  12. Why Does Google Want Open AI Standards?Amin Vahdat says the build-out is too big and too important to be any one vendor's closed proprietary stack.
  13. Why Does Google Want Data Centers in Orbit?Amin Vahdat says orbital data centers are a moonshot Google is pursuing because energy production is a fundamental constraint, and space offers far more solar…

People

More answers like this

The other side

More with Sonya Huang

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.