Amin Vahdat
Related people
Episodes
Notes
Training Data · Google's AI Infrastructure Chief, Amin Vahdat, on the Physics &…
-
How Do Long-horizon Agents Change a Data Center?
Amin Vahdat says long-horizon agents change the data center because the human pause between prompts disappears, and the CPUs, network, and storage around the accelerators have to grow with them. A person typing a prompt waits seconds, or tens of seconds, to read the reply and think of a follow-up. An agent parses the response and sends the next request in perhaps milliseconds. No human loop slows the rate of requests.
The parsing and reasoning probably run on a CPU. The CPU then has to fetch more context before the next prompt: memory on the same CPU, memory on another CPU, an SSD, or a hard drive somewhere else. Orchestration grows. Demand for accelerated compute goes up, and demand for CP
-
What Has Stayed the Same Since Google's First Tensor Processing Unit?
Amin Vahdat says the tensor processing unit, at a medium level of detail, has not really changed since the first version. A CPU instruction set is the comparison: loads, stores, adds, subtracts, and branches stayed in place while the software on top changed completely. The tensor processing unit has the same kind of stable base. The instruction set has been extended, but the primitives remain: numerics specialized for very large matrix-multiply units, a sparse core for vector operations and scatter-gather, and a remote load and store that reads and writes memory across the chip interconnect. About five or six things define it.
The primitives have carried many generations of models and many
-
How Do Google and DeepMind Change a Chip Before Tape-out?
Amin Vahdat says Google and DeepMind co-design hardware and models in the same rooms, and they will change a chip already in progress when the gain is large enough. DeepMind has brought model optimizations, including changes to transformers or to a particular kind of math, while a chip was still unfinished. If hardware support would make training or serving significantly faster, engineers and researchers sit together for a few days, a week, or two weeks. Often the chip cannot do exactly what the model team wanted. The model then shifts so most of the gain remains, and the hardware is intercepted. Sometimes tape-out waits a week or two because the benefit is worth the delay.
Several stages r
-
Why Co-design an AI Chip With the Software?
Amin Vahdat says co-design is how a stack stops leaving performance unused between the chip, the network, and the software. A system built to run on any hardware, any software, and any network can move onto new capacity overnight, with no lock-in. The cost is generality. Each layer can miss the one next to it. A layer might be worth 10 percent, 20 percent, or 2x, and multiplying the chances through the stack is a large end-to-end opportunity in intelligence per watt or goodput per watt, down to power delivery and software.
Full flexibility means the system can run anywhere, anytime. Full flexibility also leaves a significant amount of performance on the table. The case for co-design is that
-
Why Did Google Split Its AI Chips Into 8i and 8t?
Amin Vahdat says Google split the line because, by 2026, inference and serving had grown large enough to justify a chip specialized for each job. About two years earlier the question was whether 2026 should bring one chip that does inference and training together, or two chips, one further specialized for inference and one for training. The analysis produced two chips: 8i for inference and 8t for training.
Specialization is a calculus. The more a chip is aimed at one workload, the less flexible it is, and the faster and more power-efficient it gets. A large fixed cost has to be covered by enough durable demand. A workload that disappears after a month or two is a narrow window, even when it
-
What Is Goodput in AI Infrastructure?
Amin Vahdat says goodput is the useful work a real workload delivers, not the theoretical peak a chip can claim. FLOPS, and other chip-centric metrics, describe a maximum under some conditions. The performance that matters is what the workload actually delivers. A single chip rarely decides it. It depends on how two, four, eight, sixteen, a thousand, or ten thousand chips work together, plus the CPUs that feed them and the network that connects them.
Throughput is the work that is possible. Goodput leaves out the work that does not help finish the answer. Training, serving, and agentic workloads are synchronous. At a thousand, ten thousand, or 100,000 components coordinating at microsecond
-
Why Does Google Want Data Centers in Orbit?
Amin Vahdat says orbital data centers are a moonshot Google is pursuing because energy production is a fundamental constraint, and space offers far more solar power.
From a fundamentals perspective, space has something like 40% more power available because the atmosphere does not attenuate sunlight, about 1.4 times the solar capacity. In a sun-synchronous orbit, solar cells see sunlight 90% to 100% of the time, versus 28%, 30%, or maybe 35% on land. Combine the 1.4 times with 3 to 4 times more hours per day, remove batteries more or less from the equation, and the power is carbon-free with room for an enormous supply.
Cooling in space is harder. Repairs become harder, though they remain po
-
Why Does Google Want Open AI Standards?
Amin Vahdat says the build-out is too big and too important to be any one vendor's closed proprietary stack. Open standards, ideally with open source around them, are the plug-in points to support, and the result has to be one of choice.
The internet protocol won because it was an open standard and a narrow waist in the hourglass. Anything could run on top in software, and anything could run underneath in hardware. Anyone who brought the internet protocol could plug into a router port and become part of the internet. The openness let the internet explode in growth and reach across the world. Something highly specialized can still plug in when there is a better thing for the framework.
Choi
-
Are Google's Old AI Chips Still Fully Used?
Amin Vahdat says Google's seven and eight year old tensor processing units are still at 100% utilization. Older tensor processing units see significant utilization, and Google does replace them.
The depreciation lifetime is approximately six years. Once the machines are fully depreciated, the power efficiency of newer generations makes replacement and an upgrade the sensible move. Google thinks in pods rather than single chips. An 8T pod might be 9,600 chips and 152 racks. The pod comes out and a tensor processing unit 12, 13, or 14 pod goes in. The new pod's footprint might not be a perfect fit for the vacancy, and the retrofit cannot be planned many generations ahead.
Figuring out how to
-
Why Does Google Build Separate Inference Clusters?
Amin Vahdat says serving clusters have to be built because inference demand is too large to cover with older training clusters that are no longer fully used for training. Sonya Huang's picture, that the biggest, newest clusters train the latest frontier model and the rest is recycled into inference, is spot on but not the whole plan.
In a particular year, training might be centralized into a small number of large sites, with network distance kept small, so the sites sit on the same continent or even the same portion of a continent. Other continents can then be left without enough serving capacity, so specialized inference is built across the world. Training clusters will probably be used fo
-
How Does Google Get a Gigawatt of Power?
Amin Vahdat says Google's preferred model for a new data center is to stay connected to a utility, even when the site needs a gigawatt of power. A gigawatt is not a request for tomorrow followed by a question about when billing starts. Google co-plans with the utility and gives many years of notice.
Google covers the cost of transmission lines that must be built or upgraded and of additional utility base stations. The reason is billing: if the utility builds capacity for Google, other people's rates could go up. Planning can still leave a gap. In a made-up example, Google needs a gigawatt in 2028, the utility can supply a gigawatt in 2029, and can supply 700 megawatts in 2028. Covering the
-
How Does Google Replace a Failed Tensor Processing Unit Rack?
Amin Vahdat says a failed tensor processing unit rack can be replaced with another rack without moving any fiber, because the light is redirected to a spare rack and the change can be done in milliseconds.
A tensor processing unit has a torus topology that connects all the units to one another directly, and a spare rack stays available at all times. Optical circuit switching leaves the bits alone in the electrical domain and, for an input port, picks which output port receives the light. The version he started with uses MEMS switches that control mirrors in three dimensions. A box might have 128 ports or 256 ports. Each input fiber maps to an output port, and rotating the mirrors sends the
-
Why Must Google Double Token Capacity Every Six Months?
Amin Vahdat says the hardware has to double how many tokens it can generate every six months, and software will deliver as much of that capacity as the chips, or more.
Doubling the number of flops every six months is one path, not the requirement. Token capacity is a combination of software and hardware. A model optimization or a runtime optimization is a way to deliver the gain, and he expects dozens or hundreds of individual optimizations to land again and again. He does not have the exact breakdown of intelligence per watt across silicon, models, and other software. In his experience, most of the benefits come from model-side improvements, and most of the gains often come from the model
-
What Is an AI Data Center?
Amin Vahdat says an AI data center shares the basics of a non-AI data center and is not radically different. Both use concrete and an enclosure, electrical yards, mechanical yards, cooling, row after row of power, a huge network to connect large amounts of compute, and a fair amount of storage. The big difference is specialization.
A traditional data center is a 20, 25, or 30 year building investment, planned for how the site evolves over that full span. Hardware inside might last six years, so many generations have to be planned for, including servers, networking, storage, and accelerators such as GPUs and TPUs. An AI data center is oftentimes more purpose-built, with the building co-desig