What Is Goodput in AI Infrastructure?

Amin Vahdat says goodput is the useful work a real workload delivers, not the theoretical peak a chip can claim. FLOPS, and other chip-centric metrics, describe a maximum under some conditions. The performance that matters is what the workload actually delivers. A single chip rarely decides it. It depends on how two, four, eight, sixteen, a thousand, or ten thousand chips work together, plus the CPUs that feed them and the network that connects them.

Throughput is the work that is possible. Goodput leaves out the work that does not help finish the answer. Training, serving, and agentic workloads are synchronous. At a thousand, ten thousand, or 100,000 components coordinating at microsecond or millisecond granularity, one failure can stop the whole job. The team has to find which part stopped, restore a checkpoint, and restart. The worst case is starting over, which happens more on inference. Time spent recovering is still work, like redoing steps on paper after a mistake, but it is not goodput.

Accountability is delivered goodput under real failures, not a theoretical benchmark. At 100,000-accelerator scale, something fails multiple times a day, and depending on the configuration, multiple times an hour. Goodput is a Google term, and more of the industry is starting to use it.