Why Does Google Build Separate Inference Clusters?

Amin Vahdat says serving clusters have to be built because inference demand is too large to cover with older training clusters that are no longer fully used for training. Sonya Huang's picture, that the biggest, newest clusters train the latest frontier model and the rest is recycled into inference, is spot on but not the whole plan.

In a particular year, training might be centralized into a small number of large sites, with network distance kept small, so the sites sit on the same continent or even the same portion of a continent. Other continents can then be left without enough serving capacity, so specialized inference is built across the world. Training clusters will probably be used for serving in some number of years, and serving clusters are stood up at the same time.

Serving clusters are not necessarily cheaper per megawatt. Serving has more need to co-locate compute, networking, and storage, so the hardware cannot specialize as freely. Training can run as a dense, uniform deployment. Serving has to mix storage, compute, and accelerators, and it should not pile too much into one place, since workloads are served from all over the planet. Model endpoints and variations of models are spread across the world for locality. Inference sites end up smaller and probably less vertically integrated.