How Does Model Architecture Change Chip Design?

Walter Goodwin says chip design has to follow the model architectures already on the way, and a chip with a new limit can pull later architectures toward it. Models today are trained on high-bandwidth-memory GPUs and XPUs, so a useful accelerator still has to run today's autoregressive mixture-of-experts transformers faster. Fractile's planning bet is memory bandwidth, about 25 times more per chip than an HBM-based chip.

That extra bandwidth changes which designs are practical to serve. A sparser mixture of experts saves flops for the same intelligence, from about one expert in 16 toward one in 128 or one in 256, but those models are often bandwidth-prohibitive on today's HBM chips and get served at low utilization. Some attention designs trade the other way, using more flops and less bandwidth. Compute flops have scaled about a million-fold in 20 years, while memory bandwidth has scaled about 40 times. Raising bandwidth is a way to use fewer flops for the same intelligence, which multiplies how much of a model can run.