Why Does Fast AI Inference Come Down to Memory Cost?

Walter Goodwin says inference at data center scale, for thousands of users, collapses to the cost per gigabyte of the memory in use. The real elevator on capability is a multi-trillion parameter model running comfortably at many thousands of tokens per second, because that speed makes very long-running agents radically faster. A snappier chatbot is only the faster-horse version of fast inference.

Fast inference chips already have super high bandwidth memory, but the capacity is incredibly low. In deployment they are not running long context attention, so the job still flips back to a GPU. The gap is sharpest where the models could get much faster, and the technical capability is still missing. The chips need incredibly high bandwidth to memory, enough to load the weights and the state many thousands of times per second, plus economical memory. His other key focus is a path to aggressively high bandwidth from the world's lower-cost memory, DRAM.