How Do Long-horizon Agents Change a Data Center?

Amin Vahdat says long-horizon agents change the data center because the human pause between prompts disappears, and the CPUs, network, and storage around the accelerators have to grow with them. A person typing a prompt waits seconds, or tens of seconds, to read the reply and think of a follow-up. An agent parses the response and sends the next request in perhaps milliseconds. No human loop slows the rate of requests.

The parsing and reasoning probably run on a CPU. The CPU then has to fetch more context before the next prompt: memory on the same CPU, memory on another CPU, an SSD, or a hard drive somewhere else. Orchestration grows. Demand for accelerated compute goes up, and demand for CPUs, networking, and storage goes up at the same time.

Putting many CPU racks beside tensor processing unit racks or GPU racks makes it harder to specialize the building for the denser accelerator rack, which needs more networking. One option is to keep the accelerators together in one building, put the CPUs in the building next door, and put hard drives in another building. Networking between buildings then gets harder on reliability, cost, and latency. Latency can rise to hundreds of microseconds, and further once traffic queues.