Is Buying GPUs Like an Open-air Drug Market?
Erik Bernhardsson says buying GPUs at NVIDIA's GTC conference in San Jose felt like an open-air drug market, with GPUs instead of drugs. He ran around talking to about 500 cloud vendors, asking who had GPUs, and came out with a lot of deals. Getting that capacity is pretty crazy, so Modal does the deals under the hood and customers do not have to chase them.
Modal works with 25 different neoclouds. He puts Modal's share of the world's compute at 0.1% or whatever, and he does not know the exact number. The long-term goal is to control more of that compute, aggregate capacity and demand, and manage both efficiently. Modal built its own file system, container runtime, and scheduler, and does not use Kubernetes or Docker, so the world's compute can be treated as one pool.
Modal already has many thousand paying customers, and the count keeps growing. Each customer's load varies through the day and can burst, so one customer is unpredictable. Together, demand turns smooth. Customers skip capacity planning and can get 100 GPUs.
From the transcript
Maybe it's certain specialized audio models or whatever. We do a lot of stuff besides LM inference. And so that, to me, creates a lot of opportunity to basically rethink what a cloud looks like in the era of AI and deconstruct the cloud and rebuild it for this new era of software. Earlier you said about building capacity with GPUs. Yeah. What's it like doing that today? Is it as crazy as reports suggest? So it is pretty crazy.
I spent some time in March at GTC, NVIDIA's big conference in San Jose. I would describe that as like an open-air drug market, but GPUs instead of drugs. You're running around, you know, talking to 500 different cloud vendors. You're like, do you have GPUs? Like, please give. And like, you know, in the end, you come out, like you do a lot of deals.