Why Did Modal Rebuild Its Tools for Generative AI?

Erik Bernhardsson says Modal rebuilt its tools for generative AI because Kubernetes and Docker were made for a steady back-end workload. The systems expect an even number of requests and can run at low utilization, since CPUs are normally pretty cheap and cost is not the main worry. Generative AI brings very expensive GPUs, a need to manage capacity and utilization, and very large models that must load over the network into the CPU and then into the GPU. The older generation of tools was not built for that work, and a wrapper on top of them cannot deliver the new software. Modal had to start over a little bit from scratch.

Over the last few years Modal built its own file system, CPU snapshotting, GPU snapshotting, an autoscaler, a scheduler, and capacity management tools. Supplying hundreds, thousands, and tens of thousands of GPUs has always been a challenge, so Modal went multi-region very early. Modal treats capacity management as one big pool over the entire world.

Bugs, outages, and scaling issues have shown up across Modal's history. Recently uptime has basically reached four or five nines, and 100% uptime remains the aim.