Why Is Live GPU Inference Harder to Interrupt Than a Batch Job?

Evan Conrad says live GPU inference is harder to interrupt than a batch job because live customers expect the service to stay online, while a batch job can stop at any point. A batch workload can be cut with about an hour of warning, or even a minute or two. Live capacity can be taken back a month, two months, or three months out, while the operator adapts, and it cannot be pulled in a minute.

The two are priced differently because the cost of the interruption is different. A live job that goes down means paying a service-level penalty to that customer. The market is a market for breaking leases, not only a sublet of spare capacity. Power markets already split this way, with a day-ahead market and a live market. Compute has capacity that is about to arrive and capacity already in use, and those two have different prices.