Why Must a Robot Model Run on the Device?

Jason Ma says a robot model has to run on the device because the robot must infer fast enough to act. A language model or a coding agent can wait. ChatGPT can take a couple of seconds before it answers. A robot cannot hold a knife and do nothing for three seconds, and gravity does not pause while the robot juggles a ball with a tennis racket.

A call to a remote model has no latency guarantee, and self-driving a car has no guarantee either. The model that gets deployed has to stay a reasonable size so it can be served locally, on-prem. The model trained first can still be large. The best way to make a small model very smart is distillation: train the biggest model, then distill it. Robotics is getting there, and people are going to converge on that pattern.

Latency makes the whole stack matter. He builds an inference stack in-house for whatever model gets deployed and optimizes inference speed for the kind of model running.