What Is a World-action Model?

Jason Ma says a world-action model predicts the future video of the world and also turns that prediction into actions a robot can execute. Last month Dyna Robotics released Dyna-2, the first world-action model trained on more than one million hours of data. A world model predicts the future state. A world-action model uses the predicted future to choose the robot's next move, the way a person imagines how the world will change before they act. The model is more dexterous, more robust, and better at situations it has never seen.

The basic inputs are video from the robot's cameras over the past many seconds, the angles of its joints, and a language instruction for the task. Joint angles are what a pure video model does not see. The output has to include an action the hardware can run. A world-action model also outputs a future video, or a representation of that video, and uses it to guide the action. Ma compares that extra step to chain of thought: the model reasons in visual and physical space, then outputs the motion. Other robot models only output the action.