Jason Ma

Jason Ma says a world-action model predicts the future video of the world and also turns that prediction into actions a robot can execute. Last month Dyna Robotics released Dyna-2, the first world-action model trained on more than one million hours of data. A world model predicts the future state. A world-action model uses the predicted future to choose the robot's next move, the way a person imagines how the world will change before they act. The model is more dexterous, more robust, and better at situations it has never seen.

The basic inputs are video from the robot's cameras over the past many seconds, the angles of its joints, and a language instruction for the task. Joint angles are w

Jason Ma thinks people, before AlphaGo, treated Go as one of the games that AI would never solve, because the search space is larger than the total number of atoms in the world. Reinforcement learning cracked it anyway, and that is how he got into AI.

He spent time in Japan and China as a kid and grew up playing Go. AlphaGo came out when he was a senior in high school. He had played for many years and was mind blown that a game he found so complicated fell to reinforcement learning. He wanted models, systems, and agents that can self-improve. At Penn as a PhD student he worked on reinforcement learning, co-advised by advisors who were both interested in it. One advisor focused more on robot

Jason Ma says a robot can work alone for an hour when a hierarchical agent harness sits around the world-action model and strings short skills into one workflow. People plan a job in steps, notice one part is finished, and move on while something else is still running. A low-level model that only outputs the next motion would have to learn all of that reasoning at once, which is extremely inefficient.

The laundry task makes the split concrete. The washer is running, so the robot should go do something else until the cycle finishes. When the washer is done, the robot should take the towels from the wash and put them in the dryer immediately, or the wait becomes a bottleneck in the whole work

Jason Ma says Din Tai Fung put Dyna robots in many United States restaurants because the chain decided the robots create enough value to roll out across sites, not only to run one pilot. Plenty of robotics companies announce a deployment or a pilot. The hard part is not placing one robot in one restaurant. The hard part is the customer wanting robots across many sites. To his knowledge, a multi-site rollout has not happened for other companies building similar general purpose robots. Din Tai Fung is the biggest scaled customer. The robots fold thousands of napkins a day at many locations.

Din Tai Fung's quality standard is extremely high. The second fold must leave the seam of the napkin on

Jason Ma says the bottleneck for doing an hour-long full job on any task is data and hardware. A laundry demo takes one hour to finish the whole loop, and the longest duration a robot can do a full job is already in the hour range.

For some tasks, such as setting up a space, a robot might need more than a parallel jaw gripper, something dexterous like a human hand. Training a model on a hand is then bottlenecked by hardware stability. The best dexterous hands today are still not durable enough and overheat pretty quickly. A hardware limit slows everything, because knowing whether progress is real on any task eventually has to happen on the hardware. Without eval that is spontaneous and fast

Jason Ma gave the Dyna robot a torso that can lean because people use the waist and back for ordinary tasks, and a body that only travels up and down cannot match that motion. A couple of years ago he pulled a muscle while exercising. The next few days after a pulled muscle, or after sleeping in the wrong position, movement locks up and the neck gets very stiff. Daily life uses many muscles and degrees of freedom that stay invisible until one of them fails, and then tasks that usually feel natural turn awkward.

Reaching for a bottle works the same way. The arm does not act alone, and holding the upper body still on purpose feels strange. People move the torso all the time without thinking a

Jason Ma describes the robot as a semi-humanoid: the upper body is like a humanoid, and the lower body is wheels. Wheels are the practicality versus performance trade-off, because legs use a lot more energy, leave the robot a lot more unstable, and make safety extremely hard to guarantee.

The upper body has a shoulder, arms, and four degrees of freedom at the waist, enough for the upper-body motion a person can do. For legged robots right now, nothing guarantees the robot stays upright and balanced the entire time. Robot dancing videos show robots that fall over very easily.

He previously built models at DeepMind and at Nvidia and saw the upper bound of a system when the builder does not c

Jason Ma says the robot's proportions are measured against real people because the team pre-trains on human data and treats the match as the transfer path onto the machine. Hardware carries the intelligence, so the body has to serve the model.

A part is bought, rather than built, when its input and output are already clear and a mature solution exists. The black circles on the robot help it detect obstacles and move through surroundings. People have used the technology for decades on the self-driving car, so there is no need to build it from scratch. Some cameras are treated the same way. Actuators are the opposite. The team is working to own them, because design quality decides how well th

Jason Ma says a robot model has to run on the device because the robot must infer fast enough to act. A language model or a coding agent can wait. ChatGPT can take a couple of seconds before it answers. A robot cannot hold a knife and do nothing for three seconds, and gravity does not pause while the robot juggles a ball with a tennis racket.

A call to a remote model has no latency guarantee, and self-driving a car has no guarantee either. The model that gets deployed has to stay a reasonable size so it can be served locally, on-prem. The model trained first can still be large. The best way to make a small model very smart is distillation: train the biggest model, then distill it. Robotics

Jason Ma says his team generates synthetic robot data with frontier models that build synthetic environments and with a world action model that makes its own training videos. Synthetic environments, synthetic data, and simulation data are one of the next frontiers for robotics. Egocentric data and robot data are already scaling, and synthetic data is a new dimension of scaling.

In his PhD he focused on simulation-based training for robotics, and simulation was not ready. Eureka was the first paper to take a large language model, GPT-4, to self-generate environments and reward functions for reinforcement learning in simulation. The demo was a robot spinning pencils. The model lacked good vis

Jason Ma says the Dyna 2 world action model is a diffusion transformer. Building a robotics model comes down to the data and to how much the model can learn from it, and Dyna 2 uses the same transformer style as a language model, sequence to sequence and decoder only, with diffusion on top.

A diffusion model starts from random noise and, through repeated computation, turns the noise into something high dimensional with very fine detail. Image generators need sharp edges, sharp corners, and strong color contrast. Robot actions need the same precision and smoothness, so the field trains robotics foundation models as diffusion transformers. The transformer is what can take in the most data. Di

Around June 2008 when Jason made his first video theme, everyone else was focused on blogging. All the themes looked fairly similar, and none fit his needs. Jason was passionate about video, they were very visual, you could have a lot of beauty and interaction. Nobody was doing it, and it just so happened that he carved out a niche doing what he was passionate about.