What Is Egocentric Data for Training Robots?

Josh Gao says egocentric data is video of humans demonstrating and doing tasks from a first person point of view. The aim is to close a lot of the vision gap between human data and robotics data. Robots experience the world from their own point of view, so the same footage can show them what interacting with the world looks like from first person.

The bet followed a look across the field. A lot of robotics data then was teleoperation, humans controlling robots to record demonstrations, and teleoperation was very hard to scale. What they saw in GPT, together with the principles of scaling up, meant data had to grow much more significantly. An early intuition was that humans and robots share a lot of similarities. Humanoids were just starting to take off, humans and humanoids look the same, and humans could become fleet-style learning like Tesla, with many people contributing to understanding and grounding the physics of the world.