Josh Gao
Related people
Episodes
Notes
Training Data · Inside the Robot Data Factory: Mecka's Josh Gao on Sensorizing…
-
Why Is Robot Data Still Far Behind Tesla's Driving Data?
Josh Gao says robot training data is still many orders of magnitude behind the video Tesla collected for driving. Sonya Huang asks how much has been collected, and how much still has to be collected.
The field has reached the millions of hours, up from the Ego4D set of 4,000 hours two years earlier. He calls that jump a huge step up. Tesla spent more than a decade on two degrees of freedom, acceleration and direction, with millions of cars on the road and more than a few million hours of video. A robot hand has dozens of degrees of freedom. Relative to that driving record, the robotics field is still many orders of magnitude from where it has to be.
-
Why Put Sensors on People to Train Robots?
Josh Gao says Mecka puts sensors on people because a robot has to learn how hard to squeeze, and video does not capture that force. Sonya Huang asks why manipulation needs force when self-driving mostly does not. A door handle and an egg are the cases she gives.
A wrist camera can show when a hand touches an object. That view is a rough substitute. It does not measure how many newtons it takes to pull a chair open or twist a bottle cap. Closing the gap means new hardware. Mecka is building gloves people wear so a recording includes how much force a daily task takes.
He calls the broader category sensorizing humans. A phone mounted on someone for the day was only the first pilot, not the wh
-
How Many Hours of Human Data Do Robots Need?
Josh Gao says the robot-training recipe coming together looks like a million hours of human data across egocentric capture and other modalities, tens of thousands of hours of UMI data, and a couple of hours of last-mile teleoperation plus in-domain, in-distribution demonstration.
A year ago, many people argued that only UMI data would work, or that only egocentric data would work. The streams sit in one pot of soup, and every ingredient matters at some point. He does not subscribe to a single data religion, because tactile data has not entered the mix yet. Learning how much force to apply to a door handle means tactile recordings have to enter the pot. Simulation will enter too, and all of
-
Can Useful Robots Already Be Deployed Today?
Josh Gao says useful robots can already be deployed today, and people underestimate how diverse robotics already is. Humanoids are incredibly popular and will be a very popular terminal state of robotics, but many other form factors already exist.
Hotels in China and Southeast Asia already run robots that do services, cleaning, and delivery. After an Uber Eats order gets to the door, a robot takes it from the front desk to the hotel room door. Bots like that are widely deployed across the world today and are just starting to get popularized across the US and North America.
The first bots that have to get deployed are industrial cleaning bots. Work then steps up through early manipulation t
-
What Will the First Robot Tasks Look Like?
Josh Gao says the first robot tasks will be much more constrained and goal-oriented, jobs a team can hill-climb because a standard operating procedure already defines the work. Laundry folding is one example. The effort to solve laundry folding ran for a year and a half or more.
The home needs the most care by far. Benchmarks and evaluations have only started to mature, including a check that a robot does not fall on a baby. Broader robotics still arrives as a slow creep of commercial use cases. A base of data that shows generalization and scaling laws gives people the courage to scale, and in-domain, in-task data in post-training closes the remaining gap. Collecting in one exact room, incl
-
What Does It Take to Run a Robot Data Factory?
Josh Gao says running a robot data factory is more painful than a camera and a signup form, because much of the hardware sent into the world did not exist until Mecka designed and produced it. Tactile gloves are the example. Nothing off the shelf can be worn through a day, report how much force a person applies, and come back as a data set a robotics model can train on.
The factory then runs that hardware through three or four product cycles, decides what is useful, and scales production. Distribution reaches thousands of homes and thousands of businesses. The upload lines have to carry the hundreds of petabytes Mecka is dealing with. After the files arrive, they still have to pass quality
-
What Is Egocentric Data for Training Robots?
Josh Gao says egocentric data is video of humans demonstrating and doing tasks from a first person point of view. The aim is to close a lot of the vision gap between human data and robotics data. Robots experience the world from their own point of view, so the same footage can show them what interacting with the world looks like from first person.
The bet followed a look across the field. A lot of robotics data then was teleoperation, humans controlling robots to record demonstrations, and teleoperation was very hard to scale. What they saw in GPT, together with the principles of scaling up, meant data had to grow much more significantly. An early intuition was that humans and robots share
-
What Does Teaching Robots About Carbon Life Mean?
Josh Gao says teaching robots about carbon life means showing them what organic life is made of, once robots can do the tasks people do. He wrote that line on the Mecka website himself, the night before the first public announcement, or something close to that night.
The picture is from science fiction. When robotics and AI have generalized every task humans can do, the frontier that remains is organic, and it is biological. He points to Asimov and to Ender's Game. At the end of The Hitchhiker's Guide to the Galaxy, the planet is a giant organic computer trying to work out the meaning of life, or the question. He puts the human brain in the same frame. It runs on 20 watts, and people are se