How Did Lasse Espeholt's DeepMind Agent Get So Popular?

Lasse Espeholt built a reinforcement learning agent at DeepMind that utilized GPUs and scaled linearly with compute. It became the most popular reinforcement learning agent at the company.

When he joined the deep learning team at DeepMind, most people were running single-machine CPU reinforcement learning, and the scale was quite small. With colleagues he made an agent that really utilized GPUs. Lasse Espeholt says it scaled linearly with compute. He thinks that came in part because the agent was very efficient, very good at scaling, and very easy to extend and modify. At one point most of the team used the agent or derivatives of it.

After Lasse Espeholt left DeepMind to found the Google Brain Amsterdam office, he scaled the work to racks of tensor processing units. The product was called seed RL. seed RL used tensor processing units for both inference and training while keeping environments flexible, from Go to very expensive physics simulations.