Smash Notes

Back to Bringing Continual Learning to Every Company: Trajectory’s Ronak Malde, with Ronak Malde & Pat Grady.

What Is GRPO?

Ronak Malde says GRPO is the main post-training algorithm right now. Picture a player hopping across platforms in a video game and getting a reward at the end. GRPO compares repeated plays of the same task, the same level, and calls those plays a group. That group is often eight plays or 16.

People, a lot of the time, do not replay the exact same environment. They go through one rollout in the real world. The aim is a group size of one, so the model learns from that single try instead of a group of 16. With 10,000 trajectories from different tasks, the model can learn from that one behavior, and a simulated environment is no longer required.

Reward is the other half. Labeling all 10,000 traces is a lot of work. Select 100 traces that represent the larger set, have the user say what good looks like for those 100, and train a meta-grader that carries the judgment out to the full 10,000.

Transcript

Bringing Continual Learning to Every Company: Trajectory’s Ronak Malde ·

Training Data

More from this episode

  1. What Is Continual Learning?Ronak Malde tells Pat Grady and Lauren Reeder that the human brain is the model for real continual learning.
  2. How Did Harvey Train a Cheaper Legal Model?Ronak Malde says Harvey trained its own model on a legal agent benchmark built from expert data the company already holds, and that model came out six times…
  3. Why Don't Language Models Learn From Experience?Ronak Malde says language models do a task and never accumulate experience or learn anything about the real world.
  4. What Is Recursive Self-improvement?Ronak Malde says research is heading toward recursive self-improvement and continual learning, two close ways for an agent to learn from experience.
  5. How Should Product Builders Shape Self-improving Products?Ronak Malde says a product builder should take traces and experience from the real world and use them to shape and improve the product on its own.

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.