Smash Notes

Back to Bringing Continual Learning to Every Company: Trajectory’s Ronak Malde, with Ronak Malde & Pat Grady.

How Did Harvey Train a Cheaper Legal Model?

Ronak Malde says Harvey trained its own model on a legal agent benchmark built from expert data the company already holds, and that model came out six times cheaper than frontier models.

Harvey had been riding frontier closed-source models and had already built products for law firms. It also holds knowledge and nuance from 30 years in the domain, and it can see what those firms care about. Frontier models left no way to turn that data into an advantage. Training on the benchmark lets Harvey shape its own destiny, with room left to grow, and the result he describes is the power of a research lab. On that benchmark, an open-source baseline moved from around 4% accuracy to around 18%.

The same models are six times cheaper than the frontier, so Harvey can spend many more tokens to finish a task.

Grokipedia·Trajectory

Transcript

Bringing Continual Learning to Every Company: Trajectory’s Ronak Malde ·

Training Data

More from this episode

  1. What Is Continual Learning?Ronak Malde tells Pat Grady and Lauren Reeder that the human brain is the model for real continual learning.
  2. What Is GRPO?Ronak Malde says GRPO is the main post-training algorithm right now. Picture a player hopping across platforms in a video game and getting a reward at the end.
  3. Why Don't Language Models Learn From Experience?Ronak Malde says language models do a task and never accumulate experience or learn anything about the real world.
  4. What Is Recursive Self-improvement?Ronak Malde says research is heading toward recursive self-improvement and continual learning, two close ways for an agent to learn from experience.
  5. How Should Product Builders Shape Self-improving Products?Ronak Malde says a product builder should take traces and experience from the real world and use them to shape and improve the product on its own.

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.