Ronak Malde

Training Data · Bringing Continual Learning to Every Company: Trajectory’s Ronak…

• What Is Recursive Self-improvement?

  Ronak Malde says research is heading toward recursive self-improvement and continual learning, two close ways for an agent to learn from experience. Recursive self-improvement puts a model in a black box and tries to make it the best AI researcher, so the model improves itself.

  That version is, in his words, a scary world, because it leaves open whether the agents will care about human interests. Continual learning still learns from experience, but from working with humans on the tasks they want done, which keeps people in the loop. He does not want agents going rogue, and staying on that alignment path is how he would avoid it.

  ▶ at 23:11 Share Transcript

• What Is GRPO?

  Ronak Malde says GRPO is the main post-training algorithm right now. Picture a player hopping across platforms in a video game and getting a reward at the end. GRPO compares repeated plays of the same task, the same level, and calls those plays a group. That group is often eight plays or 16.

  People, a lot of the time, do not replay the exact same environment. They go through one rollout in the real world. The aim is a group size of one, so the model learns from that single try instead of a group of 16. With 10,000 trajectories from different tasks, the model can learn from that one behavior, and a simulated environment is no longer required.

  Reward is the other half. Labeling all 10,000 tr

  ▶ at 13:44 Share Transcript

• How Should Product Builders Shape Self-improving Products?

  Ronak Malde says a product builder should take traces and experience from the real world and use them to shape and improve the product on its own. Trajectory is starting with the model, and more work remains there. Eventually both the model and the harness self-improve. As products become more native to AI, the harness becomes most of the product for a lot of companies. The right to self-improve the harness and then the product is the right to improve and self-improve the entire company.

  Products and product builders then become almost like a gardener with a plant that grows on its own. The builder sets trellises in the ground to mark where the product should be shaped. A product can go out

  ▶ at 26:14 Share Transcript

• Why Don't Language Models Learn From Experience?

  Ronak Malde says language models do a task and never accumulate experience or learn anything about the real world. Agents still lack the exploring that lets a person build a mental model, and he treats that gap as one of the missing pieces for learning on their own through continual learning.

  Throw a football a hundred times and someone who has never seen one goes from having no idea what will happen to a heuristic for where it lands. Kids do the same with a soccer ball and a football, trying things until a mental model forms, and that exploration is what makes people incredibly intelligent. A Markov decision process in reinforcement learning frames the world as state-action pairs: holding

  ▶ at 20:46 Share Transcript

• How Did Harvey Train a Cheaper Legal Model?

  Ronak Malde says Harvey trained its own model on a legal agent benchmark built from expert data the company already holds, and that model came out six times cheaper than frontier models.

  Harvey had been riding frontier closed-source models and had already built products for law firms. It also holds knowledge and nuance from 30 years in the domain, and it can see what those firms care about. Frontier models left no way to turn that data into an advantage. Training on the benchmark lets Harvey shape its own destiny, with room left to grow, and the result he describes is the power of a research lab. On that benchmark, an open-source baseline moved from around 4% accuracy to around 18%.

  The sa

  ▶ at 7:40 Share Transcript

• What Is Continual Learning?

  Ronak Malde tells Pat Grady and Lauren Reeder that the human brain is the model for real continual learning. People are true continual learners because they learn online and keep updating and iterating. Agents and AI today are only pseudo-continual learning.

  On family road trips, his family went to AAA, picked up maps, and had the whole route highlighted before they left. Changing the route later meant calling AAA or going to a physical location. Agents now are very intelligent and come with mapping algorithms, but you can consult them only offline in a couple of places. Any course change still takes an entire mass batch update. The first goal at Trajectory is to democratize training models

  ▶ at 1:14 Share Transcript