Smash Notes

Back to Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again, with Rich Sutton & Khurram Javed.

What Is Continual Backprop?

Rich Sutton says Continual Backprop is ordinary backprop that also keeps planting new units with fresh random weights. Backprop starts with random weights, and that variety gets used up. Continual Backprop keeps injecting a little randomness so the network can keep learning. He and Khurram Javed published it in Nature a couple of years ago.

A naive update on one sample changes the whole model and hurts older knowledge. Khurram Javed says Cursor avoids that by training on a large batch from many users, which works only when that data exists. Most uses have a single stream, and the naive update then destroys prior knowledge. Catastrophic forgetting is totally curable with the right algorithm.

The first part of the cure is a separate step size on every weight, meta-learned so some weights move fast, some move slow, and most stay tiny. A new example then changes only the right places. The second part proposes new units without following gradients. Gradients move only in a direction already known to help, which stays slow and does not open a path to more complexity. Generate proposes the units, and backprop tests them. Together with per-weight step sizes, the hope for the next couple of years is a new generation of continual deep learning.

Grokipedia·Nature

Transcript

Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again ·

From the transcript

So how did they come up with those abstractions? That's in some sense, a crucial thing that's missing that the later half of Alberta plan answers. Can we talk about the continual deep learning part? Is it an algorithmic gap that exists today, or is it just a practical deployment infrastructure data privacy gap? Because if I wanted to do, call it naive updating of weights based on user interaction, I can do that today, right? And so what, in your opinion, is the biggest thing that we're missing to kind of get to continual deep learning? Yeah, so it's absolutely an algorithmic gap.

You can do the naive thing, but then you'll see all sorts of problems. For example, if you say, I'm going to take one sample and then I'm going to update my whole model with that one sample, you will run into this problem that now all of the previous knowledge in the model, it's impacted negatively. And the way it currently is, get around this is exactly what Cursor does.

See in transcript

Training Data

More from this episode

  1. Are Large Language Models an Example of Rich Sutton's Bitter Lesson?Rich Sutton says large language models are both a positive and a negative example of The Bitter Lesson.
  2. Why Does Rich Sutton Call Synthetic Data a Big Mistake?Rich Sutton says synthetic data is not a general method that uses computation. He calls it a big mistake, and maybe the next big lesson.
  3. Why Does Rich Sutton Say Prior Knowledge and Learning Should Be Friends?Rich Sutton says prior knowledge and learning should be friends. The Bitter Lesson opens by saying there is no need for a conflict between them.
  4. Why Do LLM Assistants Stop Learning After Deployment?Rich Sutton says the popular language-model assistants are not learning from experience. Their weights never change.
  5. Why Don't Animals Learn by Supervised Learning?Rich Sutton says no animal learns by supervised learning, because nobody supplies examples of how its muscles should twitch, and muscle movement is the…
  6. What Is Rich Sutton's Alberta Plan?Rich Sutton says the Alberta Plan, which he laid out in 2022, turns broad ideas about AI into smaller chunks. The 12 steps are those chunks.
  7. What Is Oak Lab's Biggest Ambition?Rich Sutton says Oak Lab's biggest ambition is a mind that stays coherent while it keeps changing.
  8. Why Are Big AI Labs Stuck in a Local Minimum?Khurram Javed says the big labs are stuck in a local minimum. A move toward the new algorithms will almost certainly look worse before it looks better.

People

More answers like this

More with Rich Sutton

More with Sonya Huang

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.