What Is Continual Backprop?
Rich Sutton says Continual Backprop is ordinary backprop that also keeps planting new units with fresh random weights. Backprop starts with random weights, and that variety gets used up. Continual Backprop keeps injecting a little randomness so the network can keep learning. He and Khurram Javed published it in Nature a couple of years ago.
A naive update on one sample changes the whole model and hurts older knowledge. Khurram Javed says Cursor avoids that by training on a large batch from many users, which works only when that data exists. Most uses have a single stream, and the naive update then destroys prior knowledge. Catastrophic forgetting is totally curable with the right algorithm.
The first part of the cure is a separate step size on every weight, meta-learned so some weights move fast, some move slow, and most stay tiny. A new example then changes only the right places. The second part proposes new units without following gradients. Gradients move only in a direction already known to help, which stays slow and does not open a path to more complexity. Generate proposes the units, and backprop tests them. Together with per-weight step sizes, the hope for the next couple of years is a new generation of continual deep learning.
Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again ·
From the transcript
So how did they come up with those abstractions? That's in some sense, a crucial thing that's missing that the later half of Alberta plan answers. Can we talk about the continual deep learning part? Is it an algorithmic gap that exists today, or is it just a practical deployment infrastructure data privacy gap? Because if I wanted to do, call it naive updating of weights based on user interaction, I can do that today, right? And so what, in your opinion, is the biggest thing that we're missing to kind of get to continual deep learning? Yeah, so it's absolutely an algorithmic gap.
You can do the naive thing, but then you'll see all sorts of problems. For example, if you say, I'm going to take one sample and then I'm going to update my whole model with that one sample, you will run into this problem that now all of the previous knowledge in the model, it's impacted negatively. And the way it currently is, get around this is exactly what Cursor does.