Smash Notes

Back to Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again, with Rich Sutton & Khurram Javed.

Why Do LLM Assistants Stop Learning After Deployment?

Rich Sutton says the popular language-model assistants are not learning from experience. Their weights never change. A memory of the user, or extra context, can change the state the model conditions on, and the next word can use that state. The model itself is not learning. The structuring and the new concepts that built the model were weight learning, and that should not happen only once.

He says the field does too much pre-training and post-training before launch, then does not let the model learn while someone is using it. Pre-training and post-training are fine. The disagreement is the stop. Khurram Javed says a few products, including Cursor's tab autocomplete, do update weights from a large batch of many users, which is a poor way to teach one person's own model something specific.

Transcript

Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again ·

From the transcript

Like an intelligent assistant, you know, has to become better over time, has to know what you want. Would your contention be that the current paradigm of the popular LLM-based assistants, would your contention be that these are not experiential learners or continual learners? And if so, what is the fundamental gap? Are you serious? I mean, obviously- They learn memories about me.

They're doing some in-context learning. Their weights never change. And by the way, is a small number of the weights changing sufficient or do you need all the weights to be changing? Well, so think of all the structuring and generation of new concepts that went into creating the large language models. All that is the weight learning.

See in transcript

Training Data

More from this episode

  1. Are Large Language Models an Example of Rich Sutton's Bitter Lesson?Rich Sutton says large language models are both a positive and a negative example of The Bitter Lesson.
  2. Why Does Rich Sutton Call Synthetic Data a Big Mistake?Rich Sutton says synthetic data is not a general method that uses computation. He calls it a big mistake, and maybe the next big lesson.
  3. Why Does Rich Sutton Say Prior Knowledge and Learning Should Be Friends?Rich Sutton says prior knowledge and learning should be friends. The Bitter Lesson opens by saying there is no need for a conflict between them.
  4. Why Don't Animals Learn by Supervised Learning?Rich Sutton says no animal learns by supervised learning, because nobody supplies examples of how its muscles should twitch, and muscle movement is the…
  5. What Is Rich Sutton's Alberta Plan?Rich Sutton says the Alberta Plan, which he laid out in 2022, turns broad ideas about AI into smaller chunks. The 12 steps are those chunks.
  6. What Is Continual Backprop?Rich Sutton says Continual Backprop is ordinary backprop that also keeps planting new units with fresh random weights.
  7. What Is Oak Lab's Biggest Ambition?Rich Sutton says Oak Lab's biggest ambition is a mind that stays coherent while it keeps changing.
  8. Why Are Big AI Labs Stuck in a Local Minimum?Khurram Javed says the big labs are stuck in a local minimum. A move toward the new algorithms will almost certainly look worse before it looks better.

People

More answers like this

The other side

More with Rich Sutton

More with Sonya Huang

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.