Are Large Language Models an Example of Rich Sutton's Bitter Lesson?
Rich Sutton says large language models are both a positive and a negative example of The Bitter Lesson. They scaled with computation by taking in the internet, and that produced a much more capable system. The internet is finite, the world is massively bigger than everything stored there, and leaning on that stock of human knowledge eventually holds the systems back.
The essay is not a ban on fancy algorithms. He wants algorithms that scale with computation, such as search and learning, rather than methods that scale with human input. He wrote it from decades of watching the field, including the symbolic AI era, get distracted by putting human knowledge in. He had given versions of the talk at least a year before the 2019 essay. It was not a reaction to the language-model moment.
Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again ·
From the transcript
Yeah, you're making me think about X now. I recently made a post where I tried to do the bitter lesson in 26 words. It goes something like, don't be distracted by human knowledge
as AI traditionally has been many times. Instead, focus on learning methods that will scale with computation like search and like learning. So it's really all about focusing on algorithms and improvements. It's not saying you don't need fancy algorithms. You need fancy algorithms, but you want fancy algorithms that will scale with computation.
Rather than scaling with data. Rather than scaling with human input. Yeah, and then the question, if I can anticipate it, yeah, what about large language models? Are they- Consistent or inconsistent with your essay? Yeah, and I've thought about this, and I think there's another ex post about it, but The conclusion is that it's both a positive example and a negative example of the bitter lesson.