Why Are Big AI Labs Stuck in a Local Minimum?
Khurram Javed says the big labs are stuck in a local minimum. A move toward the new algorithms will almost certainly look worse before it looks better. A different paradigm will not hit state-of-the-art performance on the first day. The labs are locked into a product, and the current paradigm still lets them keep scaling, so they cannot take a path where the results get worse first.
The new paradigm is a bet. It depends on hard technical problems that people who have worked on them for many years already consider doable, and those problems take a long time to think through. He also says he does not see research groups that even believe a much more efficient system is possible. A group that does not believe it will not work through the technical problems.
Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again ·
From the transcript
can be more efficient than that even now with the right algorithm. If we can be more efficient than that, then why aren't we? It's not like people just want to spend all their money, spend all our money. Sometimes it seems like they want to. Yeah. Doesn't it?
That's how they show they're real men, by using lots of energy. At least when I look at different research groups, I don't even see anyone believing in it's possible. And I think if you don't believe in it, you're just not going to work on the technical problems and work through them. Is it that it's not possible or it's that there's so much waste in the system? Like, which one is it? Is there someone who knows how to do it efficiently? And then there's 10 times the number of people in the same lab doing all these other things.