Is Reinforcement Learning Less Efficient Per Dollar?
Naveen Rao says reinforcement learning is more inefficient. Per unit of compute or per dollar, the model improves less, even though the training still keeps moving forward. He does not know that the shift toward it is more efficient. The aim is making models better at the tasks that matter.
For the next several years, he thinks the work is probably very important. Specialization will start in the middle layers and maybe even in the pre-training layer. He does not think the model companies are thinking very long term about performance, such as a massive breakthrough in how an AI system works that would give an advantage. In his view the big labs are not doing it as much.
One month late on a model is a disaster for OpenAI, Anthropic, Grok, or Google. The race is really fast, so they cannot afford to take the big risks just yet. XAI reached the frontier pretty fast with enough resources by taking the existing paradigm, hiring people who know how to build the pieces, and executing very fast. There was not a ton of very deep innovation in that path. Putting known pieces together quickly is different from coming up with a new paradigm.
From the transcript
Can I ask you, we've seen, you mentioned that kind of pre-training RL and then obviously a real shift has moved to post-training. With that shift of emphasis from pre-training to post-training, do you think that is an enduring permanent shift in the efficiency of model improvements and how we get there? Or will we move back to RL being the primary or pre being the primary? How does that look in terms of importance over time? Well, I don't know that it's more efficient.
Actually, I think RL is more inefficient. If you look at it in terms of how much better the model gets per unit compute or per unit dollar, it's actually more inefficient. But it does keep moving forward, right? I think the idea is that we want to make the models better at the tasks that are important. And so I think it's probably for the next several years, yes, very important.