Is Reinforcement Learning Less Efficient Per Dollar?
Naveen Rao says reinforcement learning is more inefficient. Per unit of compute or per dollar, the model improves less, even though the training still keeps moving forward. He does not know that the shift toward it is more efficient. The aim is making models better at the tasks that matter.
For the next several years, he thinks the work is probably very important. Specialization will start in the middle layers and maybe even in the pre-training layer. He does not think the model companies are thinking very long term about performance, such as a massive breakthrough in how an AI system works that would give an advantage. In his view the big labs are not doing it as much.
One month late on a model is a disaster for OpenAI, Anthropic, Grok, or Google. The race is really fast, so they cannot afford to take the big risks just yet. XAI reached the frontier pretty fast with enough resources by taking the existing paradigm, hiring people who know how to build the pieces, and executing very fast. There was not a ton of very deep innovation in that path. Putting known pieces together quickly is different from coming up with a new paradigm.
