Why Don't Language Models Learn From Experience?

Ronak Malde says language models do a task and never accumulate experience or learn anything about the real world. Agents still lack the exploring that lets a person build a mental model, and he treats that gap as one of the missing pieces for learning on their own through continual learning.

Throw a football a hundred times and someone who has never seen one goes from having no idea what will happen to a heuristic for where it lands. Kids do the same with a soccer ball and a football, trying things until a mental model forms, and that exploration is what makes people incredibly intelligent. A Markov decision process in reinforcement learning frames the world as state-action pairs: holding the ball, throwing it, and seeing where it ends up. Language models jump from state 0 to state 1 as next token predictors. They do not encode the transition function in between, the change on the way from S0 to S1.

People notice when a transition function is unknown and set out to learn it, and curiosity is that urge. In his view, listing the parts of the world still unknown, running a task to see what happens, and storing the heuristic starts to solve the curiosity problem, so an agent can keep learning from the world on its own.