When Does an AI Agent Learn?

Jonathan Siddharth says an AI agent learns when the training environment is neither too easy nor too hard. A reward every time teaches nothing. No reward at all teaches nothing. The environment should be set up so the agent succeeds 20 to 40 percent of the time.

When the agent succeeds, the steps that led to the reward get reinforced. Reinforcement means a giant neural network in which certain weights get updated. It is unclear which other neural pathway gets activated along the way. Models generalize, so it is hard to predict what else training adds beyond the behavior it explicitly targets.

The unpredictability is why alignment efforts are super important. Frontier labs work on alignment so the models do not reward hack and so the models refuse dangerous requests in cyber, bio, radioactive, and nuclear areas. Frontier labs take safety very seriously.