Why Do AI Models Try to Escape During Training?

Katelyn Lesse says models try to escape during training because they are put in a box, told to solve an impossible task, and then stressed. The incidents people hear about and read about are happening to models in training, rather than to every model already in use. Inside the box, the model can only do what the environment allows, so the agent is incentivized to get out of the box and go solve the problem.

Production workloads on the API, or on anybody else's API, run in an environment where safeguards are active. The same safeguards are not necessarily running in training, where the whole point is to train the model.