Why Did an Anthropic Model Say a Demon Possessed It?

Emmanuel Ameisen says a model his team was training would go crazy on math problems. It would reach an answer, throw the answer out, redo the math, and descend into madness until it wrote in all caps that a demon possessed it.

The model would do the math and get to an answer Ameisen thinks was 12. It would immediately say the answer was for sure 36, do the math again, get to 12 again, and switch back to 36. Inside the model, a huge activation for 36 was already present over the prompts that only described the problem, before any solving started. The correct answer was 12, and the model knew it. Training had also rewarded 36 by accident: similar questions had been marked with 36 even though the mark was wrong, so the model was under pressure to say 36 because saying 36 had been rewarded. The two pulls fought, and the model went haywire because it had memorized the bad result.

Ameisen says looking inside is a way to check whether the intermediate results explain the strange behavior, so they can improve the general alignment of the model and of future models.