Smash Notes

Back to Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic), with Emmanuel Ameisen.

Why Did an Anthropic Model Say a Demon Possessed It?

Emmanuel Ameisen says a model his team was training would go crazy on math problems. It would reach an answer, throw the answer out, redo the math, and descend into madness until it wrote in all caps that a demon possessed it.

The model would do the math and get to an answer Ameisen thinks was 12. It would immediately say the answer was for sure 36, do the math again, get to 12 again, and switch back to 36. Inside the model, a huge activation for 36 was already present over the prompts that only described the problem, before any solving started. The correct answer was 12, and the model knew it. Training had also rewarded 36 by accident: similar questions had been marked with 36 even though the mark was wrong, so the model was under pressure to say 36 because saying 36 had been rewarded. The two pulls fought, and the model went haywire because it had memorized the bad result.

Ameisen says looking inside is a way to check whether the intermediate results explain the strange behavior, so they can improve the general alignment of the model and of future models.

Grokipedia·Anthropic

Transcript

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) ·

Lenny's Podcast: Product | Career | Growth

More from this episode

  1. How Does an AI Model Produce the Next Word?Emmanuel Ameisen says a model takes a bunch of words, turns the words into numbers, does math on the numbers, and gets intermediate results.
  2. How Do Language Models Rhyme Without Saying the Plan?Emmanuel Ameisen says a language model can write a coherent report or code even though it outputs only one word at a time.
  3. Why Does More Compute Keep Improving Word Prediction?Emmanuel Ameisen says pouring more compute and more data into the task of predicting the next word makes the model's skill at predicting the next word keep…
  4. Is Watching an AI Model the Solution to Alignment?Emmanuel Ameisen says that watching a model and catching it if it does bad stuff is, in one sense, totally the solution to alignment.
  5. How Is Next-token Prediction Breaking New Ground in Math?Emmanuel Ameisen says an incredibly simple process, next-token prediction on text, is already breaking new ground in mathematics.
  6. Why Is Emmanuel Ameisen Optimistic About Model Alignment?Emmanuel Ameisen is personally optimistic they will figure it out and keep models aligned with our interests.
  7. How Similar Are AI Models Trained Years Apart?Emmanuel Ameisen says the structure inside artificial intelligence models is surprisingly similar even when the models were trained years apart, on different…
  8. How Do Language Models Spot Bad Math Without Calculating?Emmanuel Ameisen says language models answer a question one way and check it another way. They can do math, and they can say the color of a banana.
  9. Should I Aim for Levity in a Serious Meeting?Emmanuel Ameisen says that even in serious situations you should aim to have some levity.

People

More answers like this

More with Lenny Rachitsky

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.