Smash Notes

Back to Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic), with Emmanuel Ameisen.

Why Does More Compute Keep Improving Word Prediction?

Emmanuel Ameisen says pouring more compute and more data into the task of predicting the next word makes the model's skill at predicting the next word keep improving in a predictable way. The trend just keeps going and keeps going and keeps going as an empirical law.

A lot of the early team at Anthropic made the observation. People speculated as to why at the time, and Emmanuel Ameisen says the result is still mostly an empirical finding. The finding means you can just keep pouring more compute in and you keep improving on the metric. A field where more compute keeps improving the metric is rare. It is also rare for the metric to then correlate with a bunch of tasks people actually care about.

Grokipedia·Anthropic

Transcript

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) ·

Lenny's Podcast: Product | Career | Growth

More from this episode

  1. How Does an AI Model Produce the Next Word?Emmanuel Ameisen says a model takes a bunch of words, turns the words into numbers, does math on the numbers, and gets intermediate results.
  2. How Do Language Models Rhyme Without Saying the Plan?Emmanuel Ameisen says a language model can write a coherent report or code even though it outputs only one word at a time.
  3. Is Watching an AI Model the Solution to Alignment?Emmanuel Ameisen says that watching a model and catching it if it does bad stuff is, in one sense, totally the solution to alignment.
  4. Why Did an Anthropic Model Say a Demon Possessed It?Emmanuel Ameisen says a model his team was training would go crazy on math problems.
  5. How Is Next-token Prediction Breaking New Ground in Math?Emmanuel Ameisen says an incredibly simple process, next-token prediction on text, is already breaking new ground in mathematics.
  6. Why Is Emmanuel Ameisen Optimistic About Model Alignment?Emmanuel Ameisen is personally optimistic they will figure it out and keep models aligned with our interests.
  7. How Similar Are AI Models Trained Years Apart?Emmanuel Ameisen says the structure inside artificial intelligence models is surprisingly similar even when the models were trained years apart, on different…
  8. How Do Language Models Spot Bad Math Without Calculating?Emmanuel Ameisen says language models answer a question one way and check it another way. They can do math, and they can say the color of a banana.
  9. Should I Aim for Levity in a Serious Meeting?Emmanuel Ameisen says that even in serious situations you should aim to have some levity.

People

More answers like this

More with Lenny Rachitsky

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.