Smash Notes
How Is Next-token Prediction Breaking New Ground in Math?
at 48:20
Answer
Emmanuel Ameisen says an incredibly simple process, next-token prediction on text, is already breaking new ground in mathematics. Set next to the results, the method leaves two reactions: denial and curiosity. Denial says predicting the next word could never match the best specialists in a field. More evidence makes denial harder, and curiosity is the road he chose.
One early project at Anthropic makes the gap concrete. Emmanuel Ameisen spent years mastering SQL. As a side project he got Claude to learn SQL, and Claude was better than him. He will never be Claude's equal again, and Claude got there through the same simple process.
For Emmanuel Ameisen, how the simple process produces the result has real depth, and exploring the question is what he wants to keep doing.
Where this answer comes from
People
More notes from this episode
- How Does an AI Model Produce the Next Word? Transcript
- How Do Language Models Rhyme Without Saying the Plan? Transcript
- Why Does More Compute Keep Improving Word Prediction? Transcript
- Why Is Alignment Still Hard If Watching Works? Transcript
- Why Did an Anthropic Model Say a Demon Possessed It? Transcript
- How Is Next-token Prediction Breaking New Ground in Math? Transcript
- Why Is Emmanuel Ameisen Optimistic About Model Alignment? Transcript
- How Similar Are AI Models Trained Years Apart? Transcript
- How Do Language Models Spot Bad Math Without Calculating? Transcript
- Should I Aim for Levity in a Serious Meeting? Transcript
More answers like this
- Why Did an Anthropic Model Say a Demon Possessed It?
- Why Is Emmanuel Ameisen Optimistic About Model Alignment?
- Why Is Alignment Still Hard If Watching Works?
- How Similar Are AI Models Trained Years Apart?