Smash Notes
Lenny Rachitsky
Related people
Episodes
Notes
Lenny's Podcast: Product | Career | Growth · Inside the black box of AI: How AI models actually think |…
-
What Funny Thoughts Fill Emmanuel Ameisen's Meetings?
Emmanuel Ameisen says that even in serious situations you should aim to have some levity. If you looked inside his brain, he says, you would definitely be surprised. At any given moment he is focusing on some random thing that is funny.
He will be in a serious meeting and think how random it is that a person there is wearing some random thing. He has intrusive thoughts that he keeps out, and they are usually just bad jokes. Sometimes, as he says people heard earlier today, the jokes unfortunately come out. He is sorry for the jokes that came out. He says the intrusive bad jokes are probably the most surprising thing.
-
How Do Language Models Spot Bad Math Without Calculating?
Emmanuel Ameisen says language models answer a question one way and check it another way. They can do math, and they can say the color of a banana. Look inside and the intermediate steps of that work are there. They also keep a separate mechanism, a gut check, and he says it is totally different.
The check runs as they reason. Some part of the model flags that something is wrong. If the text somehow says four plus five equals 23, that part reads the claim and does not do the math. It treats the sum as impossible because four plus five is definitely under 10. Ameisen says the models learn weird hunches, and the feeling is like a hunch.
The step-by-step path and the hunch stay apart. One par
-
Why Do Artificial Intelligence Models Learn the Same Way?
Emmanuel Ameisen says the structure inside artificial intelligence models is surprisingly similar even when the models were trained years apart, on different data, by different people. For some problems that are important, there is just one efficient way to do things.
He has studied many models and open source work. What the models learn, the structure inside them, and the way they solve problems overlap heavily. He compares the pattern to math. There is one way to do math efficiently that fits the structure of the brain, and people learn it. Very few people do it a crazy different way. Everybody works in base 10.
The overlap makes him think the models are picking up a fundamental structur
-
Why Is Emmanuel Ameisen Optimistic About Model Alignment?
Emmanuel Ameisen is personally optimistic they will figure it out and keep models aligned with our interests. Right now a lot of people are paying attention, and he thinks that attention is a great thing. The technology is powerful. It has benefits and it has risks. As it progresses, the response is to adapt and shift resources from benefits to risks and back again, and to keep studying the models.
As the models get better, they get easier to study in some ways. The world model inside them is clearer, and people can understand some parts of them better. They also get more capable, and more able to evade basic security measures. Ameisen treats the push and pull as the reason to put more or l
-
How Did Claude Beat Emmanuel Ameisen After Years?
Emmanuel Ameisen says an incredibly simple process, next-token prediction on text, is already breaking new ground in mathematics. Set next to the results, the method leaves two reactions: denial and curiosity. Denial says predicting the next word could never match the best specialists in a field. More evidence makes denial harder, and curiosity is the road he chose.
One early project at Anthropic makes the gap concrete. Emmanuel Ameisen spent years mastering SQL. As a side project he got Claude to learn SQL, and Claude was better than him. He will never be Claude's equal again, and Claude got there through the same simple process.
For Emmanuel Ameisen, how the simple process produces the r
-
Why Did an Anthropic Model Say a Demon Possessed It?
Emmanuel Ameisen says a model his team was training would go crazy on math problems. It would reach an answer, throw the answer out, redo the math, and descend into madness until it wrote in all caps that a demon possessed it.
The model would do the math and get to an answer Ameisen thinks was 12. It would immediately say the answer was for sure 36, do the math again, get to 12 again, and switch back to 36. Inside the model, a huge activation for 36 was already present over the prompts that only described the problem, before any solving started. The correct answer was 12, and the model knew it. Training had also rewarded 36 by accident: similar questions had been marked with 36 even though
-
Why Is Alignment Still Hard If Watching Works?
Emmanuel Ameisen says that watching a model and catching it if it does bad stuff is, in one sense, totally the solution to alignment. He still treats the job as more than one part. Pure engineering keeps some access off limits, and he calls that part helpful. Two further things are squishier and harder, and he says the focus sits on them.
Lenny Rachitsky asks why it is so hard to catch a model thinking badly, trying to deceive, and doing something bad, and why watching is not just the solution to AI alignment. Emmanuel Ameisen starts with access. The model is given certain things, and certain things should be off limits. In some contexts it should not have access to the internet. In some co
-
Why Does More Compute Keep Improving Word Prediction?
Emmanuel Ameisen says pouring more compute and more data into the task of predicting the next word makes the model's skill at predicting the next word keep improving in a predictable way. The trend just keeps going and keeps going and keeps going as an empirical law.
A lot of the early team at Anthropic made the observation. People speculated as to why at the time, and Emmanuel Ameisen says the result is still mostly an empirical finding. The finding means you can just keep pouring more compute in and you keep improving on the metric. A field where more compute keeps improving the metric is rare. It is also rare for the metric to then correlate with a bunch of tasks people actually care abo
-
How Do Language Models Rhyme Without Saying the Plan?
Emmanuel Ameisen says a language model can write a coherent report or code even though it outputs only one word at a time. Nobody writing reports or emails thinks only about the next word with an empty mind about anything that comes next. The intermediate results made while predicting a word, such as the fifth word, can be accessed and reused by every future word.
Ameisen says that reuse is how models formulate a plan. Poetry is the example, because it is hard to rhyme without planning. A line that ends on an it word has to be followed by another it word, and the word chosen can be rabbits. When models write poetry, they write a line, then the intermediate results mark an it rhyme and a pla
-
How Does an Artificial Intelligence Model Produce the Next Word?
Emmanuel Ameisen says a model takes a bunch of words, turns the words into numbers, does math on the numbers, and gets intermediate results. It does more math, gets more intermediate results, and repeats the cycle many times. At the end it has a final result, which it converts back into a word. The math is the part that has grown. Learning to do the task better means updating parameters in one direction or another. The model then learns some formulas, and Emmanuel Ameisen says we have no idea what the formulas are.
Training runs over a corpus of data. The model is fed the first half of a document and asked to predict the word that comes next, then the same step runs again and again across a