How Do Language Models Spot Bad Math Without Calculating?

Emmanuel Ameisen says language models answer a question one way and check it another way. They can do math, and they can say the color of a banana. Look inside and the intermediate steps of that work are there. They also keep a separate mechanism, a gut check, and he says it is totally different.

The check runs as they reason. Some part of the model flags that something is wrong. If the text somehow says four plus five equals 23, that part reads the claim and does not do the math. It treats the sum as impossible because four plus five is definitely under 10. Ameisen says the models learn weird hunches, and the feeling is like a hunch.

The step-by-step path and the hunch stay apart. One part works through the problem. Another part only reads the claim and objects when the claim cannot be right.