Why Can AI Solve Hard Math but Not Accounting?
Diogo Almeida says the models people need are System 1 models, and the honest move is to say where they fail. System 1 was an unpopular name, because that kind of thinking is often used as an insult for the instinctive, mistake-making side. The shape AI actually gets compressed into is still System 1, and naming the weaknesses is useful for software by itself. Most of what people need is that kind of work.
It should be strange that models can take on hard problems in math and still not automate anything that touches accounting. Cost, speed, and the pitch that a model is good for really fast decisions are distractions beside that north star.
Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·
From the transcript
There are some things that models truly are weak at. And this is where that taste stuff comes in, right? Like what I was talking about, the reality of what you can extract from these models. The reason we call them System 1 models, among other things, and System 1 actually, people did not want us to name that it either, because System 1 thinking tends to be a bad thing.
Like, people use that as kind of like an insult, right? Yeah, it's like the part that just naturally happened from evolution, you know, from back in the day before... Exactly, instinctually, which makes mistakes, etc. Exactly. And, like, people often say that that is the mistake-making side, and I believe that the shape that actually... AI gets compressed to happens to be like very system one.