Should I Trust AI Agent Decisions Yet?
Naveen Rao says people should not trust AI agent decisions as much as human decisions in a lot of cases just yet. The element that decides which solution is good is not well understood for every task.
Becoming a neurosurgeon takes a seven-year training after med school. Learning how to throw a stitch is not what makes the job hard, and he has done that in grad school. The physical, mechanical part of being a surgeon is not the problem. Judgment about which direction to go is the problem. It takes seeing enough cases, seeing the output, and seeing how people think through problems. Intelligence, as it is defined right now, starts with a stochastic model. Prompting it several times yields different outputs and explores the distribution of possible solutions, and most of the outputs are not good. A human can think through outputs with a heuristic for what is good, sometimes as an internal monologue. A model spits out a chain of thought, then many chains of thought, and a path is taken from the ones judged good. The longer the chain of decision upon decision, the further off the rails it can go. A result can look very robust and still be completely wrong.
The same limit is where the field stands today. The picture will probably change in the future, but a few more breakthroughs are still required.
From the transcript
And he was talking about kind of the importance of chain of thought and understanding agent-led decision-making versus human-led decision-making across different functions in companies. And part of me was like, I totally get that. And that makes absolute sense. And part of me was like, that feels kind of old to think of it like that. And like, what does that mean? We don't trust agent decisions as much as we trust humans.
That feels kind of weird too. We don't. I mean, look, I agree with you. I don't think we should trust Asian decisions as much as humans in a lot of cases just yet. I mean, it's like we train for certain things for many years to build judgment. I mean, becoming a neurosurgeon, right? It's a seven-year training after med school. There's a reason for that.