Should I Trust AI Agent Decisions Yet?
Naveen Rao says people should not trust AI agent decisions as much as human decisions in a lot of cases just yet. The element that decides which solution is good is not well understood for every task.
Becoming a neurosurgeon takes a seven-year training after med school. Learning how to throw a stitch is not what makes the job hard, and he has done that in grad school. The physical, mechanical part of being a surgeon is not the problem. Judgment about which direction to go is the problem. It takes seeing enough cases, seeing the output, and seeing how people think through problems. Intelligence, as it is defined right now, starts with a stochastic model. Prompting it several times yields different outputs and explores the distribution of possible solutions, and most of the outputs are not good. A human can think through outputs with a heuristic for what is good, sometimes as an internal monologue. A model spits out a chain of thought, then many chains of thought, and a path is taken from the ones judged good. The longer the chain of decision upon decision, the further off the rails it can go. A result can look very robust and still be completely wrong.
The same limit is where the field stands today. The picture will probably change in the future, but a few more breakthroughs are still required.
