When Is a Simile Simulation Confident Enough to Act on?

Joon Sung Park says an AI simulation of a person is strong enough for a very high-stakes decision when confidence is higher than 90 percent, and usually when it is higher than 85 percent.

Users open a web interface on a SaaS platform, name who they want to talk to, and ask any number of questions. There is no limit, and the platform generates responses. The simulation is a new way to make decisions in a social setting, so the work with users includes change management: when to use a result, how to use it, where more data is needed, and how to validate it. Next to the foundation model that runs the simulation, a second model is trained only to predict accuracy. It outputs a probability from 0 to 100 for one response, using the training data and the question. In social science, a p-value under 0.05 is the usual sign that a result is ready for a decision. Customers have found confidence above 90 percent, and in regular use above 85 percent, to be robust evidence they can act on.