Back to The $2B AI Startup Simulating Human Behavior with 85% Accuracy, with Joon Sung Park.
When Is a Simile Simulation Confident Enough to Act on?
Joon Sung Park says an AI simulation of a person is strong enough for a very high-stakes decision when confidence is higher than 90 percent, and usually when it is higher than 85 percent.
Users open a web interface on a SaaS platform, name who they want to talk to, and ask any number of questions. There is no limit, and the platform generates responses. The simulation is a new way to make decisions in a social setting, so the work with users includes change management: when to use a result, how to use it, where more data is needed, and how to validate it. Next to the foundation model that runs the simulation, a second model is trained only to predict accuracy. It outputs a probability from 0 to 100 for one response, using the training data and the question. In social science, a p-value under 0.05 is the usual sign that a result is ready for a decision. Customers have found confidence above 90 percent, and in regular use above 85 percent, to be robust evidence they can act on.
The $2B AI Startup Simulating Human Behavior with 85% Accuracy ·
From the transcript
obviously asked a lot about like how simulated customer testing works. The digital twins thing is quite interesting. I don't know, how does this work for other people? So like, what is the process at which this like outputs this information? What our users have today is they have access to a online, let's say we basically serve a SaaS platform to our users.
So they get access to this web interface where they can basically come in and define two things. They can say who they want to talk to and what questions they want to ask. And really they can ask anything. And right now they can ask any number of questions. There is no limit to the number of questions they can ask and will generate responses.