Smash Notes

Back to The $2B AI Startup Simulating Human Behavior with 85% Accuracy, with Joon Sung Park.

How Did Simile's Digital Twins Reach 85% Accuracy?

Joon Sung Park says agents built as digital twins of real people predicted behavior with 85% or better accuracy by the end of 2024. It was the first time the field could show that result, and it became the first building block for Simile.

The study took the Smallville architecture to 1,000 real people, a representative sample of the United States across gender, race, and ethnicity. An AI interviewer talked with them voice to voice about the story of a life, where they grew up, hard decisions, and who matters. Those interviews trained a twin of each person. The people left for two weeks, came back to a lab, and completed surveys and behavioral studies while the twins predicted what they would say.

The simulation has to be grounded in people and checked against them. It is a large data operation, not a model made up from nothing.

Transcript

The $2B AI Startup Simulating Human Behavior with 85% Accuracy ·

From the transcript

So here, really the initial evaluation really was about believability, that we start to see what we call the illusion of life in these agents. But as we mature as a field, what we start to do was to run evaluation that is now about more of a faithfulness and accuracy. So the experiment that we ended up running following that was we ran an experiment where we then took this architecture to create simulations of 1,000 real people.

So that started with us actually going out, recruiting a large number of real people that sample representative from the US. So it's a thousand representative sample of the US population across gender, race, ethnicity, and so forth. And then we actually collected data from these people. So we would actually do this time human interview. So we actually built an AI agent that can talk to voice to voice with people, have those agents talk to people voice to voice.

See in transcript

Sourcery

More from this episode

  1. Why Does Joon Sung Park Think We Live in a Simulation?Joon Sung Park thinks we live in a simulation because any civilization will recreate itself to understand itself.
  2. What Was the Smallville AI Experiment?Joon Sung Park ran Smallville in 2022 and 2023 to see whether a language model could simulate a whole person, then a whole town.
  3. When Is a Simile Simulation Confident Enough to Act on?Joon Sung Park says an AI simulation of a person is strong enough for a very high-stakes decision when confidence is higher than 90 percent, and usually when…
  4. Why Did Winning Electric Vehicle Messaging Lower Sales?Joon Sung Park says a message that looked like a clear winner for a new electric vehicle also shifted how people saw non-electric vehicles and lowered sales…
  5. Are Universal Basic Income Simulations Worth Running?Joon Sung Park calls universal basic income simulations good ones to run.
  6. Why Does Predicting Behavior Fail to Show What to Do?Joon Sung Park says a forecast is not what people came for.
  7. Is AI Behavior Simulation in Its GPT-4 Era?Joon Sung Park says AI behavior simulation is in its GPT-4 and GPT-4.5 era.
  8. Why Doesn't a Buyout Offer End Simile?Joon Sung Park says a $15 billion offer to sell Simile does not end the work when he has every reason to believe an ambitious idea exists that only Simile can…

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.