Smash Notes

Back to The world of voice AI, with Mati Staniszewski of ElevenLabs, with Mati Staniszewski.

Why Is Voice AI Harder Than Text?

Mati Staniszewski says voice has not passed a conversational Turing test, even though text models did that a long time ago.

A useful voice agent has to decide when a person has finished, when to wait, when to ask a clarifying question, and when to go do the task. That takes speech-to-text, a judgment of turn-taking from silence and from context, and a way to speak or show text before it acts. He thinks a customer-support call can already pass a voice Turing test inside that narrow job. An interactive game, where the agent should feel like another person, has not. The ordinary failure is orchestration. A simple reply can come back at once, but a database lookup, or a step that has to authenticate the caller, still has to happen without breaking the conversation. ElevenLabs wants to pass that test for conversational agents, and he hopes the field gets there in about the next year.

Transcript

The world of voice AI, with Mati Staniszewski of ElevenLabs ·

From the transcript

So like, when do you finish sentence? When is it likely based on silence, based likely on the context? And then sometimes you want it to speak back and clarify, or at least give you the text back to clarify, and then maybe execute a set of instructions. So that problem is still very hard. So I agree with the claim that this orchestration side has not passed a true conversational agent Turing test, where it behaves as you would expect from another person, where you can say, like...

That's the simpler way of saying what I'm saying, is that we have passed the Turing test with text LLMs a long time ago, and we're actually nowhere near that on voice LLMs. So it's kind of interesting how that's a final frontier. Yeah, I feel like it's going to work in specific domains. Like in customer support, call passes the voice tuning test, works well.

See in transcript

Cheeky Pint

More from this episode

  1. How Does an Audio Model Turn Text Into Speech?Mati Staniszewski says a modern audio model predicts the next sound from the sounds and the text around it, rather than copying a human throat.
  2. How Did ElevenLabs Make Synthetic Speech Sound Human?Mati Staniszewski says ElevenLabs voices sound human because the models were trained on how speech is delivered, not only on the words.
  3. Why Can't an AI Audiobook Sell on Audible?Mati Staniszewski says an AI audiobook cannot be sold on Audible.
  4. Why Do Sales Leads Share More on a Voice Call?Mati Staniszewski saw people registering interest with ElevenLabs share more once they could speak with an agent instead of only finishing the written form.
  5. Can People Who Lost Their Voice Speak Again?Mati Staniszewski says people who lost a voice can get speech back, and he calls that the most important result of the work.
  6. Why Are ElevenLabs Voice Models Cheaper to Train?Mati Staniszewski says ElevenLabs voice models are definitely cheaper to train than language models and image and video models because they are significantly…
  7. What Is ElevenLabs Annual Recurring Revenue?Mati Staniszewski says the most recently announced ElevenLabs revenue figure was 350 million at the end of 2025, and the latest quarter added 100 million in…
  8. How Does Ukraine's Diia Service Use Voice?Mati Staniszewski says Ukraine's Diia service became a way to reach citizens during the war, and ElevenLabs added voice so people could use it by speaking.
  9. Why Do High-agency People Win From AI?Mati Staniszewski says high-agency people are the ones who win from advances in AI, and that inside organizations low-agency people will lose out.

People

More answers like this

The other side

More with John Collison

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.