Back to The world of voice AI, with Mati Staniszewski of ElevenLabs, with Mati Staniszewski.
Why Is Voice AI Harder Than Text?
Mati Staniszewski says voice has not passed a conversational Turing test, even though text models did that a long time ago.
A useful voice agent has to decide when a person has finished, when to wait, when to ask a clarifying question, and when to go do the task. That takes speech-to-text, a judgment of turn-taking from silence and from context, and a way to speak or show text before it acts. He thinks a customer-support call can already pass a voice Turing test inside that narrow job. An interactive game, where the agent should feel like another person, has not. The ordinary failure is orchestration. A simple reply can come back at once, but a database lookup, or a step that has to authenticate the caller, still has to happen without breaking the conversation. ElevenLabs wants to pass that test for conversational agents, and he hopes the field gets there in about the next year.
The world of voice AI, with Mati Staniszewski of ElevenLabs ·
From the transcript
So like, when do you finish sentence? When is it likely based on silence, based likely on the context? And then sometimes you want it to speak back and clarify, or at least give you the text back to clarify, and then maybe execute a set of instructions. So that problem is still very hard. So I agree with the claim that this orchestration side has not passed a true conversational agent Turing test, where it behaves as you would expect from another person, where you can say, like...
That's the simpler way of saying what I'm saying, is that we have passed the Turing test with text LLMs a long time ago, and we're actually nowhere near that on voice LLMs. So it's kind of interesting how that's a final frontier. Yeah, I feel like it's going to work in specific domains. Like in customer support, call passes the voice tuning test, works well.