Back to The world of voice AI, with Mati Staniszewski of ElevenLabs, with Mati Staniszewski.
How Did ElevenLabs Make Synthetic Speech Sound Human?
Mati Staniszewski says ElevenLabs voices sound human because the models were trained on how speech is delivered, not only on the words.
Audio is easy to find and usually badly labeled. It often does not say who is speaking, or what emotion and action are in the line. ElevenLabs built its own labeling team, coached on voice, and mixed automatic labels with manual ones. A speech-to-text model began as an internal tool, because models already on the market were not good enough to annotate that audio. A researcher on the team later turned that internal model into a product customers could use. Once the voices were in production, the way customers used them and corrected them became more training data.
Useful work inside the voice research kept turning into products the company had not set out to ship.
The world of voice AI, with Mati Staniszewski of ElevenLabs ·
From the transcript
And especially if it's more of a streaming real-time use case and like a voice agent setting, you need both parts to work across. So it's similar to how you would operate on the token level on the tech side. We operate on the token level on the audio side. It feels like a big part of the magic of Eleven was your voices were much more human sounding.
How did you accomplish that? So I'll kind of give you a quick synopsis of how we think about the models on the text-to-speech side today. In any model, you need architecture, you need compute, you need data. So architecture innovations were one thing. The data part was the second big thing. With audio, you will have a lot of audio data available, but frequently you will not have it annotated in the right way.