Smash Notes

Back to The world of voice AI, with Mati Staniszewski of ElevenLabs, with Mati Staniszewski.

How Did ElevenLabs Make Synthetic Speech Sound Human?

Mati Staniszewski says ElevenLabs voices sound human because the models were trained on how speech is delivered, not only on the words.

Audio is easy to find and usually badly labeled. It often does not say who is speaking, or what emotion and action are in the line. ElevenLabs built its own labeling team, coached on voice, and mixed automatic labels with manual ones. A speech-to-text model began as an internal tool, because models already on the market were not good enough to annotate that audio. A researcher on the team later turned that internal model into a product customers could use. Once the voices were in production, the way customers used them and corrected them became more training data.

Useful work inside the voice research kept turning into products the company had not set out to ship.

Transcript

The world of voice AI, with Mati Staniszewski of ElevenLabs ·

From the transcript

And especially if it's more of a streaming real-time use case and like a voice agent setting, you need both parts to work across. So it's similar to how you would operate on the token level on the tech side. We operate on the token level on the audio side. It feels like a big part of the magic of Eleven was your voices were much more human sounding.

How did you accomplish that? So I'll kind of give you a quick synopsis of how we think about the models on the text-to-speech side today. In any model, you need architecture, you need compute, you need data. So architecture innovations were one thing. The data part was the second big thing. With audio, you will have a lot of audio data available, but frequently you will not have it annotated in the right way.

See in transcript

Cheeky Pint

More from this episode

  1. How Does an Audio Model Turn Text Into Speech?Mati Staniszewski says a modern audio model predicts the next sound from the sounds and the text around it, rather than copying a human throat.
  2. Why Can't an AI Audiobook Sell on Audible?Mati Staniszewski says an AI audiobook cannot be sold on Audible.
  3. Why Is Voice AI Harder Than Text?Mati Staniszewski says voice has not passed a conversational Turing test, even though text models did that a long time ago.
  4. Why Do Sales Leads Share More on a Voice Call?Mati Staniszewski saw people registering interest with ElevenLabs share more once they could speak with an agent instead of only finishing the written form.
  5. Can People Who Lost Their Voice Speak Again?Mati Staniszewski says people who lost a voice can get speech back, and he calls that the most important result of the work.
  6. Why Are ElevenLabs Voice Models Cheaper to Train?Mati Staniszewski says ElevenLabs voice models are definitely cheaper to train than language models and image and video models because they are significantly…
  7. What Is ElevenLabs Annual Recurring Revenue?Mati Staniszewski says the most recently announced ElevenLabs revenue figure was 350 million at the end of 2025, and the latest quarter added 100 million in…
  8. How Does Ukraine's Diia Service Use Voice?Mati Staniszewski says Ukraine's Diia service became a way to reach citizens during the war, and ElevenLabs added voice so people could use it by speaking.
  9. Why Do High-agency People Win From AI?Mati Staniszewski says high-agency people are the ones who win from advances in AI, and that inside organizations low-agency people will lose out.

People

More answers like this

The other side

More with John Collison

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.