Smash Notes

Back to The world of voice AI, with Mati Staniszewski of ElevenLabs, with Mati Staniszewski.

How Does an Audio Model Turn Text Into Speech?

Mati Staniszewski says a modern audio model predicts the next sound from the sounds and the text around it, rather than copying a human throat.

Early systems tried to rebuild the vocal tract as an analog machine. Bell Labs later turned speech into a structured set of signals. Systems after that stored phonemes, the small pieces of a spoken sound, and stitched on the most likely next one. For streaming speech, today's model predicts the next sound from what came before. When the input is text, it uses that text too. A co-founder brought ideas from transformer models and diffusion models into speech so the result can be reliable, high quality, and fast. Predicting the next token, which language models already do with words, had not been possible in that phoneme space.

The voice is a separate input. The model needs a reference for accent, style, and prosody, not only the words. Older systems hard-coded a British accent or an enthusiastic delivery. ElevenLabs lets the model choose those traits, so a British sound falls out of the audio instead of being switched on. Between the text and the waveform, the path also goes through a spectrogram, a picture of pitch and energy. The work was to encode and decode those steps so context, including whether a line should sound happy, survives into the spoken result.

Transcript

The world of voice AI, with Mati Staniszewski of ElevenLabs ·

From the transcript

Thanks for having me. Let me go a place to start is describe to me how, like, I know how an LLM works at a high level. Describe to me how an audio model works. Like if we were Carpathi style looking to, you know, build a toy one from scratch. How does it work? In the early days, you tried to replicate it exactly like you would replicate it with the human body.

So you would try to completely try to reproduce an analog machine that will create a vocal tract effectively. Then that progressed into trying to create effectively digital signals for speech. Bell Labs was one of the first to try to create a structured set of signals that will represent the speech. And that is the first precursor to what we would do today.

See in transcript

Cheeky Pint

More from this episode

  1. How Did ElevenLabs Make Synthetic Speech Sound Human?Mati Staniszewski says ElevenLabs voices sound human because the models were trained on how speech is delivered, not only on the words.
  2. Why Can't an AI Audiobook Sell on Audible?Mati Staniszewski says an AI audiobook cannot be sold on Audible.
  3. Why Is Voice AI Harder Than Text?Mati Staniszewski says voice has not passed a conversational Turing test, even though text models did that a long time ago.
  4. Why Do Sales Leads Share More on a Voice Call?Mati Staniszewski saw people registering interest with ElevenLabs share more once they could speak with an agent instead of only finishing the written form.
  5. Can People Who Lost Their Voice Speak Again?Mati Staniszewski says people who lost a voice can get speech back, and he calls that the most important result of the work.
  6. Why Are ElevenLabs Voice Models Cheaper to Train?Mati Staniszewski says ElevenLabs voice models are definitely cheaper to train than language models and image and video models because they are significantly…
  7. What Is ElevenLabs Annual Recurring Revenue?Mati Staniszewski says the most recently announced ElevenLabs revenue figure was 350 million at the end of 2025, and the latest quarter added 100 million in…
  8. How Does Ukraine's Diia Service Use Voice?Mati Staniszewski says Ukraine's Diia service became a way to reach citizens during the war, and ElevenLabs added voice so people could use it by speaking.
  9. Why Do High-agency People Win From AI?Mati Staniszewski says high-agency people are the ones who win from advances in AI, and that inside organizations low-agency people will lose out.

People

More answers like this

More with John Collison

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.