Back to The world of voice AI, with Mati Staniszewski of ElevenLabs, with Mati Staniszewski.
How Does an Audio Model Turn Text Into Speech?
Mati Staniszewski says a modern audio model predicts the next sound from the sounds and the text around it, rather than copying a human throat.
Early systems tried to rebuild the vocal tract as an analog machine. Bell Labs later turned speech into a structured set of signals. Systems after that stored phonemes, the small pieces of a spoken sound, and stitched on the most likely next one. For streaming speech, today's model predicts the next sound from what came before. When the input is text, it uses that text too. A co-founder brought ideas from transformer models and diffusion models into speech so the result can be reliable, high quality, and fast. Predicting the next token, which language models already do with words, had not been possible in that phoneme space.
The voice is a separate input. The model needs a reference for accent, style, and prosody, not only the words. Older systems hard-coded a British accent or an enthusiastic delivery. ElevenLabs lets the model choose those traits, so a British sound falls out of the audio instead of being switched on. Between the text and the waveform, the path also goes through a spectrogram, a picture of pitch and energy. The work was to encode and decode those steps so context, including whether a line should sound happy, survives into the spoken result.
The world of voice AI, with Mati Staniszewski of ElevenLabs ·
From the transcript
Thanks for having me. Let me go a place to start is describe to me how, like, I know how an LLM works at a high level. Describe to me how an audio model works. Like if we were Carpathi style looking to, you know, build a toy one from scratch. How does it work? In the early days, you tried to replicate it exactly like you would replicate it with the human body.
So you would try to completely try to reproduce an analog machine that will create a vocal tract effectively. Then that progressed into trying to create effectively digital signals for speech. Bell Labs was one of the first to try to create a structured set of signals that will represent the speech. And that is the first precursor to what we would do today.