Speech algorithm can alter words spoken in audio recordings
A new piece of software called VoCo is capable of smoothly replacing a word in an audio recording of a human voice by editing a transcript of the recording.
The added words are automatically synthesised in the speaker’s voice even if they don’t appear anywhere else in the recording.
The system, which uses a sophisticated algorithm to learn and recreate the sound of a particular voice, could one day make editing podcasts and narration in videos much easier. More broadly, the technology could provide a launching point for creating personalized robotic voices that sound natural.
“VoCo provides a peek at a very practical technology for editing audio tracks, but it is also a harbinger for future technologies that will allow the human voice to be synthesized and automated in remarkable ways,” said Adam Finkelstein, a professor of computer science at Princeton.
On a computer screen, VoCo’s user interface looks similar to other audio editing software such as the popular podcast editing program Audacity or Apple’s music editing program GarageBand.
It offers visualization of the waveform of the audio track and a set of cut, copy and paste tools for editing. Unlike other programs, however, VoCo also augments the waveform with a text transcript of the track and allows the user to replace or insert new words that don’t already exist in the track simply by typing in the transcript.
When the user types the new word, VoCo updates the audio track, automatically synthesizing the new word by stitching together snippets of audio from elsewhere in the narration.
“Currently, audio editors can cut out pieces of a track of narration and move a clip from one place to another. However, if you want to add a word that doesn’t exist in the recording, it’s possible only through a painstaking trial and error process of searching for small audio snippets that might fit together well enough to plausibly form the word,” said Finkelstein.
“VoCo automates the search and stitching process, and produces results that typically sound even better than those created manually by audio experts.”
At the heart of VoCo is an optimization algorithm that searches the voice recording and chooses the best possible combinations of partial word sounds, called “phonemes,” to build new words in the user’s voice.
To do this, it not only needs to find the individual phonemes, but also find sequences of them that stitch together without abrupt transitions, as well as fit them into the existing sentence so that the new word blends in seamlessly.
Words are pronounced with different emphasis and intonation depending on where they fall in a sentence, so context is important.
For clues about this context, VoCo looks to an audio track of the sentence that is automatically synthesized in artificial voice from the text transcript—one that sounds robotic to human ears. This recording is used as a point of reference in building the new word. VoCo then matches the pieces of sound from the real human voice recording to match the word in the synthesized track—a technique known as “voice conversion” which inspired the project name VoCo.
In case the synthesized word isn’t quite right, VoCo offers users several versions of the word to choose from. The system also provides an advanced editor to modify pitch and duration, allowing expert users to further polish the track.
To test how effective their system was a producing authentic sounding edits, the researchers asked people to listen to a set of audio tracks, some of which had been edited with VoCo and other that were completely natural. The fully automated versions were mistaken for real recordings more than 60 percent of the time.
Zeyu Jin, a Princeton graduate student advised by Finkelstein, whose research interests straddle audio and machine learning, said voice conversion technologies hold promise for a range of applications beyond editing audio tracks. For instance, people who have lost their voices due to injury or disease might be able to recreate their voices through a robotic system.