Simulation software automatically creates the sounds emanating from virtual objects

While computer-generated imagery has advanced considerably in the last few decades, the sounds associated with them, such as two objects colliding, are typically just recordings that are paired with the visuals. These pre-recorded clips require manual synchronisation with the on-screen action. These clips are also restricted to noises that exist - they can’t predict anything new.

Informed by geometry and physical motion, the new system figures out the vibrations of each object and how, like a loudspeaker, those vibrations excite sound waves. It computes the pressure waves cast off by rapidly moving and vibrating surfaces but does not replicate room acoustics. Although it does not recreate the echoes in a grand cathedral, it can resolve detailed sounds from scenarios like a crashing cymbal, an upside-down bowl spinning to a stop, a glass filling up with water or a virtual character talking into a megaphone.

“There’s been a Holy Grail in computing of being able to simulate reality for humans. We can animate scenes and render them visually with physics and computer graphics, but, as for sounds, they are usually made up,” said Professor Doug James, a computer scientist at Stanford University. “Currently there exists no way to generate realistic synchronized sounds for complex animated content, such as splashing water or colliding objects, automatically. This fills that void.”

Jui-Hsien Wang, lead author of the paper, said: “I’ve spent years trying to solve partial differential equations - which govern how sound propagates - by hand. This is actually a place where you don’t just solve the equation but you can actually hear it once you’ve done it. That’s really exciting to me and it’s fun.”

“Ours is essentially just a render button with minimal pre-processing that treats all objects together in one acoustic wave simulation,” said Ante Qu, co-author of the paper.

The simulated sound that results from this method is highly detailed. It takes into account the sound waves produced by each object in an animation but also predicts how those waves bend, bounce or deaden based on their interactions with other objects and sound waves in the scene.

In its current form, the group’s process takes a while to create the finished product. But, now that they have proven this technique’s potential, they can focus on performance optimizations, such as implementing their method on parallel GPU hardware that should make it drastically faster.

“The first water sounds we generated with the system were among the best ones we had simulated - and water is a huge challenge in computer-generated sound,” said James. “We thought we might get a little improvement, but it is dramatically better than previous approaches even right out of the box. It was really striking.”

Although the group’s work has faithfully rendered sounds of various objects spinning, falling and banging into each other, more complex objects and interactions - like the reverberating tones of a Stradivarius violin - remain difficult to model realistically.