Smartphone videos combine to create 4D virtualisation of filmed events

The researchers at Carnegie Mellon University said the system could even allow actors recorded in one setting to be digitally inserted into another video with ease.

“We are only limited by the number of cameras,” said researcher Aayush Bansal, with no upper limit on how many video feeds can be used.

'Virtualised reality' is nothing new, but in the past it has been restricted to studio setups, such as CMU’s Panoptic Studio, which requires more than 500 video cameras embedded in its geodesic walls to create the effect.

But the researchers said that until now, fusing visual information of real-world scenes shot from multiple, independent, handheld cameras into a single comprehensive model hasn’t been possible.

Bansal and his colleagues worked around that limitation by using convolutional neural nets (CNNs), a type of deep-learning program that has proven adept at analysing visual data. They found that scene-specific CNNs could be used to compose different parts of the scene.

The CMU researchers demonstrated their method using up to 15 iPhones to capture a variety of scenes – dances, martial arts demonstrations, and even flamingos at the National Aviary in Pittsburgh.

“The point of using iPhones was to show that anyone can use this system,” Bansal said. “The world is our studio.”

The method also unlocks a host of potential applications in the movie industry and consumer devices, particularly as the popularity of virtual-reality headsets continues to grow.

Though the method doesn’t necessarily capture scenes in full 3D detail, the system can limit playback angles so incompletely reconstructed areas are not visible and the illusion of 3D imagery is not shattered.