‘4D goggles’ allow virtual objects to brush past wearers
Image credit: Dreamstime
Neuroscientists at the University of California-San Diego and San Diego State University have created goggles that give the wearer the sensation of being physically touched by objects in a film, game or other media experience.
The development of the goggles was inspired by a research project into the perceptual and neural mechanisms of multisensory integration. In this study, the researchers mapped the parts of the brain that integrate the sight and touch of an approaching object.
“We perceive and interact with the world around us through multiple senses in daily life,” said Dr Ruey-Song Huang. “Though an approaching object may generate visual, auditory, and tactile signals in an observer, these must be picked apart from the rest of the world, originally colourfully described by William James as a “blooming buzzing confusion”.
“To detect and avoid impending threats, it is essential to integrate and analyse multisensory looming signals across space and time and to determine whether they originate from the same sources.”
In this study, the researchers looked at the subjective sensation of synchrony as participants experienced a ball looming towards them in virtual reality (VR), while an air puff blew on the same side of their face.
When the participant experienced the puff of air at the same time the ball began moving, the air was perceived as out of sync with the ball. However, when there was a delay of 800 to 1000 milliseconds between the ball beginning to move and the air being blown (lateralised stimuli), the participants perceived the two as synchronised: as though the object had whizzed past their face, generating wind.
The researchers then used functional magnetic resonance imaging (fMRI) – which monitors brain activity by detecting changes in blood flow – to assess the responses of the participants to tactile-only, visual-only, tactile-visual in-sync and tactile-visual out-of-sync stimuli.
More than a dozen parts of the brain responded more strongly to lateralised visual and tactile stimuli than just the ball or air puff alone. The brain responded most strongly when the stimuli were synchronised, such that the participant perceived the air puff as wind blowing past after the ball.
While this fundamental research is valuable to neuroscientists, it was also used as the basis for the development of the ‘4D goggles’. The goggles add an extra sense to a film, music, video game or VR app: touch. According to the researchers, this enhances the immersive sense of presence of in-world people and objects.