Hyper-reality opera: no longer just the fat lady singing

Your journey begins in a room called the House of Subconscious. You’re wearing headphones, a small backpack containing a computer and a VR headset. As you move freely through a dream-like virtual world, from twilight to dawn, the wind blows in your face, you drift over the sea, a city emerges from under your feet. Somewhere on the imaginary horizon, an orchestra plays while a soprano wanders around you singing.

You’re inside the Linbury Theatre at the Royal Opera House, London, listening to – or rather experiencing visually, aurally, and viscerally – a newly composed operatic work. However, it isn’t like any you’ve sat on a velvet seat and watched on a stage before. “It’s not people flaying their arms around and making funny sounds,” says Simon Reveley of digital media company Figment Productions, who worked with the Opera House on realising the project. “Our stage is 360 degrees and you can move around it at will. When you’re inside the experience you’re not confined to a seat, so we needed a three-dimensional creation.” ‘Current, Rising’, due to run May 21-June 10 2021, is the world’s first hyper-reality opera.

The innovative work is inspired by the liberation of Ariel at the end of Shakespeare’s ‘The Tempest’ and was developed as part of the Royal Opera House’s innovation programme, Audience Labs, headed by Annette Mees and funded by Innovate UK. Her first concern was simple: is it opera? She came up with four defining elements of the operatic experience: it’s rooted in the human voice, it’s music-led, it’s theatrical, and it’s a shared experience. And, of course, it must be extraordinary.

“Something being ‘operatic’ is used to describe the epic,” says Mees, who believes she has found the perfect way to realise this form. “When I encountered hyper reality, it felt like a technology that was operatic and epic. You can literally transform worlds and transport people into new spaces. It felt like such a good match.”

You share the experience in the 10m×6m Linbury space with a bubble of just three. Your companions are abstract forms made from particles, each still recognisable, and you can wave and talk to each other. You stay within this world for under 15 minutes. Under Covid restrictions, the construction then must be cleaned before another group of four enters.

Reveley, who usually works with clients who want immersive dinosaur experiences, > < or theme park giant Merlin, relied upon a readily available Windows Mixed Reality system developed by Microsoft to create this new world. The system uses computer vision to process two camera feeds from the headset, builds a model of the world around that person, and estimates their position and orientation in that world.

“However, our use-case for the technology is beyond what it was originally designed for,” says Reveley. “We needed it to run repeatedly throughout the day – almost constantly, in fact – with different people donning the headset. In our operational tests we have found there were occasional aberrations, so we decided to create a secondary tracking system that runs in parallel and constantly cross-references the tracking data reported by Windows MR, called Cross-Check. By running the two solutions together, we can minimise the potential for any operational errors.”

A challenge was to ensure that this 360-degree immersive experience wasn’t the star of the show. “There’s a danger that the virtual world will overwhelm the musical experience,” he says. “We had to allow the music to be front and centre all the time.”

‘When I encountered hyper reality, it felt like a technology that was operatic and epic. You can literally transform worlds and transport people into new spaces.’

Composer Sam Fernando, one of the all-female artistic team, made music for the 360-degree space. Each instrument must be ‘miked’ individually. Sound designer David Sheppard of Sound Intermedia says the technical challenge was “to put that music into space, to conceive of the role of music not as in a traditional theatrical experience coming from in front of you. This music might be behind and around you, very close or far away. Because you’re stepping into a virtual world, sound needs to work with that, to convince you that you’re in the actual space you can see around you.”

He used binaural mixing technology, previously reserved for gaming and theme parks. Mixing software used algorithms to calculate what it would sound like if the music was captured on microphones in your left and right ear, enabling the sound to be mixed spatially, rather than just for hard and soft, a sort of sound mapping. The reverb effect was also important. “It’s the reverb that makes sound match what you see visually, and believe it,” says Sheppard.

Yet binaural recording has its limitations. “Encoding and decoding is designed on a standard-sized head and standard-shaped ears. If someone’s head is much wider, or ear canals different, they might not get as much spatial information,” Sheppard adds.

Creating the world’s first hyper-reality opera also meant creating a team that hadn’t worked together before. “It was a challenge interfacing with a team from the world of theatre and opera that had never done anything like this before,” says Reveley. “There’s a different set of skills, knowledge and approach. It’s not easy for people to ‘get it’.”

To enable them to do so, Figment designed software Hyphen. “It was a production tool to enable us to share work-in-progress and have them inside the world as we develop it. You can put the headset on in New York and look at the world we’re developing for you.”

Yet with a maximum audience of four at a time, is a hyper-reality opera economically feasible? “In the new, Covid-influenced world, cost efficiency is paramount to make these experiences viable for the future,” says Reveley. “Cross-Check is very cost-effective as both Windows MR and Steam VR are off-the-shelf, consumer-accessible products. With the world of location-based VR currently on its knees as the pandemic bites, we all need to find ways to bring down the cost of delivering these premium experiences.”

The Royal Opera House plans to develop similar pop-up operatic experiences in different spaces and places. Royal Holloway University has been brought on board to research the business model. “There’s something about the hybrid of the very old and the very new, for old audiences to find technology, and for new audiences to find opera,” says Mees.

‘Current, Rising’ is about isolation and connection and how we can join to rethink and reimagine our futures. Whatever the future holds, hyper reality will be part of it. “It’s a very new medium. We want to explore ways to produce hyper-reality experiences,” says Reveley. “Working with one of the oldest and most traditional forms was a very good way to test the boundaries of what we could use it to do.”