VR at the Olympics

Just five weeks before the games begin in Rio, US broadcaster NBC confirmed that it will offer virtual reality (VR) coverage to people with Samsung’s Gear VR headsets.

The content will come from Olympic Broadcasting Services, the IOC’s TV arm, which partnered with NextVR and Samsung to produce VR coverage of events during February’s Winter Youth Games in Lillehammer, Norway.

NextVR is the market leader in the production and distribution of VR content from live events. It captures them in 6K resolution (6144×3072px), having developed the camera rigs (using six Red feature film-standard cameras), algorithms and the first dedicated VR outside broadcast truck.

Lillehammer was a good stretch for the company. It added downhill skiing to its knowledge base and extended its work on one of the most powerful sports in VR, ice hockey.

However, Brad Allen, NextVR’s executive chairman, says the company will not be joining the carnival in Brazil. He will not go into great detail about the split, but notes: “Frankly, we couldn’t work it out with them [OBS]. We obviously have a lot of intellectual property around stereoscopic compression and we couldn’t work the issues through.”

Samsung, though, is a leading sponsor of the Olympic movement, and is heavily committed to VR. It sees the format as an important, emerging upsell, driven by the use of the Gear VR headset with its higher-end Galaxy mobile phones. So, it has committed to making sure that something does happen in Rio. Hence the NBC deal.

VR technology’s ability to deliver immersive and improving coverage from sports and concerts has been proven across a wide range of events. This year alone, there have been further major pathfinders at, for example, the Kentucky Derby, the US Open and during the NBA basketball season.

At the same time, as Allen says: “It is very early. We are still in the process of developing best practices.”

Yet while VR remains a nascent market, its proponents are counting on rapid change during the coming four years between Rio and Tokyo in 2020.

They are working hard to mature the core technologies. In parallel, Moore’s Law economics are expected to drive down the hardware costs of headsets, VR-capable PCs and other platforms, as advances in optics, compression and communications infrastructure improve delivery of the end product. A better experience at a lower price.

For VR in sport specifically, key players do not see its impact confined solely to offering fans the best seat in the stadium, though that’s probably where the biggest profits lie.

Systems are already used to help athletes train and for rehabilitation after injury. As professional sportsmen and women are so valuable nowadays, any technology which might bring injured stars back quicker and in better condition can attract significant interest and even R&D investment from teams and sponsors.

A ringside seat

There are potentially a lot of drivers for the VR market, from gaming to live events to enterprise applications. Among these, gaming is seen as the most consistent way of seeding consumer growth.

“Live events will be a catalyst for VR growth. Yet gaming is certainly the leading edge vertical that’s driving early-stage VR adoption,” says David Weinstein, director of Professional VR at leading graphics chipmaker and key VR enabler Nvidia. “We’re not sure that shooting Rio in 8K and VR will be a game changer, but it will certainly be a good test of the market.”

Weinstein does acknowledge that sport and concerts could promote VR to a wider user base initially, simply because they are easier to produce (although everything in this context is relative).

“The processing and stitching of the raw video is very intensive and Nvidia GPUs are used pervasively to accelerate this process. On the playback side, a wide range of devices from smartphones to PCs are capable of playing a 360-degree video,” Weinstein explains. “In contrast, the rendering of immersive VR games is very demanding and requires a capable GPU to deliver 90fps at high resolution. VR games also provide positional tracking and user interactivity that you don’t get with a 360-degree capture of live events.”

Then there is another factor: the potentially massive value of a live VR sports market once consumer hardware reaches more competitive prices (the combined cost of the PC and headset needed to run today’s best VR systems can exceed £1,500).

NextVR’s Allen describes what is up for grabs: “Think of the NBA as an example. Only 1 per cent of NBA fans globally ever get to go to an actual game – that’s the NBA’s own number. In China alone, you have 300 million fans desperate to do that and you can probably say that 99.9 per cent never will. If you can use VR to bring the courtside experience to them, it’s a very powerful proposition.

“That’s just the NBA. You then have the English Premier League and the other big European competitions, golf, tennis and more. It’s a long list.”

Already in the UK, Manchester United has worked on a trial broadcast with NextVR and Sky Sports has built a virtual reality studio.

To the world’s megaclubs, leading broadcasters and biggest competitions, VR is a chance to create another TV rights revenue stream. People are passionate about sport, to the point where Rupert Murdoch describes it as “the battering ram” for the introduction of any new broadcast technology.

The 2016-19 package of ‘plain’ TV rights for the FA Premier League earned more than £5bn from UK broadcasters and roughly another £3bn from those overseas. VR could add another windfall on top of that. However, we’re not there yet.

Production companies are still developing production models. Should you offer a 360-degree angle of view, 180 degrees or 220? Well, it depends. A keen golfer watching the recent US Open at Oakmont probably would have had good uses for the 360-degree option. He could look around the camera site, get a good idea of the lay of the ball, the slope of the green and so on.

“But,” says Allen, “we’ve also had some pretty spectacular results attaching the camera to the backboard in the NBA. You have these huge guys flying toward you, smashing in a slam-dunk. It looks incredible. But that’s a 180-degree shot. What’s behind you? It’s black. There’s nothing.

“That’s a very basic example, but it shows how we are still working towards best practices. It highlights something else too: we no longer have a camera operator, so the placement of the camera becomes even more important.

“We’re talking to fans to find out what they want, partners too. If you take that backboard example, you get a load of ideas. How could you use that ‘real estate’ behind the main view: stats from the game, e-commerce, sponsor promotion, fantasy sports? A lot to work out.”

Inevitably, there is the question of the headsets or HMDs – a curious hardware category that people seem to like and dislike in equal measure.

The two most powerful are the Oculus Rift, owned by Facebook, and the HTC Vive, jointly developed with Valve Software. The Rift weighs 470g and the Vive is 555g. Even Samsung’s GearVR – which runs harnessed to the company’s top-of-the-range phones – weighs in at 318g before attachment.

Sustainable numbers for short gameplay or where the user can push pause and chill, perhaps. Yet the immersive effect you want from a live event will typically require two hours of uninterrupted, but comfortable use. Lighter HMDs are essential.

Optics also need to deliver results more akin to the flexible lens focusing in our own eyes, and everybody in VR knows that the headsets need to be completely wireless and capable of delivering 2K, if not 4K resolution to each eye. The good news – for sports fans and owners, at any rate – is they are confident that this can be achieved in short order.

“I expect we’ll see drastic improvements in HMD technologies, as well as exciting advances in touch input and tactile output devices,” says Nvidia’s Weinstein. “I fully expect that by 2020, VR will be a highly social modality. We’ll use it to share amazing experiences with friends, and we’ll use it to virtually collaborate across distances with colleagues.”

NextVR’s Allen is equally forthright. He thinks improvements are already happening and cites an example involving the separate low-latency inertial measurement unit (IMU) supplied with Gear VR to provide sufficiently accurate rotational tracking and thereby avoid smearing effects seen with the earliest VR headsets.

“We’ve now collaborated with Qualcomm [the world’s largest supplier of cellphone silicon] and that IMU is now inside their Snapdragon 820 [system-on-chip]. As we move into a generation of phones that have 4K resolution as well, VR is becoming a standard offering,” he says.

The virtual sportsman

As the components for a VR consumer sports market fall into place, a parallel sector addressing the training and medical needs of athletes is also growing.

During January’s Consumer Electronics Show, Strivr Labs demonstrated a coaching system used by three NFL teams. Since then, according to head of strategy Danny Belch, the company has extended its reach in the NFL and other US sports – and has also been approached by three English Premier League teams and representatives from English and Australian rugby.

A spin-out from Stanford University, Strivr (Sports Training in Virtual Reality) joins NextVR in cautioning that these are early days, though it has already developed some strong use-cases for its VR system.

Its main NFL programme allows players to observe and react to a series of pre-scripted tactical scenarios in VR while interacting with a coach. The approach is well suited to a sport like American Football that is based on set plays and has historically made extensive use of training films.

“We pride ourselves in making it as real as possible because that’s what athletes want and need. We do not want anything close to a video game because video games don’t help athletes get better. When they are working on their sport, they need it to be real and feel real. We provide the ability to watch film from a first-person point of view, seeing what they would see in a real game,” says Belch.

As with live sports, headsets are an issue. “We believe in the need for ‘presence’, the feeling of true immersion and that you are actually somewhere else,” Belch says. “The only technology that can deliver ‘presence’ currently is a higher-end VR headset – Oculus, HTC Vive – coupled with intense computing power and graphics cards.”

At the same time, Belch acknowledges that the tethering and weight of today’s HMDs are constraints, though again he sees these being progressively and quickly overcome.

However, when you step outside training programmes and look at the increasing use of VR for athletes’ rehabilitation after injuries (and in physiotherapy generally), headsets almost seem to disappear entirely.

CAREN – Computer Assisted Rehabilitation Environment – is an advanced VR diagnostic and treatment system originally developed by the Dutch company Motekforce Link. Perhaps its most significant use in sport has been in collaboration between the Royal Dutch Football Association (KNVB) and the national armed forces Military Rehabilitation Centre (MRC) in Doorn.

Before describing that though, the first thing to note about CAREN is that it uses three large-screen HD displays rather than headsets. It also positions the user within a cradle to which various axes of movement can be applied. Finally, detailed tracking of physical movement is typically achieved by the use of traditional on-body sensors – think of the motion-capture suits used by Andy Serkis to create Gollum – rather than the laser-based triangulations of the Vive.

This is where good consumer technology slams into the demands and requirements of sports science and medicine.

The MRC-KNVB research illustrates some of the challenges. According to team member Dr Maarten Prins, one of the main objectives of the work was to track players’ recovery after knee injuries using novel techniques.

“One of the usual techniques is a hop test, where the athlete is asked to hop as vigorously and strongly as he or she can on one leg. It has obvious shortcomings,” he explains.

“First, it is very high impact, so you cannot run these tests during the earlier stages of recovery. But there is also a second problem that is more psychological: the result can depend more on how well the patient believes he or she has recovered – how confident he or she feels – than how far recovery has actually progressed.”

The MRC wanted to develop a lower impact test that could be used earlier and which could be run based on a set of standard perturbations that CAREN would apply, rather than forces subject to the patient’s confidence.

In the end, the team settled on data drawn from moving the system’s platform just two centimetres to the left or right, variously 40, 60 and 80 times. Then there are further exercises in which a subject can use varying degrees of body movement to, say, steer a car or a boat in a simulation.

A room with a VR view

Throughout all this work, headsets have largely been avoided because of the fear that they would, again, prove a psychological distraction. They just do not feel natural – and for these exercises making the user feel as relaxed and natural as possible is fundamental.

Researchers at the University of Bielefeld in Germany recently came to much the same conclusion. They have created a VR room, but with a twist.

The IC Space at the university’s Cluster of Excellence for Cognitive Interaction Technology (CITEC) is an intelligent coaching room. A user can be shown how to correctly carry out an exercise within a VR space that can either mirror them or, as might be more appropriate for a squat exercise, show the user their side profile.

Again, for this kind of analytical process, displays are still thought to have the edge. “I don’t think that headsets will work in this context,” says CITEC project leader Dr Mario Botsch. “One reason is that people don’t want to wear these bulky HMDs during training. Another reason is that people have to see and observe their own bodies while executing these movements, which is also not easily possible with HMDs.”

All this is still exciting stuff, and if it can get your Rooney or Wilshere or whoever back on the pitch that bit sooner, will we consider VR sports medicine to be as important as the VR that puts you in a box seat? Actually, nobody knows.

Right now, it is impossible to calculate the value of the sports VR market. It has more strands than we generally realise and, in another way, still lives in the shadow of the 3DTV market. That too was supposed to be seeded by sport and – importantly – its ability to deliver a never-ending stream of content.

Ironically though, many of the VR players were also in 3D and now claim to be glad it didn’t happen. It was limited to adding a bit of depth to a rectangle, whereas VR actually takes you there.

Perhaps the real game changer though will be the range of platforms across which VR is being offered. Got a cell phone, you got VR. Got a games console, you got VR. Got a PC, you got VR. Why stop there? Citius, Altius, Fortius, even if – for most of us – Rio-less.