VR and AR transforming healthcare

Read more about all aspects of Virtual Reality technology

Receiving treatment for severe burns can be a traumatic experience. Wound cleaning and bandage changes cause pain that, even with opioids like morphine, 86 per cent of patients still reported as excruciating. In 1996, after witnessing how engrossed children became while playing video games, researchers at the University of Washington’s Human Interface Technology Laboratory (HITLab) came up with the idea of providing VR games to those being treated for burns, hypothesising that immersion within a game could have positive benefits for the patients, who would concentrate more on the game and less on the pain.

Wearing a VR headset, patients were thrust into a game that by modern standards is laughably primitive. But it didn’t seem to matter. In all tests, patients reported reduced pain levels, with medics able to go about their work more efficiently and effectively.

Anecdote quickly became evidence with peer-reviewed research published by the Society of Behavioural Medicine in 2011 showing just how powerful immersive games can be as an analgesic. The technology is now being used by the US Army to help wounded soldiers while receiving treatment.

It’s just one example of how the total immersion offered by virtual reality is helping to transform healthcare. Training the next generation of students, helping soldiers to cope after a traumatic incident and even creating realistic and interactive maps of the entire human body, virtual and augmented reality technologies are coming to a hospital near you.

Facing your fears

Virtual reality seems to have been around forever, never quite breaking through to the mainstream. “The view is that it died after the 1980s,” explains Professor Mel Slater, who leads the University of Barcelona’s Experimental Virtual Environments Laboratory (Event Lab). According to Slater, that’s not the case. “There has been an absolutely massive amount of work in the past 25 years in very diverse areas from psychotherapy, to physical therapy, to sports, to medical rehabilitation, to travel, virtual meetings and studies of moral dilemmas.”

Behind the doors of healthcare facilities across the world, virtual reality technologies have been used to treat patients for years, while at the same time clinicians and researchers patiently go about developing systems that are safe and build an evidence base for the mass adoption of technology.

Since 2005, the US Army has used exposure therapy and virtual reality simulations to help treat those returning from conflict. Soldiers suffering from post-traumatic stress disorder (PTSD) are being transported back to a virtual warzone to re-experience combat.

Strapping on a headset, the soldiers work through traumatic scenarios in the safety of the virtual world, with the support of a trained therapist, by retelling their story. Using accurately rendered surroundings originally built for the Full Spectrum Warrior game on the Xbox, the realistic environment effectively allows the soldier to travel back, reliving the experience and confronting their fears in what is known as a ‘virtual Iraq’.

The work in the US is one example of how VR is used to treat mental health problems like PTSD, chronic anxiety and a host of other conditions. “We take the victim back to their memories, and build the environment in a virtual world,” says Willem-Paul Brinkman, an assistant professor at Delft University and a part of the Dutch Virtual Reality and Phobias (VRET) programme.

Brinkman and VRET colleagues at Delft University and the University of Amsterdam are using VR to help create a better future for the victims of childhood sexual abuse. Working with trained psychologists, the team are able to reconstruct scenarios from their past, allowing the victim to recontextualise their experience. “We can build a 3D environment so the participant can see the scene of their abuse from different viewpoints,” explains Brinkman. Through the relatively safe environment of VR users can begin to confront these difficult memories, often defusing them of their power.

In a more prosaic example, Brinkmann describes how VR can help those suffering from common social phobias such as delivering a speech to a large audience. Immersed in the VR world, participants can deliver their talk, with the controller able to change the parameters and stretch the participant. “We can make the audience look interested or bored,” Brinkman says. It might sound run-of-the-mill, but for those with anxiety conditions like agoraphobia, the results can be incredible and make a real difference to their lives.

Out-of-body experience

Most VR systems place us in the environment with all of our hang-ups and proclivities, but the work of Slater is taking us out of our own bodies and challenging some of the fundamentals of our own identity. “When you put on a head-tracked head mounted display and look down towards yourself, what do you see?” Slater questions. “Normally it’s a continuation of the virtual environment. However, a virtual body can be programmed into the environment so that when you look down towards yourself you can see an alternative, virtual body.”

By doing this your mind can be tricked, creating a perception that the virtual body is your own. Slater and colleagues have used this to illustrate how our perception of ourselves can be challenged. In one experiment the team put participants in white, black and purple bodies for up to 12 minutes. During that time the participants do very little; they see their body, view themselves in a mirror and watch as some people walk by. “What happens is that the implicit racial bias against black people decreases only for those in the black body,” Slater says, adding: “We have recently repeated this experiment and found that the effect lasts for at least a week.”

It may sound like fun, but the work of Slater, Brinkman and colleagues is conducted in a rigorous and controlled environment. The research is about challenging self-perception and understanding how the view we hold of ourselves can be manipulated, and the resulting influence - positive or negative - that it can have on our actions.

Augmenting the real world

During an operation, one false move could cause permanent damage. So-called virtual reality systems have been used for nearly two decades to train students to carry out certain operations, but they are more like very detailed interactive games. The benefits for training in a VR environment are clear, but what happens when surgeons start working with real patients? Here it’s not virtual reality that’s making the difference; it’s augmented reality.

“VR is all about a user being immersed in a virtual environment and getting the feeling of ‘presence’. AR is different in that it is about placing additional contextual information onto the real world,” says Daniel Anderson, a PhD student in computer science at Purdue University in Indiana and one of the increasing number of innovators pushing the boundaries of what’s possible with augmented reality technology.

Augmented reality systems like Google Glass overlay technical information on the everyday world - making our reality better. They also allow other people to see the environment from the wearer’s perspective, offering the perfect opportunity for training and development. The system was put to the test in 2014 when Shafi Ahmed, a surgeon at St Bartholomew’s Hospital in London was the first to live-stream an operation. Using Google Glass AR HMD, viewers were treated to a unique experience, with more than 13,000 students across 115 countries tuning in.

When they pick up the scalpel for themselves and start to carry out operations, trainee surgeons are traditionally supported by a mentor looking over their shoulder and providing advice. In certain circumstances reference books or information screens are also used to provide support. Processing all this information while trying to treat the patient severely stretches what psychologists call our cognitive load - essentially our total capacity for mental effort.

It’s here that AR can help. During surgery for instance, augmented reality systems enable mentors to overlay information on the patient, annotate areas and even call up relevant information, significantly reducing the cognitive load and, as a result, pressure on the surgeon.

In one exciting project at Purdue University, augmented reality systems are being developed for use in the most challenging of environments: the battlefield. Supported with a grant from the US Army, the System for Telementoring with Augmented Reality (STAR) uses touchscreen displays, transparent screens, tablets and colour and depth sensors to link battlefield medics with distant specialists.

In the heat of battle a soldier who has been shot will need urgent medical attention to stay alive, but with limited facilities and potentially limited skills, the field surgeon may be struggling. The STAR system will link the medic with an experienced mentor who can talk them through the procedure, providing real-time notes and annotations on a transparent screen, as well as giving personal support and encouragement. The surgeon sees the patient, the instruments and their own hands, but with the mentor’s mark-ups overlaid on them.

The system is much more sophisticated than just relating images, as Anderson describes: “We use computer vision algorithms to reposition the mentor’s annotations so that they appear anchored to the element of the operating field they describe.” In other words, if the organ is repositioned during the operation, the notes relating to it will move on the display.

While created for the battlefield, STAR has clear applications in civilian life too. “A general surgeon in a rural environment could use this system to perform urgent, life-saving care on a patient when transporting the patient to a major urban hospital isn’t feasible,” says Anderson.

Virtual Human

During surgery every interaction with the human body can have potentially life-long consequences for the patient. At the moment, AR systems like STAR and a number of others in development are able to reference the current environment, but what would happen is they could be used to predict the impact of any intervention by the surgeon?

Mario Viceconti is a professor at Sheffield University and director of the Insigneo Institute for in silico Medicine, which is contributing to a European project to develop the ‘Virtual Physiological Human’ (VPH). Over £200m has been earmarked to create realistic and reliable models of the human body that would provide accurate predictions for surgeons and clinicians working on some of the most challenging areas of medicine, including the neuromusculoskeletal system.

The challenge is enormous: not just to create the models themselves but to manage the processing challenge of calculating accurate and reliable results. It’s all part of a new branch of ‘in silico’ medicine, which involves the use of computers to trial new drugs or predict the impact of surgery.

Marrying the theoretical and the practical, it’s an area that Viceconti sees merging with the worlds of virtual and augmented reality. “Historically the models in real-time simulations are inaccurate. Ours are very accurate, but slow.” Conquering the computational task is simply one of processing power - something that has historically been tackled very quickly by manufacturers, he points out.

The models being developed by Viceconti and colleagues could provide surgeons with real-time feedback on the decisions they make. The challenge is a big one. “We need to merge our very accurate model with augmented reality, so a surgeon can ask ‘What will happen if I do this?’” It’s a step beyond what’s capable at the moment, but Viceconti imagines a world where a surgeon can use virtual models as reference points for all decisions made during surgery.

Take the example of a knee replacement operation. During the procedure, the surgeon needs to decide where to enter the body, cutting ligaments and tissues, as well as where to locate the prosthetic device. All this can have a real bearing on the patient’s future ability to work and function. The virtual human model could provide advice based on the actual physiology of the patient. “It could change lives,”says Viceconti.

The host of low-cost VR and AR devices being released and developed is creating a genuine public interest in immersive technologies. “I expect that now there will be a huge flowering of applications, that will seem a bit chaotic at first, but gradually settle down so that it becomes part of our lives as much as mobile phones,” says Slater.

Brinkman and colleagues see the technology increasingly being used in clinical settings, and have - in the open-source spirit - made some of their models freely available to anyone who might wish to try them out.