More than a techy fad or geeky fashion accessory, augmented reality could soon have a serious impact on the defence, medical and education industries.
As children, we come to understand our senses little by little – realising that heat or cold can equal pain, that a raised voice signifies alarm, that jelly and ice cream bring pleasure.
Augmented reality (AR) is now at a stage of development across many different disciplines equivalent to that of an infant. We are taking our first tentative steps. We understand some applications more completely than others. Yet there is no doubt that, within a few years, AR will come to play as fundamental a part in our lives as the five conventional senses do today.
"I think that augmented reality will explode as a form of the 21st century," says Chris Dede, Timothy E Wirth Professor of Learning Technology at Harvard University. "There will be an explosion of opportunities with the technology."
Professor Dede is among the pioneers of the technology from his Cambridge, Massachusetts base. With a broad remit across Harvard departments to develop new learning technologies, he has spent a decade researching and monitoring the growth of AR. What excites him most? Three things...
"It enables learning to take place in context: with AR, context is part of learning. You can be in a real ecosystem, you can see real animals, tractors etc, and they become part of learning in a way that the classroom can't.
"Second, it enables transfer. Even excellent students struggle when they get out into the real world, they have trouble transferring learning. With AR, this transfer becomes 'near' rather than 'far', because you're in the real world, but you have a scaffold from AR to help you.
"Third, unlike virtual worlds, which are expensive and technically difficult to construct, AR is relatively inexpensive and easier to author. If you have a good design, it's fairly straightforward. I can teach a middle-school student how to author AR."
Professor Dede argues that, in future, we can expect to interact with the real world through annotated AR systems, learning about the history of a churchyard or the goods in a shopping mall, for example. Although not yet a routine part of teaching and learning at Harvard University, AR technology is fast spreading through the US school system. One of his own projects, EcoMUVE, has received finance from the United States National Science Fund and explores virtual environments, studying how an epidemic can spread, for example.
"AR is where virtual reality was in 2002," says Professor Dede. He sees particular potential for the technology to support 'lifewide' learning, one of the National Education Technology missions in the US, thanks to its accessibility and its relative cost. "It is no silver bullet, but it broadens to ecology of educational opportunity."
This is particularly relevant in emerging economies, which lack the resources to build the thousands of schools and hundreds of universities they would need to replicate first world education conditions. Instead, they have mobile technology, a perfect platform for AR. "Many powerful stories are coming out from the developing world," says Dede. "It has huge implications."
At Kendal College in the UK, Dan Hodge helped to develop an AR application for the college prospectus. Further projects have sprung from this beginning, including a 'living learning' programme, with financing from JISC (the Joint Information Systems Committee – a government organisation to support post-16 learning, using new technology).
One programme helps trainee plumbers, using mobile phone applications to play videos demonstrating how to solder a pipe, for example. JISC awarded Kendal College an innovation prize in 2012 for its AR prospectus.
Launched in October 2012, the application had just £20,000 worth of funding, a tiny amount compared to the millions typically thrown at online IT projects. For Hodge, the technology offers students the chance to receive information and instruction in the place they most need it, with a person (virtually speaking) who has taught them before.
"We're living in a world where people are used to audio visual media: young people need to do two things at the same time to remain occupied. It's an enriching experience," says Hodge. He points out that 78 per cent of Japanese consumers use 'quick read' codes via their smartphones. The equivalent figure in the UK is 26 per cent, but he believes that within two or three years, AR systems will become standard in Britain, as they have elsewhere.
"In the next two years, everyone will have a smartphone and will need to understand AR, because we'll all be using the processing power of our phones," says Hodge. A recent study by Semico Research predicted that by 2017, revenues from AR applications will total more than $600bn, while in 2014 more than 860 million mobile phones will be able to handle AR applications, along with 100 million vehicles equipped with the technology.
The gap between the highest spec technologies, developed at huge cost for sophisticated professional users, and those available 'off the shelf' for mass consumption, is rapidly narrowing.
The UK's Prince Harry, interviewed following his tour of duty in Afghanistan, said of his job as an Apache helicopter pilot, firing shells at the Taliban: "It's a joy for me because I'm one of those people who loves playing PlayStation and Xbox."
The Taliban took a dim view of this nonchalance in the face of death and destruction, but it touched on a vital issue in augmented reality. The best preparation for military life in the 21st century may indeed be playing on computer games, rather than the playing fields of Eton. Prince Harry, of course, has both.
'Heads-Up' display goggles, allowing a pilot to receive detailed information and graphics on enemy (and friendly) positions, on their equipment and premises, are now in widespread use. The latest advance is AR contact lenses, allowing users to see such information without the hindrance of large glasses or goggles. 'Google Glass', developed by the search engine giant, are soon to bring a version of this technology to the masses.
At the University of North Carolina, Henry Fuchs is busy puzzling out the future of AR across a dizzying range of fields. He believes that an extended and advanced version of Google Glass, extending the range of the technology to the whole field of vision and allowing users to transmit what they can see to others, will come to challenge (or even replace) what we know as teleconferencing today.
With applications in education and business, he's researching what he calls 'telepresence' in collaboration with universities in Zurich and Singapore (a triumvirate in clear need of technological assistance for communication).
"I don't think we've seen much of the life-changing applications of the technology yet," says Fuchs, whose official title is Federico Gil Distinguished Professor of Computer Science. "It's like asking someone in 1980 what the effects of personal computing will be." He believes that personal interaction will change most profoundly, along with our experience of the world around us.
"We'll see castles which once existed superimposed onto buildings, waves of air flowing from conditioning units, the beating of birds wings." He makes it sound like a rather wonderful and beautiful world.
With his fingers in several AR pies, Fuchs has studied military AR applications and anticipates that areas such as night vision and 'seeing' what lies behind walls will be among the most revolutionary uses of the technology.
Augmented reality goes to war
Gathering information by electronic means has long been a crucial part of military intelligence, credited with winning (or at least shortening) several wars. Augmented reality is the latest technology to be employed in this field, with an ever widening importance and scope.
Unmanned drone aircraft now crisscross the skies above theatres of war, relaying information back to mission control. This can now be passed on to armed combatants, in the form of AR technology, into their goggles or through other equipment. German technology has been employed to enhance battlesuits, providing high-tech protective gear, wired up with communication and navigation aids including helmets with AR display systems, night goggles and thermal imaging equipment. Developed by the German company Rheinmetall, the system is known as Infanterist Der Zukunft ('Future Solider') or Gladius – Latin for sword. It offers protection from biological and chemical weapons, and from 'detection' – although it is not invisible.
For military operations far into the future, look no further than the headset being developed by the European Space Agency to enable astronauts to perform medical procedures without recourse to doctors back on Earth. Spaceships could sail into the far yonder, equipped with this space-age headset, using visual aids, voice control and an intuitive interface to enable the astronaut to diagnose conditions and use technology such as ultrasound to detect trauma, breakages and internal bleeding for example.
This technology would be most helpful in the case of 'Star Wars', but is simply an extension of the existing and exponentially growing field of medical AR applications, from internal explorations through surgical interventions and brain-scanning technology. Recent innovations include Simultaneous Localisation and Mapping (SLAM), which promises to map the expected path of a patient's disease, so that they can become calmer about their future medical expectations and the surgery they will need.
At the University of Chicago, students of neurosurgery are learning to use pens attached to robotic arms, manoeuvring virtual instruments to simulate 'haptic feedback'. They consequently gain a sense of space and resistance while using medical tools.
For South African computer scientist Mark Shippen, a case of malignant melanoma on his neck prompted him to research preventive measures, leading to Doctor Mole, a smartphone app which checks your skin for signs of cancer using AR to detect irregularities that may appear over time. The app archives and compares data and reminds the user when it's time to check a mole again.
At the more professional end of the scale, the VIPAR (Virtual Interactive Presence and Augmented Reality) platform enables surgeons to get help from one another using a surgical field, blue screen and digital viewpiece with two cameras. The local surgeon conducting the operation and the remote one using a virtual field can collaborate on complex surgical procedures, allowing instant transfer of knowledge and experience.
A free iPad app called Anatomy 4D, developed by Augmented Dynamics, was launched in late 2012, letting you explore an augmented reality 3D body. When the webcam detects special markers, it replaces the main image with a superimposed 3D image of a certain part of the human body, such as the digestive system.
Medical AR is especially valuable since it allows surgeons and other medical professionals to gain virtual experience of conducting procedures, armed with the best possible amounts of information, without going near an actual patient. And then when they do operate on patients, they have a wealth of assistance from technology to improve the chances of success.
Medical AR technology
With so many medical interventions taking place inside the body, or through keyhole surgery for example, without the benefit of clear natural sight, such advances are extraordinarily helpful. "The ability to see more of the inside of a patient, for diagnosis and therapy, will improve," says Fuchs at the University of North Carolina. "At the moment this is quite limited, because the displays are so cumbersome," he says.
New equipment such as transparent screens, which can be placed in between a doctor and a patient, displaying information and graphics about the person's condition, are in development, while a combination of visualisation and location tracking technology, enabling a surgeon to pinpoint and see exactly where to operate, is progressing fast.
"The situation is like mobile phones and cameras," says Fuchs. "As one technology improves, so people are willing to pay more for the other. The same is true in medical AR technology, with ultrasound and location technology improving in tandem.
"For those of us who live another 30 years, this era is likely to form a nostalgic and entertaining period: partly ignorant, partly excited by the possibilities and partly heading down blind alleys, it will be seen as the dawn of a technology that fundamentally changed the way we live and how we view the world.
"We still don't know how, but like youngsters, we're learning fast."