Augmented reality in the industrial sector

Imagine a factory environment where installation or maintaining of machines could be carried out via an iPad. Clumsy installation manuals and PC-based 3D simulation tools would become obsolete in favour of a portable version of reality straight from CAD diagrams. This could be possible with the new player in the industrial software arena: augmented reality (AR).

A common misconception is that AR is simply a sophisticated version of its predecessor, virtual reality. However, AR is actually a far more advanced technology.

There are four areas of function that must be considered to take full advantage of AR in the industrial sector. The first is the original image onto which a new version of reality must be augmented and the method used to capture that reality. The source of the image can be a still of the environment, a recorded video or a live stream, and can be captured via a multitude of optical technologies.

The second consideration is the source of the content you want to superimpose onto this reality, and through which medium. AR allows the user to overlay basic textured images on reality, or pixel-rich 3D CAD models from a local or external source such as a PLM system.

The third factor is the interface through which the augmented scene can be viewed. Most AR software for the industrial sector runs on a Windows-based, PC platform, but for portability the majority are being redeveloped to use on tablets, head-mounted displays and projectors.

The final element of AR is tracking. How often the camera captures the environment in relation to the object you want to analyse depends on the medium in which it is being streamed. A still image is a singular shot and is relatively simple to augment onto, whereas a live stream requires a more sophisticated mechanical tracking achieved through photographs at several shots per second.

But if AR is a suitable technology for the industrial sector, which markets are a suitable canvas for it to project its potential? The bulk of the market for industrial AR lies in maintenance. Christian Effert, director of Engineering Solutions at Metaio, estimates that 60 to 70 per cent of customer requests lie in this field due to the technology's aptitude for a hands-free visualisation.

Head-mounted displays, normally in the form of glasses with HD cameras linked to a control unit via cables, allow a service engineer to work hands-free with the reality augmented directly in their line of vision. Drawbacks lie in levels of performance, and also in that it is isolated. Engineers needing to engage in collaborative decision making would benefit more from using tablet, projection or laptop applications.

Prototyping tool

Large manufacturing environments produce a huge amount of digital data, particularly automotive manufacturing. For decades the automotive industry has relied on 3D simulation environments to process this data in a meaningful way to improve performance levels.

"It is useful if you can compare the digital version of your data with your actual manufacturing environment rather than having to refer back to a simulation system on your computer," says Daniel Gelder, senior vice president at Metaio. "For example, in plant planning you can augment the digital layer of the prototype into the actual prototype to identify any problems.

"A workshop environment allows engineering decisions to be made based on what is directly in front of you."

Automotive manufacturing is where Metaio cites its roots, with its founders starting their careers at Volkswagen and BMW. The company claims that every major automotive manufacturer in Europe is using some aspect of AR and VW was one of the first to adopt it. They have been using AR in their prototyping process for ten years, using the technology combined with a measuring arm to analyse steps in the design process.

AR now represents a mandatory stage in VW's production process. Gelder says: "Volkswagen has been using an AR service for their design and production processes successfully for several years to check the product against quality control for errors."

Projection is a simple technique that allows a user to reveal the inner workings of a machine to scale on the physical prototype. This method of augmentation has a whole host of potential applications, but developers claim its niche lies in training systems. Gelder says: "This works well as there are many people looking at one screen, so the projection becomes one large screen. An advanced tracking system means you can constantly drag the AR around the car prototype at a 360-degree angle if you like."

One area that AR lends itself to is machine assembly. Rather than working from manuals, virtual environments or fixed screens, which can be a relatively static way of working, AR can be transported around the factory via an iPad.

"Instead of having paper or digital manuals that are not really related to real physical environments, you can superimpose a step-by-step guide, which is the easiest way to follow instructions," says Gelder.

AR is also ideal for maintenance work in a legacy environment, which may not have been created using virtual systems. A legacy factory commissioned to build a product from its archives, for example, would not produce a maintenance model without CAD diagrams. Scanning the product in 3D creates a point-cloud that could be augmented against the reality of an existing product.

"AR presents engineers with a reliable tool to make their decisions. For a long time, the industry said 'everything has to be virtual', in CAD models, everything had to be drawn up in virtual environments. This is where AR comes in," says Effert. "You simply take a picture or stream live the new environment to find out what variations you have to prevent any failures in your planning."

Portability or performance?

Unfortunately, the trade-off for portability verses performance can become a sticking point with AR. Effert explains: "At the moment you have to choose between performance and portability; do you want to use a PC with a high-resolution camera or an iPad with a mobile camera?"

Tablet versions have been released which, despite being a more portable way to use AR, can only use internal sensors such as GPS tracking. PC versions use optical tracking in the same way as the tablet model, but are able to use external sensors such as measuring arms and improved HD cameras, which improves the accuracy.

AR applications are most commonly stand-alone, which means there is no need to network-integrate them into existing systems. However, this unfortunately means all CAD data must be imported externally.

Sensing the accuracy

AR's biggest stumbling block has always been vision. Critics have dismissed AR as 'inaccurate' compared to virtual simulation, rendering it useless for engineers. But Effert says AR software can be accurate to 1mm.

The performance of a capable vision system is crucial as it enables tracking of the object that reality is augmented onto. Historically, tracking was slow, stilted and inaccurate, so developers have begun to partner with vision system manufacturers to improve tracking capability. Metaio has partnered up with optical manufacturer Faro, who produce measuring arms to help improve the accuracy of tracking, creating a smoother, more accurate stream when augmenting over a live video.

The position of the camera to the real-life object is also crucial. Measuring arms feature a probe-tip that feeds real-time information back to the AR interface from the environment in front of it. This is calibrated with external sensors that tell the augmented application the location of its environment. Effert says improved tracking lies in collaborations with camera industry experts to improve optical sensing.

"Previously, you had to 'teach' algorithms the objects you were trying to track, along with a fixed application that monitored changes," he says. "If the user changed, you also had to change the application. New applications are geared towards the user having to do less and less preparation; you can simply hold the program up to the object and get the super precision that is necessary for automotive manufacturing. We are always trying to improve tracking for the accuracy engineers need."

The importance of exceptional optical sensor performance is evident when 3D CAD drawings are fed into a computer vision library, which can then detect objects in the camera image based on point clouds and markers. "The most important thing good AR software does for the industrial market is to connect optical or mechanical sensors, such as optical, laser scanning or measurement, and take this positioning data to allow a user to overlay digital CAD data into the live view. This allows you to visualise the CAD data on top of the real manufacturing environment."

The evidence remains to be seen whether AR is really a viable replacement for the favoured virtual reality systems that have become so commonplace in an industrial environment, or whether it is simply a dressed-up marketing tool. Can AR really become a better solution than existing PLM systems? Gelder says perhaps.

"AR will improve productivity in that it will save precious time and therefore costs in relation to decision-making, and essentially improve quality. This is primarily what AR brings to the table. Our real challenge now is to convince people to change their daily work practice; we need to convince the industry that AR is a tool that could really help them and not just a gimmick."