Hovermap lidar creates detailed images of inaccessible areas

Digging deep: mining equipment technology

Image credit: Hovermap

Brisbane, Australia, is a global hub of innovation in the mining industry, with autonomy at the heart of the latest developments.

Extracting vital mineral resources from underground has never been easy, so the mining industry relies on technology to improve productivity and economics, and enhance safety in the notoriously cyclical business.

Partially autonomous mines are the latest trend. In Western Australia, the Solomon iron ore mine operates a fleet of giant Caterpillar haulage trucks that move mineral-bearing rocks with minimal human intervention. The result has been productivity increases of up to 30 per cent, an increase in equipment utilisation, a decrease in equipment damage, and a removal of at least part of the workforce from the hazardous working environment.

“We are years away from a fully autonomous mine environment, but we are investing heavily in research to tackle many of the technical issues that we will need to solve before we get there,” says Nigel Boswell, product development engineer at Caterpillar Inc’s Australia Research Centre in Brisbane. “Cost-effectively tracking the position of all of a company’s mobile assets above or below the ground is key.”

Current underground localisation solutions are either bulky and heavy and not easily transferable to lighter vehicles, or involve the installation of many expensive beacons throughout the site that the vehicles can reference and use to navigate around the mine. In the search for a low-cost solution Caterpillar is working with Queensland University of Technology (QUT) to develop a versatile camera-based approach inspired by rodent vision.

Most robotics labs are full of toy robots, but Michael Milford’s desk is presided over by a large brown rat. Milford is a world leader in robotic vision research, based at QUT, where the workings of the rodent brain are inspiring new developments in machine vision. “All around us animals like rodents are capable of amazing feats of navigation and perception. They don’t have the largest brains or the best sensors, like the best eyes, but are capable of understanding their world, finding their way around it, gathering food and evading predators,” explains Milford.

“Most animals don’t have range sensors, nor do they store millimetre-accurate maps of their entire environment. Most mammals store maps at different scales of their world and the brain combines these to navigate. This is how we are designing our robotic vision algorithms.”

Milford’s findings were surprising to the whole robotic vision community. “Robots do not need perfect visual input from expensive high-tech cameras. Inexpensive low-​resolution cameras with sensors of only a hundred pixels are enough to navigate effectively in a mine,” says Milford. Low-res images also need less processing power to interpret, so have lower energy needs, a bonus when the underground environment does not allow you to offload any processing to the cloud.

Mines are challenging environments for electronics. Cameras and sensors have to be protected from dust, dirt and moisture, and enclosed in sealed, spark-proof containers. Complete darkness can be punctuated by dazzling lights shining from other vehicles or fixed lighting, and dust clouds can also create navigational problems. Internal passages and repetitive features like tunnel roof supports may appear very similar to one another, and of course there is no GPS underground to help out. The QUT team are training their algorithms to assess the usefulness of images by discarding those that might confuse the AI, for example, images containing glaring lights or obscured by dust clouds, and integrating information over time to better estimate the vehicle’s position.

Milford’s co-lead on the underground localisation project is robotic researcher Thierry Peynot. The project’s first phase is camera-only trials to deliver vehicle positioning accuracy to about a metre. “We are trialling prototypes on large mining vehicles in Queensland and New South Wales with the goal of having one-metre positioning accuracy in about a year from now,” says Peynot.

The second phase is now running in parallel with the camera-only tests, using multi-sensor systems combining lidar laser range finders with camera input to achieve the centimetre-scale precision needed for future autonomous operation.

Currently the cameras on the test vehicles are sensitive to normal light, but the team is looking at adding near infrared capability which may be better at clearer discrimination between very similar environments such as tunnel walls or rock faces, which can be hard to tell apart. It may also help in volumetric assessment.

“It is often important to know how much material has been removed,” says Peynot. “Broader-spectrum camera sensors may also ultimately enable qualitative assessments where you need to know what type of rock or mineral is being worked.”

Success may ultimately lead to progress towards a holy grail for the mining industry – the underground extraction and processing of ores and minerals rather than their removal to sort at the surface.


Seabed minerals

The deep ocean contains metals from silver and gold to copper, manganese, cobalt and zinc, often in concentrations greater than that of land-based mines. Extraction of these minerals is controversial, as the effect of disturbance and sediment plumes on vulnerable abyssal fauna is as yet unknown. The impact on the wider marine ecosystem could be significant.

The likely first commercial venture to extract minerals from the seabed will be Russian and Omani owned, Canada-based, Nautilus Minerals, which has ambitions to mine gold and copper deposits in Papua New Guinea territorial waters. As early as 2019, 200-tonne remotely operated mining machines supplied by SMD in the UK could be trawling the seabed in the Bismarck Sea in waters over 1.5km deep.

Seabed mining is banned in international waters, but the International Seabed Authority is developing rules to govern the exploitation of the deep ocean outside national jurisdictions.

An hour away from the city riverside campus of QUT is the Queensland Centre for Advanced Technologies, (QCAT), where Stefan Hrabar is the principal researcher for the Hovermap project.

Hrabar is using an adapted commercially available drone to map stopes – voids left after excavation – in unprecedented detail with the help of a 2.2kg payload consisting of a laser-based lidar range sensor and three cameras. Hovermap also provides the drone with collision-avoidance capabilities to keep it safe while navigating autonomously.

Stopes are too dangerous for humans to enter, so the conventional way of mapping them is to place a scanner on the end of a very long pole. This requires surveyors to work close to the stope entry, which is extremely dangerous if the stope is newly created by blasting with explosives, or very old and unmapped in recent times, which means in either case that it could be unstable.

The Hovermap team has just conducted the world’s first fully autonomous beyond-line-of-sight drone mapping trial under­ground in a number of stopes in Newcrest’s Jundee gold mine in Western Australia. In this trial the drone followed a set of waypoints, but the next step is autonomous exploration. “We hope to demonstrate the first truly autonomous drone explorations of underground mines in the first half of 2018. Nine of our Hovermap systems are already out there in the wild, working for early adopters in Canada, Japan, China and Australia. What excites these companies is the detail that we can show in the point cloud maps generated by the onboard lidar,” explains Hrabar. “We can use false colour to show the difference between reflective and non-reflective surfaces and identify the shine of silver, zinc or lead ores, all without endangering a human miner.”

Aside from ore identification in working mines, drone maps like these can help mining engineers assess just how much rock they have blasted out, and precisely where, and assist in planning the next blasting operation. In abandoned parts of working mines or in old mine workings accurate mapping can make it easier for companies to restart operations more quickly if commodity prices rise, providing better flexibility in this highly price-sensitive business.

Also at QCAT, research engineer Mark Dunn’s Energy team has developed a robust and lightweight assisted-reality headset. Assisted reality overlays sensory inputs such as sound and graphics on a real-​world situation, in a similar way to augmented reality but where the user remains fully aware of the real world around them. The headset links a local operator at the mine site with a remote expert, providing two-way audio and visual feedback via a head-​up display. The local operator retains their full awareness of the surrounding environment, critical when working in a complex industrial area.

The system is being deployed in many industries from aerospace to defence, but in remote mining sites it allows a technician or vehicle mechanic to connect with a remote supervisor or expert anywhere in the world who can offer advice on maintenance or repairs via a Windows interface.

“Using this technology, remote experts can use their hands to direct local operators more naturally, actually pointing to and showing them which tools to use to solve a problem. The expert’s experience is available instantly, providing rapid solutions to problems, boosting productivity, and assisting with training local staff,” says Dunn.

The headset development has been handed over to Australian company Fountx and was made commercially available in 2017.

The headset and other remote teleoperation technologies start to reduce the number of employees who have to be on site in often dangerous and repetitive jobs. Skilled workers will increasingly be able to live and work away from the often extremely remote areas where the mines are located, cutting the costs of transporting workers to distant sites and accommodating, feeding and entertaining them while they are there.

‘Whether a completely autonomous mining operation is 10, 20 or 30 years away is hard to say, but the level of research investment going into the sector and the ability to leverage up on the billions going into autonomous cars will ultimately make it a reality.’

Professor Ross McAree, University of Queensland

Professor Ross McAree, and his team in the School of Mechanical and Mining Engineering at University of Queensland (UQ) has been involved in driving complex mining machinery down the road to full autonomy for many years, often in collaboration with the equipment manufacturers.

One of the team’s current projects is the choreography of multiple autonomous bulldozers to remove the rocky material sitting over a coal seam in open-cast mines, known as pivot-push dozing. The research is a collaboration with Caterpillar and the Australian Coal Association and is currently on trial at the Peabody Energy coal mine in Wilpinjong, New South Wales. A team of machines is working together to reveal the coal seams below the surface.

“One of the challenges was that industry experts could not agree on the optimal approach,” comments Prof McAree, “Each had a preferred method based on personal experience. The lack of consensus required us to determine the best method, which turned out to be a combination of approaches depending on the topography of the pit and the properties of the dirt to be removed. I liken it to playing chess: you have to understand your immediate context and think through future moves. The AI at the heart of the system constantly optimises the moves of the bulldozers to achieve the best future outcome.”

The bulldozers operate autonomously for the majority of the time but each working group of machines is supervised by a human operator who can step in to operate them remotely from a nearby control room. Human intervention is only needed for exceptions: tasks outside the present autonomous capability. Examples of exceptions are where the dozers change from one section to another, remove a very large rock from the work site, and travel to and from the work site. The UQ researchers aim to include these exceptional tasks in autonomous operations in the future.

The supervisory control interface includes directional and steering controls, blade positioning control and foot brakes. The operator receives information about the bulldozer’s immediate environment via cameras on the machine, a plan view, and cross-sectional views of the machine within the work area. A microphone on the dozer also transmits sound to the control station. A single worker can supervise several autonomous bulldozers at once, obviously increasing productivity, and will usually be located in an onsite control room. Theoretically the bulldozers could be supervised from farther afield, but the availability and cost of a robust radio bandwidth limits this.

The project has established that it is possible to operate teams of bulldozers on this complex task at productivity rates that are close to the theoretical maximum. Prof McAree can envisage a future where complete autonomy can be achieved: “Whether a completely autonomous mining operation is 10, 20 or 30 years away is hard to say, but the level of research investment going into the sector and the ability to leverage up on the billions going into autonomous cars will ultimately make it a reality.”

There is no doubt that mining automation will be highly disruptive to the industry, but the expectation is that it will lead to greater efficiency and productivity. Buffeted by the boom and bust cycles of commodity prices, mining companies have strong incentives to embrace any technology that can drive down costs. Rio Tinto has just passed a major milestone – autonomous trucks have moved a billion tonnes of material in its iron ore mines in the Pilbara in Western Australia.

The autonomous trucks operate for far longer than driver-operated trucks – over 700 additional hours annually on average, cost 15 per cent less per unit haul, and there have been zero injuries associated with the autonomous fleet – a significant safety advantage. Already 20 per cent of the fleet is autonomous, and the company plans to reach 30 per cent in 2018 by retrofitting existing trucks, all controlled from a central station.

The gradual removal of human miners from a dangerous work environment where occupational health hazards abound can mean an upskilling of the labour force: former drivers of giant machines become controllers and supervisors in an air-conditioned central office. Inevitably there will be fewer jobs in an automated mine, perhaps as much as 50 per cent fewer, but the automated mine will operate 24/7. In the future mine, there will be less downtime. For example if there are no humans underground, then there is no need to ventilate after blasting to clear the air, and of course autonomous vehicles and robots do not get tired or need breaks. A future mine may even have zero ventilation.

Ventilation is a major consumer of energy in underground mining. The Kankberg gold mine in Sweden found that by introducing automation and optimisation with an ABB demand-controlled Smart Ventilation system it saved 54 per cent of ventilation energy and 21 per cent of air-heating energy.

Without human miners, the design of tunnels need only be large enough to accommodate a vehicle, which themselves could be smaller, without the need for a cab. Old workings with dangerous gas concentrations could be reopened. Areas where collapse is a real possibility could be left to the robotic miners of the future. Automation and robotics could allow the exploitation of large and deep but low-grade deposits that are currently uneconomic.

The same technologies being developed for the smart mine of the future can also be applied to open up controversial mining of the deep ocean sea bed [see box].

Further afield, Nasa’s Jet Propulsion Laboratory believes that robotic, automated mining on the Moon could support a lunar colony, and source fuel and materials for Mars expeditions. Mineral resources so necessary for our electronics, like the rare earth metals, are finite on Earth but abundant on the Moon. Even further in the future, there is the prospect of cometary mining.

Recent articles

Info Message

Our sites use cookies to support some functionality, and to collect anonymous user data.

Learn more about IET cookies and how to control them