Synchronising data with ethernet and industrial control systems

Earlier this year, Bosch Rexroth, Cisco, Intel, Kuka, National Instruments, Schneider Electric and TTTech agreed to set up a testbed to work on a way of keeping industrial control systems in sync as part of programme for the Industrial Internet Consortium (IIC) to develop technologies for Industry 4.0.

The testbed uses modified Ethernet network equipment to solve a looming problem for organisations that want to make their industrial-control systems work together across the internet - a goal of Industry 4.0 proponents. To get there, they are adopting extensions to Ethernet that have their beginnings in work to keep audio and video running smoothly in theatres, stadiums and recording studios.

As part of its plan to replace traditional coax cables with Ethernet, Gibson launched a guitar with a ruggedised computer port from technology supplier NetworkSound embedded in its faceplate. Using the digital network and the MaGIC protocol developed by Gibson Labs, each pickup under the six strings of the HD.6x-Pro guitar could send signals independently.

Back in 2003, Barani Subbiah, founder and CEO of NetworkSound explained: “After running out of the Ethernet port and into the eight-output breakout box, you can then use split mode to assign each of the six strings to a different amplifier.”

Guitarists did not find many uses for a guitar with separate audio outputs for each string and the Ethernet Les Paul faded away, but the idea of bending Ethernet to send audio reliably remained. The advantage of the MaGIC protocol over previous attempts to send audio over Ethernet was its relative openness. The guitar maker offered the protocol under a royalty-free licence, in contrast to competitors such as Cirrus Logic’s CobraNet. The company worked with Broadcom and others on proposals for an audio-capable Ethernet for home networks in the IEEE, which manages the steadily growing portfolio of Ethernet variants. The work slowly mutated into the Audio-Video Bridging (AVB) standard.

As with the digital guitar, interest in home-audio Ethernet waned. As that market became more elusive, car manufacturers discovered they needed a faster in-car network than their mainstay, the Controller Area Network (CAN). It would take ten hours to flash the 81Mb of firmware inside a fourth-generation BMW 7-series vehicle using CAN. Ethernet cut this time to a matter of seconds, easily coping with an increase in firmware to 1Gb or more.

With Ethernet adopted as the bridge between the factory and the electronic control units (ECUs) under the bonnet and the infotainment systems in the dashboard, the car makers began to work out ways in which they could work with the audio extensions developed by the IEEE to support not just digital speakers sitting in the doors but the messages sent between ECUs.

Lars Reger, automotive CTO at NXP Semiconductors, says a one-chip radio receiver can be connected to the car using Ethernet: “We are taking out €15-20 of copper cabling using these intelligent receivers.” The same network interface can be used to bring in the signals from radar units mounted around the car to spot obstacles and hazards. The network will also relay messages from vehicle-to-infra­struc­ture (V2X) communications.

“With V2X, you can see obstacles not only in front of the next car but in. I can see around corners long before I see or hear an ambulance coming the other way. And I can see the colour of traffic lights,” Reger says, adding that Ethernet can extend down to the brake-by-wire and other critical systems.

“Deterministic Ethernet is very much needed for x-by-wire technologies: steer by wire, brake by wire. The last thing you want is the car saying ‘sorry Lars I’m dealing with the entertainment systems, I will brake in a few seconds’.”

Industrial users are coming to the same conclusion as the car makers as they upgrade their systems. Historically, industrial networks and IT networks were kept separate. They even used different names. Most of the early digital networks took on the moniker of ‘fieldbus’, partly to reflect their limitations. They were generally designed for cabling simplicity, with connections from a common backbone bus being tapped off to individual sensors and programmable logic controllers (PLCs). Many users adopted a factory-oriented version of CAN, taking advantage of the huge volumes of controllers made for the automotive market, in addition to dedicated industrial protocols like Profibus and HART.

As with the in-car network, the need to move larger and larger quantities around the factory presents problems for messages that absolutely, positively have to reach the destination on time - whether to stop a robot punching a hole in the roof of a car it’s assembling or moving defective items off a production line after a machine vision system has flagged up a problem.

Machine-vision cameras create the need for higher bandwidth communication. Cameras from manufacturers such as Sony now sport gigabit Ethernet ports on their back panels to deliver data at frame rates of a hundred per second or more. If these networks were used purely for such data, there would not be an issue. However, the factory operators want to be able to have their back-end systems reach into the operational heart of their systems and pull out statistical data as well as having the convenience of working with a single type of network technology.

Paul Didier, industry solutions architect at Cisco, explained at NI Week in Austin, Texas in August that the issue is one of convergence: “We see IT networks connecting people to data at home and at work and we see OT [operational technology] networks controlling machines connecting plants and grids to each other. The cables and connectors in these networks might look the same but the objectives and the underlying technologies are quite different. To deliver the promise of the industrial IoT we need to converge OT systems and the IT infrastructure.”

In a demonstration at NI Week, Didier together with Kevin Stanton, senior principal engineer at Intel, showed how high-bandwidth traffic such as video mixed with command traffic could cause problems for real-time control. “The video traffic causes some of the control frames to be delayed past their deadlines with some of the frames being dropped entirely,” Stanton pointed out.

“Had this been a real working machine this would potentially have caused damage to the machine and certainly would have damaged the parts that it was making,” added Todd Walter, chief marketing manager at NI.

By switching on a time-​synchronisation networking (TSN) option within the Ethernet switches, the command traffic was sent over a different virtual channel from what were, in the demonstration, less time-​sensitive video packets - a reversal of the original plan behind the IEEE’s work on AVB.

“We combine time synchronisation with traffic management to deliver latency guarantees. Each switch [that handles the packets] identifies the time-critical and places it in a special queue. The switch then forwards each of these packets at specific times. What this does is create a virtual wire between each of the talkers across the entire network,” Didier says.

As well as long-term communications suppliers such as Cisco, start-ups are springing up to deliver real-time Ethernet for control systems. Nebbiolo, the target of investment from both Kuka and TTTech, aims to act as the link between real-time Ethernet in the so-called ‘fog’ layer of systems that directly support the industrial control systems and cloud-based software. By accessing real-time data passing between machines in the fog layer, software running on fast computers in the cloud can detect and flag up problems that may arise from machinery going out of tolerance.

The semiconductor industry has, for many years, been an aggressive user of real-time monitoring. Tiny flaws picked up on test equipment that probes finished wafers can indicate conditions deteriorating over time and be used to halt production or tell operators to check on equipment. Optimal Plus aims to expand the scope of its big-data analytics software out to the wider electronics assembly market.

Michael Schuldenfrei, CTO of Optimal Plus, says problems that its software detects “can stop a board moving on down the line. We can make the test process more intelligent than it has been up to today”.

Living in the cloud provides the ability for manufacturers to look across much longer time horizons than is typical today to help identify situations where productions needs to be adjusted quickly that might not show up in the pure real-time data.

“Manufacturing data is seen as increasingly valuable. They want to have the data for three, four, five years,” says David Park, vice president of worldwide marketing at Optimal Plus.

The question of where the analytics software that works on the accumulated data will run will depend on the degree of responsiveness and time criticality the algorithm has. “We’re going to need to process the data at every layer,” says Mark Potter, CTO of HP Enterprise.

“There is this notion that more and more of the compute and the trained models we use have to be reacting with real-time response. The question is how we scale so we can place the compute where it is needed. That is why we have ruggedised systems with 64 Intel Xeon cores so they can run algorithms at the edge.

“They have exactly the same architecture as systems running in the data centre and the cloud to provide the scalability. We will need edge, data centre and cloud architectures to come together.”

The link for them is a network that knows what time it is.