The real problem in designing robots that can read emotions is understanding humans.
If the broadcast of Channel 4 series 'Humans', featuring Synth character 'Anita', is anything to go by, the idea of humaoid robots emotionally interacting in society has entered the popular consciousness. Where there is awareness, there is of course a market, and companies are beginning to tap into it, either in creating robots for support and companionship, or to sell more products.
Last year, tiny startup Emoshape came close to doubling its Indiegogo funding target of $100,000 for a glowing cube that the company claimed can detect and respond to emotions with the help of a webcam. At the end of February, French company Aldebaran's foray into consumer rather than laboratory-research robots sold a limited batch of 300 in a matter of minutes to Japanese buyers.
'Pepper' the $2,000 wheeled robot was originally used by SoftBank, now the 95 per cent owner of Aldebaran, in some of the company's stores in Japan as a greeter. But both SoftBank and Aldebaran are keen to see it move into homes.
Vincent Clerc, head of Aldebaran's mechatronics lab and the lead on the project to design Pepper, explained at the DATE conference in Grenoble: "Our vision is to develop interactive robots for the wellbeing of humanity. We want the robot to be a real companion in everyday life, something that is usable by anyone.
"We want people to be emotionally connected and involved with robots. We don't want a robot to be a simple machine like a vacuum cleaner. To be connected, we have to detect intentions. We have to detect what the person is thinking." Clerc adds: "That is why they are humanoid robots. They provide the ability to have very natural and stressless interaction with each other. It's all natural. If you move your head in a certain way it means something. If you are tired, you look tired. And you want a robot that recognises when you look tired."
Engineers are tackling the problem of creating a robot able to decode emotions in several ways. At Aldebaran, one technique is voice-stress analysis. Playing a video of someone speaking nonsense in turns angrily and happily to underline the point, Clerc says: "This person is just expressing emotion without saying anything. But the robot will be able to deduce what is your state of mind, whether you are angry or not."
So far, video is the main source of data for emotion-processing machines. "With body language, the robot can see whether you are tired or apathetic or have more energy in the body," Clerc says.
Cameras have the advantage of being practically ubiquitous, providing existing devices such as phones, tablets and PCs with the ability to detect an angry or happy user.
Madmen or Sadmen?
In the commercial environment, video is driving a growing market for testing advertisements for their likely effectiveness. "Advertisers will pay a lot for this data. They want to know if you are just smirking at an advertisement or really smiling. Very often they are wrong," says Professor Rosalind Picard of the Massachusetts Institute of Technology, a 20-year veteran of emotion-detection research.
For Picard and Affectiva, the company she founded at the end of 2009 with fellow researcher Rana el Kaliouby, the move into advertising was partly a means to an end: to collect more real-world data where the researchers could compare the facial reactions to the emotions that participants said they felt. The cues are frequently extremely subtle.
Picard's team found that genuinely happy smiles and those from people suffering from frustration are surprisingly similar. The Affective software looks for combinations of 'action units'. These are codes that home in on specific movements of the facial muscles, such as 'AU4' – a furrowing of the brow during a frown of concentration.
Picard's group told subjects of the experiment they would be testing the user interface of some new software – and it would throw up frustrating problems. But many of them smiled in their frustration showing the classic 'AU6' cheek raise and 'AU12' pull at the corners of the lips. Out of context, it could easily be a smile of happiness, but it isn't. After training the software on the tiny differences between frustrated and happy smiles, it finally began to outperform humans, she says. Advertising has provided the funds to build bigger trials allowing the team to collect orders of magnitude more data.
"We had to take this circuitous path through advertising to get the data," Picard says.
Through Affectiva, the software to recognise facial movements can now be embedded in smartphone and tablet apps as well as embedded systems through a software development kit (SDK). One result was a 'smile sampler' designed for US sweet maker Hershey. Designed for use in stores, the sampler dispenses a small block of chocolate in return for a smile.
Eyeris is another company that grew out of TV analytics, using its EmoVu player to track predicted emotions as videos play. But the company has found recently that carmakers are keenly interested in being able to detect emotions using cameras mounted in the dashboard or under the rear-view mirror.
According to Eyeris CEO Modar Alaoui, car makers are looking to use the technology for safety: to detect when drivers are not paying attention to the road or suffering from road rage. Researchers have suggested using technology like this to divert incoming phone calls to voice mail if the driver is showing signs of stress or increased concentration to avoid distracting them while driving in difficult traffic conditions.
Although training on increasingly large data sets has reduced the confusion from similar facial movements with very different meanings, some of the best clues to state of mind may not come from facial expressions but from body-worn sensors – and they could save lives. Arup experimented with body-worn sensors over a decade ago to diagnose sick buildings, using the same kind of sweat-based stress test used in classic lie detectors to estimate discomfort.
Picard's group has found intriguing ways in which the body's sweat response to changes in the brain can provide potentially life-saving results. A chance decision by a student to put electronic bands on both wrists of a relative with epilepsy showed that the skin conductance can differ dramatically. Twenty minutes before a grand-mal seizure, the conductance on one wrist spiralled upwards while the other did nothing. Since then the group has found other ways in which the differences in the two sides of the amygdala – a key part of the brain for processing emotional responses – can result in changes in sweat on either side of the body.
The result of the work is Picard's second startup, Empatica, which closed a crowdsourcing fundraiser with five times more money than the initial target earlier this year. For each sensor wristband it sells, it will donate one to a child suffering from epilepsy. External stimulus can prevent sudden death during seizures. The idea is that the band triggers an alarm that can be used to alert someone close by. Potentially, it could alert a robot to attend to the sufferer.
Up to now, the signal-processing software used for emotion processing has been designed to run on conventional microprocessors and graphics processor units (GPUs). The most time-consuming part of the deep-learning techniques used in many detection systems, such as those used by Affectiva and Eyeris, are today run on high-end cloud servers. The trained networks are then downloaded and run on the embedded chips in smartphones.
But the rise in interest in deep learning for an array of applications, from processing the input from cameras on cars to watch out for potential hazards to emotion detection, is leading companies to build compute engines that are more efficient at handling the algorithms.
Karim Arabi, vice president of engineering at Qualcomm, which makes processors for smartphones, says the company is working on a 'neuroprocessor' engine that will be more power efficient and so more suitable for portable devices than conventional processors. "The next decade will be quite active with all these new technologies," he says.