When the care robot recognises feelings

He is 60 cm tall, has cuddly fur, looks like a seal and lets residents cuddle his fur: Paro is a robot that lives and works in the three day care facilities and two outpatient assisted living communities of Ambulante Krankenpflege in Tutzing. The robot from Japan is equipped with sensors and artificial intelligence and reacts to being spoken to and touched, moves, opens its eyes and makes sounds. In Japan, the country that produces the most robots, robots that interact with humans enjoy a high level of acceptance in the care sector. „The appearance of such robots is designed to make them appear emphatic. But in the end, they reel off routines“, says Christian Rembe, Professor of Measurement Technology at the Institute of Electrical Information Technology (IEI) at Clausthal University of Technology.

In the Keiko project– the acronym stands for „cognitively and empathically intelligent collaborative robots“ – he wants to develop a robot that can respond to people's emotions in an interdisciplinary team with the universities of Göttingen and Duisburg-Essen. It should be able to recognise people's intentions in order to work with them flexibly, proactively and safely. The initial focus is on industrial applications, but according to Rembe, the project team also has the use in care in mind. This also applies to the recently launched Fluently research project led by Roboverse Reply. However, the Emocare project funded by the Federal Ministry for Economic Affairs and Energy in the Central Innovation Programme for SMEs (ZIM) focused on an emotional robot specifically for the care sector. Offenburg University of Applied Sciences and the Freiburg-based software development company Dr Hornecker worked together on this project.

„We are still at the very beginning of the development of emphatic robots,

Rembe emphasises. Interactions between people are very complex and place special demands on our brains. Humans are constantly observing the facial expressions, gestures, tone of voice and actions of their counterparts. They also assess intentions, moods and level of knowledge.

Care robot learns to assess its counterpart

The team at Clausthal University of Technology wants the robot to be able to psychologically interpret the behaviour of humans and recognise their attentiveness and ability to control the joint task. „In the first step, we want to find out how the robot can recognise its counterpart's willingness to accept an object. This involves measuring the human's attention“, says Rembe. The person recognises without thinking, for example, whether the person is tired or mentally absent, whether they are concentrated or attentive. In terms of care, this means that the robot recognises whether the patient is ready to eat or whether they are alert when food is handed to them - and can then adjust the pace of feeding, for example.

In a preliminary project for Keiko, the researchers at Clausthal University of Technology have already shown that the individual analysis of sensor data from certain modalities is not sufficient for the robot to be able to capture a complete picture of the situation. Rather, several sensor types must be combined with each other. The fusion of the collected data using innovative data processing mechanisms such as AI is necessary for a semantically correct interpretation of the available data," says Rembe.

In the Keiko project, the researchers want to use pupil dilation as an expression of emotions alongside posture, facial expressions, gestures and tone of voice. „Pupil oscillations show the influence of activities on brain networks. Normally, brain activity can be detected with EEG measurements, but the wiring of a patient required for this is of course not practical in nursing care," explains Rembe. The team is therefore working on demonstrating a connection between pupil oscillation and brain waves. Cameras in the robot will then take high-resolution images of the pupil's oscillation in the micrometre range. And from these oscillations, the robot can then draw conclusions about the emotional state of the person.

Data can now be processed in real time

„Image processing has made great progress in recent years. Large amounts of data can also be processed in real time so that the robot can react quickly, says Rembe. Nevertheless, the image processing required by the Keiko robot is not available off the shelf. Keiko may also use laser Doppler vibrometry, which the IEI has already trialled in other projects: This involves a laser beam measuring vibrations generated by the human body without contact, such as slight tremors.

In the Fluently project, the project partners are also focussing on multimodal sensor technology: "Speech content, tone of voice and gestures are interpreted via a smart interface," explains Marek Matuszewski, Manager at Roboverse Reply. „The translation into robot instructions is done via pre-trained machine learning models that have been customised to the individual user and individual process. The interface enables industrial robots to be accessible for every skill profile.“ Several technologies are used, such as programming in natural language and gestures: the cobot recognises and interprets voice commands and gestures from the operator and translates them into low-level robot programming tasks so that they can be executed immediately. Dynamic adaptation based on machine learning is also used: it helps to adapt the robot's behaviour based on contextual information relating to the state of the human operator and the specific task in progress.

AI helps to check the emotional state of the counterpart

The third technology used by Fluently is predictive analytics: This means that the current scenario is analysed to predict workflows and assign process parameters and tools to specific tasks and components. „And finally, AI helps to constantly check the health and emotional state of the other person and take countermeasures as soon as their cognitive and physical stress changes. The cobot can suggest support and/or modified routines," says Matuszewski.

In the Fluently project, the first step is to focus on language: the tone of voice is used to recognise whether the user is tired or stressed. According to Matuszewski, the translation work between man and machine is outsourced to separate modules: "One module is carried out by the human, the other by the robot. The human end device can recognise speech via sensors. The device collects this information from the human operator and processes the data internally with the help of cloud-based services. The input is then transmitted to the module on the cobot. This module interprets the received data, translates the commands into instructions for the cobot and sends them to the robot controller. At the same time, the module collects the robot's input, processes it and converts it into an understandable form for the human operator. This takes place in the audio area via speech generation, visually via a display and haptically via so-called vibrotactile feedback. And cameras can display the body position of the person.

„Without AI, the project would not be realisable“, Matuszewski clarifies. „Fluently's AI framework distributes functional modules across the global central cloud and the distributed physical Fluently units. These are both human and robotic devices.“ Matuszewski admits that the AI for the Fluently robot needs a lot of training data: „For this, an extensive training campaign is undertaken that covers all application cases as well as unforeseen events such as new product versions or tools. In addition, the Fluently system continues to learn in the field in order to be able to react to new situations. There are advantages to deploying some AI models on the device and accessing the cloud at the same time.

Image processing must match the work assignment

In contrast, the Emocare project has already produced a prototype of an emotionally active care robot, which has been tested at the St. Carolushaus nursing home in Freiburg and which Dr Hornecker now wants to develop into a commercial product. Here too, facial expressions, voice and gestures were used as measuring instruments to record emotions. The primary sensors are cameras that capture the carer's face. „The system then searches for markers on the face. We based this on the Facial Action Coding System developed by the US psychologist Paul Ekman, who has worked scientifically on the identification of universal emotional expressions," explains Managing Director Dr Achim Hornecker. The facial features that are assigned to certain emotions are coded from the images. And the AI is trained on the position of the features/extracted data. „However, we have found that image processing in care robots encounters more difficult conditions. These include poor lighting and poor camera viewing angles.

Care robot to recognise emotions via voice

However, the Emocare team had not thought about one challenge in advance, namely the dialogue system: "Systems like Alexa are useless for this purpose for data protection reasons, but also because they only recognise simple key words," says Hornecker. „However, AI technology is continuing to develop: chat GPT or derivatives now promise a natural dialogue.“

According to Hornecker, vision solutions for the application of the Ekman method are available off the shelf, but were fundamentally redeveloped by researchers at Offenburg University of Applied Sciences in the project. The voice module focussed on extracting emotions from the melody of speech. „This also works via pre-processing, in which frequency patterns are extracted. Here, too, there are corresponding standard solutions available on the market; they are mainly used in call centres. Emotions can be inferred from the change in frequency bands. But there are factors that influence the result, such as when the person is cold“, says Hornecker.

In contrast to the Fluently developers, he believes that the care robot will not require a cloud connection once it is in use: „The main feature of modern AI such as Chat-GPT is that it works with pre-training. You do not necessarily have to retrain the AI“, says Hornecker. However, he misses a cost-effective standard robot system on the market that works independently of robot manufacturers, something like a personal robot, analogous to a PC: the more people use it, the cheaper the robot becomes. This will come in the future. Open source robotics plays into the cards, but construction plans for this are not yet available for download.

Article from "medizin & technik" from 20 April 2023