Author: Rob Matheson | MIT News Office<\/p>\n
MIT Media Lab researchers have developed a machine-learning model that takes computers a step closer to interpreting our emotions as naturally as humans do.<\/p>\n
In the growing field of \u201caffective computing,\u201d robots and computers are being developed to analyze facial expressions, interpret our emotions, and respond accordingly. Applications include, for instance, monitoring an individual\u2019s health and well-being, gauging student interest in classrooms, helping diagnose signs of certain diseases, and developing helpful robot companions.<\/p>\n
A challenge, however, is people express emotions quite differently, depending on many factors. General differences can be seen among cultures, genders, and age groups. But other differences are even more fine-grained: The time of day, how much you slept, or even your level of familiarity with a conversation partner leads to subtle variations in the way you express, say, happiness or sadness in a given moment.<\/p>\n
Human brains instinctively catch these deviations, but machines struggle. Deep-learning techniques were developed in recent years to help catch the subtleties, but they\u2019re still not as accurate or as adaptable across different populations as they could be.<\/p>\n
The Media Lab researchers have developed a machine-learning model that outperforms traditional systems in capturing these small facial expression variations, to better gauge mood while training on thousands of images of faces. Moreover, by using a little extra training data, the model can be adapted to an entirely new group of people, with the same efficacy. The aim is to improve existing affective-computing technologies.<\/p>\n
\u201cThis is an unobtrusive way to monitor our moods,\u201d says Oggi Rudovic, a Media Lab researcher and co-author on a paper describing the model, which was presented last week at the Conference on Machine Learning and Data Mining. \u201cIf you want robots with social intelligence, you have to make them intelligently and naturally respond to our moods and emotions, more like humans.\u201d<\/p>\n
Co-authors on the paper are: first author Michael Feffer, an undergraduate student in electrical engineering and computer science; and Rosalind Picard, a professor of media arts and sciences and founding director of the Affective Computing research group.<\/p>\n
Personalized experts<\/strong><\/p>\n Traditional affective-computing models use a \u201cone-size-fits-all\u201d concept. They train on one set of images depicting various facial expressions, optimizing features \u2014 such as how a lip curls when smiling \u2014 and mapping those general feature optimizations across an entire set of new images.<\/p>\n The researchers, instead, combined a technique, called \u201cmixture of experts\u201d (MoE), with model personalization techniques, which helped mine more fine-grained facial-expression data from individuals. This is the first time these two techniques have been combined for affective computing, Rudovic says.<\/p>\n In MoEs, a number of neural network models, called \u201cexperts,\u201d are each trained to specialize in a separate processing task and produce one output. The researchers also incorporated a \u201cgating network,\u201d which calculates probabilities of which expert will best detect moods of unseen subjects. \u201cBasically the network can discern between individuals and say, \u2018This is the right expert for the given image,\u2019\u201d Feffer says.<\/p>\n For their model, the researchers personalized the MoEs by matching each expert to one of 18 individual video recordings in the RECOLA database, a public database of people conversing on a video-chat platform designed for affective-computing applications. They trained the model using nine subjects and evaluated them on the other nine, with all videos broken down into individual frames.<\/p>\n Each expert, and the gating network, tracked facial expressions of each individual, with the help of a residual network (\u201cResNet\u201d), a neural network used for object classification. In doing so, the model scored each frame based on level of valence (pleasant or unpleasant) and arousal (excitement) \u2014\u00a0commonly used metrics to encode different emotional states. Separately, six human experts labeled each frame for valence and arousal, based on a scale of -1 (low levels) to 1 (high levels), which the model also used to train.<\/p>\n The researchers then performed further model personalization, where they fed the trained model data from some frames of the remaining videos of subjects, and then tested the model on all unseen frames from those videos. Results showed that, with just 5 to 10 percent of data from the new population, the model outperformed traditional models by a large margin \u2014 meaning it scored valence and arousal on unseen images much closer to the interpretations of human experts.<\/p>\n This shows the potential of the models to adapt from population to population, or individual to individual, with very few data, Rudovic says. \u201cThat\u2019s key,\u201d he says. \u201cWhen you have a new population, you have to have a way to account for shifting of data distribution [subtle facial variations]. Imagine a model set to analyze facial expressions in one culture that needs to be adapted for a different culture. Without accounting for this data shift, those models will underperform. But if you just sample a bit from a new culture to adapt our model, these models can do much better, especially on the individual level. This is where the importance of the model personalization can best be seen.\u201d<\/p>\n Currently available data for such affective-computing research isn\u2019t very diverse in skin colors, so the researchers\u2019 training data were limited. But when such data become available, the model can be trained for use on more diverse populations. The next step, Feffer says, is to train the model on \u201ca much bigger dataset with more diverse cultures.\u201d<\/p>\n Better machine-human interactions<\/strong><\/p>\n Another goal is to train the model to help computers and robots automatically learn from small amounts of changing data to more naturally detect how we feel and better serve human needs, the researchers say.<\/p>\n It could, for example, run in the background of a computer or mobile device to track a user\u2019s video-based conversations and learn subtle facial expression changes under different contexts. \u201cYou can have things like smartphone apps or websites be able to tell how people are feeling and recommend ways to cope with stress or pain, and other things that are impacting their lives negatively,\u201d Feffer says.<\/p>\n This could also be helpful in monitoring, say, depression or dementia, as people\u2019s facial expressions tend to subtly change due to those conditions. \u201cBeing able to passively monitor our facial expressions,\u201d Rudovic says, \u201cwe could over time be able to personalize these models to users and monitor how much deviations they have on daily basis \u2014\u00a0deviating from the average level of facial expressiveness \u2014 and use it for indicators of well-being and health.\u201d<\/p>\n A promising application, Rudovic says, is human-robotic interactions, such as for personal robotics or robots used for educational purposes, where the robots need to adapt to assess the emotional states of many different people. One version, for instance, has been used in helping robots<\/a> better interpret the moods of children with autism.<\/p>\n Roddy Cowie, professor emeritus of psychology at the Queen\u2019s University Belfast and an affective computing scholar, says the MIT work \u201cillustrates where we really are\u201d in the field. \u201cWe are edging toward systems that can roughly place, from pictures of people\u2019s faces, where they lie on scales from very positive to very negative, and very active to very passive,\u201d he says. \u201cIt seems intuitive that the emotional signs one person gives are not the same as the signs another gives, and so it makes a lot of sense that emotion recognition works better when it is personalized. The method of personalizing reflects another intriguing point, that it is more effective to train multiple \u2018experts,\u2019 and aggregate their judgments, than to train a single super-expert. The two together make a satisfying package.\u201d<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"