When you walk into a room and instantly sense conflict brewing among a huddle of people, or a friend tells you, “I’m fine,” but you have a strong intuition that they are not, you’re reading an unspoken language. Your mind collects cues from dozens of tiny facial expressions in the eyes, the mouth, the nose, the smile, the gaze of the other person. You’re also listening to the sound of their voice: its pitch, timbre, brightness, and loudness, sighs or gasps, pauses between words.
Reading emotion in this way is inherently multisensory. It’s also essential to building and maintaining social relationships. We make these calculations about emotion dozens of times a day. But because different parts of the brain tend to read faces and voices, it hasn’t been clear how the brain integrates these different signals to decode what another person is feeling.
Read more: “How Emotions Connect Your Body and Brain”
Recently, a team of scientists decided to try to map this process out in the gray matter. They invited 24 people to a hospital in Brussels and scanned them with fMRI while they watched and/or listened to 20 brief video clips of four different actors expressing five different emotions: disgust, fear, happiness, sadness, and neutrality. The emotional content of face and voice always matched. Then the team used pattern analysis to see which region of the brain carried information about which emotions.
They found that deciphering an emotion from faces and vocal cues isn’t a tidy process, or a series of isolated steps. Instead, the brain processes emotional data across a broad network of face- and voice-selective brain regions at once. Almost all of these regions rely on the same neural “signal” to carry emotional meaning, regardless of its sensory origin. The sole exception was the occipital face area, which used distinct patterns for voices and faces. Also, while combining vocal and facial cues changes how the brain responds, it doesn’t make the brain any better at deciphering the underlying emotion. The team published their results in the Journal of Neuroscience.
The authors propose that emotion processing is shared across networks rather than siloed because faces and voices typically provide complementary information about emotional meaning. Earlier work, they point out, suggests that “face and voice regions constitute interacting networks specialized for socially meaningful information rather than strictly unimodal sensory systems.” The findings are consistent with studies that show impairments in emotion recognition often affect a person’s ability to decode both facial and vocal expressions.
Still, faces may play the dominant role in the understanding of emotion. When participants were looking at faces only, information about the emotions they were detecting still showed up in areas of the brain that respond preferentially to voices. But the reverse was weaker. The authors suggest this is because facial expressions tend to be recognized more reliably than voices, and—according to earlier work—are more often used to detect the underlying emotion when face and voice conflict.
A couple of caveats: A few important areas of the brain, including the amygdala, which helps to process emotion, and certain face-selective regions, weren’t reflected in the data. Moreover, the number of participants in the study, and the number of faces and vocal cues they were tested on, were small, so more data is needed.
But if the findings hold up, they suggest that the unspoken language of emotion is pretty universal. The brain reads what we see and what we hear as a single tongue. 
Enjoying Nautilus? Subscribe to our free newsletter.
Lead Image: Tada R / Shutterstock