Brief
Study finds brain integrates sight and sound to decode emotions
Researchers discovered that seeing an emotive face lights up voice‑processing areas and vice versa, highlighting a multisensory basis for emotion perception.
By Felo News Desk · Published
Scientists at Ghent University have shown that the brain does not treat visual and auditory emotional cues as separate streams. In an MRI study, participants viewed or heard actors expressing emotions such as happiness, disgust and fear, and the researchers observed overlapping activation in brain regions traditionally linked to face and voice processing.
What happened
The experiment involved 24 volunteers who were scanned while watching video clips, listening to audio clips, or experiencing combined video‑audio clips of actors portraying distinct emotions. Each stimulus type triggered the brain areas normally dedicated to the opposite sense: visual clips activated voice‑processing zones, and audio clips activated face‑processing zones. Moreover, the same emotional categories produced similar patterns of activation regardless of whether they were presented visually or auditorily.
What the reports add
According to Scientific American, the findings challenge the long‑standing view that facial and vocal information are processed in isolated cortical modules. The study suggests that the brain’s emotion‑recognition system is organized around the emotional content itself rather than the sensory modality. The article also notes that the research aligns with earlier work on brain plasticity in people who are blind or deaf, where sensory‑specific regions repurpose to process emotional cues from the remaining sense.
What was said
"We don’t only see a face that is sad," said lead author Stefania Mattioni, a postdoctoral researcher at Ghent University, as quoted by Scientific American. "We normally also hear the sound of the person crying, for instance. So, the brain is actually structured to interact with this kind of multisensory stimulation." Mattioni added that the regions appear to be guided more by the emotion than by the modality. Stefania Benetti, an assistant professor of psychology at the University of Trento who was not involved in the study, told the outlet that "emotion recognition… depends on a multisensory network that integrates what we see and hear" and that this integration offers an evolutionary advantage by reducing ambiguity in noisy environments.
How it came about
The study builds on prior investigations into how the brain adapts to sensory loss, such as the finding that voice‑selective areas can process facial expressions in people born deaf. It also follows recent Felo News coverage of brain‑related research, including a October 18 article on the youth protein TIMP2’s role in rejuvenating microglia and a October 16 report on cortisol’s effect on emotional memory. Together, these works illustrate a growing focus on how emotional processing is intertwined with broader neural mechanisms.
Key facts
- Seeing an emotive face activates brain regions normally dedicated to processing voices. (scientificamerican.com)
- Hearing an emotional voice activates brain regions normally dedicated to processing faces. (scientificamerican.com)
- The same emotional categories (happiness, disgust, fear) produced similar brain activation patterns regardless of visual or auditory presentation. (scientificamerican.com)
- The study involved 24 participants scanned with MRI while viewing, hearing, or both viewing and hearing emotional clips. (scientificamerican.com)
- Lead author Stefania Mattioni is a postdoctoral researcher at Ghent University in Belgium. (scientificamerican.com)
Sources
- [1] scientificamerican.com — originally reported as “The brain doesn’t process emotion the way we thought”
