How multilingualism alters multisensory perception

We know that different people can perceive sounds and images like the blue/gold dress in divisive ways depending on how their brain processes them. But how can bilingualism alter the way that light and sound information are delivered to the brain?

First, the gold/blue dress “melted the internet.” Then, Yanny/Laurel “divided America.” Illusions like these ignite interest and controversy because they violate the intuition that our perceptions are direct, unfiltered versions of the external environment. Color, after all, shouldn’t be a matter of opinion. Which is why it blows our minds when someone insiststhat the gold dress we see is blue, or that what we clearly hear as“Yanny” sounds to them like “Laurel.”

The reality is that sensory perception can be nudged, distorted, and radically transformed by anything from the languages we speak to the surrounding visual context.

The reality is that sensory perception can be nudged, distorted, and radically transformed by anything from the语言我们说话to thesurrounding visual context– something that is dramatically illustrated by the strength of perceptual illusions. Like a virus, a well-designed illusion contains just the right combination of features to essentially hi-jack the “machinery” of the perceptual system, including learned heuristics and innate functions, and make it do what it always does – interpret sensory input based on (usually) reliable cues. Unlike a virus, perceptual illusions are generally harmless (other than inciting the occasional internet war), and can even provide a glimpse into the inner-workings of our minds. They can also shed light on how our personal experiences, such as experiences with different languages, shape how we perceive the world around us.

A neutral density filter held in front of a landscape
Credit: Robert Emperley [CC BY-SA 2.0]
To illustrate, consider the audiovisual illusion calledthe McGurk Effect.When we hear a speech sound, such as “ba,” but see an incongruent lip movement, such as “ga,” the brain tries to reconcile this discrepancy, often resulting in the perception of a completely different sound, such as “da.” Our research shows thatbilinguals are more likely to experience the McGurk Effect than monolinguals, suggesting that multilingual experience alters multisensory integration. One potential explanation is that bilinguals may need to (at least initially) rely more on visual information to make sense of complex, multilingual speech. Indeed, there is evidence thatbabies growing up in multilingual households attend to the mouths of speakers to a greater degree than infants in monolingual households.如我们最近所述review paperpublished inBehavioral and Brain Functions,these early differences in how bilinguals and monolinguals attend to speech-relevant inputs continue to shape how, and even how well, individuals process sensory information throughout their lives.

Utilizing neuroimaging tools such as EEG and fMRI, researchers have demonstrated that bilingualchildrenandadultsshow visible differences in processing of auditory stimuli compared to their monolingual peers. These effects of bilingualism have been observed in neural regions devoted tohigher-level cognitive functions, such as those responsible for controlling attention, as well as in the evolutionarily ancientauditory brainstem,which can encode sounds so faithfully that it is possible to play back recognizablespeechandmusicusing the electrical signals generated by the brain.

Bilingualism can therefore alter bottom-up processes that deliver light and sound waves up to the brain, as well as top-down processes that guide how we interpret and organize this information. It is possible that bilingualism generally increases audiovisual integration due to stronger encoding of sensory stimuli (think, a more sensitive mic or camera). However, because bilingualism also impacts top-down functions, the influence of language experience on sensory perception is likely to be more nuanced. Specifically, bilinguals may integrate sensory information differently due to more precise control over whether inputsshouldbe combined. Indeed, researchers have found thatauditory and visual stimuli need to be timed more perfectlyfor bilinguals to experience thedouble-flash illusion, whereby hearing two tones transforms a single flash of light into what appears to be two flashes. In other words, bilinguals are better able to determine when sights and sounds likely originated from different sources (based on features such as timing), and subsequently suppress inputs that would distort their perception.

Over time, language experience can even alter the physical structure of the brain. For instance, it has been shown that bilingual experience can lead togreater brain matter density and volumein regions associated with sensory processing, such as theprimary auditory cortex, as well as executive function, such as theprefrontal cortex.The behavioral correlates of these physical changes can be significant, as greater gray matter in a region of the primary auditory cortex predictsbetter speech perception.Increased gray matter in the prefrontal cortex is additionally associated with enhancedcognitive controlAttentional Network Task→ → ← → →).

Given the importance of both sensory and executive functions for language learning, one of theconsequences of bilingualism is a greater capacity to continue learning new languages, thereby perpetuating the cycle of neural reconfiguration that can begin with exposure to multilingual speech.

The film maker, Federico Fellini, famously said that “a different language is a different vision of life”. Through a combination of behavioral and neurological investigations, it has now been confirmed that learning a different language can literally change what you see.

View the latest posts on the On Biology homepage

Comments