We’ve discussed synesthesia many times before on Cognitive Daily — it’s the seemingly bizarre phenomenon when one stimulus (e.g. a sight or a sound) is experienced in multiple modalities (e.g. taste, vision, or colors). For example, a person might experience a particular smell whenever a given word or letter is seen or heard. Sometimes particular faces are associated with specific colors or auras. Synesthesia is relatively rare, but the people who experience it are genuine: their perceptions are consistent and replicable.
But one question researchers haven’t been able to nail down is exactly how synesthesia occurs. Consider the relatively common form of synesthesia, where colors are perceived along with words. One synesthete consistently sees the color green when she hears someone say “neat.” Does the synesthetic experience occur when she first detects the word, or only after she understands its meaning?
A team led by Gary Bargary has figured out a new way to test when a synesthetic experience occurs by relying on the McGurk Effect. In the McGurk effect, the word you “hear” someone saying changes depending on what you see. This movie gives a quick demonstration of the phenomenon:
In the first clip, I superimposed the sound of myself saying “neat neat peat peat” over video of myself saying “neat peat neat peat”. What most people think they hear is “neat meat peat peat.” You can see the actual recording of what I said in the second part of the clip. Because my mouth makes a similar movement when I say “p” and “m”, the combination of the audio “neat” with a video “peat” makes viewers think they heard “meat.” Listeners use both the audio and video information to decide what I’m saying, and they get it wrong! Did you experience the illusion? Let’s make this a poll:
After a few people have responded, we’ll have a good idea of how many people experience the effect.
This offers a unique opportunity to see if synesthesia results from the actual sound a person hears, or their perceived meaning of the word. Bargary’s team assembled a list of words that produce reliable McGurk effects. Here’s a sampling:
Then they played three videos of each word (in completely random order) to 12 synesthetes who had previously demonstrated an association with words and colors. In one, the full-McGurk effect was present (e.g. the visual said “peat” while the audio said “neat”) In one, the viewer saw the video only, and white noise was played. Finally, in one, the face was pixellated and only the audio was heard. On a separate computer monitor was a large array of colors, and after the participants saw each video, they clicked on the color that they experienced with it. Then they told the experimenters the word they thought they had heard/seen. Here’s a typical set of responses:
As you can see, the synesthetes chose very different colors depending on whether or not the McGurk illusion was presented. Respondent S21 perceived light blue when only the visual was shown, green when they only heard the word, and blue when the full illusion was shown. Interestingly, when they saw the visual alone, some of the synesthetes reported no color seen, even when they had identified the word. No one said they saw perceived a color when they couldn’t figure out what word was being said.
But maybe these colors aren’t really that different — you might argue that green and blue are in the same family. So the researchers undertook a systematic analysis of the results, comparing the respondent’s consistency in choosing a color when they didn’t experience the McGurk effect (e.g. they just perceived “neat” both with audio-only and audiovisual) to the times when they did experience the McGurk effect version of the same word (e.g. auditory “bay” compared to audiovisual “day”). Remember, not everyone experiences the effect every time they see it, as (hopefully) our poll above demonstrates. This graph summarizes the results:
The graph plots the difference in colors chosen by nine participants. The researchers calculated the color difference by calculating how far apart the colors were in an RGB color space like this (Source: Wikipedia):
The farther apart the colors, the bigger the color difference. As you can see, for each of these respondents except one, the difference was significant: They chose colors that were more different from each other when the McGurk effect was experienced compared to when it wasn’t experienced.
Bargary’s team says this means that synesthetic perceptions occur late in the process of sensing and perceiving words. The McGurk effect results from integrating inputs from multiple senses (vision and hearing), and the synesthetic perception must occur after that has happened — otherwise, the synesthetes’ responses wouldn’t have been different when they didn’t experience the McGurk effect.
Bargary G, Barnett KJ, Mitchell KJ, & Newell FN (2009). Colored-speech synaesthesia is triggered by multisensory, not unisensory, perception. Psychological science : a journal of the American Psychological Society / APS, 20 (5), 529-33 PMID: 19476587