Synesthesia and the McGurk effect

ResearchBlogging.orgWe've discussed synesthesia many times before on Cognitive Daily -- it's the seemingly bizarre phenomenon when one stimulus (e.g. a sight or a sound) is experienced in multiple modalities (e.g. taste, vision, or colors). For example, a person might experience a particular smell whenever a given word or letter is seen or heard. Sometimes particular faces are associated with specific colors or auras. Synesthesia is relatively rare, but the people who experience it are genuine: their perceptions are consistent and replicable.

But one question researchers haven't been able to nail down is exactly how synesthesia occurs. Consider the relatively common form of synesthesia, where colors are perceived along with words. One synesthete consistently sees the color green when she hears someone say "neat." Does the synesthetic experience occur when she first detects the word, or only after she understands its meaning?

A team led by Gary Bargary has figured out a new way to test when a synesthetic experience occurs by relying on the McGurk Effect. In the McGurk effect, the word you "hear" someone saying changes depending on what you see. This movie gives a quick demonstration of the phenomenon:

In the first clip, I superimposed the sound of myself saying "neat neat peat peat" over video of myself saying "neat peat neat peat". What most people think they hear is "neat meat peat peat." You can see the actual recording of what I said in the second part of the clip. Because my mouth makes a similar movement when I say "p" and "m", the combination of the audio "neat" with a video "peat" makes viewers think they heard "meat." Listeners use both the audio and video information to decide what I'm saying, and they get it wrong! Did you experience the illusion? Let's make this a poll:

After a few people have responded, we'll have a good idea of how many people experience the effect.

This offers a unique opportunity to see if synesthesia results from the actual sound a person hears, or their perceived meaning of the word. Bargary's team assembled a list of words that produce reliable McGurk effects. Here's a sampling:

i-7d089c2fe6b7e7556d88c3a9898ba195-bargary1.gif

Then they played three videos of each word (in completely random order) to 12 synesthetes who had previously demonstrated an association with words and colors. In one, the full-McGurk effect was present (e.g. the visual said "peat" while the audio said "neat") In one, the viewer saw the video only, and white noise was played. Finally, in one, the face was pixellated and only the audio was heard. On a separate computer monitor was a large array of colors, and after the participants saw each video, they clicked on the color that they experienced with it. Then they told the experimenters the word they thought they had heard/seen. Here's a typical set of responses:

i-08896a653080bff6eedcde8341473cb0-bargary2.gif

As you can see, the synesthetes chose very different colors depending on whether or not the McGurk illusion was presented. Respondent S21 perceived light blue when only the visual was shown, green when they only heard the word, and blue when the full illusion was shown. Interestingly, when they saw the visual alone, some of the synesthetes reported no color seen, even when they had identified the word. No one said they saw perceived a color when they couldn't figure out what word was being said.

But maybe these colors aren't really that different -- you might argue that green and blue are in the same family. So the researchers undertook a systematic analysis of the results, comparing the respondent's consistency in choosing a color when they didn't experience the McGurk effect (e.g. they just perceived "neat" both with audio-only and audiovisual) to the times when they did experience the McGurk effect version of the same word (e.g. auditory "bay" compared to audiovisual "day"). Remember, not everyone experiences the effect every time they see it, as (hopefully) our poll above demonstrates. This graph summarizes the results:

i-183f0a1ff5bae1fffb20073495a7fc79-bargary3.gif

The graph plots the difference in colors chosen by nine participants. The researchers calculated the color difference by calculating how far apart the colors were in an RGB color space like this (Source: Wikipedia):

i-1a3d065d0fe33a05825a3fda9b4b41b5-bargary4.jpg

The farther apart the colors, the bigger the color difference. As you can see, for each of these respondents except one, the difference was significant: They chose colors that were more different from each other when the McGurk effect was experienced compared to when it wasn't experienced.

Bargary's team says this means that synesthetic perceptions occur late in the process of sensing and perceiving words. The McGurk effect results from integrating inputs from multiple senses (vision and hearing), and the synesthetic perception must occur after that has happened -- otherwise, the synesthetes' responses wouldn't have been different when they didn't experience the McGurk effect.

Bargary G, Barnett KJ, Mitchell KJ, & Newell FN (2009). Colored-speech synaesthesia is triggered by multisensory, not unisensory, perception. Psychological science : a journal of the American Psychological Society / APS, 20 (5), 529-33 PMID: 19476587

More like this

Very clever study!

However, I was wanting to say for some time now: shouldn't that be "A Journal of the Association for Psychological Science" these days? A good thing too, considering the many members (myself included) and contributors (I tried!), who are not American.

this is interesting. but in the case of grapheme/color synesthsia may not be so useful. i wonder how/what the linkage between the sight (of written letters) and sound (of spoken words) compares to sight versus sound of lips. being a grapheme/color with slight sound/color (associative) synesthete myself, i'm not sure how i would have scored the words verus sounds myself ... i heard "neat meat pete pete" (but also have experience with poor video quality in youtube). for me, perhaps, the sound is dominant over the sight of the speaker, and the first letter of a word gives the predominant color to the whole word (though it's not that simple. the color does not come from the sound, as such, however, but from the translation from heard word to spelled word (graphemes), which imparts a color to the word which is related to but not at all identical to the color of a non-word sound (such as music). and complicatingly, the number 5 has a totally different color from the word five. so i'm not sure of the utility of this test for associative synesthetes.

so, grain of salt.

of course, i don't think synesthesia is bizarre.

also, in my schema, "p", "r", and "a" are similar colors but don't have anything to do with one another in color space. "p" is pink, "n" is brown, and "m" is blueish ... so where would that put the differences on the color scale metric? the problem with this test is that the synesthetic color experience does not, or does not necessarily, scale with the RGB scale in any meaningful way.

I have spatial-sequence synesthesia (but I also associate the days of the week with colours). I have often wondered about the "sensory" implications of synesthesia. Below is a repost of a discussion I began in an online forum a couple of years ago . . . would be happy to get feedback.

****
I am curious about the âsensoryâ aspect of synesthesia. In its strictest meaning, sensory perception requires external stimulation of a sense organ, but I find that there is a very clear internal process in my synesthetic experiences that is completely separate from any external input of information.

For example, when I think of October, I know that it occupies the space between about 2:30 and 3:30 on my elliptical map, but I do not need to hear or see the word âOctoberâ for this locating to occur. I know that October is in that spot even in recollecting events that have occurred in Octobers past. The same goes for imagining, say, what this yearâs upcoming Thanksgiving dinner will be like (Iâm giving myself away as a Canadian, I know): I foresee the dinner occurring in that 3:30 locale.

What Iâm getting at is that there is no sensory perception required for this mapping to occur. Unless remembering and predicting are forms of sensory perception, it seems like the process is independent of immediate sensory input.

Interesting article. Synesthesia is also fascinating because logically it should not be a product of the human brain the trend has been for increasing separation of function anatomically.

"Does the synesthetic experience occur when she first detects the word, or only after she understands its meaning?"

Dave, like Julia, I would say it is past the sound detection because it is grapheme/color, not sound/color synesthesia, so it is important to know WHAT the word is, to know what letters it is made up of, but the meaning is irrelevant.* I also perceive '4' and 'four' as distinct colors, even though they sound the same. I read the paper and the authors of the study explain this in depth.

Unlike Julia, my words are mainly colored by their vowels, so the the McGurk Effect on color choices would hardly have been noticeable in my case.

*with rare exceptions like the months or days of the week, which often, though not always (about half the time) have unique colors unrelated to the color the word would normally have.

i think i'm a grapheme-color synesthete, so for me, each numerical digit has a color. its actually really useful to me because numbers combinations and things are really easy for me to remember because when i think of them, they have a distinct color combination in my mind that i picture.
for instance, people's birthdays are really easy for me to remember because when they tell me them, i file that person away in my brain as the colors in their birthdays. like if someone's birthday is July 12 (7/12) then i know them as being a yellow-white-pink person. i mean i remember birthdays and numbers REALLY well because of it, and if i forget the exact date of their birthday, i can guess it pretty accurately just by the color aura that they have in my brain
it's so weird how our brains do things like this..i definitely wanna learn more about synesthesia. nice article!

I believe synesthesia is much more common than researchers realize. The color-sound mix is easy to identify but others can be subtle. I've known for a long time that I experience a mix of taste and sound but it wasn't until a couple of years ago that I read about kinesthetic synesthesia and realized I experience that for dates. I had to learn about it before I could become aware of my internal process. I think as awareness of the many types of synesthesia increases we will hear from more and more newly self-discovered synesthetes.

I don't know much about synesthesia. But I do know that I experience it. I'm not sure what you call it, but it's a color-music synesthesia. When songs are played/sung in a key that contains flats, it is a "blue" key. When songs are played/sung in a key that contains either sharps or nothing (like C, or F which has one flat, which sorta screws up my theory), it is a "red" key. I have experience this since at least junior high, if not before (but junior high is when I started to understand key signatures). It probably doesn't have as much to do with flats or sharps themselves than it does the place the key begins in the chromatic scale. Interesting it'd be called chromatic, right, because chromatic is related to color as well... I also have perfect pitch, so I don't know if that's related or not.

I was a little confused by the neat peat stuff, since that's not the kind of synesthesia I experience, but I thought it was interesting nonetheless, and I wanted to share my synesthesia experience with you in case it interests you.

I know someone who experiences synesthesia - where he hears music and sees colour. I don't have this. I have CAPD, an auditory processing disorder. So, I'm used to depending on watching people's faces and using context to figure out what is being said.

This is what occurred when I watched:

1. neat, ng/m/neat?, peat, peat. neat, neat, peat, peat
2. neat, neat, peat, peat. neat, neat, peat, peat

Note "ng/m/neat?" is where he says people hear "meat." I /did/ think I heard meat, based on his lips, but it sounded too nasalised (I actually /heard/ "ngeat" even as I turned it into "meat" in my head), and I was very very confused by the fact the lips didn't match perfectly with the syllable. It was like it was out of synch or something. At first I figured I heard him wrong, began comparing what I'd seen to his voice as I watched the next word come forth, couldn't figure out why the sound was off so badly that it didn't match the sound correctly, and ultimately concluded that he'd said "neat" after all and the audio-video track was screwed up and I'd lost track of where it was with the words. I based my choice of "neat" on context, because he never said he was going to say "meat" but /had/ mentioned the word "neat" and it was very distinctively spoken as the first word, prior to "ngeat."

After I got myself all confused, I found myself reading below the video as I listened to and half-watched the second part, and realized he /had/ said "neat" and that he was superimposing his speech on top of an original recording of "neat, peat, neat, peat."

So: In short, I /did/ do what he said most people do, as a very fast "fix" to an unclear auditory message, but due to the synchronization problem, I wasn't 100% certain I heard him correctly even then. Then I pulled a CAPD compensation and found "neat" was the only obvious word he could have used in that strange context, so I fixed what I heard with what I thought I should have heard.

If he /had/ said "meat" with the same synchronization issue, I may have gotten that wrong, especially after I'd heard the whole thing.

Very interesting article.

Check out episeattle.com

The Existential Psychoanalytic Institute of Seattle is a contemporary psychoanalytic society, research institute, and clinic dedicated to the exploration and treatment of the human condition through a synthesis of existential analysis, traditional psychoanalysis, phenomenology, and critical philosophy.

The condition has a very long history if one includes scriptural tuaditions: about the time when Moses received the Ten Commandments, the people "heard with their eyes" and "saw with their ears".

By Robin Datta (not verified) on 19 Jan 2010 #permalink