Cool Audiovisual Illusions: Spatial and Temporal Ventriloquism Effects

In keeping with the theme of illusions that result from crossmodal interactions, this week's illusion is the ventriloquism effect, first reported by Howard and Tempelton in 1966. As you can probably tell from the name, the ventriloquism effect involves visual information influencing where we think sounds occur, like the moving mouth of a ventriloquist's dummy influencing where we perceive the voice to be coming from. In a typical setup for the ventriloquism effect, participants are exposed to a sound and asked to point to where they think the sound occurred. Participants are then exposed to either no flash of light, or a flash of light at the location of the sound or a flash of light at a different location. When participants see the flash at a location other than that of the sound, they tend to point to a position offset from the actual location of the sound toward the flash. In fact, when the distance between the sound and flash is small (a few degrees of the visual arc), they often point to the position of the flash. This is often called "visual capture," because it's as though the visual stimulus "captured" the auditory stimulus.

Interestingly, if people are exposed to sound and a visual stimuli that are separated by a few degrees for an extended period of time (20-30 minutes), a "ventriloquism aftereffect" is produced. For example, Gregg Recanzone2 presented participants with a sound and a flash at the same time, over and over again (2500 pairings in total) for 20-30 minutes. The flash was 8° off from the sound, either to the right or the left. After this training phase, participants were then exposed to a sound, and asked to turn their heads to its location. For each auditory stimulus in this test phase, participants' tended to turn to a position about 8° off from where the sound was, and in the opposite direction from the flash they'd seen in the training phase. In other words, if the flash had been 8° to the right of the auditory stimuli in the training phase, participants tended to perceive the auditory stimuli in the test phase as being ° to the left of where they actually were. The participants' brains seemed to be adjusting for the disparity between the visual and auditory stimuli, so that when the auditory stimuli were presented alone, their brains corrected for the disparity when they didn't have to.

One explanation for the ventriloquism effect is that it occurs because visual acuity for spatial information is much better than auditory acuity for spatial information3. We're just better at locating things that we see than things that we hear, and this may lead to our brains utilizing visual over auditory information when the two are competing, leading to visual capture. One interesting prediction of this view is that for dimensions in which auditory acuity is greater than visual acuity, "auditory capture" will occur. There is a great deal of evidence that auditory acuity is greater than visual acuity for temporal information. This led several researchers to look for a temporal ventriloquism effect in which auditory stimuli "capture" visual stimuli. For example, Recanzone4 presented participants with a series of light flashes and beeps. Participants were told to attend either to the flashes or the beeps, and to estimate the rate at which they were flashing or beeping. When participants attended to flashes while the beeps were at a different frequency, participants' estimates were biased in the direction of the beep rate. For example, if the beeps occurred at a slower rate than the flashes, then participants reported that the flash rate was slower than it actually was. Participants who attended to the beeps were not biased by the flash rate. Recanzone also produced a temporal ventriloquism aftereffect by exposing participants to flashes and beeps at a different rate for 20-30 minutes. Participants were then exposed to flashes alone, and asked to estimate the rate at which the flashes occurred. In this test phase, participants perceived the flash rate to be faster than it actually was if they had been exposed to beeps that occurred at a slower rate than the flashes in the training phase, and their estimates were slower than the actual flash rate if they had been exposed to beeps that occurred more frequently than the flashes in the test phase.

¹Howard, I.P., Templeton, W., B. (1966). Human Spatial Orientation. New York: Wiley.
²Recanzone, G.H. (1998). Rapidly induced auditory plasticity: The ventriloquism aftereffect. Proceedings of the National Academy of Sciences, 95, 869-875.
³Recanzone, G.H. (2003). Auditory influences on visual temporal rate perception. Journal of Neurophysiology, 89, 1078-1093.
⁴Ibid.