What we hear, and how it affects what we see

In movie fight scenes, punches often miss by a foot or more, but when sound effects are added, and the punchee adds an effective-looking recoil, we're convinced that the punch is "real." We've posted on this phenomenon before: when a "click" sound is played as two animated balls pass by each other, it's perceived as a "bounce."

This type of sound effect, where the sound occurs at a critical moment in an animation or movie, has been explored quite extensively. But is it possible that other types of sound might affect how we see motion? Adam Ecker and Laurie Heller realized that they could apply sound to the research stimuli of Daniel Kersten. In Kersten's research, a ball moves around a virtual 3-D box, and the appearance of a shadow causes viewers to view the identical ball movement in different ways. What if sound is added to the moving ball? I've modified a couple of Kersten's demos, adding sound to suggest different types of movement.

In the first two examples, the motion of the ball and shadow is consistent with with the sound effects. In the third example, the shadow suggests that the ball is moving along the ground, but the airplane sound effects might suggest a lift-off and landing. In the final example, the tire skidding suggests motion along the ground, but the shadow again suggests flight. Which input matters most in motion perception? Ecker and Heller designed a series of comprehensive experiments to find out.

Ecker and Heller, being less lazy than I am, chose more realistic sound effects (ones which, unfortunately, aren't available in the standard clips bundled with iMovie): the sound of a marble rolling along the floor, or the sound of a basketball bouncing. They also used different versions of movies based on Kersten's original stimuli:

The white line shows several of the possible paths the balls took. In each of these cases, the video was synchronized with the marble-rolling or ball-bouncing audio. In the marble-rolling condition, an additional sound was added when the ball "bounced" off the back of the box. Twenty-two observers watched dozens of movies with the shadows removed, each time indicating whether they saw the ball rolling or jumping. As you can see from the figure, some of the paths, like (a) and (b), make more sense for rolling than for jumping (where paths (c) and (d) are the more probable paths the ball would take).

Ecker and Heller actually systematically varied the paths, modifying the curvature so that it appeared more or less like a rolling or jumping ball. Overall, respondents identified nearly 80 percent of the trials with the "roll" audio as "rolling," while rating just 20 percent of the trials with "jump" audio as rolling. In every case, it was significantly more likely that viewers would identify the trial in a manner consistent with the audio -- even when the curvature of the path suggested the opposite: a pure parabola would still be rated as rolling if it was accompanied by a rolling sound.

But it's possible that participants were only listening to the sound and ignoring the display. In a new experiment, Ecker and Heller "stretched" the box where the ball appeared, making it three times deeper than it had been previously:

The image on the left is what experimental participants saw; the image on the right demonstrates that the "rolling" path" is much longer than the "jumping" path. Now, instead of asking whether the ball jumped or rolled, experimenters simply showed two displays, and asked observers which was moving faster. When the identical motion was shown with "roll" and "jump" audio, observers said the ball with the "roll" audio was moving faster 72 percent of the time.

Ecker and Heller conclude that depth perception is influenced not only by visual phenomena such as shadows, binocular disparity, and object size, but also by sound. When the participants were asked how they made the determination that one object was moving faster than the other, they simply indicated that they "experienced" a speed difference -- there was no active consideration of whether the ball was rolling or jumping. It's as if the perceptual system unconsciously combines both audio and visual inputs to represent motion in three dimensions.

Ecker, A.J., & Heller, L.M. (2005). Auditory - visual interactions in the perception of a ball's path. Perception, 34, 59-75.