Cognitive Daily

More on sound and vision

Today’s reading is “When Sound Affects Vision: Effects of Auditory Grouping on Visual Motion Perception,” by Katsumi Watanabe and Shinsuke Shimojo of Caltech (Psychological Science, 2001).

In 1997, Sekuler, Sekuler, and Lau discovered a fascinating effect that I’ve attempted to replicate using the crude resources available to me (iMovie). I’ve made two movies (quicktime required) that (perhaps with a little imagination) replicate the effect. First watch movie 1:

movie 1

It looks like the two balls pass through each other, right?

Now, make sure you have the sound turned up on your computer and watch movie 2.

movie 2

You should hear a clicking sound, right as the balls meet. Now it looks like the balls “bounce” off each other, right? But the display was identical—the only thing I changed was to add the sound to accompany the “collision.” The simple soundtrack changed your visual perception of the event. Nearly every observer Sekuler et al. showed similar movies to saw it the same way.

Watanabe and Shimojo wanted to know how this effect works. Does any sound at the time of “collision” cause the bouncing effect, or do only certain types of sounds work? So they asked observers to do the same task, but instead of playing a single sound, they played three separate clicks—one before the collision, one at the time of the collision, and one after. Now observers saw the balls pass through each other again.

Next they played a series of tones, but they changed the pitch of the tone simultaneous with the collision. Again observers saw the balls “bouncing” back to their original positions. Watanabe and Shimojo believe this is due to a phenomenon called “auditory grouping”—we don’t hear the series of clicks as separate, disconnected sounds, but as a single group. By changing the pitch of one of the sounds, they cause observers to ungroup the sounds, and this in turn causes the bouncing effect to reappear.

The bouncing effect is yet another example of how our auditory system is linked to the visual system. It’s part of the reason that staged fistfights and swordfights seem “real”—we’re often fooled even by fake punches that “miss” by a foot or more, as long as they’re accompanied by an appropriate sound effect.

Comments

  1. #1 Bill Banks
    October 21, 2005

    Motion and sound is great. Shimojo took it off of his website, but I’m glad I found yours.

    Do you have a version i can download to show in class?

    I could make it myself but generally I find stealing is easier than doing it myself.

    Bill Banks

  2. #2 Dave Munger
    October 22, 2005

    Bill—

    You should be able to download these videos just by right-clicking. Let me know if it doesn’t work for you.

  3. #3 Bill Banks
    October 23, 2005

    Thanks for your response. I’d not thought of the right-click. I mainly use Macs and only one of them has a 2-button key.

    Meanwhile I discovered that I can open directly from Quicktime, just by opening the URL in Quicktime.

    Bill

  4. #4 Dave Munger
    October 23, 2005

    Actually, I’m mainly a mac user myself. In that case, ctrl-click should do the trick.

  5. #5 Mikhail Zvyagintsev
    October 28, 2005

    Hi, Dave! What do you think – Would It be interesting to try to get opposite effect – when visual stimuli affect of perception of auditory one? Actually, I consider to try it…

  6. #6 Dave Munger
    October 28, 2005

    Mikhail–

    You should definitely try it—sounds like a great idea! Let us know how it turns out.