Cognitive Daily

This is a guest post by Martina Mustroph, one of Greta’s top student writers for Spring 2007.

When you’re typing, your senses of touch, hearing, and sight align. You feel, see, and hear your fingers touch the keyboard. Now imagine that you are outdoors and you feel a drop of water hit your hand. If you are like me, then it probably immediately occurs to you that it was a raindrop, so you stretch out your hand to see if more will come, and you look up at the sky for menacing clouds. Let’s say the sky is blue and clear as far as you can see. Now your senses of touch and sight are at odds: your sense of touch just told you it was raining, but your sense of sight said it was not. In this case, you don’t go running for cover; you choose to go with the information you get from your sense of vision and not the information you got from your sense of touch, probably because you only felt that one drop.

But what if you don’t get much more information from one sense than from the other? A team led by Jean-Pierre Bresciani showed people flashes on a screen. At the same time, a device tapped these people’s right hand a certain number of times. The screen was set up so that it obscured people’s right hand from their field of vision, and the flashes occurred where the right hand would be. People could only feel–but not see–their hand being touched. People were instructed to either count the number of flashes they saw on the screen or the number of taps they felt on their hand, but they were never told to pay attention to both simultaneously. The number of taps given differed from the number of flashes by plus or minus one. Why just one? It goes back to the rain example: when you have tons of information (miles of blue clear sky) from one sense telling you something and little information (a raindrop) from the other sense, it’s easy to pick one sense over the other. Likewise, if you are shown 2 flashes but feel 5 touches, it’s easy to dismiss the information you get from one sense over the other.

Now let’s look at how people performed. First, people were only given taps or only given flashes to see how much their counts would vary when only one sense was stimulated. People’s counting of taps was less variable than their counting of flashes. Here we need to distinguish between variability and accuracy. If Tom is repeatedly shown 2 flashes but keeps counting 3 every time, Tom’s count is reliable (it has low variability), but that doesn’t make it accurate. Variability and accuracy are not the same thing. Keep in mind that at this point, people are only shown flashes or are receiving taps. When this is done, touch is the more reliable sense. Then people were given both flashes and taps simultaneously. They were never told to pay attention to both flashes and taps, yet apparently, people do automatically pay attention to both. How do we know this? Even though they were told to focus just on the taps or just on the flashes, their accuracy of counting changed whenever the second number of events differed from the one they were focusing on (in other words, the added taps or added flashes messed them up). Take a look at this graph of the results:

The graph shows just one case: when three flashes were shown. In this case, as you can see, people believed they saw more than three flashes when they were tapped four times, and less than three flashes when they were tapped twice, so the number of taps affects the perception of flashes. The pattern also worked when two or four flashes were shown. When people are given one more tap than the number of flashes, suddenly their count of the number of flashes rises, once again even though the actual number of flashes they saw did not change at all. If the taps were not messing people up, the line on the graph should have a slope of 0; it would be a straight horizontal line.

Did their counting variability also change? Yes! Take a look at this graph:

People are significantly more consistent (although less accurate!) at counting the number of taps (but not flashes) when they are given flashes to go along with the taps-and here is the amazing part–even when the added flashes differed in number from the taps they were counting! This means that we are automatically processing the added background stimulus, because if we weren’t, people’s counts should not change when the second stimulus is added. Yet the range in the reported numbers of perceived taps decreases when both flashes and taps are given simultaneously.

One final thing: At the start of the experiment, Bresciani and his team found that when people are only shown flashes or only given taps, their counting is very reliable. When people are shown both flashes and given taps at the same time, there is another interesting thing that happens. Even though you’re paying attention to flashes, feeling a tap messes you up. It only sort of works the other way. The effect of touch on vision is more pronounced than the effect of vision on touch. When you’re paying attention to taps, vision only rarely messes you up. Seems strange, right? Actually, that’s explained by touch being the more reliable sense. We appear to give more weight to the more reliable sense. The fact that vision, the less reliable sense, still affects people’s counting of touch, the more reliable sense, means that we automatically process both, but then treat each sense with a weight corresponding to its relative reliability. If we just blocked out the information we get from one sense, then when counting taps, people should not be less accurate when flashes are added, but their count is affected by them, meaning that they do process the flashes.

It seems like this is the optimal way for our brains to work. Going back to our rain example: Ever felt a drop of water land on your hand and look up to indeed see a few clouds overhead that you just hadn’t noticed? In this case, you conclude that it may indeed rain soon. What you really did in coming to that split-second conclusion is combine information from both senses to assess the situation in front (or above) of you, and that combination of information from both senses helps you come to an appropriate decision.

Bresciani, J-P., Dammeier, F., & Ernst, M.O. (2006). Vision and touch are automatically integrated for the perception of sequences of events. Journal of Vision, 6, 554-564.

Comments

#1 Michael Chermside
June 15, 2007

Interesting article. I have one quibble. To explain why touch interferes with vision more than vision interferes with touch, you write “[it is] explained by touch being the more reliable sense. We appear to give more weight to the more reliable sense.”

I disagree. I doubt it has anything to with “reliability” of the senses, but is more likely to have to do with what part of the brain is processing them. Touch is a more “primitive” sense (in the sense that it evolved before sight did). It is also more “critical”. If you see one lion and you feel another, the one to worry about is the one you’re feeling!

Aside from that minor change in interpretation of the results, I find this an interesting idea. Thanks for posting it!
#2 Ken Watts
June 15, 2007

It would be interesting to see if the results would differ if the visual cue was more appropriate to the touch cue: for example, if instead of a tap, it was a tiny shock, or instead of a spark on the screen it was a CG picture of something touching the hand.
That way, one could assess how much our previous connection in experience between two such events affects the weight we give to the respective senses.
#3 Zachary
June 16, 2007

I would be interested to see how the the experiment performs with flashes and beeps of sound. It seems that tactile and visual information go together. You can usually see something touching you, and you can usually touch something you see.

I would imagine focussing on hearing would interfere with vision significantly. When I used to go to the pediatrician for those simple hearing tests (where you hear beeps in either ear of various frequencies) I would stare off into the distance and focus on the beeps, trying to visualize where I heard them. Similarly, if I was out in the woods playing tag with friends as a little kid and heard a twig snap, I would often stand still and listen intently for where the sound came from. Then move my eyes to where I perceived the sound, not the other way around.

Hearing shares a similar set of spatial features as vision. Tactile sense spatial area is limited to your immediate body, whereas both vision and hearing are projected into your surrounding environment.

I’d be interested to see how those results came out
#4 Tony Jeremiah
June 16, 2007

Interesting study. There’s been similar research in development psychology concerning the developmental of intermodal perception in children. As examples, newborns can learn associations between the sights and sounds of toys; can relate lip movements with specific speech sounds; link emotional tone of speech with facial expressions; match voices with faces on the basis of gender; and in relation to the study reported here, are particularly sensitive to the temporal and rhythmic sequences of sights and sounds, which is an important precursor to social and language communication. (I wonder how autistic children would perform in the study reported here).

I’m curious as to whether the study controlled for the possibility of a temporal sequence confound since there are two possible temporal sequences for presenting taps and flashes differing by 1. As an example, you can have 3 simultaneous flash-taps followed by a tap; or you can have a tap followed by 3 simultaneous flash-taps. The former might lead to reporting of a greater number of flashes (while the latter might lead to accurately reporting the correct # of flashes), if one’s short-term memory system focuses on the last piece of information stored in short term memory (i.e., the tap).

Additionally, it is known that different sensory systems vary in terms of how long sensory information persists in their respective memory systems. For example, visual information last a shorter time in visual memory while auditory information lasts longer in auditory memory. So on a practical level, you are more likely to be able to remember a sequence of letters if presented to you auditorily than if presented visually. Touch stimuli might also leave a longer memory trace than a visual flash, which might explain the differences reported in the study. So in this instance, I’d bet that if a person were asked to remember the sequence in which their fingers are tapped by recalling which ones were actually tapped vs. recalling the sequence by watching it in a video presentation, the former is likely to lead to greater sequence recall.

So memory differences might play a strong role in the data reported here.
#5 roseindigo
June 17, 2007

“So memory differences might play a strong role in the data reported here.”

Memory differences also play a strong role in the way people learn. Some individuals learn better by hearing, some learn better by reading, some by writing, some learn best by doing; and some use a combination of all of them, but favor one or the other. I think in particular teachers needs to understand that when teaching children. They don’t all learn in the same way because they have memory differences.

I have noticed that for myself the best way to learn has been to read something, and then write it or draw it. Once down on paper it is also forever in my memory bank and can be recalled at will. With certain tasks only “doing” them repeatedly will store them away to be retrieved at a later date, such as when using any sort of machinery.
#6 Tony Jeremiah
June 17, 2007

Hi Roseindigo,

Your comments are consistent with Kolb’s learning styles and Gardner’s concepts of multiple intelligences.

In general, the principle that memory differences are involved also applies to what mode people best learn by (hearing, reading, writing, doing), because these different learning modalities activate different neural systems in the brain which must ultimately report to the brain’s memory system. As many neural network models imply, individual differences in “memory” for each modality is likely realized as the strength of the neural connections between each modality and the memory system. These strength differences might explain why there are individual differences in relying on particular modes of learning. Additionally though, the differences in neural connection strengths themselves can be a product of biological factors, the learning process itself (i.e., a person may come to depend on a particular mode of learning due to environmental demands for a particular modality to be used), or a combination of the two.

Furthermore, different tasks are better suited for different learning modalities. As an example, one is more likely to learn more about driving a car, riding a bicycle, or practicing medicine by actually doing these things rather than reading, writing, or hearing lectures about them. Although one could argue that more academically oriented persons might learn these skills faster through reading, writing, and hearing, maximum ability at the tasks associated with these skills is realized through doing them.

But still, the entire process boils down to memory since learning (and differences in people’s ability to learn and method of learning), boils down to variations in the encoding, storage, and retrieval of information.
#7 roseindigo
June 18, 2007

Thanks Tony, for the great explanatory post. It fills out the lecture I heard about this subject which was given by a psychologist. I’m a botanical artist, and once I’ve really looked at a flower, analyzed it, and put it down on paper with paint, I can usually retrieve the shape and colors from memory at a later time. But yes, I do agree that there are some things one can learn best by “doing”.
#8 rogue77
June 30, 2007

I seem to recall reading that vision is strongly dominant over other senses. Hearing: the ventriloquist effect, in a movie or TV the voice seems to come from the actor’s mouth even with a mono sound system. Touch: when looking through glasses that shift the picture of your hand away from its real location, you feel it where you see it. Maybe the senses work differently for location and for counting or timing?