Clicking on the image below will take you to a short Quicktime movie. Make sure you have your sound turned up, because I've recorded a few sentences that play along with the movie. Your job is to determine, as quickly as possible, if each sentence is grammatically correct -- while you focus your vision on the animated display.
This demonstration replicates part of an experiment conducted by a group of researchers led by Michael P. Kaschak. The researchers showed similar animations to a group of volunteers and asked them to make similar judgments about spoken language. The question: does our reaction time differ when the animation corresponds to the movement described in language?
In the demonstration you just tried, the first two sentences were distractors. In sentence 1, the motion was "towards," but the animation was moving down. Sentence 2 was ungrammatical. The two sentences we're interested in are 3 and 4. Sentence 3, "The leaves fell from the tree," describes a downward motion, just like the motion in the animation. Sentence 4, "The balloon ascended into the clouds," describes upward motion, opposite the animation.
Kaschak et al. have good reason to suspect that in the case of these last two sentences, the animation may indeed affect how quickly you can process the sentence. We recently reported on data that suggests that impairment of motor control of the hands may also impair our ability to visualize the same motion. Further, memory for visual objects also appears to make use of the visual system. Kaschak's team points to other research showing that understanding sentences also involves a "sensorimotor simulation" of the action the sentence describes.
But prior to Kaschak's team's experiment, no one had tried to measure how quickly people process language when the motion they are viewing corresponds to an animated display. If viewing motion affects language processing, then there are two possibilities for how the two activities interact. It's possible that watching motion that corresponds to the motion in a sentence will cause viewers to process language faster (i.e. participants will respond faster to "the leaves fell from the tree" when the animation is moving down). Another possibility is that viewing the animation burdens the same region of the brain that is needed to process the language, so when the animation is moving down, then sentences describing downward motion will be processed slower.
Kaschak et al. showed participants four different animations depicting basic motions: moving lines showing up or down motion, and a spinning spiral that could show movement towards or away from the viewer. During each animation, 10 sentences were read: 2 corresponding to the direction of the movement in the animation, 2 in the opposite direction, and 6 distractors. The test questions were always grammatically correct so that each participant was performing an equivalent task. The distractors included some nongrammatical sentences (to keep the task realistic) and some grammatical sentences describing movement that did not correspond to the animation (like Sentence 1 above).
Respondents took an average of 369 milliseconds to respond to sentences that matched the direction of the animation, but only 330 milliseconds to respond to sentences that described movement in the opposite direction. The difference was statistically significant: people take longer to process sentences that match the movement of an animation than they do to process sentences that don't match it. Kaschak's team reasons that we must be using the same region of the brain to process the motion itself as we do to process the language describing that motion.
Note that these results are only for animations showing a very generic sort of motion. There's little doubt that if we saw an actual leaf falling, or balloon ascending, we'd be able to process that language very quickly. Yet the simple concept of "downward motion" does appear to distract from our ability to process a simple sentence describing a particular sort of downward motion.
We weren't able to measure how quickly you processed the sentences in our demonstration, but did you notice anything different about trying to assess the "down" sentence compared to the "up" sentence? Let us know in the comments.
Kaschak, M.P., Madden, C.J., Therriault, D.J., Yaxley, R. J, Aveyard, M., Blanchard, A.A., & Zwaan, R.A. (2005). Perception of motion affects language processing. Cognition, 94, B79-B89.
- Log in to post comments
I had no difference in reaction time for either of the last two cases. In both cases, the sentence registered as "gramatical" before it was even completed. In other words, I parsed "balloon ascended" as gramatical as "into the clouds" was being spoken.
I didn't register "towards the tree" as gramatical, however. In my version of English, the phrase should be "toward the tree" and "towards" sounds incorrect. This is a regional variation. The American Heritage College Dictionary (Third Edition) has the following usage note on "toward" vs. "towards."
"Some critics have tried to discern a semantic disticntion between toward and towards, but the difference is entirely dialectal. Toward is more common in American English; towards is the predominant form in British English."
For what it's worth, I was born and raised in Wisconsin. I wonder what the pattern of usage of "towards" is throughout the United States.
Kelly, towards/toward might be a good topic for a Casual Friday.
I think it's very difficult for an individual to measure their own reaction time for a task like this one (and my online programming skills aren't quite sufficient for me to automate the task).
Perhaps I was biased (since I created the demo), but I felt like it was definitely more difficult to process the "down" sentence than the "up" sentence.
I didn't measure my own reaction time, but my sense was that I took longer in the "up" sentence. I suppose this is why we try to avoid single trial studies.
I love this sensorimotor simulation stuff!
I believe Zwaan & colleagues did some cool studies recently, where participants read sentences like "... he turned the volume down" while turning a self-paced reading window in a counterclockwise (matching) or clockwise (mismatching) direction. It was found that people turned (& read) faster when the turning motion matched the motion implied by the sentence.
It's incredible that such subtle things can influence our reading and processing speed. Thanks for the summary!
PS: I think Zwaan's name was mispelled in the reference. :)
I enjoy exploring cognitive differences through another sort of filter as I'm neurologically atypical (I am autistic). Autistic people process information in strikingly different ways. In this case, it's largely impossible to really "see" the motion _and_ listen at the same time. For example, as the first sentence was spoken, my eyes immediately shot off to the left so I had to correct. By the third sentence, I realized that while I was viewing the image, I wasn't "seeing" it. Many people like me cannot hear and see at the same time. We can only do one or the other.
Thanks for the comment Kathleen -- that's fascinating.
I felt like there was a kind of 'expense' of noticing 'not' or 'opposite' for sentence 4 that was on the order of the 'expense' of attending to the grammar. That is, trying to see, parse and hear at the same time all drew from the same pool of attention, and noticing a grammatical error 'took' about as much time as noticing the reverse of the pattern set up by the correspondence of imagined movement in sentence 3.
So i wonder, is there also the third possibility, besides the alternatives you mentioned (motion with or against effecting response time)? Could parsing a sentence also be using the part of the mind that is looking for patterns between image and sentence. So the order of the sentences can set up transient relationships between the sentences and animations. When those relationships (between sound and image) change, one might find oneself taking a few extra milliseconds to register that. I would guess the study randomized the correspondences to neutralize that effect.
I also appreciate Kathleen's comment, and feel that i often attend to sight and sound in rapid alternation, and only synthesize them intermittantly and even then one sense tends to lead the other.
Is there any reason that the study was done with listening to a sentence and seeing, rather than reading a sentence and hearing, say a rising tone or something binaurally moving closer or farther away?
I didn't notice the difference at all. That is to be expected since as mentioned, the difference between same direction response and opposite direction response was 39 millisecs on average... I don't know if we can even notice that difference while thinking.. I'm kinda suprised it was considered to be statistically significant.
interesting study, though.
Cat: I just noticed your comment about the misspelling -- you're right, and I fixed it.