Eye movement, visual memory, and peanut butter sandwiches

i-eca0cf2af9fc3ac4445c7dff7d8aab70-research.gifExperiments on change blindness have revealed striking limitations in visual memory. Take a look at the video below, for example (click to play -- and note that the video is contained in a java applet that may take a while to load -- but it's worth it!).

The woman is giving directions to one "construction worker," when two other "workers" carry a door in between them. A new worker (all of them are actually actors hired by the experimenters) is substituted during the hubbub, and the woman continues giving directions as if no change has occurred. This work has been duplicated in a variety of different environments.

i-497b29b35c8d949871898763b68a348f-change.jpg

How is it that visual memory is so inefficient? If we can't remember the person we were talking to 2 seconds ago, how can we even function in the world?

Recently, vision researchers have begun to take a different approach to studying visual memory. Instead of looking at isolated events, out of context, they are analyzing real-world activities, with real goals.

A team led by Mary M. Hayhoe performed such a study on volunteers as they made peanut butter and jelly sandwiches. The participants wore headgear that simultaneously tracked the movement of their eyes and videotaped the scene before them. After the task was completed, the eye-tracking data was mapped onto the video, and the precise movements of the eye could be tracked to a precision of 1/30 of a second.

Here's an example of one such video (click on the image to play):

i-60c0c6f0d479cc53d1a3488db73697db-vmem.jpg

The small white circle is an indication of the focus of visual attention. Believe it or not, this small area is the only part of your field of view that you can have in precise focus at any time: it corresponds to the region called the fovea inside your eye: a tiny field with a much higher density of cones -- vision receptor cells -- than the rest of the eye. You build mental pictures of scenes through saccades -- quick eye movements between areas of visual interest.

At the beginning this video, the sandwich-maker's eye is fixated on the knife as she sets it down on the plate. But before she's even let go of the knife, she saccades first to the jelly jar, then to the lid. As soon as she grasps the lid, she saccades again to the jar, where she sustains fixation until she's successfully screwed the lid back on.

Now take a look at this movie, which shows how the participant first explores the sandwich-making apparatus:

i-5b50d008b426465d63d63269be6dd8c3-vmem2.jpg

As soon as the bread is revealed, she quickly fixates on it. Then her gaze shifts to the plate, the knife, the peanut butter, and the jelly. Once the dropcloth is out of the way, she fixates again on the bread in preparation for the task ahead.

Hayhoe et al. analyzed fixations on a frame-by-frame basis, and found that many fixations lasted less than 100 milliseconds. Take a look at this chart of one participant's fixation durations:

i-622fba3225754c9f08dab051786e1866-vmem.gif

A large portion of the fixations last less than 200 milliseconds, with some as short as 60 milliseconds. This is surprising because several studies have previously found that simply preparing the eye to saccade takes 200 to 250 milliseconds. If many fixations last less than half that time, it suggests that saccades must often be planned in advance -- even several saccades at a time. In the second video above, you can see that the left hand moves to pick up the jelly lid nearly a second before the eye actually saccades to the lid. In the meantime, the eye has fixated on the plate, then the jelly, suggesting that saccades may be planned well in advance.

If this is true, Hayhoe and her colleages argue, then perhaps visual memory isn't as limited as once was thought. In order to plan several saccades in advance, visual memory must maintain a detailed representation of the location of the relevant fixation points.

It is possible that the objects aren't actually maintained in visual memory. Perhaps the rough focused image of the scene in front of the person serves the function of memory; perhaps we don't need accurate visual memory because the very scene we might need to remember is right in front of our eyes. While they acknowledge this may be partially true, Hayhoe's team points to other research showing that objects are more easily found when they have been fixated a few seconds earlier. This is consistent with the evidence shown in the third video above, where the participant first locates each of the objects she will need when making the sandwich.

So though change blindness represents an important limitation of visual memory, Hayhoe et al.'s study demonstrates that the visual system does appear to retain key information about a scene, when that information is relevant to the task at hand.

Apparently, a peanut butter sandwich is a powerful motivator for visual memories!

Hayhoe, M.M., Shrivastava, A., Mruczek, R., & Pelz, J.B. (2003). Visual memory and motor planning in a natural task. Journal of Vision, 3, 49-63.

More like this

Since yesterday I talked about blinking I figured that today I would re-post something from the old blog about another important eye movement that impacts our day to day life as much or more than blinking - The Saccade. Without further ado, here's the old post (with some edits including a…
When we look at a scene, it often seems as if we perceive it all at once. Yet in fact, we are physically able to accurately view only a tiny portion of the scene at a time. Take a look at this image of a human retina (the back of your eyeball). The portion of the eye responsible for accurate vision…
Take a look at the short movie I've linked below (Click on the picture to play. QuickTime required). The movie shows a virtual gripping device (two red balls) lifting rectangular objects and placing them on a conveyor belt. Do you notice anything unusual happening as the objects are being moved…
Cataract 3, Bridget Riley, 1967. In the 1960s, the British artist Bridget Riley began to develop a distinctive style characterised by simple and repetitive geometric patterns which create vivid illusions of movement and sometimes colour and often have a disorientating effect usually described…

I'm trying to figure out whether there's an explanation here for something I do with surprising frequency in sandwich building: I find myself about to try to put the jelly lid on the peanut butter jar, or even the ketchup lid on the mayo jar.

re lids:
Aa I get older (57+) I sometimes put the coffee into the cup, not the plunger. Stimulus generalisation?: "put it into a nearby container"? - accurately into the cup!

Very interesting. I'm wondering how this might be related to reading and reading difficulties. I tutor kids with dyslexia, and I'm always looking for more information to help them. Could this research reveal a problem with tracking, or not properly sequencing the items, or general visualization? Or is this relevant to reading at all?