An Example of How Not to Use Imaging Data: Participants' Prejudice or Researchers'?

Back in May, a study by Mitchell, Macrae, and Banaji (of Implicit Association Test fame) was published in Neuron that made the following claim (from the abstract):

We observed a double dissociation such that mentalizing about a similar other engaged a region of ventral mPFC linked to self-referential thought, whereas mentalizing about a dissimilar other engaged a more dorsal subregion of mPFC. The overlap between judgments of self and similar others suggests the plausibility of "'simulation'' accounts of social cognition, which posit that perceivers can use knowledge about themselves to infer the mental states of others.

The study got some attention in the blogosphere (I didn't see it in the popular press), and then died. Or at least I'd hoped it had, but I just saw it pop up again over at BRAINETHICS in a post titled "The roots of prejudice." I'd hoped the studied had died because a careful reading of it (oh hell, it doesn't have to be all that careful; the study is that bad) shows that methodological flaws make it impossible to draw from it the conclusions that the authors do. But I'll get to that in a moment. First, I should probably describe the study.

The study starts out as an attempt to test a hypothesis of simulation theory (a competitor of "theory theory" in the domain of theory of mind). I'm working on a post on simulation theory, so I won't bore you with its details yet. For now it will be sufficient to say that simulation theory argues that, when we are trying to figure out what other people are thinking/feeling, we "put ourselves in their shoes" and simulate our own thoughts and feelings in the other individual's context. The results of such simulations are our inferences about other people's thoughts and feelings. Mitchell et al. argue (or, well, they state) that simulation theory predicts that "mentalize in a different way when the other is perceived to be similar to versus dissimilar from oneself." They focus on the medial prefrontal cortex (MPFC), because previous imaging studies have indicated that this part of the brain is involved in thinking about the mental states of others. Mitchell et al. report that a particular part of the MPFC, the ventromedial prefrontal cortex (VMPFC), has also shown an association with thinking about close friends as well as self-referential thinking, (important from a simulation theorist's perspective). The first sign that things are going to go awry comes at this point in the paper. They want to contrast the VMPFC with another part of the MPFC, the dorsomedial prefrontal cortex (DMPFC). Mitchell et al. write:

However, an important clue regarding the functional distinction between [the VMPFC and DMPFC] has recently
come from studies of self-referential processing, which have consistently observed selective engagement of ventral mPFC during tasks that require reporting one's own internal states. This ventral mPFC region (typically, within a few millimeters of the axial plane of the genu of the corpus callosum) has been linked to a variety of self-referential tasks (Northoff et al., 2006) such as reporting on one's preferences or personality (Johnson
et al., 2002; Kelley et al., 2002; Macrae et al., 2004; Moran et al., 2006; Schmitz et al., 2004; Zysset et al.,
2002), reflecting on one's current affective state (Gusnard et al., 2001), or adopting a first-person perspective
(Vogeley et al., 2004). Typically, dorsal mPFC activation has not been modulated during self-referencing
tasks.

Waaaaaait a minute. As someone who's very interested in both theory of mind and self-referential thinking, I happen to have read each of those papers, and so I know that this paragraph is just... wrong. First of all, the Gusnard et al. study argues that reflecting on one's current affective state activates the DMPFC1. And the Northoff et al. paper (which is a review of pretty much all of the others cited) states quite clearly that while the VMPFC is associated with self-referential thinking, in that self-referential tasks almost always produce a decrease in activity in the VMPFC, the DMPFC is also associated with self-referential thinking, which generally produce an increase in activity in there2. Nobody knows what those increases and decreases mean, really, but what they do not do is show that the VMPFC is involved in self-referential thinking while the DMPFC isn't. So before it's even started, the focus of the imaging analysis (on the VMPFC and DMPFC) is based on a misrepresentation of past research. (By the way, the Northoff et al. review also showed that posterior parts of the PFC are also associated with self-referential thinking; why not look at those too?)

Anyway, I was supposed to describe the study before criticizing it. Let me take a few breaths... OK, I'm ready. For the study, fifteen Harvard undergrads (9 male, 6 female) were presented with descriptions of two indivuals, along with pictures (taken from internet dating sites... hahahaha). One of the individuals was described as liberal, and the other as a fundamentalist Christian who was politically conservative (they were told that the descriptions came from the dating site too!). Participants were then strapped in to the fMRI machine, and peresented with 66 "opinion questions." Each question was presented individually, along with one of the pictures of the hypothetically dating-site individuals or the word "me," and participants were asked to rate on a 4-point scale how much the pictured individual or they themselves agreed with the question. Naturally, pictures of their brains were being taken during this phase of the experiment. Finally, in order to determine whether the participants were closer to either the liberal or conservative-side of the sociopolitical spectrum, they took a version of the Implicit Association Test.

It's at this point that things begin to fall apart. First, the participants, being Harvard undergrads, were heavily biased towards the liberal end of the spectrum. So Mitchell et al. decided to do something entirely arbitrary and divide the group in half, based on the median, with those on one side (those whose IAT scored indicated that they were the closest to the liberal end of the spectrum) of the median as "similar to liberal" and those on the other side as "dissimilar to liberal." There's no indication that these participants were actually closer to conservative than liberal in the IAT data. Mitchell et al. just decided to make them conservatives for the purpose of their anlysis.

Here, in essence, is what they found: activity in the VMPFC increased when rating how much a similar-to-self person -- the hypothetical liberal dating-site person, if you liberal, and the hypothetical conservative if you were conservative -- agreed with the question, while activity in the DMPFC went up when rating how much a different-from-self person agreed with the question. The correlations between the activity in the two regions and IAT scores are shown in this figure (p. 658):

Now, from a simulation theory perspective, this finding is perplexing. If anything, the pattern in the data is the opposite of what we'd expect from simulation theory. Since activity in the VMPFC always goes down when we're thinking about ourselves, we'd also expect it to go down if we're trying to figure out what other people are thinking by simulating how we would feel about it. Since Mitchell et al. predict that thinking about similar individuals mental states will be done through simulation, we'd expect to see a decrease in VMPFC activity in this case. But they found an increase in activity for similar individuals. But this doesn't phase Mitchell et al. Furthermore, as the abstract suggests, they draw even stronger conclusions from the data. In the discussion, they write:

To the extent that members of a social group other than one's own are viewed as dissimilar from oneself, the current results suggest that perceivers may actively deploy a different set of social-cognitive processes when considering the mental states of someone of a different race or ethnicity than a member of one's own ingroup. As such, prejudice may aris in part because perceivers assume that outgroup members' mental states do not correspond to their own and, accordingly, mentalize in a non-self-referential way about the minds of people from different groups. Without a self-referential basis for mentalizing about outgroup members, perceivers may rely heavily on precomputed judgments--such as stereotypes--to make mental state inferences about very dissimilar others. This view suggests that a critical strategy for reducing prejudice may be to breach arbitrary boundaries based on social group membership by focusing instead on the shared similarity between oneself and outgroup members. (p. 660)

But does the data really license anything like this speculation, or even the conclusion that same and different individuals recruit different mental processes in different brain areas? The Neurocritic offered a simpler, and perhaps more plausible interpretation:

It couldn't have anything to do with merely agreeing or disagreeing with "the other" now, could it? It really has to do with extracting the putative mental contents from the brain of a hypothetical individual, now does it? Not different affective states engendered by the agreement or disagreement? Not semantic knowledge about how certain "types" of people may act in certain situations? Hmmmm?

Does the first bar graph [the left graph above -- Chris] mean that liberals are a little less hostile to conservatives than vice versa? Does the other bar graph (below right) mean that the "Not Like Me" area in liberals is equally activated by "self" and "conservative other"?? What DOES it all mean?

Furthermore, despite Mitchell et al.'s attempt to arbitrarily divide the participants into "similar to liberal" and "dissimilar to liberal," as the Neurocritic notes in comments to his/her post, only three of the participants actually produced IAT results that would indicate they are actually more similar to conservatives than to liberals. This makes it very difficult to interpret the correlations as presenting a "same" vs. "different" distinction (as an anonymous commenter at The Neurocritic put it, "we're talking about analyzing functional imaging data using what is essentially RT noise").

Now, ordinarily I wouldn't post about a paper like this, because it hasn't received a lot of attention in the media, and thus most people are unlikely to have heard about it. However, after Paul Bloom's Seed Magazine article about the seductive nature of imaging studies, which some people seem to believe lend an air of "scientificness," I thought this would be a great example. Particularly since at least one brain scientist at BRAINETHICS appears to have been seduced by this particular study. That's partially understandable. Some of the problems with this study (the misuse of past research, for example) are subtle. Others, however, are basic methodological problems: an insufficient sample size (only 3 conservatives!), the use of a largely mysterious measure (the IAT) to classify individuals, and a task that doesn't distinguish between a range of possible processes in the brain regions the experimenters looked at. So it appears that the fact that it was an imaging study really was the seducing factor (it also appears to have seduced the reviewers and an editor at NeuroImage). So, let this be a lesson for you. And go read other posts at The Neurocritic, for further lessons.

¹Gunsard, D.A., Akbudak, E., Shulman, G.L., & Raichle, M.E. (2001). Medial prefrontal cortex and self-referential mental activity: Relation to a default mode of brain function.
²Northoff, G., Heinzel, A., de Greck, M., Bermpohl, F., Dobrowolny, H., and Panksepp, J. (2006). Self-referential processing in our brain - A meta-analysis of imaging studies of the self. Neuroimage,31(1),440-457.