The Frontal Cortex

Voodoo fMRI

Posted by Jonah Lehrer on January 5, 2009
(8)
More »

I just wanted to draw attention to two fantastic blog posts that describe a new paper by Edward Vul, a grad student at MIT, and colleagues at UCSD. The first post comes from Vaughan over at MindHacks:

I’ve just come across a bombshell of a paper that looked at numerous headline studies on the cognitive neuroscience of social interaction and found that many contained statistically impossible or spurious correlations between behaviour and brain activity.

Social cognitive neuroscience is a hot new area and many of the headline studies use fMRI brain imaging to look at how activity in the brain is correlated with social decision-making or perception.

This new analysis, led by neuroscientist Edward Vul, was inspired by the fact that some of these correlations seem to good to be true, and so the research team investigated. The abstract of their study is below, and it’s powerful stuff – indicating that many of the results are due to flawed analyses.

If you’re not familiar with neuroimaging research it might be useful to know that what a ‘voxel’ is before reading the abstract.

Essentially, brain scanners digitally divide the scanned area into a block of tiny boxes and each one of these is called a voxel (think 3D pixel).

This allows the scans to be analysed by comparing the activity or tissue density in each voxel to another measure – which could be the same voxel during another scan, or it could be something entirely different, such as a measure of emotion or social decision-making.

The Neurocritic has much more. One noteworthy aspect of this paper is that it calls out a few dozen brain scanning studies for very public criticism. (Vul is a brave grad student.) After reading his analysis, I’m rather amazed that these many of these studies got through the peer-review process, and ended up getting published at such eminent journals!

This latest paper continues what might be called the “fMRI backlash,” although the backlash is really being driven by people within the brain scanning field who are simply trying to improve the quality of the average fMRI paper. Here’s how I described the backlash in a Globe article earlier this year:

The scanners, these critics say, excel at measuring certain types of brain activity, but are also effectively blind when it comes to the detection of more subtle aspects of cognition. As a result, the pictures that seem so precise are often deeply skewed snapshots of mental activity. Furthermore, one of the most common uses of brain scanners – taking a complex psychological phenomenon and pinning it to a particular bit of cortex – is now being criticized as a potentially serious oversimplification of how the brain works. These critics stress the interconnectivity of the brain, noting that virtually every thought and feeling emerges from the crosstalk of different areas spread across the cortex. If fMRI is a window into the soul, these scientists say, then the glass is very, very dirty.

(8)
More »

Comments

  1. #1 Glenn
    January 5, 2009

    Just wondering if you watched the 60 minutes episode last night? The one in which both the journalist from CBS and the reseachers made fMRI imagery out to be the “next great mind-reader,” with one researcher making the unbelievable assertion that in just 5 years we’ll be reading people’s intentions with brain imagery.

  2. #2 Crusty Dem
    January 5, 2009

    OMG NO!! fMRI pseudo-scientists utilized bogus math to get the “correct” results?!?!?! That’s unpossible!! I can’t believe that the pressure of getting multi-million dollar grants to run multi-million dollar machines applied to fuzzy (at best) social science with complex statistical methods could ever lead to results that were anything but completely accurate..

    Of course, I’ve seen far worse behavior from post-docs trying to get faculty positions. Still, the whole field is a steaming pile. My opinion may be clouded by the fact that I have known several fMRI researchers and have found them to be well-versed, scientifically-literate snake-oil salesmen. But then, easy success usually leads to later problems…

  3. #3 bruce hood
    January 6, 2009

    Thanks for posting this…. I always thought that there was far too much cherry-picking going on using this technique and this BOLD paper (sorry) has really shown how bad the situation has become. Anyway, I have always been deeply suspicious of neophrenology.

    http://brucemhood.wordpress.com/2008/11/29/neophrenology/

    bruce

  4. #4 Thad
    January 6, 2009

    I saw the 60 Minutes episode, and my wife and I laughed as the computer “guessed” the thoughts of the producer. Some guess. The computer screen in the clip showed that the anaylsis had a 50-50 chance to guess each thought fMRI pattern, though we were led to believe the computer can sort through a large number of different patterns at the same time.

  5. #5 joneilortiz
    January 7, 2009

    But if all those studies Vul reviews are indeed fundamentally flawed (and not simply ‘off’), then the force behind them must have been, and must still be, cultural. When Vul picks apart and ultimately dismisses Takahashi’s 2006 study that purported to demonstrate how “Men and women show distinct brain activations during imagery of sexual and emotional infidelity”, the next question should be, ‘what motivated and created, out of thin air and through relatively sophisticated means, a study that assumed the highest legitimacy afforded knowledge today?’

    It is at moments like these that social scientists, the most equipped to intervene, ought to step in to show how, in addition to a science, cognitive neuroscience can also be an apparatus, an ideology, and a conduit for far-ranging, deep-seated biases. Strictly scientific solutions won’t be enough to solve these methodological problems; we’ll also need to understand the kinds of thinking that produce them.

  6. #6 Phil from Clifton
    January 8, 2009

    You can’t generalize a field by citing a few outliers, or some reports by the mass media who are simply trying to sell ad space. fMRI is a valuable clinical tool that has been proven useful in assessing whether metabolic activity in the body falls within nominal margins. Sure, anyone (ESPECIALLY post-docs) can take science data and turn it into science fiction with a sleight of hand (and a word processor…and a gullible publisher) but that does not invalidate an entire field of study. The only reason to be concerned is that the general public believes anything it sees on Fox News; I submit that you guys should be more selective about whose reports you believe. -Phil.

  7. #7 C.P.
    February 10, 2009

    Some of the research I’ve read that utilizes fMRI seems incredibly far fetched. I’m reminded of the joke by Brian Regan in which a whale researcher is recording whale sounds and claims to have figured out what the whales are saying. As it turns out, the sounds were from a squeaky door on the cabin of the boat.

  8. #8 Matthew Lieberman
    March 24, 2010

    For anyone interested, there was a public debate on Voodoo Correlations last fall at the Society of Experimental Social Psychologists between Piotr Winkielman (one of the authors on the Voodoo paper) and myself (Matt Lieberman). The debate has been posted online.

    http://www.scn.ucla.edu/Voodoo&TypeII.html