As a graduate student, I observed the nascent field of functional magnetic resonance imaging and thought to myself with some amusement "modern phrenology! Now with big, fancy, expensive equipment!" Count me among those who have never been terribly impressed with fMRI, and certainly not with its applications in what is known as social neuroscience.
Now we have this:
Late last year, Ed Vul, a graduate student at MIT working with neuroscientist Nancy Kanwisher and UCSD psychologist Hal Pashler, prereleased "Voodoo Correlations in Social Neuroscience" on his website. The journal Perspectives in Psychological Science accepted the paper but will not formally publish it until May.The paper argues that the way many social neuroimaging researchers are analyzing their data is so deeply flawed that it calls into question much of their methodology. Specifically, Vul and his coauthors claim that many, if not most, social neuroscientists commit a nonindependence error in their research in which the final measure (say, a correlation between behavior and brain activity in a certain region) is not independent of the selection criteria (how the researchers chose which brain region to study), thus allowing noise to inflate their correlation estimates. Further, the researchers found that the methods sections that were clearing peer review boards were woefully inadequate, often lacking basic information about how data was analyzed so that others could evaluate their methods. (Read Vul et al.'s entire in-press paper here.)
There's been a flurry of response in the blogosphere to this not-yet-published paper. There are many interesting things about it, including the changing way in which scientific debate is being conducted. Several of my Sciblings have posted in response to the controversy. I particularly like Ed Yong's post; if you weren't skeptical about fMRI before reading it, you should be afterward. (Side note: Ed Yong is a damn good writer!)
What interests me in all this, however, is the debate over words. Consider this:
Tor Wager, a Columbia University cognitive neuroscientist, whose work was not mentioned in Vul's paper but who helped prepare one of the rebuttals, says that it was important to respond both publicly and swiftly. "The public and the news media operate on sound bites, and the real scientific issues are quite complex." His complaints focus not only on the content of Vul's paper, but also on the authors' dictionâ--âspecifically, the title, and its use of "voodoo."
"When the conversation gets complexâ--âand with statistics it always isâ--âmany blog readers will form opinions based on very simple things," says Wager. "Like words such as 'voodoo correlations.' There's no reason to use such loaded words when making a statistical argument. The argument should be able to stand on its own."
Should it? Vul notes:
He and Kanwisher had previously written a similar paper discussing the statistical point on its own, and it went largely unnoticed.
Maybe a good argument can't really stand on its own against established dogma, without a little help. In any case
...the editor of Perspectives, Ed Diener, in conversation with Vul and his rebutters, has decided to strike the word "voodoo" from the paper's name. Yet for Wager and social neuroscientists, it feels like a hollow victory that's come too late, and they find themselves wondering why Diener and the reviewers approved the title in the first place. According to Wager, the paper has made grant administration officers more wary, and it has affected the peer review process: "Everyone knows about the voodoo thing now, even though it's getting taken out of the journal article," he says. "The idea is out there, and it's hard to correct."
Are these bad things? If peer reviewers were letting papers slide through without adequate attention to things like the possibility of nonindependence errors, that would be a bad thing for everyone, right? So increased scrutiny can only improve the quality of published work, no? Isn't that what we all want? Shouldn't an argument be able to stand on its own, Mr. Wager? Or do some social neuroscience arguments only "stand on their own" when we don't pay too close attention to the methods and statistical arguments?
Mr. Wager objects to language like "voodoo" because it creates an "emotional effect" that affects public perception. The implication here is that the type of language Mr. Wager approves of creates NO emotional effect and has NO affect on public perception. And yet this is crazy. Using language that is more traditionally associated with scientific discourse creates its own emotional effect - it creates the effect that the speaker is speaking without emotion, is completely rational and objective, has no vested interests or biases, and can be trusted, even when some or all of those statements are unfounded. The discourse of science most assuredly has an affect on public perception, and we would be crazy to pretend that we did not want to have such an effect. What Mr. Wager is really upset about is that the perception that bucket loads of cash should be poured indiscriminately into social neuroscience because those folks know absolutely what they are talking about and their work is incredibly important has been disrupted.
If we take the definition of voodoo as "characterized by deceptively simple, almost magical, solutions or ideas" then the use of the word voodoo in Vul et al.'s title is really not all that unreasonable, if a bit unusual, of a word choice. It's a bit disingenuous to focus in critique so much upon the use of this word, to complain about how it interjects "emotion" into what was (erroneously) supposed to have been previously an emotion-free space. Ed Vul has exposed the biases existing in an entire field of research, and his critic is upset because he used an "emotion"-laden word in his paper title? There's a lot of emotion present here, for sure, but it isn't in the word voodoo.
- Log in to post comments
I think you've mischaracterized Wager's argument.
Out of a 7 page rebuttal, only 1 paragraph deals with the use of the term "voodoo" and in that the complaint is made explicitly against "fraud" connotation of the word. The 'emotional' content of the word is not even mentioned in the rebuttal.
Clearly the connotation is the phrase "voodoo economics". A very apt determiner, really! I can't see why anyone would find it inappropriate.
Y'all are so culturally insensitive. Vodou is an old and facinating religious/mythical belief structure, and to liken fMRI to it is an insult to vodou practioners.
JT, I can hardly have mischaracterized Wager's argument, at least with regards to his complaint about the use of the word voodoo, seeing as how I was quoting his own words, at least as quoted in the Seed summary of the controversy. He did indeed make the complaint about the injection of emotions into the situation. As I stated in my post, my interest in this controversy lies particularly in this aspect, in the idea that normally there is no emotion, no bias, no prejudicing of the lay person's opinion, in what we think of as "normal" scientific discourse, but that somehow all that was injected by the use of an "emotional" word like voodoo.
I was reading Wager's rebuttal yesterday. I don't think he (or his other authors) understand the use of rhetoric. Their first argument against Vul & Kanwisher is that they V&K aren't specifically criticizing social neuroscience. Despite V&K acknowledging this in their manuscript. It's a pretty bad start to a criticism.
I'm all for provocative titles. A few years ago I had a philosophical article published in an ecology journal. There was another philosophical article in the same issue. Mine was called "The Anarchist's Guide to Ecological Theory. Or, we don't need no stinkin' laws." For a time this was the most downloaded paper from the journal, and the other philosophical article was only third. The best explanation for the difference I can see is the appeal of my title.
Well, the excerpt you quoted has Wager objecting to "loaded words" and if you look at his rebuttal, he points to a specific loading of the word "voodoo" in the context of science writing, and that is "fraud".
You then proceed to say "It's a bit disingenuous to focus in critique so much upon the use of this word..." which implies that the 'focus' of Wager's objection is to the word, when, in fact, almost all of his rebuttal deals with statistics.
Anyhow, I guess that's almost tangential to the more interesting (to me) discussion. Regardless of who is correct about the voodoo=fraud issue, does that make it out of bounds? You've posited that striving for emotion in scientific discourse is desirable in some cases-- what is the limit then? If a scientist thinks another field or another's work is fraudulent, shouldn't that scientist be able call it out in public? Or will it just devolve into shouting matches. "You're a fraud!" vs. "You're a Ingorant Sensationalist!"
I've got an example in mind...
I subscribe to the "Eureka Alerts" newsfeed which monitors press releases on a wide variety of engineering and science issues. 95% of them are basic, incremental science, but many headlines screams out about how this particular piece of research is going to change humanity, or has some topical pun or twist. It's gotten to the point where I just don't read them that much-- or I end up parsing the exaggerated headline to take a guess if what's contained has something worthwhile. So I'd say that this example of adding emotion into science writing doesn't really change all that much. When everyone's shouting, I just turn the volume knob down.
Or the global warming issue has more than its share of personalities who consider each other frauds, conspirators or willfully blind. I'm not sure the appeals to emotion have helped folks understand the issue-- there tend to be short bursts of media attention whenever a contentiously titled paper emerges, and often the media coverage is bad enough that the headlines are discussed instead of what lies beneath.
Of course, emotion is part and parcel of this, so denying that it exists isn't useful either. It's not liked masked loathing is really much better than loathing.
or if this:
http://pages.cs.wisc.edu/~kovar/hall.html
were titled "Equipment deficiencies in an experiment to determine Electron band structure" or something like that, it's easy enough to read between the lines.
But I think it's reasonable the way things have played out -- leave the emotion to the blogs, online self-publication, and so on, and have the more formal publications be more measured.
--jt
A slight clarification on the 2nd sentence of my 3rd graph (should've looked at the preview longer)
The question I was trying to pose, is "If you replaced the word 'voodoo' with 'fraud', would that cross any lines"?
--jt
I think you are mostly missing my point. Which is, that it is disingenuous to pretend that normal scientific discourse completely removes all emotion and bias from our work and discussion of that work. It may serve to mask it in some ways, and to heighten some kinds of emotion at the expense of others but it does not completely remove it; this is impossible and, I would argue, not at all desirable. So complaining about a word like voodoo on the basis that it injects emotion into a discussion is not a sensible response to Vul et al. It does, however, reveal the emotions at stake in the discussion. Weiger wishes to claim the moral high ground of "rational", "emotion-free" scientific discourse while painting Vul with the tarbrush of emotion-laden, non-scientific rhetoric. All on the basis of the word voodoo, regardless of the validity of Vul's argument - which, we see, went ignored when it was put forth in the normal mode of scientific discourse.
It's interesting that you equate the word voodoo with fraud. I am not sure that Vul is accusing researchers of fraud, which would imply intentional scientific deception, no? Isn't he talking more about making mistakes and misuse of statistics, incorrect interpretation of data? Which would make the science wrong, but not necessarily fraud. People are pursuing "deceptively simple" ideas and being deceived by them. But not necessarily committing fraud.
I think I may see some of the misunderstanding. It wasn't me who brought up "fraud", it was Wager himself in his rebuttal (and I think it makes sense to talk about Wager's point of view in the context of the rebuttal)
http://www.scribd.com/doc/11471802/Correlations-in-social-neuroscience-…
(as linked too by the SEED magazine article you took Wager's quote from, hyperlink on the text "one of the rebuttals")
The very first words of the rebuttal are "The word 'voodoo' when applied to science carries a strong and specific connotation of 'fraudulence' as popularized by Robert Park's (2000) book, 'Voodoo Science: The Road from Foolishness to Fraud'"
And it's important to note that the discussion of the connotations of "voodoo" serve as the motivation for the rebuttal, not the substance of the rebuttal. So I'd say it's a mischaracterization to say that the connotations of the word were the 'focus' of Wager's objection to Vul et al.
I don't think Wager's trying to remove "all emotion" from the conversation in an effort to tar Vul. He is trying to point out that the emotion Vul has practical consequences, e.g. funding, out of proportion to the accuracy of the accusation. Combine that with the fast speed of the news cycle, and the slow speed of journal discourse and you can create a mess. He then goes on to attempt to rebut Vul's points. So his argument does NOT rest on the idea that Wager=rational Vul=Emotional. His argument is that Vul=wrong for reasons x,y,z, and that the tone of Vul's paper is a pain in the ass.
In another realm, think of the meme "Al Gore invented the internet".
http://www.snopes.com/quotes/internet.asp
It turned out to be false, but the bite it took out of his image would have occurred regardless of the accusations veracity. Similarly, Wager objects to the idea that the phrasing of a paper title, as opposed to the value of its content could affect his funding or public perception of his work.
Now the objection to raise here is that the phrasing of the paper ALWAYS affects how it is perceived, so Vul can do what he darned well pleases to get his opinions and data out there. And I think it's an interesting question-- how much should press releases and paper titles use Madison Ave. style techniques to be noticed?
I'd say that the fact that discourse is always emotion-laden doesn't mean that there shouldn't be rules and customs. It seems to me that the general rule "Let's not call each other frauds (unless you really mean it)" is a decent one. The only moral-high ground Wager's claiming is "Try not to be a jerk".
As I mentioned in the last post, I think there's more room to be a "jerk" online. Here, where the turnaround is hours, or days (and sometimes faster), there is ample time for a response to settle itself out before it enters the public consciousness. With the more deliberate pace of journal publications, I think a correspondingly more measured emotional tone is called for, because the pace of correction is so slow (if it happens at all), and the visibility is arguably greater.
As for the definition of fraud, I think there's a fine line between believing (maybe foolishly) in your data/data processing, being willfully blind, and intentionally deceiving.
--jt
Zuska,
I think you are letting our own, admitted bias against fMRI distort your view of this argument. Sure there are problems with the field, but that doesn't mean that every criticism is correct. One problem with the Vul paper is that they use emotional language to mask that their own analysis is rather lacking. If you read past the complaining about the term "voodoo" in the Wager paper, you'd read valid criticisms about selective picking of data, unclear methodology, assumptions about other people's methods that were without merit, misleading simulations, and conclusions not supported by the data.
At the very least, I hope you can agree that if you are going to fill an article with emotional terms and try to grab as much attention as possible, you better make absolutely sure that your own arguments are solid.
BTW, you realize that Ed Yong's post was about a completely different article and a completely different topic (I commented there too if you want to read my take on that article)
Yeah, my mistake on that link. His post got caught up in the Buzz on the SB main page, which was where I found it, and I had it in my mind when I was working on this post.
I think there are plenty of valid arguments to be made pro and con about the critique and the rebuttals both (some of which have been made here), but I am interested in a very small piece of what is going on, which might more properly be identified as the subtext (oh, that word!) of part of the rebuttal. It's kind of a side issue to the main points of the rebuttal I suppose but it is what interested me most.
The trouble here is that it's hard to separate the emotional argument from the scientific criticisms. This isn't a perfect analogy, but it's like finding at a group of feminist writers who all made the same logical fallacy. After that, inflating the significance of the fallacy, using emotional words to attract extra attention and then tar all feminist theory for not rooting out this one group.
If the original paper was rock-solid correct, I don't think there would have been as much complaining about the use of emotional words.
For anyone interested, there was a public debate on Voodoo Correlations last fall at the Society of Experimental Social Psychologists between Piotr Winkielman (one of the authors on the Voodoo paper) and myself (Matt Lieberman). The debate has been posted online.
http://www.scn.ucla.edu/Voodoo&TypeII.html