I have to take this opportunity to express a bit of disappointment in one of my fellow SB'ers.
When I encounter a study that seems to confirm my biases, as a skeptic, I try very hard to be even more skeptical than usual, because I would hate to be caught trumpeting a weak or bogus study as evidence supporting a belief of mine. That would be very embarrassing to me. At the very least, although I might not always succeed, I usually try to be very candid about limitations of studies that I cite. Unfortunately, yesterday, Bora (via Archy) failed to heed that rule. Indeed, he clearly let his politics overwhelm his critical thinking skills when discussing a study. Even Tom Tomorrow fell for it, but I've come to expect that of him, given how utterly unskeptical and downright credulous he's been in the past about, for example, the pseudoscience of the mercury militia. No surprise, either, that Daily Kos gleefully lapped up this twaddle and that the Kossacks are presently merrily gloating over it, with only a precious few comments expressing any critical thinking or skepticism about it at all. Worse, even Sara Robinson over at the usually sane Orcinus fell for this dubious study hook, line and sinker (as did a fellow skeptic Southern Fried Skeptic). It's obvious that the suppression of skepticism when it comes to comforting studies is not a phenomenon restricted to the right.
Here's news story about the study that Bora posted about and that's spreading throughout the liberal blogosphere:
A collective "I told you so" will ripple through the world of Bush-bashers once news of Christopher Lohse's study gets out.
Lohse, a social work master's student at Southern Connecticut State University, says he has proven what many progressives have probably suspected for years: a direct link between mental illness and support for President Bush.
Lohse says his study is no joke. The thesis draws on a survey of 69 psychiatric outpatients in three Connecticut locations during the 2004 presidential election. Lohse's study, backed by SCSU Psychology professor Jaak Rakfeldt and statistician Misty Ginacola, found a correlation between the severity of a person's psychosis and their preferences for president: The more psychotic the voter, the more likely they were to vote for Bush.
But before you go thinking all your conservative friends are psychotic, listen to Lohse's explanation.
"Our study shows that psychotic patients prefer an authoritative leader," Lohse says. "If your world is very mixed up, there's something very comforting about someone telling you, 'This is how it's going to be.'"
The study was an advocacy project of sorts, designed to register mentally ill voters and encourage them to go to the polls, Lohse explains. The Bush trend was revealed later on.
How bad is this? Let me count the ways.
From what I can tell based on the story, this study reeks of really bad design and post hoc analyses, and, sadly, Bora and much of the liberal blogosphere fell for it hook, line, and sinker just because it reinforces their own political beliefs. Let's take a brief look at the reasons this is almost certainly a worthless study. First, here's the first:
Rakfeldt says the study was legitimate, though not intended to show what it did.
"Yes it was a legitimate study but these data were mined after the fact," Rakfeldt says. "You can ask new questions of the data. I haven't looked at" Lohse's conclusions regarding Bush, Rakfeldt says.
"That doesn't make it illegitimate, it just wasn't part of the original project."
Aarrrgh! That last statement causes intense pain to my brain! Did this guy ever take a course on the scientific method and statistical analysis? Worse, he's basically admitting that he hasn't looked at Lohse's conclusions about a correlation between psychosis and voting for Bush. Unbelievable! If Lohse were my graduate student, I'd have slapped him down hard for publicizing his new conclusions before I had had a chance to look at them in detail and decide whether I thought they were a valid extension of his original thesis work. After all, my reputation would be on the line as much as his. I might even boot him from my lab.
Rakfeldt is only partially correct at best. The post hoc analysis of data gathered for another purpose and data dredging (a.k.a. data snooping, which occurs when a given set of data is used more than once for purposes of inference or model selection) used to come up with this result don't make the study "illegitimate." However, they sure as hell make its conclusions very, very suspect at best, particularly given that there were only 69 patients examined in the study. If Lohse had wanted to do data mining correctly, he'd have to use a technique that we use when looking for patterns in huge data sets of microarrays. Basically, he needs to divide the dataset randomly into two subgroups. Then he needs to use one group for hypothesis generation and then test the hypothesis on the other group. Only if the second group supports the hypothesis generated is it reasonable conclude that the hypothesis might be valid. Ideally, one should then validate this hypothesis again on an entirely different dataset unrelated to the first one. It doesn't sound to me as though Lohse did any of that, and, even if he did, the number of patients was almost certainly too small to draw valid conclusions about such an issue.
Worse, Lohse completely ignores one major potential explanation for his result. Note that the study looked only at the 2004 election (when Bush was running as an incumbent) and that the other study cited by Lohse as consistent with his results was a 1977 study in which psychotic patients were found to prefer Richard Nixon over George McGovern in the 1972 election (an election when Nixon was running as an incumbent). Couple that with another observation in Lohse's study:
The study used Modified General Assessment Functioning, or MGAF, a 100-point scale that measures the functioning of disabled patients. A second scale, developed by Rakfeldt, was also used. Knowledge of current issues, government and politics were assessed on a 12-item scale devised by the study authors.
"Bush supporters had significantly less knowledge about current issues, government and politics than those who supported Kerry," the study says.
Did Lohse ever consider that the true correlation he had detected is not what he thinks it is (i.e., a correlation between support of President Bush and psychosis) but rather a correlation between psychosis and not knowing much about current issues, government, and politics? In other words, did he even bother to consider th alternative hypothesis that, the more psychotic the voter, the more likely he or she wouldn't know anything about the other candidate in an election, who is not an incumbent? Perhaps the idea that psychotic patients prefer "comforting" or "authoritarian" choices is correct, but a perfectly reasonable alternative explanation could still be: The more psychotic the patient, the more likely that patient is to vote for the incumbent because the incumbent is known to him, already the President, and therefore a more comfortable choice. This could easily be tested by looking at data from the 1996 and 1992 elections, when incumbents were running, and comparing to the 1988 and 2000 election, when no incumbents were running. (I do note that these might not be the best comparisons, because both George H. W. Bush and Al Gore ran as sitting Vice Presidents in 1988 and 2000, respectively.) Or, even better, Lohse could design a prospective study for the 2008 election, which would actually test the hypothesis, rather than relying on crappy post hoc analysis and data dredging of a small study designed to look at an entirely different hypothesis.
Instead, he appears to have published by press release before his work was even accepted by a peer-reviewed journal.
In any case, the story suggests that Lohse didn't even bother to consider this rather obvious alternate hypothesis. (Believe me, I'm not a social scientist or psychologist; so if this alternate hypothesis is obvious to me it should be mind-numbingly obvious to a real social scientist.) No, Lohse zeroed right in on the correlation he wanted to zero in on, in much the same way that antivaccination zealots zero right in on vaccines as the "cause" for the alleged lower rate of autism in the Amish while ignoring all sorts of other alternative hypotheses about the Amish that could explain such a lower rate (if it even exists). Lohse may be correct that psychotic people prefer an authoritarian, but it could also just be that psychotic people know less about issues and candidates and therefore do what most people ignorant of the candidates and the issues do: Vote for the incumbent. Lohse may be correct, but we'd never know it from this amazingly flawed study. His failure even to consider alternate hypotheses, coupled with the post hoc analysis and data dredging, guarantee that.
No one could ever accuse me of being a fan of our current President, but unless there's something that is not being reported that makes this study more scientifically sound than it appears at first glance in the the news report, my tentative (and probably correct) conclusion is that this study, to put it bluntly, sucks. And sucks hard.
It's exactly the sort of study that provokes snorts of justifiable derision from scientists in the "hard" sciences, and that's exactly the reaction I had when I read the story about this study--that, and a little bit of embarrassment for Bora, Orcinus, Tom Tomorrow, and all the others who were so quick to buy into this because it confirms their preconceived notions.
ADDENDUM: Here's the only other skeptical take on this study that I've been able to find thus far. Unfortunately it is rather partisan.
ADDENDUM #2: I've followed up on this piece.
They can snow all their clients / by calling it "science" / although it's only sociology! — Tom Lehrer
I did not trumpet it. I reported on its existence, placed it in the context of previous research, and provoked the comments to hammer at it.
As the old adage says, "Never lets facts stand in the way of a good story." Thanks for doing the research on this.
But it was such a good story dammit!
Which you just did.
Oh, please, Bora. Don't insult my intelligence.
You tried to make this heaping pile of poo sound more credible by doing that, and you pointed out at the end that Lohse was a "Reagan revolution" fanatic, as if that made the study more credible as well. Nowhere did I detect even a hint of skepticism or criticism of the study's methodology in your post, nor did I hear you inviting criticism of it.
The "commenters", not "comments"....
For someone actively seeking bias, one can see it. I am favorable to the Sulloway research I linked in the beginning. Some of the more recent MRI studies are much more suspect (Chris of MM has attacked them with glee). I cannot be biased pro or con the latest study in any way since I have not even had a chance to SEE IT yet, so I just reported what was in the media. This is blogosphere - it is my job to bring media reports to the attention of other bloggers so they can make their own opinions and blog about it, as you just did. And you did it well - some of the quesitons you posed I thought of myself: small sample size, correlation between knowledge and Kerry-preference (soemthing seen in many other polls of the general population, and certainly nothing for Republicans to gloat about). The incumbency factor in comparing two elections is something I missed because I was not around during the Nixon reign.
Oh, there is nothing wrong with data mining if done well (what you donna do if an unexpected pattern emerges in the data - ignore it?), and we have no idea if the people are not planning on a follow-up for 2008.
Stephanie Miller had a funny take on this today. Voice Deity Jim Ward prevented to be a bunch of famous psychotics (Manson, Murphy, Lechter) and when they would say things you might expect them to say would simply reply:
"Well, OK, then, we'll just put you down as voting for George Bush."
It was on the "Stand-up news segment."
I wasn't "actively seeking" bias. The very obvious faults of the study were so glaring that I was thinking of them in rapid succession as I was reading the story; very little thinking about it was necessary to come up with glaring holes. It's possible that the actual study itself might be less of a stinkbomb than the story makes it sound, but I tend to doubt it. Also, remember that this guy publicized it before the election.
As for the data mining issue, that was my point. This doesn't look like valid data mining. It looks like data snooping:
Bingo!
Bingo!
These guys used data from a set that was designed to examine another question entirely and tried to shoehorn it into their hypothesis about "authoritarian" preferences, while totally ignoring at least one equally if not more plausible alternate hypothesis.
There are proper statistical procedures to avoid the data snooping bias or "data dredging" while data mining. I could be wrong, but, s far as I can tell, Lohse didn't follow them. And even if he did, 69 patients almost certainly represent too small a number to be a valid sample or a good cross-section of the study population. Data mining usually requires large numbers of subjects.
Er, what exactly is "partisan" about what I wrote? As I said, I didn't vote for Bush - in fact, a look through my archives will show that I backed Kerry in 2004 - so I certainly wasn't writing from a pro-GOP viewpoint. There's also nothing "partisan" about saying that those who are excused from the full rigors of the law because of diminished responsibility should not be pushed to the poll booths as if they were in full possession of their wits. Finally, I must point out that Chris Lohse claims to be a Reaganite Republican, so being sharply critical of his shoddy little exercise doesn't necessarily say anything about one's political association.
In summary, your accusation of my being "partisan" has no worthwhile basis as far as I can tell. The most that can reasonably be said against me is that my tone is too abrasive for the taste of some (which, imho, is too bad for them).
It's also worth nothing that Connecticut went to Kerry by around 10%. If the majority of the patient population didn't vote for Bush, you'd see a pro-Bush trend even without any pro-incumbent bias if patients simply voted more randomly the more severe their symptoms were.
Was there any attempt to correlate voting patterns with the nature or severity of the symptoms/delusions, rather than just the diagnosis of 'psychotic patient'? Were they even active at the time? If so, how?
I might also vote Bush if a voice inside my head was telling me the undeniable truth that the world would be destroyed if I did not...
Y'know Orac, you mention that even though you're not a social scientist, you could see right through the crap in this study. Last I checked, Bora isn't a social scientist either, maybe you could cut him a little slack, since non-social-scientists sometimes have difficulty smelling the BS through the potpurri the researchers use to mask it.
With regards to the study, if there were any truth to it, you'd have seen one party or the other trying to capitalize on it, given how close some of these midterm elections were predicted to be. Either you'd have Republicans trying to convince psychiatric patients not to take their meds (hey, maybe the Scientologists are part of the vast right wing conspiracy), or maybe Democrats trying to encourage people to get treatment for mental illness. Also, given the potential impact of these elections on Medicine, you'd think that if this were true, the APA would have been all over it.
One other thing, though, that no one has mentioned is whether they controlled for income, education level, religiousity, or other variables that some studies have shown may affect party affiliation. If this were a serious study, they would have attempted to do so. This is especially important as it is not unthinkable that severity of psychotic symptoms could also affect those variables.
Of course the most obvious caveat with this study is that even if the results *are* correct, they have very limited applicability. "Most psychotics vote Republican", even if true, is not at all the same statement as "Most Republican voters are psychotic"! This study, even if correct, says little to nothing about Republican voters in general.
Of course, the statement that there are bad reasons to do or believe something says nothing about whether or not there are also any *good* reasons to do or believe the same thing.
It's not a worthless study, just an overextended interpretation. If most of the patients voted for Bush, then that means *something* (unless it's just luck, but I hope they were at least capable of enough statistics to rule that out), although as you point out it is not exactly clear what. Psychotics - at least that group of them - apparently *did* prefer Bush in that specific election.
I think you did a pretty good job pointing out the problems with that logic yourself. First of all, the sample size was only 69. Statistically, that's very small; you want around 1,000 to make solid estimations about the general populace, and they have to be randomly selected.
Which brings us to the other problem: these patients weren't selected randomly. They're just from three Connecticut locations, so they really shouldn't be used to extrapolate about the population as a whole. With such a small, localized sample, there's way too much possibility of other confounders.
Orac, I'm skeptical of the study too. My own criticism focuses on the few comments on the relevant thread on Kos that seem to be written by people with any sort of expertise about the subject, all of which express skepticism due to disagreement with previous studies.
When a topic is subject to many studies, a 5% statistical significance means nothing. Back when the Bell Curve was published, some researcher did a survey of peer-reviewed studies about race and IQ. There were seven, of which one showed a statistically significant racial difference (and the rest showed no differences or small differences in either direction). A political hack, like the people who wrote the Bell Curve, could easily seize that one study and scream, "Peer-reviewed research confirms my position."
Statistical significance is not irrelevant. Its just that there needs to be replication with several data sources. After all no one considers an experiment a true success until it has been replicated, right?
And the fact that the sample is non-random shoots this thing full of holes. A sample of 69 might be enough if the effect was very large, but if the sample is non-random its almost worthless.
Actually, my point was that this study was so obviously bad that you don't need training in social sciences to smell it out.
I would counter that the "interpretation" is so overextended as to render the study utterly worthless.
I don't think so: it reveals an interesting small-scale phenomenon, and therefore points to an interesting area for further research: trying to replicate the results on a larger scale and with a less localized sample.
Of course, if it turns out that a larger scale study *doesn't* produce the same results, then the original study will turn out to have no applicability outside its own narrow area and thus be ultimately pointless, but it's too early to conclude that. It could be the first sign of a genuine large-scale pattern, which would not be worthless at all even if it doesn't convincingly establish that pattern by itself.
In short, it does not support any firm conclusions, but as a *preliminary* result pointing to possible areas for larger scale followups, it may have some value.
This does not, of course, excuse the irresponsible reporting about it.
Chris,
I'd argue that unless they came up with some way to control for incumbency and increased randomness vs true preference, both of which don't seem possible based on what little we know of their data set, there is no possible way they could support their conclusion. The statistics don't even matter if your study design can't measure what you're looking for.
A version of the mGAF is available here here for the curious. mGAF is not a metric of psychosis, but of functional impairment. A patient could get a low mGAF score (lower = more impairment) without any psychosis being present. For example, a depressed patient who makes a serious suicide attempt or an anorexic patient who has severe medical problems as a result of his or her condition may not exhibit any psychosis, but would still get an mGAF of 8-10.
It's also worth noting that the outpatient population with a given mGAF would not be representative of the total population with a given mGAF because patients who pose a treat of bodily harm to themselves or others would be more likely to be in inpatient psychiatric treatment.
coturnix wrote:
"I did not trumpet it. I reported on its existence, placed it in the context of previous research, and provoked the comments to hammer at it."
You took an unpublished study whose credibility is already in question and went on about it under the title 'You gotta be nuts to vote for Bush!'. I'd call that "trumpeting".
I'm a liberal who hates Bush and also a person who does not have any background in statistics, and even I thought that it sounded like this person decided very prematurely to publish findings that would have to be borne out by much more specifically designed and larger-scale and more random studies.
It also bothered me because people who suffer from mental disorders are made into a sort of joke, and it offended many bloggers who take mental health meds as sort of exacerbating the current state of public opinion about their ability to do such things as vote.
What is the practical application of such a study?
Does this mean that Democrats need to target psychotics as a voting bloc in the next round of elections?
Do Republicans need to work with psychiatric hospitals to "get out the psychotic" vote?
Ugh.
This study makes my head hurt, not only because of its mediocre design, but because I can't think of any good applicable use of said study. Well, other than casting aspersions on those who support a particular political party.
actually anonimouse the glass is emptier than you think. This study looked at voting for Bush, not the Republicans. Since there is a 100% chance of Bush not being relected as president in 2008 this study has no relevance whatsoever. The perfect marriage of crappy methodology and useless conclusions.
According to Oliver Sacks, lunatics don't always side with Republicans.
Was there any attempt to correlate voting patterns with the nature or severity of the symptoms/delusions, rather than just the diagnosis of 'psychotic patient'? Were they even active at the time? If so, how?
I might also vote Bush if a voice inside my head was telling me the undeniable truth that the world would be destroyed if I did not...