A second crank finds Ioannidis

This time it’s Steve McIntyre representing for the anti-global warming cranks following the HIV/AIDS denialist lead and using John Ioannidis’ study to suggest science is bunk. Never mind that this research is primarily focused on medical studies. Never mind that the study wouldn’t even exist if replication in science didn’t identify in the first place. Cranks like to latch onto anything that they think is embarrassing to science out of the mistaken belief that it makes their nonsense more believable.

It’s funny, I was sure they would have picked up on this stuff years ago, but the critical event was clearly the publication of his findings in the Wall Street Journal (clearly the go-to paper for cranks). However, despite not appearing in the editorial page, it’s still a pretty poor analysis I’m sorry to say. The point of this research isn’t to say that medical research is bunk, or sloppy, the point is to understand that an over-reliance on statistical significance and emphasis on positive results will inevitably result in false-positives entering the literature, through no fault of the authors or the editors.

For an excellent analysis that is easy to understand, check out Alex Tabarrok’s discussion of the research. His analysis also benefits from actually discussing what Ioannidis’ research means for medical science, and how it can reform to diminish these effects.

What can be done about these problems? (Some cribbed straight from Ioannidis and some my own suggestions.)

1) In evaluating any study try to take into account the amount of background noise. That is, remember that the more hypotheses which are tested and the less selection which goes into choosing hypotheses the more likely it is that you are looking at noise.

2) Bigger samples are better. (But note that even big samples won’t help to solve the problems of observational studies which is a whole other problem).

3) Small effects are to be distrusted.

4) Multiple sources and types of evidence are desirable.

5) Evaluate literatures not individual papers.

6) Trust empirical papers which test other people’s theories more than empirical papers which test the author’s theory.

7) As an editor or referee, don’t reject papers that fail to reject the null.

Steven Novella also addresses this research and Tabarrok’s analysis and emphasizes the importance of prior probability in determining whether a study is reliable (Tabarrok’s #1). Simply put, hypotheses shouldn’t be tested simply because they can be thought of at random. There should first be some biological plausibility for the effect. This becomes hysterical when Novella expands his analysis to address what the research says about complementary and alternative medicine studies.

But there are other factors at work as well. Tabbarok points out that the more we can rule out false hypotheses by considering prior probability the more we can limit false positive studies. In medicine, this is difficult. The human machine is complex and it is very difficult to determine on theoretical grounds alone what the net clinical effect is likely to be of any intervention. This leads to the need to test a very high percentage of false hypotheses.

What struck be about Tabbarok’s analysis (which he did not point out directly himself) is that removing the consideration of prior probability will make the problem of false positive studies much worse. This is exactly what so-called complementary and alternative medicine (CAM) tries to do. Often the prior probability of CAM modalities – like homeopathy or therapeutic touch – is essentially zero.

If we extend Tabbarok’s analysis to CAM it becomes obvious that he is describing exactly what we see in the CAM literature – namely a lot of noise with many false-positive results.

Tabbarok also pointed out that the more different researchers there are studying a particular question the more likely it is that someone will find positive results – which can then be cherry picked by supporters. This too is an excellent description of the CAM world.

The implications of Ioannidis’ research, therefore, is not to undermine or abandon scientific medicine, but rather to demonstrate the important of re-introducing prior probability in our evaluation of the medical literature and in deciding what to research. As much as I am in favor of the Evidence-Based Medicine (EBM) movement, it does not consider prior probability. I have said before that this is a grave mistake, and the work of Ioannidis provides statistical support for this. One of the best ways to minimize false positives is to carefully consider the plausibility of the intervention being studied. CAM proponents are deathly afraid of such consideration for they live in the world of infinitesimal probability.

Considering scientific plausibility would also kill, in a single stroke, the National Center for Complementary and Alternative Medicine (NCCAM) – which is ostensibly dedicated to researching medical treatments that have little or know scientific plausibility.

The irony is that cranks have started to cite this article in some snide attempt to disparage science as a whole, but in reality this research is mostly about why you shouldn’t trust crank methods. Instead of cherry-picking results the emphasis should be on literatures. Instead of testing modalities with no biological plausibility and publishing the inevitable 5% of studies showing a false-positive effect, one should have good theory going in for why an effect should occur. Finally, Ioannidis’ study would not exist if it weren’t for the fact that the literature is ultimately self-correcting, and the false-positive effects that inevitably make it in ultimately are identified.

Comments

  1. #1 Bill H
    September 19, 2007

    I sometimes wonder who are the “cranks”.

    Should scientists be held to high standards (at least as high as accountants)?

    Should there be open access to methods and data?

    Recently McIntyre found a problem in the method used to make adjustments to some data, he had to “reverse engineer” what NASA had done and he told them what they had done. NASA waits for the firestorm to explode and then says it’s much ado about nothing. NASA also recently made changes in which data set it used McIntyre also spotted this. Is he really a crank for expecting high standards and open access?

    Information about global warming is improved because of those you dismiss as cranks.

  2. #2 MarkH
    September 19, 2007

    Oh yes, I know. When he discovered that flaw I wrote about it and saluted him for at least looking at the data.

    But then what happened? Did he just say, wow, I altered one year’s record so that now 1998 is within to margin of error for the hottest year in the US to being, well, still within the margin of error for the hottest year in the US? Did he emphasize that it was just the record for the US and not the world? Of course not. He hyped up that completely irrelevant little mistake and every crank on the internet rapidly jumped on it and acted as if it showed something significant about climate change.

    Yes he looks at data, but that makes him no less a crank. He’s a classic toiler, hunting, endlessly for something, anything to challenge the theory he dislikes. And then when he does find something insignificant, act as if it overturns the science.

    Then, examine his response to Ioannidis’ research. Do you not find it funny that he did the exact same thing as the HIV/AIDS cranks in regard to this article? Purposefully misunderstanding it to suggest the science he dislikes is BS?

    McIntyre is a classic crank.

  3. #3 Boris
    September 19, 2007

    If McIntyre stopped at suggestions for bettering data collection, that would be one thing, but his new baby is to suggest that Hansen deliberately changed data sets to make 1998 the warmest year in the US once more. This despite the fact that Hansen has never claimed 1998 was the warmest year. If McIntyre’s supposed to be the skeptics best man, why is he peddling conspiracy theories?

    The ClimateTurth movement is moments away.

  4. #4 booger
    September 19, 2007

    Um, I don’t understand #7. Can that be re-written in the affirmative?

  5. #5 MarkH
    September 19, 2007

    #7 could read: Allow papers to be published that do not show an effect or significant result (negative papers) if they are well constructed and sufficiently powerful.

    It’s been a criticism a lot of late, that we need to start allowing for the publication of more negative results, with the caveat that it must be clear the study design was good, and the study wasn’t negative because of some flaw.

  6. #6 Bill H
    September 19, 2007

    Hansen changes data sets and tells no one, why?

    Why does it take an email from McIntyre to get NASA to put that up on their site?

    Don’t you realize that the actions of Hansen give science a black eye in this case? If there is no jiggery pokery going on why what until McIntyre blogs and emails about it?

  7. #7 Davis
    September 19, 2007

    If science is bunk, as these cranks seem to suggest, then how would we even know scientific results were wrong? Last I checked, the only thing overturning scientific research was more scientific research.

  8. #8 Andrew
    September 19, 2007

    The cranks see scientific research being overturn by scientific research, but they misunderstand it. Somewhere along the line, the crank decides that one field of science is overturning another field.

    It’s only a hop, skip, and jump from that until they convince themselves that their rantings are a field of science. They then further delude themselves by creating truly bad “journals” that accept only articles that agree with their viewpoints.

    That is not say the current academia and journal system is without flaws. It is difficult to present good science when it is truly radical. The academic world today requires a slower evolution of ideas. Years of research won’t be scrapped overnight because of a single dissident study.

    I think this slow evolution of ideas is preferable and necessary to the alternative. At a cost of possibly silencing radical theories (rarities throughout the history of science), we manage to ignore crackpots still be arguing about light waves moving through ether.

  9. #9 Boris
    September 19, 2007

    If there is no jiggery pokery going on why what until McIntyre blogs and emails about it?

    Who knows? But it’s an odd conspiracy to switch datasets with no significant effect.

  10. #10 stewart
    September 19, 2007

    I’d also add to point 6, that papers written by opponents of a theory that corroborate that theory are the strongest support. Hence, McIntyre’s review of the GISS data, finding the modern climate trends are consistent, is the most ringing endorsement I can imagine. Similarly, the EPR thought experiments to prove the implausible paradozes of quantum physics, are strong evidence of quantum distance effects.

  11. #11 SLC
    September 20, 2007

    Re negative results

    The comment about publishing negative results is considerable merit. It should be recalled that what turned out to be, arguably, the most important experiment ever performed in the entire history of science, namely the Michaelson/Morley experiment, had a negative result.

  12. #12 Paul
    September 20, 2007

    Mark, You fail to understand what McIntyre is up to. His mission is to get scientists that do global warming research to make their data and methods available so that the scientific process can work properly. He has said repeatedly that he thinks that global warming is real.

    Mann and his collaberators of hockey stick fame, failed to make their data and methods available for replication and nobody ever checked their work until McIntyre came along. Even after McIntyre successfully debunked their work, they still resisted full disclosure.

    Jim Hansen publishes global temperature data, also without providing his data and methods. Not only that, he is a climate modeler and uses his temperature data to validate his models. It wasn’t until McIntyre discovered Hansen’s error that he disclosed his data and methods or proper peer review.

    The real cranks are those scientists, like Hansen and Mann, who use their work to promote a political agenda, but expect everyone to just take their word for it and insist that they and their work are above reproach.

    McIntyre has no political agenda. Unlike Hansen and Mann, he gets no funding from any side of this debate. He’s just a guy performing the due diligence that should have been performed in the first place by Mann and Hansen.

  13. #13 Bill H
    September 20, 2007

    What McIntyre and other so called cranks are doing is trying to make sure that the science is sound. For this effort they are called denialist and cranks.

    Someone trying to prove the greenhouse effect doesn’t exist is a crank.

    Someone trying to prove that CO2 isn’t a greenhouse gas is a crank.

    Someone trying to prove that global warming is worse than it really happens to be is a crank.

    Someone who asks how you came up with these numbers so that he can understand them isn’t a crank.

  14. #14 Winnebago
    September 20, 2007

    LOL! One simple post on Mc’s crankery and the Climate Frauditeers wind up the conspiracy whirlitzer.

  15. #15 minimalist
    September 20, 2007

    I’m hearing slide-whistles and kazoos, too. The traveling kook circus is in town, yay!

    McIntyre has no political agenda. Unlike Hansen and Mann, he gets no funding from any side of this debate.

    The latter does not prove the former.

    Further, I was under the impression that Hansen and Mann receive public funding from science agencies. It’s pretty amusing whenever the deniers try to equivocate between that and the funding deniers receive from energy companies acting purely in their own finincial interest. Create the idea that since there are two “sides” of the debate, the motives of one mean that the other “side” obviously must have equal but opposite motives.

  16. #16 MarkH
    September 20, 2007

    I agree, classic conspiratorial nonsense. Next.

  17. #17 Boris
    September 20, 2007

    It wasn’t until McIntyre discovered Hansen’s error that he disclosed his data and methods or proper peer review.

    This is wrong. How do you think MCI found the error in the first place. He looked at the data.

    As for MCI, he is the ultimate crank. Take for example when he was blocked by NASA for causing problems while data scraping. They unblocked him and told him when to download and he still whined. Then, he ran to the right wing/denialist website Townhall.com where the headline proclaimed “NASA Blocks Blogger from Data.” He’s baiting the Rush Limbaugh types.

The site is undergoing maintenance presently. Commenting has been disabled. Please check back later!