Science Based Medicine is a site we highly recommend with experienced scientists and practitioners in charge. In other words, it’s run by adults. But scientists often disagree about things. This is apparently a secret to non-scientists and many reporters who assume that when two scientists disagree, one is lying or wrong. But it’s true nonetheless. Whatever the subdiscipline, there are disagreements. If you pick up almost any issue of Science or Nature you will find plenty of them, usually (but not always) couched in polite language in the Introduction or Discussion section of a paper or in the Letters. So it’s not surprising that I disagree with Val Jones’s piece from last week: Science-Based Medicine 101: How To Establish A Source?s Credibility. Actually I don’t just disagree. I think it is quite wrong headed.
Jones’s piece is meant as a guide to lay readers about how to establish a source’s credibility. Right off the bat we are in trouble here, because there is an immediate confusion between credibility and being right. Most credible scientists are wrong at times, so if credible doesn’t mean “right” what does it mean? Presumably it means something like, how seriously should you take this person? Let’s look at the indicia of credibility as given in this guide:
In medical research, I like to think of credibility in three categories:
1. The credibility of the researcher: does the researcher have a track record of excellence in research methodology? Is he or she well-trained and/or have access to mentors who can shepherd along the research project and avoid the pitfalls of false positive ?artifacts?? Has the researcher published previously in highly respected, peer reviewed journals?
2. The credibility of the research: does the study design reflect a clear understanding of potential result confounders and does it control for false positive influences, especially the placebo effect?
3. The credibility of the journal that publishes the research: top tier journals have demonstrated a track record of careful peer review. They have editorial boards of experts who are trained in research methodology and are screened for potential conflicts of interest that could inhibit an objective analysis of the research that they review. The importance of careful peer review must not be underestimated. Some say that the quality of a product is only as good as its quality control system. Top tier journals have the best quality control systems, and the articles they publish must undergo very careful scrutiny before they are published.
Oh, my. Where to begin? Let’s start with track record of the researcher. More and more the real work in science is being done by post docs and graduate students who often get their names first (which is fair). But somewhere in the list (usually last, the second best place to be in biomedical articles) is usually the head of the lab or the principal investigator of the grant and they are often listed as “corresponding author” (because they’ll still be there after the post doc moves on). They are also often the ones who get the credit (“Dr. Smith’s lab”). How much guidance and input they had in the work depends. Sometimes it’s a lot. Sometimes they barely know what’s in the paper. One thing for sure. Looking at the name and record of the first author or the “senior author” is not a sure way to gauge credibility. Ask the numerous lab heads who have had to retract papers after fraud or misconduct by one of their students or post docs was uncovered.
More importantly, some of the best work is done by those fresh out or still in training. There are Assistant Professors out there who have already lost touch with the latest techniques because they spend their time writing grants and teaching and the real work is being done by their grad students and post docs. That goes triple for full professors like me. It’s great to think about what to do and to have people who will try to do it, but if I had to do a lot of it myself, I’d be helpless. I don’t have the time to learn these new techniques and spend the tens or hundreds of hours to master them. And don’t have to. I have students and post docs to do it for me.
Indeed, if track record were enough, grant reviewing would be much easier. We wouldn’t have to look carefully at the Methods. But we do look carefully at them because science is ever renewing itself and the best research often uses things that haven’t been done before. And if a lay person is having trouble reading a paper, how is he or she going to be able to judge whether previous papers are any good or not? There are scientists, some quite eminent, who specialize in the LPU, the “least publishable unit.” You take your stuff and chop it up into the smallest pieces and publish each as a separate paper. Very good for your resume but very bad practice. Another way to bulk up your resume is to find some technique and then keep turning the crank on it with different model systems or different data sets or different populations. It’s really just one big (and often very uninteresting) paper.
As for advising the lay reader to check the study design to see if it exhibits a clear understanding of potential confounders and “controlling for false positives” (I’m not even sure what that means, but if it means correcting for multiple comparisons, this is very, very deep methodological water that biostatisticians can’t agree on, so how is a layperson expected to?), if a lay person could do this they wouldn’t be lay people. I spend years teaching grad students how to do this. If you are able to do this you don’t need any other criteria. It’s just another way of saying, is this good science or not? And it’s not easy and there will be many disagreements. Regarding “placebo effect,” it betrays a very parochial view of science (that it involves randomized clinical trials), but I don’t think many scientists really understand what is involved and are very likely to make extremely serious errors in interpreting results (e.g., that a result that an effect is not statistically significant means there is no effect). One of my colleagues calls the worship of the randomized clinical trial “a methodolatry.” I like it.
So I think this is all either bad or useless advice, but the one that is worst is the one that says, look to see if it is in a “top tier journal.” OK, now I need to declare a conflict of interest. I am co-Editor-in-Chief of a peer reviewed scientific journal. Our journal is in the top quarter in impact factor in a specialty field of over a hundred journals, but it isn’t one of three or four journals listed as “top tier” by this piece. It cannot possibly be that only papers published in these journals are reliable or that papers in these three or four journals are on average more reliable that many other top specialty journals and I doubt that Dr. Jones would claim this. But then how is a lay reader to know the quality of a journal that doesn’t have a press office to get its name in the news all the time?
Moreover there is very little about these journals that insures reliability. It’s just false that they have better quality control than other journals. They often have much shorter review deadlines and that doesn’t make for careful reviewing. I have done peer review for most of the journals listed as “top tier,” I do peer review for many other journals, I assign papers to others for peer review and I make editorial decisions about whether to publish based partly on the peer review process. I have also published over a hundred papers and book chapters myself. So it galls me to hear someone say that you can judge credibility by where something is published (and I’m not even touching the reliability of peer review, which is another subject). The “top tier journals” are very good but they are top tier mainly in the sense of top visibility or top profile, not because they publish the top science. Most good science is published in journals like ours: high quality journals catering to a scientific specialty and read by specialists. We don’t depend on advertising and don’t have big staffs of editorial assistants but we do the same (or better) peer review processes and attract the same quality of papers as the top tier journals. Those journals are interested in high visibility and attract papers that use novel methods or have surprising findings or are the first papers on a subject. The swine flu papers are a good example. They were given expedited handling by the big journals (Nature, Science, New England Journal) but the papers were very preliminary and often not very informative, scientifically. But they were first and everyone was hungry for every bit of information. Those journals were the right place to publish them because of their visibility and reach. But they weren’t any more reliable for that. On strictly scientific grounds, they were mostly of very routine quality, or worse.
Some of these journals also give preference to big drug trials. These trials make news and they sell drugs. The journals make money from advertising (guess who the advertisers are) and they make a lot of money by selling reprints of the papers to drug companies who then send them for free as a form of marketing to tens of thousands of doctors. These top tier journals have also been grossly manipulated in the past and now are trying to come up with safeguards to clean up their act. The top tier journals have been purveyors of fraud more often than other journals because they were the right place for it to have an impact. But the idea that being “top tier” makes them more reliable just isn’t the case, either as a thought experiment or as an empirical fact.
So if these aren’t the right indicia of reliability, what are? There is no answer to this question (and certainly not the answer given in the post in question). Science is a process of sifting and winnowing and often whether work is reliable or not isn’t known for some time. It has to be tested, cross-checked and fit into an existing body of information. As one of my colleagues is fond of saying, “Real peer review happens after publication.” Most science reporting these days is quite terrible, little more than regurgitating the press release from a university’s media relations outfit. If you are a lay reader interested enough to look at the actual paper, then you are very far ahead of the game. Most lay readers are at the mercy of a reporter or a press release and there is no good way to tell which of these are credible.
That means most lay readers have to depend on others who look at the literature with a critical and informed eye. There are some extraordinary science journalists out there who are able to do this by providing the reactions of others in the field. The Perspectives, Commentaries and News sections of the top tier journals are very good at that, as well. Then, there are the science blogs, of which Science Based Medicine is one of the best. We try to do the same kind of critical appraisal here at Effect Measure on certain subjects like influenza, and there are many, many more science blogs (such as those produced by our publisher, Seed Media Group at scienceblogs.com).
If you are a layperson interested in a particular subject, these days the internet can be a helpful resource. Of course there are the Jerry Springers and National Enquirers in the blog world, too. Maybe I should write a post on how to judge the credibility of a blog. But that’s another story. Don’t get me started.