There was an interesting story today on Morning Edition about new research on potential bias in nutrition studies funded by industry. Dr. David Ludwig of Children’s Hospital in Boston led a team that analyzed 206 nutritional studies published between 1999 and 2003. More than half of these studies were at least partially industry-funded (in particular, funded by purveyors of milk, fruit juices, and soft drinks).
Ludwig’s analysis of this body of literature had one set of researchers look at the 206 nutritional studies just from the point of view of their scientific conclusions (with no information about the sources of funding) and another group of researchers look at them just in terms of their funding sources. What they found was that, in the aggregate, industry funding was highly correlated with the reporting of scientific results that were favorable to the funder’s financial interests — indeed, the industry-funded studies were eight times more likely to report such favorable results than were the studies that were independently funded.
It’s worth noting that Ludwig’s study doesn’t claim conscious manipulation of the findings on the part of the researchers. Rather, it suggests bias — almost certainly unconscious bias. And unconscious bias is pretty hard for even the most honest scientist to turn off (because it’s unconscious). One of the useful things a study like Ludwig’s can do is to get scientists in the habit of examining their own work for creeping bias. Am I framing the questions my experiments are trying to answer in a way that might stack the deck in favor of my funder’s product? Am I enthusiastically publishing negative results as well as positive ones? Given Ludwig’s findings, it’s not unreasonable to ask scientists to try to set up additional experimental controls on their own potential biases.
Of course, the guy they interviewed from the National Dairy Council, senior vice president of nutrition and product innovation Dr. Greg Miller, thought Ludwig’s study was flawed, but at least in the parts of the interview that were broadcast he didn’t point out any specific flaws with the study. Rather, he noted that NDC demands that the researchers it funds hew closely to the scientific method, and that these studies get published in prestigious and peer reviewed scientific journals. And that’s good, as far as it goes, but since we’ve discussed many examples of fakery (whether confessed or alleged) that have appeared in peer reviewed journals, I think it’s fair to say that scientific peer review is not definitive evidence of the soundness of reported findings. Peer review cannot, with perfect accuracy, screen out the papers that are knowingly fabricated or misleading. Thus, it seems overly optimistic to point to scientific peer review as a tool that effectively irradiates industry-funded research against contamination by unconscious bias.