Are fakers outliers or bellwethers?

Well, the new digs here at ScienceBlogs have thin walls (GrrlScientist, will you please turn down that stereo!), which means that sometimes we get sucked into the conversations our neighbors are having. And, almost as if this were the complex at Melrose Place (shut up!), a lot of us have been chattering about the same people, notably Hwang Woo Suk.

So, for example, I've been hearing Chris Mooney telling his guests that, peer review or no peer review, the community of scientists will always include some fakers. Through the air-vent, I've got PZ Myers musing on how detection (or not) of the fakers could be connected to how well-established or cutting-edge the faked research seems to be.

As it happens, I've always taken thin walls as an excuse to poke my head into a conversation, so here's my take on fakers and the mechanisms within the tribe of science for dealing with them.

The title of Chris's post is There Will Always Be Fakers, which is a claim I'm hearing more and more lately. Indeed, here's what Derek Lowe says about fakery in science:

[S]cientists do indeed realize that fraud happens, because every working scientist has seen it. For starters, most large academic departments have tales of grad students or post-docs whose work could never be trusted. And all of us in research have run into papers in the literature whose techniques won't reproduce, no matter what, and the suspicions naturally grow that they were the product of exaggeration or wishful thinking. The number of possible publications sins alone is huge: yields of chemical reactions get padded, claims of novelty and generality get inflated, invalidating research from other labs doesn't get cited.

It's painful for me to admit it, but this kind of thing goes on all the time. And as long as the labs are staffed with humans, we're not going to be able to get rid of it. The best we can do is discourage it and correct it when we can.

The main issue that gets disputed when talking about fakers in lab coats is just how common they are. Are they a tiny proportion of the tribe of science, or are they as common as (say) cheaters in a business ethics class? We don't know. To the best of my knowledge, there haven't been any studies that have turned up reliable data on the incidence of fakery, the proportion of the scientific literature that is rendered unreliable by such fakery, etc. Which means a scientist who says that only a tiny number of fraudulent papers make it into the scientific literature is, essentially, making a claim with no data to support it.

Even if human nature makes it inevitable that some scientists are drawn to dishonesty, remember that science strives to be a fact-based community, making fakery much more of a problem than poor table manners or B.O. Given the particular epistemic task to which scientist set themselves (building a good picture of how the various bits of the universe work), there's no level of fakery (except "none") that you can really label "acceptable" -- at least, it's not acceptable to let it continue and infiltrate the literature, grant applications, and the like. Scientists want the facts. Those who try to pass off made-up stuff as facts are undermining the enterprise and Must Be Stopped. Take away their licenses to practice science. Zap their funding. Shun them. Assign them a minder. But whatever the means of stopping them, once fakers have been detected, they cannot be allowed to continue their wrong doing, or else they'll wreck it (the body of reliable scientific knowledge) for everyone else.

OK, so how do you detect the cheaters? Peer review is supposed to be one mechanism: your manuscript doesn't get published until it survives the scrutiny of a subset of your scientific peers. These peers are supposed to figure out whether the experiments you describe are good ones, whether your analysis of your data is appropriate, whether your interpretation of the results is resonable. Presumably, if they sniff out results that are too good to be true (including figures or graphs that look doctored), they're supposed to ask some hard questions about them. The fact that people who know a lot about systems like the one you work with will be subjecting your manuscript to this kind of scrutiny is supposed to deter people from cheating.

But there are a couple factors that can make this more complicated than you might expect. First off, the more cutting-edge your research is, the less likely that there is a large number of other scientists in your field who have had success with the kinds of systems you study or the kinds of things you're doing with them -- which means, there will be fewer scientist who could give the same kind of well-grounded peer review of your manuscript. Indeed, the reviewers may be in a tricky spot: they could be credulous ("I've never seen anything like this before, but it's all so new that maybe they really did see it.") or curmudgeonly ("It doesn't fit with what I know, so I don't like it, not one bit!"). The latter mode may keep new findings that really are valid (and useful, and important) out of the literature, while the former may let crap into the literature. Simply because we have less experience with the newer lines of research, these may be more likely, when they hit the journals, to be riddled with errors. Some of these errors are inevitable when scientists are working in new areas; less experience means it's harder even for the researchers to distinguish the good results from the errors. But some of the problems could be the result of fraud.

In areas of research that are not at the very frontiers of science, peer reviewers have more collective experience to draw on in order to evaluate manuscripts. They have more of a gut feeling for results that seem reasonable and those that seem fishy. So, you might think it would be harder to fake results the closer the area of research gets to "textbook" knowledge. But, as PZ writes:

Hwang Woo Suk's results were not at all unexpected, did not contradict any accepted scientific concepts, and were dramatic because they represented a methodological breakthrough. In a way, it was almost a "safe" category in which to cheat: lots of people are trying to transform adult cell nuclei into totipotent stem cells, it looks like a problem that's just going to require a lot of trial-and-error hammering to resolve, and what Dr Hwang did was steal priority on a result he could anticipate would be "replicated" (or more accurately, done for the first time) in short order. This is one of the hardest categories of science to police, I would think. It's frontier science all right, but it's only one step beyond the textbook.

In other words, working in a realm of science where most of the scientists agree about how things go (or how things will go once technique is perfected) may make it easier to cheat. Just as peer reviewers know what a reasonable result should look like, so do researchers ... and they might be tempted to take the short-cut of just making up reasonable looking results rather than trying to wring them out of the experimental system.

What a lot of non-scientists don't realize about peer reviewing of scientific manuscripts is that it does not generally involve the reviewers attempting to replicate the experiments reported in the manuscripts. Peer reviewers who happen to be working on the same experimental systems may try to get the same results, but seeing a papar in the journal is no guarantee that this has happened.

But, it's only a matter of time before someone does try to replicate the results reported in the journals, right? If someone has managed to get faked results past peer review, they are bound to be exposed eventually, right?

Depends how many other scientists see the experiment and/or results reported in the paper as a useful starting point for further work. It also depends on how finicky an experimental system it is. Sometimes reproducing an experimental result is hard because you don't have the same skill in the lab as the team who reported it. Or, you can afford the new, reliable equipment (or high-quality reagents) that team used. Or there might be subtle but important experimental factors not reported in the paper (because they weren't recognized as important) that need to be just so for the experiment to go. (For the chemical reactions I studied, reactor geometry and stirring speed were the two variables that were underreported in the literature, and this made months of my life suck.)

Also, scientists might abandon an experimental system if they don't have the patience required to obtain special reagents from very busy scientists in competing research labs.

The bottom line is that there are plenty of published results that don't undergo anything like a straight replication by other research groups. Given that there seems to be very little professional reward for replicating other groups' published results, this probably won't change any time soon. And this means that the literature may have lots of problems -- whether the result of fraud or honest mistakes -- mixed right in with all that science-y goodness.

It seems, therefore, that rooting out problems (fakery and honest mistakes) might be better accomplished before the manuscripts even go out to the journal editors. Chris suggests that ratting out your collaborator/colleague/underling/boss might be the best way to make happen. But, there are power dynamics to navigate. How much standing does a student have when it's the boss who's faking data? Does the department chair believe some newbie (still learning how to use all the equipment, for goodness sake) over an esteemed and highly productive member of the department? Should scientific collaborators subject each other to surprise inspections? Sure, science is an enterprise in which the community's fortunes rise or fall based on the honesty of its members, and to a certain extent that makes each scientist his or her scientific sibling's keeper. But is a presumption of guilt really going to be the thing to help the scientific community get fakery under control?

A response I like better would be taking steps to strengthen scientists' awareness of their connectedness to each other. Rather than a consciousness-raising that makes each scientist aware that every other scientist is potentially a lying rat-bastard, there needs to be a consciousness-raising about what the whole point of scientific activity was supposed to be in the first place: not tenure, awards, big buckets of grant money, or dates, but a better and better understanding of how the stuff in the universe works. You may be able to get the tenure, the fame, the bucks, and the booty with lies, but you can't get the better understanding with fakery. And maybe that means the scientific establishment needs to pull back from its current structure of staging gladiatorial battles for limited resources (tenure/fame/bucks/booty).

You might wonder if the number of scientific fakers detected in recent months is a testament not to the vigilance of the scientific community in keeping its knowledge base sound, but to an increase in the number of "scientists" opting to cut to the chase by seeking the rewards of science (tenure/fame/bucks/booty) without getting mired in the slow, difficult processes involved in actually doing science. Maybe bigger rewards for scientific achievement make the temptation to fake such achievements harder to resist. (I often buy a lottery ticket when the jackpot tops $80 million.)

The scientific community needs to get itself to the point where it is absolutely unthinkable for any of its members to sacrifice the overarching goal (getting good knowledge) for an external reward. It needs to create and maintain an environment where fakery does nothing good (even in the short term) for anyone who might perpetrate it.

The trick, of course, is figuring out how to get there from here.