In the U.S., the federal agencies that fund scientific research usually discuss scientific misconduct in terms of the big three of fabrication, falsification, and plagiarism (FFP). These three are the "high crimes" against science, so far over the line as to be shocking to one's scientific sensibilities.
But there are lots of less extreme ways to cross the line that are still -- by scientists' own lights -- harmful to science. Those "normal misbehaviors" emerge in a 2006 study by Raymond De Vries, Melissa S. Anderson, and Brian C. Martinson [1]:
We found that while researchers were aware of the problems of FFP, in their eyes misconduct is generally associated with more mundane, everyday problems in the work environment. These more common problems fall into four categories: the meaning of data, the rules of science, life with colleagues, and the pressures of production in science. (43)
These four categories encompass a lot of terrain on the scientific landscape, from the challenges of building new knowledge about a piece of the world, to the stresses of maintaining properly functioning cooperative relations in a context that rewards individual achievement. As such, I'm breaking up my discussion of this study into two posts. (This one will focus on the first two categories, he meaning of data and the rules of science. Part 2 will focus on life with colleagues and the pressures of production in science.)
It's worth noting that De Vries et al. have not identified these categories of problematic behaviors from first principles. Rather, they sought scientists' perspectives:
Interestingly, those who know science best -- scientists themselves -- are almost never asked to describe the behaviors they regard as most threatening to the integrity of their work. ...
Using data from a series of focus groups, we describe the kinds of behaviors working scientists believe to be most threatening to the integrity of the research enterprise. (43)
To the extent that scientists themselves identify these behaviors as threatening to the integrity of their work, it's probably worth taking them seriously. At the very least, scientists' perceptions about who engages in problematic practices may influence their choices about who to collaborate with and whose work they can trust.
One of the take-home lessons of this research, as we'll see, is that top-down policies meant to prevent (or at least discourage) scientific misconduct miss the mark:
While it is true that the definition of misconduct as FFP describes "actions that are unambiguous, easily documented, and deserving of stern sanctions" and offers the added benefit of being similar to definitions used in other countries, we believe that policies intended to reduce misconduct must be informed by what researchers see as behaviors that hamper the production of trustworthy science. (44)
As we'll discuss at length below, the authors of this paper suggest it would be better to direct policies toward addressing behaviors that worry scientists, not just the agencies funding their research. I would further venture that it will probably take more than sanctions to really address the worrisome behaviors. Scientific communities and their members will also need to be involved in encouraging good behavior and discouraging bad behavior.
De Vries et al. will also draw some lessons for those who make research ethics the focus of their own research:
[S]cientists' reports on the types and effects of misbehavior in research serve to highlight a blind spot in the field of research ethics. It is a curious fact of the organization of intellectual life that research ethicists focus largely on the protection of human and animal subjects of research; the behavior of researchers (apart from their direct treatment of subjects) has yet to capture their imagination. This lack of interest may be the result of the ordinariness of misbehavior; we were told by one researcher that study of the poor behavior of researchers is far less intellectually stimulating than the conundrums of consent and conflicts of interest. (44)
There's an interesting question lurking here: are people who make the study of research ethics the focus of their professional activities primarily directed towards following their own interests, or primarily directed at being a resource to those conducting scientific research?
If the former, they need only worry about which problems they find intellectually captivating -- but they should not be surprised or offended if working scientists proceed to ignore all of their scholarly output.
I'm willing to bet that most of the people who study research ethics for a living hope that at least some of their scholarly output will be of use to scientists trying to conduct responsible research. To the extent that members of the field are serious about their work being useful and relevant to scientific researchers, attending to the "normal misbehavior" -- and what consequences it might have for the scientific community and the body of knowledge it works to produce -- seems like a good idea.
Are scientists interested in the subject of research ethics? The response De Vries et al. got when recruiting subjects for their study suggests that they are:
More scientists than could be accommodated volunteered to participate. We restricted the groups to no more than 10 participants, and, in order to minimize participants' reluctance to discuss context-related issues, we constructed the groups in such a way that participants were from different academic departments. The groups represented considerable diversity in race/ethnicity, gender, and disciplinary affiliation. (44)
As I'm not a social scientist, I can only speculate about the methodological advantages of focus groups rather than questionnaires. My hunch is that focus groups work better for open-ended questions, and that maybe interaction with other people on a focus group might make subjects more reflective than they would be whipping through a questionnaire.
Any social scientists want to weigh in on the pros and cons of focus groups?
It turns out that the focus group participants (and De Vries et al., in considering the information the focus groups have given them) latch onto one of the issues that I find most intellectually engaging, namely the interaction between the epistemic project of science and the ethical considerations guiding scientific activity:
[T]he everyday problems of scientists are often associated not just with ordinary human frailties, but with the difficulty of working on the frontier of knowledge. The use of new research techniques and the generation of new knowledge create difficult questions about the interpretation of data, the application of rules, and proper relationships with colleagues. Like other frontiersmen and -women, scientists are forced to improvise and negotiate standards of conduct. (44-45)
In other words, even if scientists were all on the fast-track to sainthood, they would still run into difficulties figuring out the right thing to do simply in virtue of the ambiguities inherent in building new scientific knowledge.
My guess is that, in most vibrant professional communities, community members are thinking (and talking) pretty regularly about what practices are good or bad from the point of view of their shared activity. That science is constantly pushing past the edge of what we know, trying to build good knowledge on the frontier, just means that the reflection and negotiation within the scientific community has to happen more frequently to keep the edges of the enterprise connected to the middle.
What specific issues did the scientists in the focus groups raise?
[Respondents] were troubled by problems with data that lie in what they see as a "gray area," problems that arise from being too busy or from the difficulty of finding the line between "cleaning" data and "cooking" data. (45)
The distinction between "cleaning" data and "cooking" data is important.
Even if there's unclarity about what you've found and what it means, in these gray areas scientists seem to have a bit more clarity about the mindset a scientist should adopt, and the kinds of pitfalls she should try to avoid. In sorting out the experimental results, a scientist should try to be objective. She should make as sure as she can that her result is robust -- so she'd be surprised is another scientist couldn't reproduce it, not surprised if she could.
Objectivity is hard even in the best conditions. Some of the less-than-objective calls scientists make in gray areas may be motivated by the practical goal of getting something published. As one of the focus group members put it:
"Okay, you got the expected result three times on week one on the same preparation, and then you say, oh, great. And you go to publish it and the reviewer comes back and says, 'I want a clearer picture,' and you go and you redo it -- guess what, you can't replicate your own results. ... Do you go ahead and try to take that to a different journal ... or do you stop the publication altogether because you can't duplicate your own results? ... Was it false? Well, no, it wasn't false one week, but maybe I can't replicate it myself ... there are a lot of choices that are gray choices ... They're not really falsification." (45)
Presenting iffy data is not the same as making up data from whole cloth. Presenting iffy data as being solid, however, is at least misleading.
Undoubtedly, there is also something worrisome to the scientist making this questionable call about standing by the iffy data. Once your results are published, you know, or ought to know, that there may well be other scientists trying to reproduce those results. Those scientists have no personal stake in giving you the benefit of the doubt if your results don't hold up.
If you know you're presenting an overly rosy view of how robust the results are, you also know that you're taking a risk that others will expose the flakiness of those results.
Some of the focus group participants pointed to gray areas that are, arguably, grayer, including the problem of separating data from noise in systems no one else has studied:
How do scientists actually clean their data? They often rely on their experience, cleaning out unanticipated findings and preserving what they "know" they would find (45)
We can't help being influenced by (indeed, guided by) what we think we already know. This makes it harder for a scientist to believe that unexpected results are reflections of how things really are, rather than the result of a faulty instrument, a contaminated culture, the wrong reagent.
If the purpose of science is to generate new knowledge, the meaning of the new data generated in that quest will necessarily be difficult to discern, requiring interpretation, inference, the sorting of "good" and "bad" data, decisions about the use of controls and statistical data. Scientists are aware of the opportunity to slant the data that these decisions afford and they remain unclear about the best ways to make and report these decisions. (45)
I think it's worth noting that the scientists here aren't just concerned about peers who take advantage of the uncertainties. Rather, these focus group participants seem to want a firmer sense about how to make better decisions about ambiguous data. There might even be the will to have regular conversations within the scientific community about tough calls.
What's the alternative? No one knows quite how other scientists made the calls, or how carefully they tried to avoid bias. This state of affairs makes scientist-to-scientist communication harder, both on the transmitting end and on the receiving end.
Another issue of concern to focus group participants was an excess of rules and regulations concerning various aspects of their research activities:
The work of scientists is increasingly governed by layers of rules intended to, among other things, protect animal and human subjects, to prevent misuse of grant funds, and to control the use of harmful materials. Our respondents noted that this plethora of rules -- many of which they find to be unnecessary and intrusive -- can actually generate misconduct (45)
Humans love rules, don't they?
There's much we could discuss here which I will defer so we actually make it all the way through this paper. Briefly, let me suggest that the question of which rules are unnecessary and intrusive is a matter of opinion.
Indeed, I wonder how many of the practices that scientist accept and take for granted at this stage of the game are accepted and taken for granted because of the prevailing rules and regulations. In the absence of these rules and regulations, would the behaviors of scientists all default to some agreed upon standard position? If so, why? Maybe this would come about by way of pressure from members of the scientific community. Do we have good evidence that the scientific community would be active in exerting such pressure, or that this kind of pressure would be sufficient to prevent some of the outcomes the rules and regulations are meant to prevent?
If, in the absence of rules, the behaviors of scientists did not all default to some agreed upon standard position, would scientists be prepared to deal with a disgruntled public, whose members might take issue with scientists' animal use, human subject use, toxic waste disposal choices, and so forth, and holler for their tax dollar to fund something other than scientific research?
Being accountable to the public that funds scientific research has something to do with the fact that the rules and regulations were imposed, after all.
However, it doesn't sound like the focus group participants were arguing that scientist be released from all rules. The real issue for scientists in the study doesn't seem to be that the rules and regulations exist, but that they command so much time and attention that they crowd out discussions of more nuts-and-bolts issues involved in doing good science. As a focus group participant says:
"I think it's really unfortunate that there's so many rules about how we use radioactivity, how we use these animals, but there really aren't many guidelines that train scientists on how to design experiments, how to keep notebooks. And it's almost as if young scientists think that, 'Oh, this is a pain, you know, let's just do it and not think about it, and you're just pestering me and you're expecting too much.' And it's extremely frustrating as someone that's running a lab. "(46)
Federal regulations (and scary penalties for breaking them) may grab the attention of trainees. But trainees may end up seeing the guidance from the PIs training them as just more burdensome rules. Indeed, PIs may bear some of the responsibility for this if they are communicating the message that the rules from the feds and the institution are necessarily burdensome.
If you're already in a bad mood about the very existence of rules, you may be inclined to do the absolute minimum to comply with them -- or at least, to appear to be in compliance.
But given that the guidelines that are part of a good scientific training are important to making successful scientists, it seems worthwhile to develop a better strategy for conveying them. PIs can motivate their own expectations and the prevailing regulations from the point of view of what they help scientists accomplish in their research, not on the basis of what penalty befalls those who flout them.
In the next post, we'll look at what scientists had to say about life with colleagues and the pressures of production in science. We'll also look for the take-home message from this study.
________
[1] Raymond De Vries, Melissa S. Anderson, and Brian C. Martinson (2006) "Normal Misbehavior: Scientists Talk About the Ethics of Research" Journal of Empirical Research on Human Research Ethics 1(1), 43-50.
- Log in to post comments
Beautifully thoughtful and well-written post, Janet.
It's simply a fact that--Open Science Eleventy Point Bajillion notwithstanding--the only reasonable way for science to progress is for scientists to assess their own data and discard what they have good reason to consider "bad" from analysis and publication. There is simply no way for any but an exceedingly small number of these decisions to be scrutinized by anyone other than the scientists making them, and even then only after the fact when a conclusion becomes important.
There is no way to keep things on the up and up other than trusting individual scientists to make these decisions appropriately. And that means training young scientists on what is and and isn't an appropriate basis for excluding collected data. I spend a substantial amount of time with my own trainees on detailed explicit discussion about their decision-making processes in this context.
And the fact remains that even I, the PI, cannot possibly directly assess every decision about data-exclusion made by my trainees. Without trust, the entire scientific enterprise grinds to a halt.
It's simply a fact that--Open Science Eleventy Point Bajillion notwithstanding--the only reasonable way for science to progress is for scientists to assess their own data and discard what they have good reason to consider "bad" from analysis and publication.
Utter bullshit. There is no reason not to provide the data to anyone who is interested -- uninterested parties need only skip the "supplementary data" section. But I've lost count of the number of times I've had issues with a paper or figure and been completely unable to dig any further, because there's no actual data provided.
Yes, most people won't care to check the second and third replicates; sure, most people will be content with "data not shown". So what? The scientists most closely involved with the work are going to want the details, and there's no good reason not to provide them -- unless, of course, the author is lying. Read the Martinson papers and then tell me that such lying is not a problem.
There is no way to keep things on the up and up other than trusting individual scientists to make these decisions appropriately.
Oh hey, let's just scrap that whole Office of Research Integrity boondoggle then shall we?
Without trust, the entire scientific enterprise grinds to a halt.
Damn straight -- so earn my trust, show me the fucking data!
that means training young scientists on what is and and isn't an appropriate basis for excluding collected data
Depends where you're excluding it from. If you're hiding it away so no one but you ever sees it, you're doing something very wrong. The data you excluded from whatever analysis, and your reasons for doing so, should be reported.
Again, when papers had to be written out longhand by a special caste of scribes and distributed to their intended audience by carrier pigeon, there were good reasons for "data not shown" and no supplementary section. There are no such reasons now that primary distribution is electronic.
Yeah, bill, that all sounds nice. But who the fuck has the motherfucking TIME for that shit? I sure as fuck don't. Working scientists don't have the time to wade through other scientists' raw data. This is why OPEN SCIENCE ELEVENTY POINT BAJILLION is a complete waste of time.
Maybe enthusiasts like you will waste your time dredging through scientists' unfiltered raw data, but actual real scientists are too busy actually doing science for that shit.
actual real scientists
Actual real scientists do the experiments they claim to have done, and don't whine if asked to show the results.
Maybe your field is better off than mine, but I routinely come across published experiments that should not have made it past the review process, and figures or claims that I strongly suspect are bogus but can't examine because no data is shown, I'm just supposed to "trust my fellow scientists".
If, instead, I could click through to the raw data behind that spurious claim, well, it wouldn't have been made in the first place. The mere fact of a community expectation that such information be available would clean up the published record immensely.
You complain about time -- how much time is wasted reading published bullshit that skirts the edges of FFP, repeating experiments that half a dozen shitweasels have already done but didn't publish because it didn't fit into their "story", fighting an entrenched dogma that's based on garbage work, and so on and on?
And whose time am I wasting, exactly? If you've done the experiments, it shouldn't cost much time to make the data available. If you don't want to read through it, don't -- no time wasted there.
Why are you defending practices that do no one any good and provide cover for malcreants and incompetents?
Oh, and I'm not talking about your notebook. I'm talking about the data behind the claims in your published papers. I'm talking about something you claim, in publishing, to have produced -- so what's the big deal? If it exists, show me; you had to work it up, the same as the stuff you're showing, no? Or did you glance at it, ignore that band there that you "just know" is an artefact, wave your magic no-see-um wand over the fact that there was no control this time, put that one down as "repeated" and call it good?
(OK, I actually assume that you don't do that, or I wouldn't be talking at you. But I know a lot of motherfuckers who do. And I don't see why you're providing them with cover.)
Wrong. One only works up data to the extent necessary to determine whether it is valid--i.e., relevant to testing one's hypothesis--or not. As soon as one realizes it is invalid, analysis immediately ceases. In many cases, this occurs as soon as one sees the most superficial outcome of a positive or negative control.
The aggregate amount of time and effort it would take scientists to "work up" and "make available" data from failed experiments--which make up the vast majority of all experiments performed in all labs all over the world--vastly outweighs the amount of time and effort expended on retrospectively looking into situations that appear to have possibly been mishandled.
OPEN NOTEBOOK ELEVENTY or whatthefuckever might make some sense for certain fields, such as highly repetitive high-throughput churning out of extremely stereotyped datasets that can be analyzed and prepared for being made available in an automated fashion. It is absolutely absurd for low-throughput hypothesis-testing oriented bioscience research that relies on the development and testing of novel techniques and measurements.
If one of my trainees told me that she completely wasted the time and effort continuing to analyze a set of measurements in which an experimental failure was already apparent so that she could upload it to the Internet for the perusal of enthusiasts, I'd rip her a fucking new one. We are working scientists and we have limited time and resources available to DO FUCKING SCIENCE. We can't afford to waste it on pipe dreams of OPEN SCIENCE ELEVENTY enthusiasts.
PP,
How do you assess the reliability of someone's work? Chemistry methodology (notably organic synthetic methodology) is notorious for methodologies that are unpredictable - there's lots of methodology that only works for some things (with known limitations), and some that only works if you look at it the right way while stirring counterclockwise and singing the lyrics to "Shine On, You Crazy Diamond" in a perfect falsetto. I don't need the output of every experiment the authors ran, but it would be nice to know how often I can expect to get the stated yields (or even how often I can expect not to get any of the desired product). I know there are issues of the competence of the person repeating the experiments, but if the limitations of a method are unclear (how much the outcome depends on the experimenter), it's hard to know how useful the method is and how well to trust the claims of the authors.
When someone shows pretty pictures in Nature or Science or JACS of nanothingies, how often do they observe the desired patterns or shapes? Lots of time, pictures are selected to give pretty images - but if the paper is advertising a method to facilely make patterns, it would be good to know how often it works (and how much the authors know on how to make it work).
The extreme example would be the LaClair synthesis(?) of hexacyclinol - the author provided no experimental procedures at all and three spectra (one of which was a reproduction of the literature spectrum of the desired product). Without the procedures and data, there is no way to validate or invalidate the work - in the case of a total synthesis, it is unlikely to be done anyway, but there isn't even the theoretical possibility of doing so. Another example would be most of BMCL - there are no procedures for many of the articles, and no way to validate that the observed compounds actually have the activities shown (because you don't know exactly how they made the compounds, and so can't verify the bio data unless you have them lying around - you probably would have to make them yourself from scratch).
It doesn't seem that an overabundance of data is the problem here. If providing it all is too much to ask, some sort of summary ("we got this result four of four times we tried it") would help.
Are you being deliberately obtuse? Seriously, are you jerking my chain here? You'd do that, I know you would.
Read my lips^H^H^H^H electrons: I'm not talking about Open anything.
I'm talking about the published scientific record. When you say, "the experiment was repeated three times with similar results", show those results. When you show a graph, put a table of the data in the supplementary section. Don't give me any of this "data not shown" crap. There's simply too much documented fraud and "normal misconduct" now for your kum-bah-yah "let's all just trust each other" to work. (Jesus, and you call my ideas naive!)
Right -- and those experiments don't go into the published record. Making them available is the Open Science/Open Notebook Science argument, which we have had many times and which we. are. not. having. here.
Again (and again, and again) I'm not talking about the data that isn't valid (say, because of a control readout). I'm talking about valid data whose existence is claimed but never proven in things like "representative results" and "data not shown". There's a whole different argument to be had about the decisions regarding which data are or are not valid, and we're not having that argument.
In the simplest possible terms, I'm advocating that each and every claim made in the scientific literature should be backed with hard data.
OK. We have, indeed, been talking past each other. Never mind. There's nothing to see here.