If you read Alex Bradley's guest post calling into question the claims of the recent Science paper stating the existence of microbes that can substitute arsenic for phosphorus in their DNA you might be wondering what to take away from it all, if the scientists can't even agree on whether the study is valid or not. I would like to take this opportunity to reflect on the scientific process and hopefully explain why this type of intellectual discourse demonstrates the strength of the scientific process rather than being an example of scientists not knowing what they are talking about.
Here's the deal. In terms of the scientific process, this is how major discoveries happen, or dont.
- Scientist thinks they have made major new discovery
- Scientists writes up said discovery and submits paper to peer-reviewed journal (the bigger the claim, the fancier the journal)
- If editors of journal like the paper, they send it off to 3 or so experts in the field who review it, and provide the editor with feedback as to whether they feel the work is valid or not.
- Eventually, lets say the paper makes its way through the reviewers and gets published.
- Hold a massive press conference hyped up with hints of extraterrestrial life and blow things totally out of proportion. Hey, how'd that get in there?
- Now the fun begins.
- The vultures start to circle, and by that I mean other scientists start to pick at the paper (again, the bigger the claim, the more criticism typically surfaces).
- Original author might perform additional experiments that address these claims.
- Other scientists might carry out experiments that disprove the claims.
- Original claim either does, or does not, stand the test of time.
This paper makes a very bold claim. It claims that a bacterium grown in the lab under very particular chemical conditions eventually started building its DNA in a way previously assumed to be impossible. Please note that the claim of the paper is NOT that this organisms does this in nature, (although the authors do imply this, there is no evidence for it), it is merely claiming that the switch itself was shown to happen, and therefore is possible in nature.
Understandably there are many scientists out there (chemists in particular) who don't like it for many of the reasons that Alex pointed out (see link above) in the previous post on this blog. I am not a chemist, and don't feel qualified to determine the validity of the paper's claims. I do feel (perhaps naively) that errors as fundamental as the ones pointed out previously should have been (and may have been) addressed during peer review, and it is possible that Wolfe-Simon and her team have done more work than the space of the short Science communication allowed. I imagine we will be seeing more thorough publications from them soon which may strengthen their case. I am certainly looking forward to these publications.
I hope it turns out that they are correct. It would certainly be very cool, and would likely reinvigorate astrobiology research and funding for astrobiology research. Which is something I would very much like to see. Also, if it ends up that this claim is erroneous, I fear serious negative impacts on the astrobiology field because of all the fanfare that NASA gave the publication of this paper. I would blame NASA for this though, not Wolfe-Simon and her team. Scientific findings that have not had the chance to withstand the test of at least a bit of time do not generally get their own press conferences. Basically, I worry that NASA shot themselves in the foot by touting the enormity of this discovery too early.
In summary this study is, at this point, neither bad science nor good science. It is unproved science. The jury is still out. Publishing potentially controversial findings is part of the scientific process. Big name journals like Science tend to publish the higher profile claims, and inherent in that is that these claims are sometimes disproved. That does not mean the the journal is publishing bad science or is in someway flawed. It does not mean that the science itself is bogus. Even when findings are disproved, they often inspire other studies that make groundbreaking discoveries. That is, they have an inherent value to science even if the claims do not stand the test of time. Sometimes the initial way an experiment was run actually does support one claim. This is why we repeat experiments and do follow up experiments to confirm. There are always more tests you could do, but scientists walk a fine line between definitively proving their claims, and publishing exciting results so that others can hear about them. Obviously you need to be confident in your data, but you also need to consider the benefits of publishing quickly.
The point is that we will have an answer, and we will be able to confidently say that the answer is correct. Either this microbe can do what has been claimed, or it can't, and there are some simple analyses that can be done to determine which is the case. We just need to be patient and see what happens... and realize that this IS how science happens.
[NOTE: Because this post is now a magnet for spam, commenting has been closed. If you want to leave a comment, please send it to me via e-mail: webeastiesblog (at) gmail.com]
I disagree. In good science, we do follow up experiments to test, not to confirm. The experimenter should be the first one to ask "how might I be fooling myself?" Before publishing.
As Rosie Redfield wrote, "There's a difference between controls done to genuinely test your hypothesis and those done when you just want to show that your hypothesis is true."
Sometimes the experiments are too hard or will take years, so publishing a partial result will advance the field. That doesn't seem to be the case here.
I don't agree on the fact that a published article is a unproved science. In my opinion of what is "good science" an article should bring experimental proved statements and not theories that has to be proven later. The scientist should performe all the experiments that are logically required for demonstrating their statements, nothing less or more. The rest of comunity test the statements by using different models in order to verify wheater the hypotesis is true universally.
You're both right, and you're both wrong.
Don - The philosophy of science has long said that scientists should only be working to disprove their hypothesis, but this is rarely the case in actual fact. Yes, controls are vitally important, and should always be designed to to make sure the observations you are making are actually telling you what you want them to tell you. You may also be right that the proper controls were not done in this particular case, and that's quite a lapse. But most of the time, when scientists in the real world design experiments, they are out to see if something is true, not to see if it's false. I feel like I'm belaboring the point, but it bears repeating in a different way: in a perfect world, your controls will have the same effect as attempting to prove yourself wrong, but that's not typically the mentality.
Mario - You're right that the best science is experimentally proven, but it always depends on your standard of proof. Some people may be utterly convinced based on the material presented (presumably the referees and, we hope, the authors were), and others may bring different opinions or different expertise (as Alex did in his guest post here). I think that's what Heather was getting at: science doesn't end at peer review. There will be people who disagree, sometimes vehemently, and the field will move forward because of these disputes. Eventually, we'll approach consensus.
Please be careful with the word "prove"!
Often only 2, unfortunately.
Kevin,
I think we need to distinguish process from outcome.
You're right, of course, that much research does not systematically aim to test alternative hypotheses. Moreover, "good science" (that which will become widely accepted) can emerge from sloppy practices, and "bad science" can fool even very careful researchers. But following these "best practices" ups the odds that the eventual conclusions will stand up over time. I fear that the hegemony of ScienceAndNature and the era of science by press release are obscuring the importance of these practices in the minds of early-career scientists.
I expect that this will be a career-altering event in Wolfe-Simon's career. But if the criticisms I've seen are correct, it won't be in a good way, and she will wish she had done more of the easy controls before claiming success in her quest. We shall see.
I don't agree on the fact that a published article is a unproved science. :/
Well, usually unproved science should not be published, that is true. But I agree with the author that this case is way too far to be considered bad science. Is past bad science, is unproved science, i.e. fiction (at least for now).
Now, I am curious to see which kind of effects it will have on Wolfe-Simon's career, a point that a commenter brought up. Indeed, I read somewhere that the name given to the strain (GFAJ-1) stands for Give Felisa A Job.... very interesting I would say.. Sloppy paper, out too fast too soon, too much advertisement, and a lot of pressure to find a position alltogether??
catastrophic..
Science can be defined as doing research about nature, then publishing it so it can be criticized. A fair number of capable people do not become scientists because they don't want to face this criticism of their work.
Tests, experimental or otherwise, cannot confirm an hypothesis. They can only, at best, support the hypothesis,or reject it. A properly designed test, in this case, would be an attempt to reject the null hypothesis, "Ain't no Arsenic replacing Phosphorus in that DNA."
There is often, maybe always, going to be a conflict between publishing and doing more. This conflict happens in the laboratory or field setting where people are trying to decide when to stop collecting data or using as yet untried tests and publishing, it happens when a paper is submitted (editors may ask for more, or even on occasion, less) and during the referee process.
It is fairly common for a publication to be defined as the application of a limited number of methods to a particular problem, with other methods (often done in a different lab that specializes in them) being reported in another paper.
This question is so ubiquitous and so often problematic that it really isn't fair to go all crazy mad at a research team for publishing some of the potential results and not all. There is no such thing as "all" anyway, and for some sorts of findings it isn't always clear as to what should come next.
Within the field of palaeoanthropology, one of the most influential and important papers ever written (of the last century) was by Isaac and Harris, and it consisted of a table with labeled rows and columns and nothing in the cells. The paper essentially said "what we've got to do is to take a table kinda like this one and go out and get the data to fill it it." The point of the paper was to organize the future of Early Stone Age African Archaeology .... an entire continent's worth of data, and it was brilliant.
I worked on a paper a few years back in which a dozen specialists worked on one problem, from all the angles we could think of, with at least three specialists working different angles on, essentially, on set of soil samples. We bet that problem to death, and at that time and since no one has proposed one iota of methodology or analysis that we didn't think of.
Those are two extreme ends of a scale for which there is no appropriate a priori place that the ideal paper should be. The belly-aching about this paper not having been done correctly is nothing more than an Internet hater-meme that was born of wild anger for non-science web sites like Gawker taking a very sparse pre-press release announcement and turning it into a trailer for a remake of ET.
In the end, the scientists report that A is in for P in places where it has never been seen before. Their assertion is suggestive and unproven, yet they claim that the next paper they've got in the works will shore that up. All the criticisms that are coming out now about the actual paper (and not the structure of the paper, what they shudda said, or what Gawker.com said) are going to impact that research in a large and helpful way. It may well be that the commentary out there now will even kill the research, or it could shore up what they are doing so when the February paper comes out it is very convincing that there is something astonishing going on here.
Either way, valid actual critique of the paper is good, and either way, valid actual criqtique of the paper would not have happened had it not been published.Papers are published to be critiqued. Saying that "this paper is being critiqued, it should only have been published when it is unassailable" is not correct and indicates very little understanding of the process.
Did the paper deserve a pres conference? Probably not, especially that one. Better to have timed the paper with an appropriate scientific meeting, where there would be lots of press conferences on various works, and use that more natural forum for a modest event.
But no, there is no valid way to say that a paper is too done or not done enough with respect to whether or not certain instances of a wide range of possible methods have or have not been applied.
Had I been in charge I would have put a version of this paper in PLoS ONE with no press conference, let the criticisms fly, and if things worked out with demonstrating A in for P for long lengths of DNA backbone, had a big party in February with that second paper.