Uh-oh...poor science alert!

Carl Zimmer has some interesting stuff on this.

Couldn't they come up with a better acronym than PNAS? I mean you would pronounce it "penis" wouldn't you?

Whoa there. I read the original paper, and while I agree with Nick that the conclusion of a "step-by-step" explanation of the flagellum is unwarranted by their methods, I don't see anything immediately wrong with the *data* in the paper.

It's one thing to be critical of a paper, but it's another thing to have a complete pile-on by a blogosphere which is largely not qualified to understand the methods. Let's see what Nick's explicit criticms are before we decide that he is correct, and the two authors and three reviewers are in error.

pablo: I've heard two variations: the formal "P-N-A-S" and very informal "pee-nAs". I'm sure you can imagine the etymology of the latter.

Echoing Myrmecos, it looks like Nick's primary complaint is a minor wording mistake about homology in the discussion. If that is it, this is a trivial error (though nice to fix it before publication if possible).

No, there may actually be a huge methodological flaw, a lot of false homologies rife with artifacts. Hang tight, PT will have more. It's looking ugly right now.

We should also remember that just because it was published in PNAS doesn't necessarily mean it's good. Remember, Linus Paullng's "vitamin C cures cancer" woo was originally first published in PNAS. Although plebians like you or I have to run the usual peer review gauntlest, any member of the academy can submit his or her own work or "communicate" a nonmember's work and pick the reviewers:

In addition, an Academy member may ''communicate'' for others manuscripts that are within the member's area of expertise. Before submission to PNAS, the member obtains reviews of the paper from at least two qualified referees, each from a different institution and not from the authors' or member's institutions. Referees should be asked to evaluate revised manuscripts to ensure their concerns have been adequately addressed. The names and contact information, including e-mails, of referees who reviewed the paper, along with the reviews and the authors' response, must be included. Reviews must be submitted on the PNAS review form and the identity of the referees must not be revealed to the authors. The communicating member is required to include a brief statement endorsing publication in PNAS. In the journal, these papers are distinguished as ''Communicated by'' the responsible editor.

An Academy member may also submit his or her own manuscripts for publication. Beginning May 1, 2007, members must secure the comments of at least two qualified referees. Referees should be asked to evaluate revised manuscripts to ensure their concerns have been adequately addressed. Members' submissions must be accompanied by the names and contact information, including e-mails, of knowledgeable colleagues who reviewed the paper, along with the reviews, the authors' response, and a brief statement endorsing publication in PNAS. The reviews must be on the PNAS review form and should not be from the authors' own institution. The Academy member must be a corresponding author on the paper. In the journal, these papers are distinguished as ''Contributed by'' the responsible editor.

In other words, Academy members and their friends can in essence publish anything they want in PNAS. In actuality, it's amazing that PNAS is as good as it is; it could be (and sometimes is) a dumping ground for leftovers from the labs of Academy members that couldn't get published elsewhere. Fortunately, Academy members are by and large fantastic scientists (or they wouldn't be Academy members), and they seem to take pride in PNAS. If they didn't, the journal could easily turn into a pile of crap.

Echoing Myrmecos, it looks like Nick's primary complaint is a minor wording mistake about homology in the discussion.

the author's use of the single gene model for flagellar evolution is a minor wording mistake?

that must be one hell of a misprint.

Nick's primary complaint seems based on the author's acceptance of this model without sufficient evidence to support it.

...and that was just scratching the surface.

Nick is very well versed in this issue, and is more than qualified to comment on it.

This paper was a direct submission, so it went through the normal peer review process rather than the "PNAS member free ride."

I will echo what a few have said about PNAS, it can and indeed does publish good papers but has a flawed peer review process compared to almost every other top journal, namely the fact that members of the academy can bypass rigorous review for their direct submission. If it went through the proper review process then I guess the reviewers werent thorough enough with their reviewing if they let that comment slip about all the flagellum genes being derived from from a single gene (that might be true technically but only in the sense that everyone in the USA is an African-American).
Peer review, even when working properly has still one major flaw, apparent only to those working at the coal face of research. This is the fact that manuscripts, when submitted, get sent out for review to an 'expert in the field' - someone who really should know what they are talking about regarding a particular topic. The trouble with this is that, in a competitive scientific environment, the person with the most knowledge of a particular topic is likely to be someone who is working on exactly the same problem.
There is an unspoken assumption that scientific intergity will ensure that everything is done above board but one wonders whether this is not an inherent flaw built into the system, particularly when you realise that a reviewer may recieve a good paper that, if accepted, would most likely result in a competitor getting the limited future funding rather than the
reviewer. It is a veritable turkeys voting for thanksgiving question. For the non scientists I should let you know that papers generally aren't rejected by such reviewers but 'delayed' (the reviewer asks for very difficult or time consuming experiments to be done such that the paper takes much longer to be completed - and funnily enough gives the reviewers group the time to publish first).

I apologize. I have not looked at this closely. My initial impression was that a sentence in the discussion said "All proteins in the flagellum show homology" when Nick was wanted it to say "All proteins -of this subunit- in the flagellum show homology", and other comments led me to believe the data in the paper focused on the sub-unit, thus making this a rather modest language issue. I have not read the original paper, and so I withdraw my largely ignorant criticism.

Also, as PNAS is getting a bit knocked here lately, I wanted to make some defense:

First, NAS members have a limited number of contributed and communicated articles per year (two, I believe), so PNAS is not just the journal of the old boy network helping each other out.

Second, contributed articles are reviewed by someone of you choice. This is not as good as anonymous peer-review, but a less stringent publication outlet for NAS members is not necessarily bad.

Third, both communicated and edited articles require at least two anonymous peer-reviews. The difference between the two is (ideally) sponsorship. With edited articles, you toss your paper into the general pool with no NAS members endorsing you. With communicated articles, you have an NAS member endorsing your paper.

Yes, one could cheat by having a friendly NAS member communicate the paper and get you easy, cursory reviews, but there is a limit to the NAS short-cut: you can publish your "Insane Rantings I" and your "Insane Rantings II" every year, or if your set up with the tenure, you can hook up your friends with "Professor A's Insane Rantings" and "Professor B's Insane Rantings" every year. But your name is on it, and reputation means a lot in science.

So much, in fact, that a famous non NAS member sees little distinction between communicated and edited articles.

Ultimately, the PNAS editorial board approves everything. The communicated track simply treats NAS members as professionally equivalent to staff editors.

Of course, I'm especially sensitive now because my upcoming PNAS paper was communicated -- but none of us knew the communicating NAS member. He had interest and expertise in the field, and we cold-emailed him with the manuscript. He loved it, sent it to anonymous reviewers (I have a theory on who one of them was, but I really don't know) and they loved it, and now it will be in PNAS soon.

It's as though we threw the paper into the general submission pool, had an NAS member like it and recommend it for editorial priority. Is that bad?

Regardless, the flagellum paper in discussion was edited, meaning it was straight-up peer reviewed without NAS member endorsement or tinkering. If it is bad, it was a general peer-review failure and not due to NAS privilege (more so than any other journal is susceptible).

Short summary: Whether this is a bad paper or not, don't try to blame it on the particular culture at PNAS. Even if you dislike the NAS sponsorship/deference there, this was an *edited* paper! If it is bad, standard peer-review failed just as certainly as Cell or Science or Nature or JMB's peer-review has.

I think it's interesting that people complain about PNAS letting this paper go through without peer review (which it did not -- neither author is a member), and yet appear willing to accept Nick's non-peer-reviewed comments at face value. I personally look forward to Nick publishing his critique in a peer-reviewed journal, to which Liu and Ochman and others with expertise can reply. And I would point out that not everything that gets done in science is about ID, nor is an aggressive blog criticism going to stop IDers from using information as they see fit. Case in point, some IDers are very happy with Nick's rhetoric:

There are many other detailed criticisms that can be made. Happily, a good start has been made here by Nick Matzke, who refers to "canine qualities" of the paper. By this he means that "there is nothing for it but to suck it up and declare this paper a dog." Of the claim that all flagellar proteins come from one gene, Matzke writes: "Frankly it is a flabbergasting thing to say, and I can't understand how it got published." Also: "There is just no way that the flagellum evolved by diversification of a single gene." However, Matzke misses the main methodological flaw in the research, which concerns their interpretation of sequence homologies in terms of descent with stepwise modification.
Jennifer Cutraro wrote that the study "not only answers an important question about the evolution of complex structures but also provides additional ammunition to counter arguments from evolution's foes." ID scientists are opposed to the 'just-so' stories and unwarranted extrapolations which are often found associated with evolutionary biology. By overstepping the mark so badly, perhaps this new paper will help people to understand the importance of paradigms in science and why critical issues are continually being overlooked.

http://www.arn.org/blogs/index.php/literature/2007/04/19/sequence_simil…

If the paper is flawed, the scientific community will make this clear in due course. I agree with being critical of published papers, and it should go without saying that publication does not prove accuracy. Peer review is only a first step. Comments, further analysis, and new research that supports or refutes the original report are how the quality of a study ultimately will be assessed. The issue will not be decided on hastily prepared blog posting for or against a new paper, although I am sure that rapid online commentary could have a role. However, blogs will not take the place of the normal channels, and they will have little credibility among scientists if the primary concern with a publication clearly is political. Yes, ID is a problem and people who write papers relevant to the issue should be cognizant of that. But it is totally backwards to expect the response to ID to drive science. It also remains dubious that aggressive in-fighting will be any more effective than other possible approaches such as providing a reasoned rebuttal that is cool-headed and based strictly on data.

You guys have gone completely overboard on this one! Matzke's first post is based on the wording of the dscussion (hardly a method flaw). His second is on structure and homology (largely). Homology and structure are very difficult to correlate. Single amino acid changes can completely alter a structure. I don't buy the argument. Rerun the BLAST searches, use a different method for phylogeny construction and see if supports the original conclusion. Sorry to be largely repeating TR Gregory but this is really irresponsible. Its one thing to argue about the wording of the discussion or general conclusions, quite another to start screaming about method flaws and that the paper is sunk. All of the sequences are available and BLASTing is free. Do some science!!

We evolutionary biologists complain furiously when the creationists claim to find serious "problems" in evolutionary research they have little or no expertise in. But Matzke and Myers are doing exactly the same thing here.

Howard Ochman is one of the very best people working in this difficult field, and any claim that his work is deeply flawed should be treated sceptically until other experts have had time to evaluate it carefully.

pablo: I've heard two variations: the formal "P-N-A-S" and very informal "pee-nAs"

It's usually "puh-nass" around here.

PZ, this post was pretty uncool. You should have waited for the full comment and reply to be released. To be fair, you should contact the authors with your criticisms, allow them to respond, and post the entire exchange here. Pharyngula is too widely read and respected for this kind of superficial and one-sided trashing of papers that you disagree with. It's a bad precedent you're setting.

I giggle internally every time I see PNAS in print, and I'm not sure I could suppress a outright guffaw if I heard someone say "puh NASS" or "P N A S" because they both sound enough like penis, and there seems to be the attempt to not sound like penis, so it gets exponentially funnier. Regardless of the augustness of the body behind the publication, you're asking for whatever you get with letters like that. "journal of", "transactions of", "Thus spake" could avoid the 'problem', but what fun is that?

Dave,

When my brother was little, he went to Princeton Nursery School, or, "PNS." I was too young to get the joke at the time, but my mother distinctly remembers how funny yet disturbing it was to hear the kids sing their school song, the chorus of which went, "I love PNS..."

I pronounce it as a single syllable that starts with the pm/tm/km sound. The mouth only opens during the N. It's a bit difficult, though.

Not being used to pronouncing all unstressed vowels the same, I've never noticed the similarity... I just like pronouncing abbreviations as words. :-)

It's usually "puh-nass" around here.

One of my fellow postdocs, long ago, used to refer to it as "Pain in the ass".

David Marjanović:

Is your favorite novel Pnin?

When I think of the initials PNAS, I always think: "Pain in the Ass". It's never occurred to me that it might sound like penis. I find that pretty funny.

Sorry I can't add anything of substance to the main point of the post or the commentary here. This is not my field.

That's pretty low-budget, PZ. If I understand correctly, you're using the huge audience that your blog commands to trash a paper that you haven't fully read and comprehended, without contacting the authors or giving them the right of response, based on something that you read on another blog.

In general, the blogging world is a good thing for science, and I accept the possibility that the paper may be mistaken - but this sort of public tarring is a disgrace. If you've got a serious criticism of the paper, publish it or take it to the author. Don't post these sort of "I heard that..." insinuations: "If half of what I'm hearing about the paper is true ... well, it's going to be rather thoroughly sunk."

Science is an open process, but it's not the sort of pitchfork and torch mob justice that you are trying to promote here.

-Crow

Afterthought:

Perhaps I was too harsh in my assessment of the harm that this sort of thing causes to the authors of the PNAS paper. Given that your readership seems more titillated by the pronunciation of PNAS than engaged in the content of the paper, perhaps Renyi Liu and Howard Ochman have little to be concerned about.

-Crow

The plan was to hold off until Nick and others had completed their analysis. However, we're already seeing people invoke the Liu and Ochman paper as a powerful new tool in the arsenal against ID, and we're also already seeing the IDists seizing on its flaws to repudiate all evolutionary explanations. I've seen enough of the analysis to be convinced. This post was essentially a quick slap of a bandage to stop a potential hemorrhage.

I'd like Liu and Ochman to be right, and I promise that if they somehow fix the problems in their work and the result holds up, I will put a post up with a big title all in bold caps saying "LIU AND OCHMAN WERE RIGHT", OK? I'm just seeing a lot of evidence that says that's going to be highly unlikely.

Never heard of Pnin. I don't read novels much.

indeed. I still don't see where the critics of Nick's initial analysis have addressed the issue of the single gene model espoused in the paper.

this seems a blatantly obvious issue, not some semantic misintrepration.

Nick has studied this issue for years, AND published an excellent review on the subject himself (which was even cited in the paper under the thumb).

He DOES know what he's talking about.

the responses in defense of the paper here seem more knee-jerk than thoughtful, and themselves are likely to lead to the IDers claiming "circling the wagons" than anything in the paper.

think about that.

I for one hardly blame the PNAS review process, as I've read enough articles in any given journal to know that while the peer review process in general is mare than sufficient, occassionally things slip through the cracks.

happens with every journal, including science and nature.

As I stated over on PT, the main thing here will be to hear what the authors themselves have to say, then things will get clearer.

PZ said: "...we're also already seeing the IDists seizing on its flaws to repudiate all evolutionary explanations."

The IDists have not said anything about the data in the paper as far as I am aware, but they have certainly been seizing on Nick's unilateral declaration that it is "a dog" that somehow snuck through peer review.

Ok, latecomer here. Can someone please explain why this paper one way or the other would be important to the ID group?

uber,

one reason only:

it looks at the evolution of flagellum, the flagship of the ID concept of irreducible complexity.

NONE of them understand even half the actual research that went into publishing the paper, as evidenced by both Paul (whoa nelly!) Nelson in comments to Nick's post over on PT and William (WD40) Dembski over on UD.

ergo their commentary so far has actually been more humorous than instructive.

Nick's been far more concerned with the actual paper itself than their commentary, but after the dust settles, I'd bet he will circle back to pop their bubbles for them.

er, not that that hasn't already been done in several places already, and not that it's all that difficult or anything.

I think it would be unfortunate for the perception of Panda's Thumb if this were a shoot first ask questions later situation. Also, the seemingly prevailing blogospheric "wisdom" about the quality of papers in PNAS is pretty ridiculous. It sounds like first year graduate students trotting out some received wisdom to crap on the journal. PNAS is not that different from any number of journals, which I can assure you publish their own crap as well.

We are attempting to redo the entire portion of the Liu-Ochman study that produced Figure 3 and the homology results that led the authors to the "all flagellar genes from one" conclusion. This will take a little time. In the meantime, here are the results of a preliminary attempt to replicate just the FliC homologies for E. coli, reported in Liu & Ochman's Figure 3.

I pronounce it as a single syllable that starts with the pm/tm/km sound. The mouth only opens during the N. It's a bit difficult, though.

Not being used to pronouncing all unstressed vowels the same, I've never noticed the similarity... I just like pronouncing abbreviations as words. :-)

Never heard of Pnin. I don't read novels much.