The saga of the journal comment.

Recently, Steinn brought our attention to some of the difficulties involved in getting a scientific journal to publish a "Comment" on an article. He drew on a document (PDF) by Prof. Rick Trebino of the Georgia Institute of Technology School of Physics detailing (in 123 numbered steps) his own difficulties in advancing what is supposed to be an ongoing conversation between practicing scientists in the peer reviewed scientific literature. Indeed, I think this chronology of exasperation raises some questions about just what interests journal editors are actually working towards, and about how as a result journals may be failing to play the role that the scientific community has expected them to play.

If the journals aren't playing this role, the scientific community may well need to find another way to get the job done.

But I'm getting ahead of myself. First, let's look at some key stretches of Trebino's timeline:

1. Read a paper in the most prestigious journal in your field that "proves" that your entire life's work is wrong.

2. Realize that the paper is completely wrong, its conclusions based entirely on several misconceptions. It also claims that an approach you showed to be fundamentally impossible is preferable to one that you pioneered in its place and that actually works. And among other errors, it also includes a serious miscalculation--a number wrong by a factor of about 1000--a fact that's obvious from a glance at the paper's main figure.

3. Decide to write a Comment to correct these mistakes--the option conveniently provided by scientific journals precisely for such situations.

Now, of course we might guess that the authors of the original paper are just as sure of their correctness as Trebino is certain that they've erred. And, as obvious as it is to Trebino that they've made significant errors in their paper, Trebino could be the one that's mistaken. However, in the scientific discourse, there is supposed to be an ongoing back-and-forth between scientists, raising objections to assumptions, calculations, and interpretations, and responding to the objections. Publishing a paper is not supposed to bring that exchange to an end, but rather to bring it to a larger slice of the scientific community with something relevant to add to the exchange. In other words, if you read a published paper in your field and are convinced that there are significant problems with it, you are supposed to communicate those problems to the rest of the scientific community -- including the authors of the paper you think has problems. Committed scientists are supposed to want to know if they've messed up their calculations or drawn their conclusions on the basis of bad assumptions. This kind of post-publication critique is an important factor in making sure the body of knowledge that a scientific community is working to build is well-tested and reliable -- important quality control if the community of science is planning on using that knowledge or building further research upon it.

Noting that, at least in theory, the authors of the original paper would want to know about the flaws in their paper (and perhaps recognizing that his own judgment that the paper was flawed might change in the light of additional details from the authors that did not appear in the paper), Trebino wrote to the authors privately to request some of the details that were not provided in the paper. He received no response.

Is it surprising that the authors did not respond to a fellow scientist's request for further information? Scientists are supposed to share information with each other, on the theory that they are engaged in the shared enterprise of building a body of reliable knowledge to be shared by the scientific community. Since what gives findings the status of knowledge is that they survive the scrutiny of other scientists, they are also supposed to provide the details that are needed for scientists to perform such scrutiny. And while there is surely a competitive element in the manner science is practiced, presumably scientists understand themselves to be involved in a quest for truth about the bits of the universe they study -- meaning that a failure to help provide details that would be useful in pursuing a particular piece of truth may come across to others as obstruction.

Surely, scientists who opt out of participating in the follow-up discussion privately don't have firm ground from which to object if the follow-up discussion is then conducted publicly.

Of course, one of the most effective ways to conduct that discussion in public is to harness the power of the journal that published the paper at the center of the discussion.

15. Write a Comment, politely explaining the authors' misconceptions and correcting their miscalculation, including illustrative figures, important equations, and simple explanations of perhaps how they got it wrong, so others won't make the same mistake in the future.

16. Submit your Comment.

17. Wait two weeks.

18. Receive a response from the journal, stating that your Comment is 2.39 pages long. Unfortunately, Comments can be no more than 1.00 pages long, so your Comment cannot be considered until it is shortened to less than 1.00 pages long.

19. Take a look at the journal again, and note that the title, author list, author addresses, submission date, database codes, abstract, references, and other administrative text occupy about half a page, leaving only half a page for actual commenting in your Comment.

20. Remove all unnecessary quantities such as figures, equations, and explanations. Also remove mention of some of the authors' numerous errors, for which there is now no room in your Comment; the archival literature would simply have to be content with a few uncorrected falsehoods. Note that your Comment is now 0.90 pages.

21. Resubmit your Comment.

22. Wait two weeks.

23. Receive a response from the journal, stating that your Comment is 1.07 pages long. Unfortunately, Comments can be no more than 1.00 pages long, so your Comment cannot be considered until it is shortened to less than 1.00 pages long.

The journal's insistence on this immovable length limit is galling in light of the fact that, in the self-same journal, Trebino found a number of published Comments that ran to three pages. It's almost as if the official rules on Comment length were either not official rules, or were bendable if one could only come up with a magic word.

After a wait of a few months:

31. Receive the reviews of your Comment.

32. Notice that Reviewer #3 likes your Comment, considers it important that the incorrect paper's errors be corrected and recommends publication of your Comment as is.

33. Notice that Reviewer #2 hates your Comment for taking issue with such a phenomenal paper, which finally debunked such terrible work as yours, and insists that your Comment not be published under any circumstances.

34. Notice that Reviewer #1 doesn't like it either, but considers that its short length may have prevented him from understanding it.

35. Also receive the topical editor's response, pointing out that no decision can be made at this time, but also kindly suggesting that you consider expanding your Comment to three pages and resubmitting it along with your responses to the reviews.

Perhaps, then, allowing authors (whether of Comments or articles) enough space to lay out their assumptions, methods, data, and conclusions, and to explain what they're saying in sufficient detail that someone else could actually understand what they are saying might be a good idea all around. Communication is hard enough without arbitrarily strict page limits.

Might Trebino have felt a renewed sense of optimism as he revised his Comment and prepared his responses to the reviews? I'm guessing nothing that another few months of waiting couldn't deplete:

41. Receive the second set of reviews of your Comment.

42. Notice that Reviewer #3 continues to like your Comment and continues to recommend its publication.

43. Notice that Reviewer #2 continues to hate it for taking issue with such a phenomenal paper, which finally debunked such terrible work as yours, and again insists that your worthless Comment not be published.

44. Note further that Reviewer #2 now adds that your Comment should under no circumstances be published until you obtain the important details from the authors that you confessed in your response to the reviewers you were not able to obtain and are not ever going to.

45. Realize that Reviewer #2's final criticism inevitably dooms your Comment to oblivion until such time as the authors provide you with the important details, your best estimate for which is never.

Here, notice that the unwillingness of authors to participate in a private discussion of their published work is being used as a reason to obstruct a public discussion of their work. Seemingly, this gives authors the option of halting any productive discussion of their work once it has been published. This does not fit easily at all with the picture of the community of science as engaged in ongoing discussion and scrutiny, checking results and interpretations even after they have been published and sharing important information with the community. How is science supposed to be self-correcting if recalcitrant authors can take discussion of their results off the table at their whim?

But, the plot thickens:

47. And, in an absolutely stunning turn of events, note also that Reviewer #1 writes further that he has also somehow secretly obtained from the authors the important details they neglected to provide in their paper and refused to send to you. Even better, using them, he has actually checked the relevant calculation. And he finds that the authors are wrong, and you are correct.

48. Realize that it is now no longer necessary to respond to the impossible criticism of Reviewer #2, as Reviewer #1 has kindly done this for you.

49. Add a sentence to your Comment thanking Reviewer #1 for his heroic efforts in obtaining the authors' important details and for confirming your calculations.

50. Receive the editor's decision that your Comment could perhaps now be published. Unfortunately, Comments can be no more than 1.00 pages long, so your Comment cannot be considered further until it is shortened to less than 1.00 pages long.

51. Point out to the editor that most Comments in his journal are two to three pages long. Furthermore, it was the editor himself who suggested lengthening it to three pages in the first place. And Reviewer #1 strongly recommended leaving it that long.

I don't know about you, but at this point I need to pause for some deep, cleansing breaths.

More editing for length ensues. More time passes.

80. Receive a response from the senior editor that, while your Comment is now short enough and properly formatted, over the many modifications and shortenings that have occurred, its tone has become somewhat harsh. For example, a sentence that originally read, "The authors appear to have perhaps accidentally utilized an array size that was somewhat disproportionate for the corresponding and relevant waveform complexity," has evolved into: "The authors are wrong."

Can you imagine that this continuing exchange with the editors might have an effect on the tone of a Comment?

83. Wonder whether your Comment has finally been sent to the authors for their Reply, or instead was lost, trashed, or sent back to the reviewers for further review and possible rejection. ...

90. Realize that you had stopped carefully reading the journal, and, as a result, had missed the "Erratum" published by the authors on the paper in question six months earlier, shortly after you submitted your short-lived three-page version of the Comment.

91. Note that, in this "Erratum," the authors actually admitted no errors and instead reported new--similarly incorrect--numbers, which they concluded "do not change any conclusions" in their original paper.

92. Feel old, as you can remember the days when Errata involved correcting old errors and not introducing new ones.

93. Note also that, in their "Erratum," the authors have actually responded to some highly specific criticisms of their errors you mentioned in the three-page version of your Comment--criticisms that you had removed when shortening it to meet the journal's strict 1.00-page limit. Criticisms the authors couldn't possibly have known about in view of the journal's strict confidentiality rules for submitted papers, unless this version of your Comment was somehow leaked to them...

It would seem, then, that the original authors did want to be involved in the follow-up conversation about their published findings, sort of. They just preferred not to provide particular details requested by other scientists, or to answer the specific questions raised by those scientists. And someone involved on the editorial end of the journal -- an editor, a reviewer, or someone else with access to the submitted but not-yet-published Comment -- decided that it was OK to help the authors dodge participating in an actual discussion with give and take, one over which the authors did not have most of the control.

More deep, cleansing breaths.

108. Receive a phone call from the senior editor, who takes advantage of this opportunity. He has suddenly remembered that your Comment's tone is a bit harsh. He is concerned that the authors, who appear to be highly motivated and quite crafty, will complain loudly and aggressively about the obviously preferential treatment your Comment is clearly receiving from the journal and make his life miserable. He objects to nearly every sentence in your Comment, in each case, insisting on a considerably longer sentence. He insists that you not say that the authors are "wrong" and suggests instead "perhaps mistaken." He also insists on replacing the word "so" with the unforgivably long "therefore."

Some thoughts for the put-upon senior editor here: Loud and aggressive complaining happens, and working out ways to deal with it might be a useful life skill. Publishing the Comment and inviting the authors submit their own response to it would certainly diffuse worries about preferential treatment. And throttling serious communication between scientists seems like the opposite of what a scientific journal ought to be doing.

119. Encounter a journal representative at a conference, who kindly mentions that the one-page version of your Comment was, in fact, sent to the authors for their Reply. And, after a series of delays, they have submitted it. But, unfortunately, it is extremely contentious and will be rejected unless toned down significantly. It's as if, for some reason, they want it to be rejected.

120. In preparation for the final phase of the Comment process, write to the editor asking if you will be able to see the Reply to your Comment and make minor modifications in view of it, as allowed by most journals.

121. For once, obtain a quick response: "No."

122. Finally receive notice from the editor that the authors' official Reply to your Comment has been reviewed and processed. Unfortunately, it was not found suitable for publication and so was rejected. And because, for maximum reader enjoyment, it is the policy of this journal that a Comment cannot be published without a Reply, your Comment cannot be published. This decision is final.

I'm going to need something stronger than deep, cleansing breaths here.

While the authors of the original paper seem not to be behaving well here at all, what makes me pig-biting mad about this chronology is the journal policies which seem designed to give too much power to the authors of a paper to shut down follow-up discussion of their work if they so choose, whether by refusing to reply to a submitted Comment, or by submitting a reply which they have reason to believe will be judged "unpublishable". Journal policies like these do not advance the ongoing scientific discussions by which science's "self-correcting" mechanisms work. I'm inclined to think that if a journal has policies that get in the way of science working as it should, scientists would be well-advised to dump that journal in favor of others with less stupid policies.

Indeed, Trebino expresses the central concern pretty calmly:

Nearly everyone I've encountered who has written a Comment has found the system to be heavily biased against well intentioned correcting of errors--often serious ones--in the archival literature. I find this quite disturbing.

The idea that the journal here seems to be missing is that they have a duty to their readers, not just to the authors whose papers they publish. That duty includes transmitting the (peer reviewed) concerns communicated to them about the papers they have published -- whether or not the authors of those papers respond to these concerns in a civil manner, or at all. Indeed, if the authors' response to a Comment on their paper were essentially. "You are a big poopyhead to question our work!" I think there might be a certain value in publishing that Reply. It would, at least, let the scientific community know about the authors' best responses to the objections other scientists have raised.

The scientific community, of course, will draw on multiple sources of data to make its own judgments about who the real poopyheads are.