Your Review of Grant Review Sucks

. . . and so does the current grant review system. And manuscript review sucks, too. And your argument is based on flawed data.

A couple of months ago I posted on an article in Cell that drew parallels between the NIH grant review process and the TV show American Idol. As someone new to the grant writing game (and who has never seen what happens when they get reviewed), I wasn't sure if the metaphor was appropriate or not. Michele Pagano's article also advocated prescreening proposals, switching to an electronic system, and holding review panels over teleconference. These suggestions are intended to streamline the grant review process and decrease costs.

Three letters published in the most recent issue of Cell take Pagano to task for his ideas. The first, by Karl Munger, argues that screening proposals using a presubmission process will favor established scientists over more junior researchers. Additionally, Munger believes that teleconferencing reviews will only lead to more Simon Cowells, and we cannot put fair review at stake with funding levels so low. Then he goes on to rail on the NIH's new electronic submission system, which appears to be piss-poor (especially in relation to the one in place at the inferiorly funded NSF). The new NIH system is only compatible with the Windows operating system, has additional forms, is still being updated, and is too hard to figure out. Hopefully they'll have it all straightened out when I submit my first NIH grant in five years or so.

One of the other replies to Pagano's piece goes after the idea that grant review stands to learn something from manuscript review. In his correspondence, Steven Dowdy argues that the anonymous nature of manuscript review allows for reviews that are much harsher than needed. To remedy the problem, Dowdy suggests that journal editors hold monthly teleconferences with all reviewers to discuss the reviews from the past month. He hopes that this will prevent erratic reviews by forcing reviewers to own up to their criticisms. Obviously, Pagano's suggestion that grant review move to a more anonymous system modeled after manuscript review does not sit well with Dowdy.

Finally, in the third letter Ivo Dinov argues that if a Simon Cowell can hijack a review panel, it is the panel's fault not that of the NIH. A good panel should stand up for themselves in the face of a bully. On top of that, Dinov questions the assumption that the NIH needs to streamline its grant review process. He points to the fact that grant review currently makes up less than 0.2% of the annual NIH budget. There is no need to switch to teleconferencing as it would not save much money in the grand scheme of things.

For more on NIH grant review see this post at Effect Measure and this one from Orac.