Interesting NIH grant tidbits

Thanks for the fun articles!

Found this article in WIRED... future Woo Friday?

http://www.wired.com/medtech/health/news/2007/04/bacteriahacking

Being a past member of an NIH study section, I had the "honor" to be responsible for triaging a proposal by a big-name scientist who got used to the fact that every proposal he had submitted was funded. In response to his proposal being triaged, he wrote a scathing letter to the chairperson of the study section. For the next round of proposal reveiw session, I was relieved of my duties on that study section. I was transferred to another study section that was not related to my area of expertise and thus I asked to be relieved of serving on it. I was never called again to serve on any NIH study section.

Interesting, that your reaction is "I find it nice to know that occasionally even the gods of science have a bump in the road to funding."

My reaction is the process is so broken that it is a crap-shoot just to get considered.

Doctors (reportedly) have a "thing" about "evidence based medicine". Shouldn't grant reviewers have a "thing" about "evidence based grant evaluation"?

I guess making conclusions without evidence is only "scientific misconduct" if it is a "scientific" conclusion, not a "funding" conclusion.

As scientists we know without proper data, correct conclusions cannot be drawn. Can correct conclusion be drawn on grant proposals without reading? Obviously not, but that is what NIH is pretending to do. Were a scientist to draw conclusions without data, they would be accused (and guilty) of scientific fraud.

Optimum allocation of scarce resources requires wisdom, not fairness, cronyism, politics or a count of hot topic buzz words. The scientific worth of a research proposal cannot be expressed by a single number. Ranking proposals and funding the highest 10%, is akin to ranking dietary nutrients and consuming only the highest 10%. A diet sure to result in ill health and eventual death. Is there such a dearth of good research ideas that 90% must be discarded? Are all scientific paradigms correct? Very likely not, but because "high risk" paradigm breaking proposals are not funded, we may never know.

How did this happen? No doubt it is akin to the apocryphal boiled frog effect. A frog placed in hot water will jump out, but a frog in cold water that is gradually heated will stay until it dies. A system that selects for the highest pain threshold doesn't produce the best science, only the most pain tolerant researchers. No doubt successful researchers (and heat tolerant frogs) see no problems with the status quo until their threshold is reached, and they either jump out of the pot, or die. How much of the frog (and scientific) "gene pool" is lost by selecting for only one characteristic, unrelated to excellence or creativity?
Do I have a solution? Unfortunately no. The status quo, and no doubt any modification acceptable to the "leaders" of a field will favor the competition skills of those leaders. Scientific peers are good for judging what Kuhn calls "normal science" . They are not good at judging paradigm breaking conceptual advances, particularly in the formative stages before there is overwhelming proof, which nearly always requires funded research. We are selecting and rewarding competitive prowess, as if scientific progress is just like a foot race. No doubt, in time, what remains of scientific research will be.

Can correct conclusion be drawn on grant proposals without reading? Obviously not, but that is what NIH is pretending to do.

What on earth are you talking about? I never said anything of the sort. In your eagerness to launch off on a tirade, you obviously neglected your reading comprehension skill.

Just like every submitted grant, triaged grants are read in detail by at least two reviewers and a discussant. In addition, the study section members will at least read the abstract, and many will read the entire grant in less detail than the grants assigned to them to be primary reviewers on. The reviewers write up detailed evaluations (2-4 pages), and the discussant a brief (less than 1 page) evaluation. All three assign priority scores to a grant. If the scores of the reviewers and discussant fall below the 50th percentile greater than 3.0 on a scale of 1 to 5, where 1 is the best and 5 is the worst), then at the very beginning of the meeting the chair proposes to triage the grant application, and a vote is taken. If any member of the study section objects to the grant's being triaged, it is placed in the list of grants to discuss. If there is a big discrepancy in the priority scores between any of the reviewers (one rating it very highly and one very low, for instance), the grant will usually be discussed.

As for the feedback given to the applicant, for triaged grants, the raw reviews are sent. For grants discussed and given a priority score under 3.0, the study section coordinator synthesizes the reviews and summarizes the discussion of the study section and sends that to the applicant as the summary statement.

Really, you sound as though you don't know how NIH study sections work. Having served on a study section, I know its weaknesses, but in my experience the members take their duties very seriously and try hard to give investigators, particularly young investigators, a fair shake. It's not perfect, but I haven't been able to think of a better system.

Finally, regarding the issue of giving an overall score to a grant. How do you propose otherwise to rank grants and determine which ones get funded? Really. I want to know. There's only so much money to give out, only enough to fund around 12% of the applications these days. How do you separate the wheat from the chaff? A scoring system of some sort is needed. In fact, without a scoring system, I would postulate to you, things would be even worse, because it would be easier for program directors to dole out money based on personal likes and dislikes, without having to justify some sort of objective score.

or it will decrease the mobility of investigators looking for a better situation, leading to investigators staying put for longer (or at least as long as the grant continues).

With you so far... But is that good or bad? My gut says it's a Good Thing, but my gut has been wrong before...

- JS

Daedalus2u,

Very well said!

I've gotten PO'ed at grant reviews that make it clear the reviewer didn't get past the abstract, but after cooling off I remember it's important to put yourself in the shoes of a reviewer. Given that basically 90-95% of all packets get rejected at the first look, differentiating is not an easy job.

And seeing one of the previous rants, I'd like to add that personality does come through.

Actually I was thinking more about the NIH's own words.

http://www.niaid.nih.gov/ncn/grants/basics/basics_c2.htm

Where they say

"Most reviewers will likely scan your application, reading only your abstract, significance, and Specific Aims."

If they are going to pass judgement on it without reading it, they don't need all the detail. They could have a "pre-screening", with just an abstract significance and specific aims, so that the 90% plus of the effort that goes into these proposals isn't simply wasted.

I don't doubt that the reviewers are trying their best, and I am not critisizing them. They are in an impossible situation. They are forced into triage because they don't have the resources to do otherwise. When patients are triaged, the ones that don't make the cut don't get care and might die. The careers of those researchers who don't make the cut might die too. We know that we can't judge which discoveries are truly important until years later. How can the worth of research be judged before it is done by those who don't understand it?

Would a proposal to investigate the hypothesis that Helicobacter pylori causes ulcers have been funded before it was "proven" in today's climate? I don't think so.

The recent JAMA article showed that antioxidants increase mortality. Would a proposal attempting to look for and explain why have been funded a year ago?

I had already written this tirade and didn't want to "waste" it. Some of it is my frustration at having a "far out" research concept that is unfundable because it is far out, not because it is wrong, or unimportant or unsupported by facts, logic and data.

I've gotten PO'ed at grant reviews that make it clear the reviewer didn't get past the abstract, but after cooling off I remember it's important to put yourself in the shoes of a reviewer

Of course, that's why framing your proposal in the abstract is so critical. One mistake newbies often make is that they don't spend enough time honing the abstract to a razor's edge of their best evidence and persuasion. Ditto the Specific Aims page. You have to grab the reviewer right off the bat at those two pages to make him enthusiastic about reading the rest of your grant.

If they are going to pass judgement on it without reading it, they don't need all the detail. They could have a "pre-screening", with just an abstract significance and specific aims, so that the 90% plus of the effort that goes into these proposals isn't simply wasted.

What do you think that the two primary reviewers and one discussant do? They are, in essence, prescreening the grants assigned to them for the entire study section! The rest of the study section generally only skims the grants that aren't assigned to them as primary reviewers or discussants because to do otherwise would be a ridiculous amount of work. Remember, to review an R01 grant can take anywhere from 4-10 hours each and an R21 between 3-6 hours per grant (at least for me; I'm still rather new at this--old hands can do somewhat faster), and each member is usually assigned 8-10 or so grants as primary reviewer. A typical study section will go over 50 grants in one of its meeting. It's just not feasible for every reviewer to read every grant in excruciating detail. This is why assignments are made.

Grants that score in the lower 50th percentile by the combined score of the primary reviewers are proposed for triage. The study section then votes. If anyone, primary reviewer or any other study section member, objects to triaging a grant, it will likely be discussed. If there is a big discrepancy in the scores given by the primary reviewers, there will be a discussion and the one scoring it better will either relent or will insist that the grant be discussed. (Of course, that's only likely to happen when the reviewer feels really strongly about it.)

As for funding, I don't know what funding mechanism that you used, but the R01 mechanism is designed for large, 5-year projects that are based on a lot of preliminary data. And lots of preliminary data are required for them. However, the NIH is coming up with more mechanisms to test more "out there" ideas, but they are usually smaller and for less time (usually one, or at most two, years). There is one mechanism (I think it's the R21) that doesn't even require preliminary data (although I must admit I've never seen one funded without at least a modicum of supporting data). There are also small business support grants designed to help startup companies and other small biotech companies do innovative research. I don't know what sort of project you sent in, but you should look carefully at the reviews; they tell you what needs to be "fixed" to make the grant more competitive.

I am not with an institution, all my research has all been self-funded so far. I think any funding agency would require animal data, and I don't have the resources to do any animal data unless I get funding. My n of 1 human data doesn't "count".

Part of the problem for me, is that the field I am in, nitric oxide physiology is unusual in that some of the "leaders" in the field have misconceptions about it. Sort of like how the ulcer field was when ulcers were thought to be due to "stress". No amount of "framing" was going to get anyone to consider an infection mechanism for ulcers seriously. They all "knew" ulcers were due to stress. That is the problem I am having, everyone "knows" that nitic oxide comes from nitric oxide synthase. Who every heard of bacteria living on the skin and making physiologically important nitric oxide?

I completely understand that reading every proposal isn't feasible. Triage is all well and good for the screeners and reviewers, but how long does a grant application take to prepare? Probably at least 20 times what it takes you to review it. Who funds that? No one does. Either the researcher works nights and weekends, or has grad students and postdocs who do.

Unfortunately, the NIH grant reviewing system is gears towards triaging scientific ideas that stray from mainstream thinking and methodology. The Helicobacter pylori example is just but one of numerous others that never see one NIH dollar.

Being at the twilight of my scientific career, I can attest to the endless attempts I have made to get the reviewers of my proposals to take them seriously and not triage them because my ideas were outlandish. After 25 years of scraping for small amounts of money from numerous sources, with the majority of the burden falling on my own department (thanks to a great chairperson who believed in the ideas), I have managed not only to show that were not outlandish, but that they are of paramount importance with implications in many areas. Of course, now, when the work has been published in dozens of papers and is part of normal science, it is too late to apply for NIH grants to fund them, since they are just old news.

Writing a grant proposal should not be different from writing a scientific paper. In both cases the writer must persuade the reviewers that the written material is worthwile, either to be funded or to be published. A grant proposal does not need to have a jingle at the openning to attract the reviewer.

Are you sure that it's such a good thing that an 82 year-old is getting his first R01? I think it's great that people continue to do research into their 80s and 90s - they have an enormous well of experience to draw on. And by that point, they usually have a well-established lab and may have associate scientists working under them. However, if you don't even start really building a lab until you're ~78 (I assume that's probably when he started), you ought to make sure you're in very good health. Because if you die, the people you have working for you might find that they've been stranded by your death. And whatever preliminary work you've done may not be carried on to completion in your absence (something that is usually not a problem with established scientists).

To clarify a bit on the "New Investigator" thing for Brian and others. This is not a synonym for "young investigator" or "recently appointed investigator" or the like. It simply means that the PI has never been PI of a major (R01 or equivalent) NIH research award. They could have been supported to the tune of millions by some other agency. They could have been hired as senior muckety-muck after a long career outside of the US. The may have worked in private industry, etc (not sure if NIH intramural positions/funding count or not but that's another issue.

But as to someone 82 getting an award...well, everything that seems bad about this is basically illegal for the study section and the Institute to consider, isn't it? Age-ism and all. Of course it is perfectly ok to practice age-ism against the younger end of the distribution...