Posted by Abel Pharmboy on December 2, 2008 I just learned this from the Great One, writedit:
NIH grants for FY2010 funding will be scored by review panels on a 1-to-9 scale instead of the current 1-to-5 scale. (Which for you, the applicant, is multiplied by 100 for your priority score.).
Guaranteed that we’ll all still stay between 1.1 and 2.5 for 40% of the grants and it won’t make a difference until people like me die.
Sorry. I’m just very cynical today.
This particular change isn’t gonna do jack diddly shit.
ha!
/checks when due to rotate off study section
Regarding Orac’s question about rationale, the new scoring appears to be an attempt to spread out scores – correct me if I’m wrong, NIH CSR readers.
The announcement states that current scoring allows for only 41 increments (1.0 to 5.0 in 0.1 steps). The new system of 1 to 9 would conceivably give 81 increments (and the scores will be multiplied by 10 instead of 100 to give a priority score of between 10 and 90 instead of 100 to 500 – I think that’s a ruse to get reviewers away psychologically from what a 150 means vs a 190).
(Aside: I know of some late 70s-early 80s data from patient pain scoring scales that a 1 to 10 scoring scale has less precision than a 10 cm line where a patient indicates their magnitude of pain which is then measured to the millimeter. However, 1 to 10 seems to be used in all clinics I’ve been in three states – seems the more precise scale is used for clinical trials. I say this because there may initially be some).
However, this 81 increment scoring will only be after averaging the panelist’s individual scores – reviewers will individually score applications in whole numbers, providing them with only 9 increments instead of the current 41.
Are you still with me?
So, I would submit that the new scoring system provides less flexibility to the reviewers. I certainly understand that currently the priority scores of grants bunch up around a panel’s unspoken but well-recognized approximate historical payline after being percentiled (for the casual observer: panels are only charged with judging scientific merit on a relative basis, with funding decisions/cutoffs left to NIH’s Program component that is independent of the governing org for review. Program also has the discretion to fund borderline grants over those in the fundable range based on programmatic priorities.) Very small differences in the average score determine fundable vs. nonfundable. The new scoring will potential expanding that tight bottleneck by two-fold. But I guarantee (p<0.05) that scores will pile up around a still-very tight range.
In the best case scenario, I see the range being useful for the first one or two rounds under the new system, before reviewers develop a psychological sense of what scores will translate into the approx payline. However, the same argument could be used to say that those first two rounds will be more arbitrary.
I’d been avoiding reading the details because I knew it would just piss me off… guess I better suck it up. your description is not encouraging Abel.
ok, I’m back. This thing is mind bogglingly teeth achingly stupid. NOTHING will change. We’re not talking moving deck chairs, we’re talking changing the flower arrangements in the upperclass lounge.
Read the review/recommendation report from the last OMG downturn in NIH funding that is cited in the current notice.
http://grants.nih.gov/grants/peer/rga.pdf
That will start your blood boiling about how long they’ve known about the supposed problems this current change is supposed to address.
There seem to be a few goals in the present notices (there’s another one on the review criteria) that one can discern.
1) changing the wording, every so slightly, in the description of the review criteria. this does zero to address the problem of individuals applying different concepts and weights within and between the criteria.
2) scoring the individual criteria which will be reported on summary statement. AHAHAHAHAHAAH! what a joke. all this does is cut down the explanatory text and provide some random meaningless number which may or may not have influenced that individual reviewers overall score and may or may not influence other reviewers. yet it provides some fake objectivity/precision to the applicant and to program- precisely the kind of fake impression that is directly addressed and criticized in defending the new scoring scale!
3) overall score is still the thing. no change whatsoever here.
4) combating score spreading. the logic, from what can tell, is mind numbingly dumb. the present scale has too many increments so providing fewer increments will combat score spreading. no it won’t.
5) breaking the tyranny of faux precision. now this is actually a good idea and may be the real intent, slipped in amongst all the fake “based on psych principles of decision making” hand waving. It could have been accomplished before, of course, but is anti-score-spreading in nature. for arguments sake instead of having ten scores in the range from 148 to 152 they will all have the score of 2. Instead of program being forced (apparently, see comment in the old report) to consider the 148 to be better than the 150 which is better than the 152, they’ll just shrug at the lack of difference. Of course, anyone with half a brain knows this anyway! so this is what? further justification for program to do its job in selecting amongst approximately equivalent grants? but….but….this is what they are already SUPPOSED TO BE DOING!!!! ARGH!
I’m still on slow burn but 4 should read “improve score spreading” at the end.
December 2, 2008
December 2, 2008
December 3, 2008
December 4, 2008
December 4, 2008
December 4, 2008
December 5, 2008