The NIH peer review system is still the best at identifying innovative biomedical investigators

This recent Nature paper makes the controversial claim that the most innovative (interpreted as best) scientists are not being funded by NIH. Not surprisingly, it is getting a lot of attention in the popular media. The title and introduction make it sound like there is a pervasive problem biasing the funding enterprise against innovative scientists. To me this appears counterintuitive given how much innovation, relative to other funding agencies around the world, comes out of NIH funded researchers (here is a recent example) and how many of the best biomedical investigators in the world elect to work for NIH funded institutions. The authors use data to justify their conclusions but I do not find it very convincing.

First, the paper defines innovative/non-conformist scientists as those with a first/last/single author paper with 1000+ citations in the years 2002-2012. Obvious problems with this definition are already pointed out in the comments of the original paper but for argument's sake I will accept it as useful quantification  The key data point the authors use is that only 2/5 of people with a first/last single author 1000+ citation paper are principal investigators on NIH grants. I would need to see the complete 2x2 table for people that actually applied for grants (1000+ citations or not x got NIH grant  or not) to be convinced. The reported ratio is meaningful only if most people with 1000+ papers are applying for grants but the authors doen't report how many are retired, or are still postdocs, or went into industry, or are one-hit-wonders. Given that the payline is about 8%-15%, the 40% number may actually imply that NIH is in fact funding innovative people at a high rate.

The paper also implies that many of the undeserving funding recipients are connected individuals that serve on study sections. The evidence for this is that they are funded at a much higher rate than individuals with 1000+ citation papers. But as the authors themselves point out, study section members are often recruited from the subset of individuals who have NIH grants (it's a way to give back to NIH).  This does not suggest bias in the process, it just suggests that if you recruit funded people to be on a panel, that panel will have a higher rate of funded people.

NIH's peer review system is far from perfect but it somehow manages to produce the best biomedical research in the world. How does this happen? Well, I think it's because NIH is currently funding some of the most innovative biomedical researchers in the world. The current system can certainly improve, but perhaps we should focus on concrete proposals with hard evidence that they will actually make things better.

Disclaimers: I am a regular member of an NIH study section. I am PI on NIH grants. I am on several papers with more than 1000 citations.

This entry was posted in Uncategorized. Bookmark the permalink.
  • http://twitter.com/caseybergman Casey Bergman

    What is the evidence that the NIH produces "the best biomedical research in the world" and what is your metric for the "best" science?

  • Steven Salzberg

    I wholeheartedly agree with your points - and actually the problem with the recent Nature article goes deeper than that. I requested the authors' original data and re-analyzed it. I found that 75% of highly-cited papers were supported by NIH. This stands in sharp contrast to the authors' report of 40%. My letter to the editor at Nature will appear in next week's issue. However, they cut my letter to just 185 words (from 340) so it is a really bare-bones description of what I did. I may post the whole thing on my own blog after the Nature issue comes out.

  • Pingback: Does NIH fund innovative work? Does Nature care about publishing accurate articles? | Simply Statistics