This recent Nature paper makes the controversial claim that the most innovative (interpreted as best) scientists are not being funded by NIH. Not surprisingly, it is getting a lot of attention in the popular media. The title and introduction make it sound like there is a pervasive problem biasing the funding enterprise against innovative scientists. To me this appears counterintuitive given how much innovation, relative to other funding agencies around the world, comes out of NIH funded researchers (here is a recent example) and how many of the best biomedical investigators in the world elect to work for NIH funded institutions. The authors use data to justify their conclusions but I do not find it very convincing.
First, the paper defines innovative/non-conformist scientists as those with a first/last/single author paper with 1000+ citations in the years 2002-2012. Obvious problems with this definition are already pointed out in the comments of the original paper but for argument’s sake I will accept it as useful quantification The key data point the authors use is that only 2/5 of people with a first/last single author 1000+ citation paper are principal investigators on NIH grants. I would need to see the complete 2x2 table for people that actually applied for grants (1000+ citations or not x got NIH grant or not) to be convinced. The reported ratio is meaningful only if most people with 1000+ papers are applying for grants but the authors doen’t report how many are retired, or are still postdocs, or went into industry, or are one-hit-wonders. Given that the payline is about 8%-15%, the 40% number may actually imply that NIH is in fact funding innovative people at a high rate.
The paper also implies that many of the undeserving funding recipients are connected individuals that serve on study sections. The evidence for this is that they are funded at a much higher rate than individuals with 1000+ citation papers. But as the authors themselves point out, study section members are often recruited from the subset of individuals who have NIH grants (it’s a way to give back to NIH). This does not suggest bias in the process, it just suggests that if you recruit funded people to be on a panel, that panel will have a higher rate of funded people.
NIH’s peer review system is far from perfect but it somehow manages to produce the best biomedical research in the world. How does this happen? Well, I think it’s because NIH is currently funding some of the most innovative biomedical researchers in the world. The current system can certainly improve, but perhaps we should focus on concrete proposals with hard evidence that they will actually make things better.
Disclaimers: I am a regular member of an NIH study section. I am PI on NIH grants. I am on several papers with more than 1000 citations.