04
Jan

Does NIH fund innovative work? Does Nature care about publishing accurate articles?

Tweet about this on Twitter93Share on Facebook30Share on Google+13Share on LinkedIn11Email this to someone

Editor's Note: In a recent post we disagreed with a Nature article claiming that NIH doesn't support innovation. Our colleague Steven Salzberg actually looked at the data and wrote the guest post below. 

Nature published an article last month with the provocative title "Research grants: Conform and be funded."  The authors looked at papers with over 1000 citations to find out whether scientists "who do the most influential scientific work get funded by the NIH."  Their dramatic conclusion, widely reported, was that only 40% of such influential scientists get funding.

Dramatic, but wrong.  I re-analyzed the authors' data and wrote a letter to Nature, which was published today along with the authors response, which more or less ignored my points.  Unfortunately, Nature cut my already-short letter in half, so what readers see in the journal omits half my argument.  My entire letter is published here, thanks to my colleagues at Simply Statistics.  I titled it "NIH funds the overwhelming majority of highly influential original science results," because that's what the original study should have concluded from their very own data.  Here goes:

To the Editors:

In their recent commentary, "Conform and be funded," Joshua Nicholson and John Ioannidis claim that "too many US authors of the most innovative and influential papers in the life sciences do not receive NIH funding."  They support their thesis with an analysis of 200 papers sampled from 700 life science papers with over 1,000 citations.  Their main finding was that only 40% of "primary authors" on these papers are PIs on NIH grants, from which they argue that the peer review system "encourage[s] conformity if not mediocrity."

While this makes for an appealing headline, the authors' own data does not support their conclusion.  I downloaded the full text for a random sample of 125 of the 700 highly cited papers [data available upon request].  A majority of these papers were either reviews (63), which do not report original findings, or not in the life sciences (17) despite being included in the authors' database.  For the remaining 45 papers, I looked at each paper to see if the work was supported by NIH.  In a few cases where the paper did not include this information, I used the NIH grants database to determine if the corresponding author has current NIH support.  34 out of 45 (75%) of these highly-cited papers were supported by NIH.  The 11 papers not supported included papers published by other branches of the U.S. government, including the CDC and the U.S. Army, for which NIH support would not be appropriate.  Thus, using the authors' own data, one would have to conclude that NIH has supported a large majority of highly influential life sciences discoveries in the past twelve years.

The authors – and the editors at Nature, who contributed to the article – suffer from the same biases that Ioannidis himself has often criticized.  Their inclusion of inappropriate articles and especially the choice to require that both the first and last author be PIs on an NIH grant, even when the first author was a student, produced an artificially low number that misrepresents the degree to which NIH supports innovative original research.

It seems pretty clear that Nature wanted a headline about how NIH doesn't support innovation, and Ioannidis was happy to give it to them.  Now, I'd love it if NIH had the funds to support more scientists, and I'd also be in favor of funding at least some work retrospectively - based on recent major achievements, for example, rather than proposed future work.  But the evidence doesn't support the "Conform and be funded" headline, however much Nature might want it to be true.

  • http://twitter.com/ecoPeffs Mel Peffers

    Great read. Thanks for posting the whole letter.

  • http://twitter.com/DrHubaEvaluator George Huba

    Why not analyze all 700 papers and really get to the bottom line? I agree with your points completely but I wonder why you would not analyze all 700 and have something that policy makers could work with rather than holding back and only analyzing 125.

    • Steven Salzberg

      Just a matter of time, but I agree looking at all 700 would be nice. I only had a few days and Nature only publishes letters if they are sent in very, very soon after the original article. I downloaded the PDF files for all 125, and read the abstracts plus acknowledgments. And note that the original study only analyzed 200 papers, from which they further chose a subset of 158.

  • jwoodgett

    Rather than look at such statistical outliers, would it not be more informative to evaluate the degree to which Nobel, Lasker, Gairdner, etc. awardees were supported by NIH in the publications cited for the relevant awards? Indeed, just the list of NIH awardees who are Nobel Laureates should give pause to the idea that NIH fails to support innovative science: http://www.nih.gov/about/almanac/nobel/

  • Mike

    you are obviously one of those mediocre tyrants sitting in study sections...

  • mike

    you are obviously one of those mediocre tyrants sitting in study sections

  • S

    Do you know how many of your 125 sampled papers overlaped with their 200 sampled papers?

    • Steven Salzberg

      no, but you can get my list from the Nature site, or by emailing me, and you can get all 700 from Ioannidis by emailing him, if you want to check.

  • Pingback: NextGenSeek’s Stories This Week (03/01/13)

  • Pingback: Is Steven Salzburg Defending the Indefensible? « Homologus

  • homolog.us
  • Pingback: Is Steven Salzberg Defending the Indefensible? « Homologus