A recent news article in Science suggests that peer review of grants submitted to the NIH National Heart, Lung and Blood Institute (NHLBI) does not reliably predict the projects likely to have the greatest impact. This conclusion was reached after a systematic study of proposals submitted to NHLBI beween 2001 and 2008 found no correlation between the ‘priority scores’ of funded projects and the number of citations and publications those projects generated. A similar study used an alternate metric, time to publication (the rationale being that more significant studies would be published faster, I guess) of NIH-funded clinical trials, and again found no correlation.
One could argue that all the aforementioned criteria are inadequate measures of scientific impact, and it’s certainly not hard to think of others, including the impact factor of journals the research was ultimately published in, its contribution to translational initiatives, and extent of dissemination of the resulting products. It’s also impossible to rule out the possibility that projects which were ultimately funded had a greater impact than those which weren’t (had they been funded).
But to me, the simplest and most obvious explanation for the study findings is that there is a threshold past which all projects have similarly high merits, and any differences between the priority scores of studies which exceed the threshold are stochastic and meaningless. If that threshold is less stringent than the priority score cutoff for funding, a comparison of priority scores of funded proposals would have no discriminating power, consistent with the studies above. Taking this a step further, one could hypothesize that some of the higher-scoring but not funded projects would have had, on average, similar impact to those which were ultimately successful in securing funds.
A fun experiment which will never happen would be to compare outcomes (based on a variety of metrics) from research proposals selected based on priority scores being in the 90th percentile or higher (comparable to current R01 paylines at NHLBI), versus ones selected at random from the top, say, 50% of proposals. Of course, the idea of randomly deciding which projects (above a certain threshold) to fund would be unpalatable to most if not all scientists, but I’m not convinced that it would be functionally very different to the status quo.