Forum – NSF: Peer reviews peer-review process

Review criteria at the National Science Foundation: a more coherent path for discovery and a unifying metric for evaluation.

By Sheldon H. Jacobson

Sheldon H. Jacobson

The National Science Foundation (NSF) is the largest investor in non-medical academic research in the nation. Created in 1950, NSF is a U.S. federal government agency, funded by taxpayers, whose mission is “to promote the progress of science; to advance the national health, prosperity and welfare; to secure the national defense.” By investing in basic and applied research in the sciences and engineering, mostly in academia, it has served as a catalyst for economic growth and facilitated societal well-being. It has enabled the discovery of new knowledge and spearheaded innovations that have helped shape the economic and social fabric of the United States. Together with the National Institutes of Health, the NSF has enabled the United States to become a world leader in scientific innovation and advancement.

NSF makes funding decisions based on a peer-review process. The peer-review process typically calls upon academic and industrial experts to evaluate the creativity of proposed research and the value of the knowledge that would ensue from its support. Over the years, NSF has used a variety of criteria to evaluate proposals for funding. Prior to 1997, there were four peer-review criteria: 1) research performer competence, 2) intrinsic merit of the research, 3) utility or relevance of the research, and 4) effect on the infrastructure of science and engineering. The theme of these criteria centered on “advancing the frontiers of knowledge.”

Under concerns raised by the efficacy of the review process based on these four criteria, the National Science Board (NSB), the entity that sets policies for NSF, recommended changes resulting in a reduction from four down to two peer-review criteria: intellectual merit, to capture the knowledge created by the research, and broader impacts, to capture the societal benefits that would accrue from NSF supported research investments. This change was enacted into law by the 114th Congress, with the American Innovation and Competitiveness Act. As with many actions, the laws of unintended consequences may yield the dominate effect.

As an academic who has been supported by NSF funding, has sat on NSF review panels, led an NSF-sponsored workshop on enhancing broader impact, and served as a program director at NSF as a rotator (while on leave from my university faculty position), I appreciate what NSF does, and value the contributions that it has made to our nation’s well-being. Indeed, I have personally benefited from the existing NSF funding model and peer-review process. I also believe that having two criteria for evaluating research proposals for support is fundamentally flawed, and limits the potential value of what NSF can achieve.

When presented with two objectives, human nature inherently prioritizes them, based on internal biases or external values. Given two criteria to assess the value of a research proposal, researchers and reviewers implicitly place a priority on one criterion over the other. This is manifested in one criterion being optimized (the primary) and the other criterion meeting a threshold or being used as a tiebreaker (the secondary). With few exceptions, intellectual merit serves as the primary and broader impacts serves as the secondary.

The result of this hidden hierarchy is that the full value of NSF investments is not being fully realized. Does this mean that NSF is funding low-quality research? Most definitely not. NSF receives more quality proposals than it can support. What it does mean is that the collective value of research being supported by NSF can be elevated to better serve its stakeholders.

NSF Logo

The fundamental metrics with the two review criteria are sound. What is required is a single cohesive criterion that embodies both intellectual merit and broader impacts, so that they are not considered as disparate concepts within proposals, but rather, a single unifying ideal to strive for. One possible way to express this ideal is with the criterion, advancing knowledge for society. This criterion embodies the intellectual merit criterion for advancing the frontier of knowledge, both fundamental and applied, and the broader impacts criterion for societal benefits. The name of the criterion is less critical than the need to identify a unifying measure for evaluation.

The societal benefit of new knowledge may vary by the scale of the societal footprint impacted and the time horizon over which it would be realized. The most basic research (such as in theoretical physics, chemistry, computer science, mathematics and the methodological foundations of operations research) would be covered under this umbrella, since fundamental knowledge would benefit a more focused aspect of society (at least initially, namely, members of a particular field), and be realized over an extended time horizon. Applied research (such as in many areas of engineering, computer science, the social sciences and operations research applications) would be covered in their benefit to a larger footprint of society, over a shorter time horizon.

By unifying the merit criteria into a single objective, researchers are guided to think more holistically about their research proposals, limiting the either/or trap that researchers and reviewers inadvertently fall into. To reiterate, the fundamental components of intellectual merit and broader impacts are sound and should be preserved; what is needed is the incentive to coalesce them into a more coherent path for discovery and a unifying metric for evaluation.

Recasting the NSF peer-review criteria into a single cohesive criterion is a step to enhance the value of NSF investments. To test this hypothesis, it may be worthwhile to conduct a pilot study within NSF to evaluate how researchers and reviewers respond to a single criterion peer-review model. Such a pilot study would either inform the NSB on one possible avenue to enhance the peer-review process, or provide evidence that the existing two criterion model has sufficient merit to be retained.

NSF recognizes that the peer-review process can be improved. Although the NSB could not predict the consequences of its actions in 1997, creating two evaluation criteria may have inadvertently created a schism and confusion in the peer-review process. I believe (with a pilot study supporting or refuting this belief) that a single review criterion best serves NSF, the communities who benefit from NSF support and most importantly, the taxpayers who support the opportunities that NSF makes possible with their tax dollars. Growing federal budget deficits in the United States and the need to prioritize federal tax dollar investments are incentives to revamp NSF funding criteria in this manner. By taking this simple, albeit important, step forward, NSF may position itself for even greater successes, and will permit it to retain its status as the gold standard for scientific discovery across the globe.

Sheldon H. Jacobson (shj@illinois.edu) is a founder professor in engineering at the University of Illinois at Urbana-Champaign. He served as a program director for Operations Research at the National Science Foundation from 2012-2014.

Acknowledgement

This article was written with support to the author from the National Science Foundation (CMMI-1629955). Any opinions, findings, conclusions or recommendations expressed in this material are those of the author and do not reflect the views of the United States government or the National Science Foundation.

Visit these Advertiser Sites