Future directions for broader impacts at the NSF

Can investments made by the National Science Foundation be enhanced via “broader impacts” contributions?

Can investments made by the National Science Foundation be enhanced via “broader impacts” contributions?

By Sheldon H. Jacobson, Jerome F. Hajjar, Andrew Johnson, Erick Moreno-Centeno and Siqian Shen

On May 26-27, 2016, a group of 30 researchers and administrators in the areas of mechanical, industrial and civil engineering met to discuss a game plan to uncover how investments made by the National Science Foundation (NSF) in these areas can be enhanced via “broader impacts” contributions. Through a series of presentations on broader impacts contributions by academic thought leaders in these engineering research communities, plus a vibrant discussion on the issues related to broader impacts criterion, several points came across that would add value to NSF’s basic research investments.

“Intellectual merit” and “broader impacts” criterion both require significant attention if NSF is to continue to contribute to the economic and social well-being of the United States, and advance the nation’s worldwide position of preeminence and leadership in science, technology, engineering and mathematics. Intellectual merit and broader impacts both have a significant impact on the development of human capital. In a society that demands a growing number of highly trained, technology literate and skilled people, the need to grow and nurture such talent domestically, across all facets of society, is critical to our nation’s economic wealth and societal stability.

Grass-Roots Effort

A grass-roots effort must be used to organically grow the emphasis on and enhance the value provided by broader impacts contributions within the NSF target community of current and future principal investigators (PIs). Dictating such an emphasis from the top at NSF will only have limited value and a short life span. At the same time, the implementation of evaluating broader impacts must be done at the review panel level, through communication by program directors to their panelists, so that proposal evaluations appropriately reflect innovations captured in both intellectual merit and broader impacts, with funding decisions made accordingly.

Indeed, researchers must be incentivized to embrace broader impacts as a critical element of a proposal, and reviewers must be educated to embrace broader impacts as a substantive component in the evaluation process, not just an add-on to the intellectual merit contributions. Intellectual merit and broader impacts cannot be viewed as either/or entities, but rather, complementary and synergistic contributions.

The objective of the workshop was to outline a roadmap for enhancing broad impacts activities in NSF grants within the Division of Civil, Mechanical and Manufacturing Innovation (CMMI). In order to elicit ideas from the workshop participants, several questions were posed. The questions served to initiate discussion on how innovative ideas can be facilitated from the research community and how institutions can work with PIs to realize more significant broader impacts by leveraging institutional resources.

Key Points from the Workshop

The key points that came across during the workshop are as follows:

1. Encourage NSF to provide a framework for thinking about broader impacts that avoids a list of prescriptive activities. Such a framework will enable PIs to define more clearly their project’s broader impacts and will make reviewing and evaluating broader impacts more consistent. We propose that broader impacts be classified along three dimensions: (i) immediacy to the intellectual merit impact, (ii) societal benefit, and (iii) outcome time horizon. Immediacy to the intellectual merit impact can be intrinsic (high immediacy), direct (moderate immediacy) and extrinsic/far-reaching (low immediacy). Societal benefit can be in the domain of information/communication, global leadership, people, economic, health and national interests. The outcome time horizon can be classified as short term (concurrent to grant period), medium term and long term. The classes on all three categories are neither exhaustive nor rigid; broader impacts may fall outside this structure or may fit in several.

2. Encourage NSF to clarify and emphasize to all of its stakeholders that broader impacts can be the impact of the intellectual merits of the proposal and broader impacts are not necessarily achieved through extra activities beyond the intellectual merits activities. NSF does not require proposals to have broader impacts activities beyond the broader impacts intrinsic to the intellectual merits.

3. Improve communication of broader impacts between NSF and its stakeholders (program managers, reviewers/panelists, principal investigators, Congress and the general public). Specifically, both the evaluation of broader impacts in proposals and the reporting of broader impacts to the general public should be emphasized.

4. According to the type of societal benefit they bring, broader impacts can be classified as: information/communication, global leadership, people, economic, health and national interests.

a. Information/communication: Any activity that brings new insights onto existing and anticipated societal issues or promotes and protects professional standards. Such information may lead to better informed societal decisions, which may in turn enhance economic and societal well-being. Also, any activities that promulgate societal benefits and/or inform population sectors that improve quality of life and enhance public literacy. This may include communication and outreach via social media, websites, blogging, videos and traditional media (radio, newspaper, television).

b. Global leadership: Any activity that strengthens the United States’ world leadership (e.g., economic, technological, productivity, sustainability). The broader impacts in this category generally, but not necessarily, will be a direct or indirect result of broader impacts achieved in other broader impacts categories. An example of broader impacts in this category is that of maintaining and growing leadership in strategic technologies such as information systems.

c. People: Any activity that educates people or increases diversity in the human pool of skills resulting from the research. For example, this can be bringing STEM activities in K-12 environments, enhanced opportunities for underrepresented populations or training a more capable workforce. Any activities that enhance and enrich human capital fall within this classification, including the impact of policies on the lives of U.S. citizens.

d. Economic: Any activity that directly or indirectly leads to economic value being created or facilitating the creation of economic value. For example, this can be enhanced productivity, job creations or cost reductions.

e. Health: Any activity that enables the development of new health devices and/or medicines. This includes any activity that directly or indirectly leads to a better quality of life or societal well-being as well as activities that bring new insights onto existing or anticipated health issues. Such contributions are not of a clinical nature, but rather, include societal impacts related to health.

f. National interests: Any activity that brings advancements aligned with national interests, such as national security (e.g., reduces or eliminates the threat of attacks via nuclear, biological and chemical weapons) or national stability (e.g., enhances the viability and stability of national systems including energy supplies, transportation systems, financial markets and/or the environment).

5. Emphasize that the proposed classification framework is neither prescriptive nor exhaustive. Specifically, the aforementioned classes are neither exhaustive nor rigid; broader impacts may fit in several classes or in none. As such, this classification framework is only meant to: (i) facilitate researchers to frame the broader impacts of their project and to think of additional broader impacts that they may not realize their project encompasses; (ii) make explicit the diverse and vast broader impacts attained by the activities funded by NSF.

6. Invigorate efforts to track, document and disseminate the medium- and long-term broader impacts. A first step is to allow PIs to append/update the final project outcomes/abstracts with broader impacts. A second step entails requiring PIs to report the broader impacts achieved five or 10 years after the project end date. Moreover, NSF should implement a highly prestigious and highly broadcasted/disseminated broader impacts award. There can be one such yearly award for each type of societal benefit broader impacts category, one for each immediacy to intellectual merit broader impacts category, and one for each outcome time horizon category, as well as career-long awards (with visibility similar to the PECASE awards) targeting researchers with lifelong achievements of broader impacts. These awards will not only incentivize PIs to report their broader impacts, but also help to disseminate them.

7. Intrinsic and direct (typically short-term) broader impacts are inextricably linked to intellectual merit; in contrast, extrinsic/far-reaching broader impacts are not. The workshop participants believe that a possible approach to incentivize extrinsic/far-reaching broader impacts activities would be via a two-phase procedure. This procedure would work as follows: First, evaluate and select proposals based on strong intellectual merit, as well as intrinsic and direct broader impacts. Subsequently, PIs of funded proposals can propose additional extrinsic/far-reaching broader impacts activities and request an additional funding supplement. The size and review procedures for such broader impacts supplements would be analogous to those of the REU supplement. This two-phase approach will allow PIs to have more information, time and resources to pursue more difficult and substantial tasks in terms of innovating educational forms and reaching out to general public. Moreover, since these activities are funded, it will be easier to track their results and also to hold PIs accountable.

The workshop provided an environment for spirited discussion and sharing of ideas to promulgate the value of broader impacts activities. This resulted in the aforementioned ideas and guidelines to facilitate quantum leaps forward to enhance NSF’s investments in basic research.

The outputs from this workshop serve as recommendations. NSF will ultimately decide which to adopt and implement. With these recommendations in hand, NSF is now positioned to take bold steps to elevate the expectations for broader impacts activities in the proposal that they support. The academic research community can respond by using their creativity and talents to enhance their efforts on broader impacts, and demonstrate that intellectual merit and broader impacts can both be transformative, substantive and impactful.

Sheldon H. Jacobson is a founder professor in Computer Science at the University of Illinois. He has a longstanding commitment to broader impacts, serving as the director for operations research at the NSF from 2012-2014.

Jerome F. Hajjar is the CDM Smith Professor and department chair in the Department of Civil and Environmental Engineering at Northeastern University. He is committed to fostering a broad understanding nationally of the seminal impact for society of NSF-sponsored research and education.

Andrew Johnson is an associate professor in the Department of Industrial and Systems Engineering at Texas A&M University and a visiting associate professor at Osaka University. His international collaborations have helped him understand the value of broader impacts not only locally, but also globally.

Erick Moreno-Centeno is an associate professor in the Department of Industrial and Systems Engineering at Texas A&M University.

Siqian Shen is an associate professor in the Department of Industrial and Operations Engineering at the University of Michigan. She is currently an associate director for the Michigan Institute for Computational Discovery and Engineering (MICDE).


The authors thank Dawn Tilbury for her contributions to this effort. This article was written with support from the National Science Foundation (CMMI-1629955). Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the U.S. government or the National Science Foundation.