Assessing the Impact of Programs, Departments, and Institutions
In this section, we'll explore approaches that research leaders can use to identify, understand, and assess the range of impacts being generated by the multiple projects and people within a program, department, or institution. We'll cover:
- Defining your organization's societal impact goals
- Impact descriptors to help you communicate about your organization's impacts
- Approaches to assessing cumulative impacts within an organization
- The role of societal impacts in academic reward and advancement structures
At each step, you'll have an opportunity to draft an impacts assessment for your organization.
When you structure your program or institutional assessment around societal impact, it is important to define for staff and researchers what societal goals you want to contribute to. The goals should focus on your organization’s contributions to solutions to problems for particular people, communities, or places in the world. Societal goals related to positive changes you hope to contribute to outside of the academic literature, in the lives of people and our environment.
Some institutions have committed to making contributions to the UN Sustainable Development Goals (UN SDGs) as a way to define their societal goals. Others have defined specific, regionally appropriate challenges they are dedicated to addressing. In some cases, institutions adopt goal definitions aligned with funding or federal government goals and objectives.
While no one set of societal goals is perfect for every organization, by selecting and defining a set of goals, programs and institutions can help their staff and researchers find common purpose in working toward positive contributions in their communities, regions, and worldwide.
Below are some examples of societal goals you can explore to find the best fit for your program or institution.
| Scale or Focus of Goals | Goal Set | Institutional Examples |
|---|---|---|
| Global-scale goals | United Nations Sustainable Development Goals | |
| Municipal/Regional Government Goals | Southern Arizona’s Prosperity Initiative Municipalities in Southern Arizona jointly defined a set of goals and objectives for the region in four broad areas: Education, Critical Family Resources, and Asset Building and Infrastructure. | Programs or organizations could align their work with these goals to support ongoing regional efforts. Working with local agencies offers opportunities to engage directly with local decision makers and community members. |
| Funding Agency Goals | The National Science Foundation NSF requires that all applications and grants address the broader impacts of the research being proposed to answer the question “How does your research benefit society?” | Programs or organizations could align their work to these impact goals, particularly if a large portion of their research funding is tied to NSF. However, be aware that other funders do not use the same categories as NSF. See the section on Funder Definitions and Expectations for more information. |
| Community-scale Goals | When working with a specific community, ensuring that their goals and priorities are centered in the project is important. Collaborative, community-based goal setting is important in this approach. Below are several resources that can help guide your program or organization through that process. Community Autonomy and Place-Based Environmental Research: Recognizing and Reducing Risks. CLEAR Lab Book: A living manual of our values, guidelines, and protocols. | Programs or organizations that focus their work with particular communities can engage in a process of collaborative goal-setting with their community partners to ensure that research activities are aligned with local needs and priorities. |
YOUR IMPACT ASSESSMENT PLAN
In the template linked here, write the goals of your program, organization, or institution using an example from above or outlining specific organizational goals and who or what will benefit from the work done toward these goals.
Be sure to define a time frame, such as projects that have been active in the last 5 years.
| Organization's Goals | Projects/Activities Addressing the Goal | Research Partners |
|---|---|---|
| Brief statement of the goal. | Which projects/activities are focused on this goal? List and/or count. | Who is your organization collaborating with as part of this research? |
| UN SDG 6: Clean Water and Sanitation | 6 projects | 8 different neighborhood throughout the state state water resources agency 2 county water management offices 3 municipal sustainability offices |
| UN SDG 12: Responsible Consumption and Production | 3 projects | 2 local neighborhoods industry partner |
| UN SDG 14: Life Below Water | 4 projects | conservation NGO (with direct connections to community partners) local watershed management group state wildlife agency |
Want more?
- Beyond the Academy Network (https://beyondtheacademynetwork.org/)
- Transforming Evidence Network (https://www.pewtrusts.org/en/projects/evidence-project/transforming-evidence-network)
- APLU Public Impact Research (https://www.aplu.org/our-work/2-fostering-research-innovation/public-impact-research/)
Impact descriptors use our understanding of the ways that new knowledge or research is used in practice to help us describe the many ways our research is contributing to positive changes in the world or for our external research partners.
Identifying types of impact
| Category | Description | Impact Example | Impact Example |
|---|---|---|---|
| Conceptual | Research contributed to changes in people’s knowledge about or awareness of an issue | Your external partners report that the research findings are being discussed regularly at agency meetings; they are thinking about the implications for their program and how they might incorporate the findings into their practices | Your project involved collaboration with a local neighborhood organization to regularly test water quality in a stream running through the neighborhood. It has contributed to community members understanding pollution rates and pollution sources. They demonstrate this knowledge by doing presentations for other neighborhoods and are able to answer technical questions about water quality posed by their peers. |
| Capacity-Building | Research contributed to enhancing the skills, expertise, or resources of an organization or group of people | Through your research project, you created an online professional development course to help teachers learn to incorporate climate science into their high school classrooms. You have offered the class three times to cohorts of 20 teachers. In a follow-up survey, teachers reported that their students engage well with the material and their test scores have improved. | Your project developed a plan to share new research instruments with a smaller, R2 university in your region. As a result of having access to the instruments, faculty and students at the R2 have developed and held an advanced lab skills course for 40 students so far; published research findings based on work completed in the lab; and have just submitted their own funding proposal that will involve use of the lab space. |
| Connectivity | Research contributed to new or strengthened relationships, partnerships, or networks that endure after the project ends. | You were originally approached by a resource management agency for technical assistance 5 years ago. Since then, the agency has asked for your assistance with analyses two more times and you just collaborated on a successful funding proposal that will allow you to study an issue of significant concern for the agency. | In the first phase of your project, you worked closely with the sustainability office of your local municipal government to create an analysis of air quality specific to the region. After completing the report, your municipal partners shared the document in their network of sustainability offices. Since then, three more municipalities have reached out to ask to collaborate on analyses for their regions. |
| Instrumental | Research contributed to tangible changes to plans, decisions, practices, or policies | You worked with staff of a local wildlife management agency throughout your project. When it came time for them to update their species management plan, they cited your report and journal article in the plan. They also asked you and your co-investigator to review their plan to ensure that the research findings were explained accurately. | Throughout your project, you collaborated with a small group of high school science teachers to help them incorporate cutting-edge planetary science research into their AP science classes. By the end of the 3-year project, all 5 teachers had developed curriculum that met state science standards and they were actively using in their classrooms. |
| Social and/or Environmental | Changes to social and/or ecological systems, such as improvements in health and well-being or in ecosystem structure and function that result from changes in policy, practice, or behavior. | Your team worked closely with a local farmers' collective that focuses on growing local and drought-tolerant crops. As part of the project, your team helped the collective create a new marketing strategy. Over the course of three years, farmers in the collective saw a 15% rise in income from their farm products. | In collaboration with a local neighborhood, your research team installed a green stormwater capture site in a local park. The project gave your students as well as community members opportunities to practice landscape design and hydrology skills. The team has been monitoring the site for 2 years since its installation and has noted an increase in bird biodiversity as well as lower web bulb temperatures in the area immediately surrounding the site. |
Want more?
Meadow, A. M., & Owen, G. (2021). Planning and Evaluating the Societal Impacts of Climate Change Research Projects: A guidebook for natural and physical scientists looking to make a difference. http://doi.org/10.2458/10150.658313
Edwards, D. M., & Meagher, L. R. (2020). A framework to evaluate the impacts of research on policy and practice: A forestry pilot study. Forest Policy and Economics, 101975. https://doi.org/10.1016/j.forpol.2019.101975
Meadow, A. M., Owen, G., Joshi, N., & Lodge Otto, E. (2024). Combining Impact Goal and Impact Descriptor Frameworks to Elucidate the Societal Impacts of Research: A Pilot Study. Research For All. https://doi.org/10.14324/RFA.08.1.03
Planning Your Assessment
To make the most of the time and resources you put toward an organizational impacts assessment, it is important to prepare your staff, researchers, and leadership ahead of time. Below are some tips for planning and completing a robust assessment that will help you understand your organization better - and help you communicate about your accomplishments to a range of interested audiences.
Introduce your assessment framework to program staff and researchers.
- Define your program/institution’s societal goals (use the resources above to help with that process).
- Share impact categories and examples to familiarize the organization with the terms.
Determine how you will use the information gathered through the impacts assessment.
Some examples include:
- Primarily for internal evaluation and strategic planning
- Demonstrating your contributions toward institution-wide goals
- Demonstrating your impact in a way that goes beyond required reports on standard academic metrics.
- Building trust with community partners by reporting back on projects and other collaborations.
Purpose of Assessment Key Questions Audience Format Organizational Planning and Internal Assessment Are we meeting our organization’s goals and expectations? leadership and members of our organization internal report, presentation(s) at meetings and strategic planning sessions Five-year organizational or academic program review How do our impact goals and outcomes align with the institutions goals? institutional leaders Complete date will be used internally for management decisions, but primary outputs will be a brief written report (2 - 4 pages) and presentation at leadership meeting (10-15 minutes maximum). Public-facing communications for external partners and other community members; a way to demonstrate that we are responsive to community priorities and want to work effectively with communities. How do our impact goals align with those of our external partners? members of the public who have been involved in our projects or have supported our organization in other ways Regular newsletters highlighting 1 - 2 projects each time. Stories should be about 500 words per project. We may do presentations or meetings with specific partners when we want to provide more detailed information about particular project or topics. In all cases, the information should be geared to an engaged and insightful public audience.
Commit to regular data collection
- Your timeline should align with your use plan to ensure that data contributed as part of the assessment process is put to productive and transparent use in the organization.
- Regular data collection, for example as part of annual reporting procedures, can help keep staff and researchers familiar with the processes. But sometimes, new impacts do not emerge every year, so a 2 - 3 year cycle may be more efficient.
- The first time you conduct the assessment will take more time.
Create a data collection plan and tools
- Example of data collection survey ( https://forms.gle/2scfBTVhyuUSwgBt6)
- Will your staff and researchers need some professional development in order to respond to the survey? If so, who will deliver the training materials and when?
Create an analysis plan.
Questions to consider:
- Who will collect and compile the data?
- What is the procedure if data collection surveys are incomplete?
- How will you ensure that respondents have provided sufficient evidence of their impact?
Presenting and Sharing Impact Examples
- Narrative case studies that follow a set outline can help readers gain an appreciation for the work involved in generating the impacts as well as the significance of the impact to the people involved. Case studies provide an opportunity to share quotes or other feedback from people who have benefited from the research or project.
- The UK Research Excellence Framework Case Study Database allows you to browse and search for impact case studies submitted to the REF 2021. REF Impact Case Studies come from all research disciplines and provide examples of a wide range of research impacts. The case studies can help provide a template and inspiration for how to frame examples from your organization. https://results2021.ref.ac.uk/impact
- Examples from UA https://impact.arizona.edu/research-impacts-project)
- Descriptive statistics that illustrate cumulative contributions to societal goals can help you quickly communicate about the focus areas of your organization’s work. However, it is important to make sure that you have robust evidence of the ways in which your work has moved the needle on the goals so that your assessment doesn’t become a box-checking exercise.
Create a summary of impacts
Some examples of how to convey your summary include:
- A bubble diagram that provides a quick overview of the contributions to the UN SDGs by one research center. The size of the bubbles reflects the number of projects contributing to that SDG. However, this kind of overview does not provide information about how and under what conditions projects are contributing to the SDGs, so it is not as robust as a narrative case study.
A combination of brief narrative and visualization of impact categories can highlight the ways that different projects have contributed to changes for people, communities, and the environment. In the example below, we see how four different projects generated impacts in a number of different categories. The short summaries in the red ovals help to balance the quantitative information (number of red ovals) with a description of the content of the changes. This approach could get very complicated with a large number of projects; it is best for highlighting 3-5 key projects within one program such as projects all funded by the same source or all focused on one societal goal.
Image
- A heat map that crosswalks two frameworks to show how many different ways your programs are contributing to large societal goals. In the example below, which shows a selection of SDGs being tracked by one institute, we can point to particular strengths in addressing SDG13 - Climate Action and SDG11 - Sustainable Cities and Communities. Projects within this organization have contributed in a number of different ways (all the impact descriptor categories) to each of those goals. On the other hand, the organization has not been contributing as much to SDG9 - Industry, Innovation, and Infrastructure. If this is a goal that is important to the program or to its institution, this analysis could help leaders course-correct to allocate more resources to working toward SDG2.
Indirect Impact Assessment
- In some cases novel data collection from researchers is not possible. New tools for scientometric analysis of the use of research can help to provide researchers and research programs with some insights into how their research is used by policy makers.
Overton is a subscription database of 13 million+ policy documents. You can identify policy documents from around the world that cite or refer to a specific publication or researcher by searching using a DOI or an individual’s name and institutional affiliation. It is also possible to search multiple names or publications at one time. Overton will find all the citations to a piece of research and generate a report or a set of Powerpoint slides to help you share the findings. At the University of Arizona, please contact the Office of Societal Impact about conducting a search using Overton. Some examples of presenting cumulative data from Overton follow.
Image
Image
Sage Policy Profiles is a free tool that individuals can use to track their policy impact, (https://policyprofiles.sagepub.com/). Researchers can set up an individual account will will allow them to create a summary of which research outputs have been cited in policy documents as well as providing links to those documents. A sample of a researcher’s profile page follows.
Image
- Finally, Altmetric (https://www.altmetric.com/) is a free tool that tracks the academic and policy citations of a particular article as well as its appearance in social media and traditional media. While Altmetric’s policy database is not as complete as the Overton database, it can provide you with real-time information about where a piece of research is being shared.
In order for faculty, researchers, and SI/BI professionals to do societally impactful work, they need support and recognition from their employers and supervisors. In academia, what is valued is often reflected in the criteria for promotion, tenure, and career advancement. Universities, research funders, and scientific professional societies are increasingly open to incorporating criteria related to engaged research and societal impact into their reward structures. However, large-scale institutional change is challenging and takes time. Here we share some resources for academic leaders who would like to take steps to broaden performance review criteria to include, recognize, and reward societal impact alongside existing measures of scholarly impact.
- A 2023 white paper from the Impact Funders Forum which “provides an overview of promising attempts to reform and/or strengthen promotion and tenure (P&T) systems to reward the societal impact of research.” The authors note a number of innovations already underway in several US universities including
- Campus- and system-wide (i.e., in public university systems) reforms to faculty advancement guidelines, criteria, and language.
- Formalized roles and review processes to build institutional capacity for implementing faculty advancement guidelines in university departments, schools, and colleges.
- Capacity-building for faculty in developing P&T cases.
Evaluation of candidates should recognize and value:
- the diversity of scholarship within and among these disciplines and the associated qualities that constitute an outstanding research record;
- interdisciplinary contributions both at established interfaces between distinct disciplines and at new interfaces between previously separate areas and modes of inquiry; and
- engaged scholarship, which is reflected in efforts and products that are often outside the traditional measures of research excellence within disciplines.
- The National Science Foundation has supported an effort to reform promotion and tenure guidelines to incorporate innovation and entrepreneurship. The Promotion and Tenure - Innovation and Entrepreneurship (PTIE) initiative is a global movement to support the inclusive recognition of innovation & entrepreneurship (I&E) impact by university faculty in promotion, tenure & advancement guidelines and practices.
- The Association of Public and Land-grant Universities published a set of recommendations for Modernizing Scholarship for the Public Good. Among the recommendations are that higher education institutions should:
- Explicitly value a variety of different forms of scholarship and impact
- Link promotion criteria to institution-wide mission and definitions
- Structure documents to elevate engagement & equity work
- Expand who counts as a “peer” during review
- Provide training and support for reviewers
While institutional changes are underway, research program and other academic leaders can encourage faculty and staff researchers to use the tools outlined above [link to Researcher section] to document and describe the impacts of their work to be prepared to discuss engagement and impact in annual reviews and promotion dossiers.