Participatory Research Evaluation Framework

What is the Participatory Research Evaluation Tool?

The Participatory Research Evaluation Framework is a methodological framework designed to capture and evaluate the societal impacts of participatory sustainability research. Participatory research involves non-academic stakeholders (e.g. governments, businesses and citizens) in the research process with the aim of developing solutions to complex societal problems.

An important element of the evaluation framework is that, in addition to concrete research results, it also focuses on societal changes. Examples of such changes are improved decision-making, strengthened networks, and sustainable transformations. The framework therefore focuses on measuring effects on both short-term results (e.g. improved knowledge) and long-term results (e.g. behavioural changes). It distinguishes four main categories of securities:

1. Useful products (e.g. technologies or action plans)
2. Strengthened capacity (including knowledge sharing and skills development)
3. Network effects (e.g. building trust and partnerships)
4. Structural changes (including policy changes and behavioural changes)

The framework links the effects to specific characteristics of the participatory research process in order to find a possible explanation for the effects of the project. These characteristics have an influence on the aforementioned effects. These features include:

• The representativeness of participants (make sure that all relevant groups are involved);
• The degree of interaction and collaboration between researchers and stakeholders;
• Transparency in how the input of stakeholders is incorporated into the research.

This framework is especially suitable when you are conducting research aimed at developing solutions to complex societal problems in collaboration with non-academic stakeholders. This includes research within sustainability themes such as climate change, urban planning or public health, in which policymakers, companies, communities and civil society organisations are closely involved. Depending on the objectives of the research and the parties involved, the framework can be used for several reasons:

• Evaluation of societal impact: It can be used by researchers and funding organisations to determine what societal impacts a particular participatory research project has had.
• Improving the research process: By linking the participatory characteristics of the research to the measured effects, researchers can gain insight into what works and what doesn’t. This can help to better design and optimise future participatory research projects.
• Assessment of programs or policies: Governments and policymakers can use the framework to assess the effectiveness of participatory research and determine whether it has contributed to policy change or the resolution of societal problems.
• Stakeholder reporting: Researchers can use the framework to communicate their results to a wider audience, including the non-academic stakeholders involved in the research.

How do you use the Participatory Research Evaluation Tool?

The figure shows the steps for using the Participatory Research Evaluation Framework. The use of the framework takes place through four steps in which different research methods are used.

The following steps are distinguished:

1. Identification of projects (if applicable): The first step is to select relevant research projects aimed at developing solutions to complex sustainability challenges. It can be one specific project, but also a program consisting of several projects. It is important that these projects are carried out in cooperation with stakeholders. After this identification, documents, such as reports and minutes, are collected to gain a good understanding of the participatory activities within the project.
2. Document analysis: In this step, the participatory elements of the project are analysed, including events (e.g. workshops, focus groups). Specially for events, both the process (e.g. number of events and motivation of participants to participate) and the quality of the events (representativeness of the relevant target groups) should be analysed. This information is visualised in a storyboard, which provides a chronological representation of the participatory activities within the project. This storyboard helps identify potential societal impacts that can be attributed to the project.
3. Survey and analysis: In order to make a detailed evaluation of the impacts, a survey is conducted among the stakeholders involved. This survey includes questions about their involvement in the project and its perceived effects, such as improved knowledge or networking. Statistical methods, such as correlation and regression analysis, can be used to examine how specific features of the participatory process are related to the measured societal impacts. This could for example makes it possible to determine whether increased collaboration between stakeholders has led to stronger network effects or improved decision-making.
4. Interviews and in-depth analysis: In addition to the survey, interviews will be conducted with project participants to gather additional information about the observed effects. These interviews often provide more in-depth insights into how the participatory activities have contributed to specific societal outcomes. This qualitative data is then analysed to further validate the attribution of impacts to specific participatory elements of the project.

One of the challenges of using the framework is the time lag between the end of the project and the visible societal impacts, such as policy changes or structural transformations. It is therefore essential to use the framework both during and after the project to measure both short-term and long-term effects. Combining different methodological approaches (e.g. collecting real-time data during the project and conducting post-project evaluations) can help to get a more complete picture of the societal impact.

Example: Project Planlt Calgary

One case study mentioned is the “PlanIt Calgary” project, which used participatory methods to integrate sustainability principles into urban development. This project involved stakeholders such as municipal authorities, citizens and experts in interactive workshops and survey processes. A specific outcome was the development of visualised scenarios for urban patterns, which formed the basis for further policymaking. The project demonstrated how participatory events could lead to improved decision-making capacity and broader societal engagement with sustainability goals.

What is the origin of the Participatory Research Evaluation Tool?

The framework is based on an extensive literature review and a workshop with stakeholders, including governments and companies. The aim was to create a systematic and practically applicable scheme that can capture the societal effects of participatory sustainability research.

Participatory Research Evaluation Framework has its origins in a number of scientific approaches and methodological traditions that focus on involving non-academic stakeholders in the production of knowledge and solutions. It arose from criticism of traditional academic research practices that often have little direct societal impact. The development of the framework took place within the broader context of new research approaches, such as transdisciplinary research. Within these types of research, scientists work closely with non-academic stakeholders to develop solutions to complex societal challenges. The framework builds on previous efforts to capture the broader societal impact of research, but provides a more standardised and applicable approach that is suitable for different types of participatory sustainability research.