My dissertation seeks to answer the following questions: What are the requirements for evaluating the research impact created by Dutch Universities of Applied Sciences (UASs) research? How can these requirements be applied within the context of the policy goals of Dutch UASs? The goal of this dissertation is to determine recommendations for evaluating the impact made by UAS research, and how we can implement them within the current policy and organisational landscape, uncovering the impact created by UASs and making it visible.
The initial phase of this research aimed to discover the governing principles and criteria essential in assessing the impact of UAS research. An extensive literature analysis was undertaken, however, very little is written about evaluating the research impact of UAS research itself. The results were drawn from diverse sources including transdisciplinary and multidisciplinary evaluations, general impact evaluation literature, and practical applications within UAS.
The analysis provided several recommendations while recognizing the philosophical assumptions that influence these evaluations (Greenhalgh et al 2016). The two recommended philosophical assumptions for evaluating UAS research are realist, which seeks to unveil the context-mechanism-output-impact configuration; or performative, grounded in actor-network theory, advocating for evaluating research impact by scrutinizing the broader consequences of interactions between research and society (Raftery et al 2016, Greenhalgh et al, 2016).
Further, these recommendations underscore the importance of conducting real-time, formative evaluations that foster learning and improvement. Advocating for a flexible evaluation structure that resonates with the dynamic nature of practice-oriented research, these recommendations caution against the constraints of a prefabricated linear logic model linking objectives, input, output, and impacts. The recommendations also emphasize the importance of a co-production approach, involving stakeholders from the outset. Currently, there are no established frameworks that conform to all these recommendations, while ASIRPA (Joly et al 2015), PIPA (van Drooge and Spaapen 2017), and Contribution Mapping (Kok and Schuit 2012) each meet several of the recommendations and could provide a starting point for evaluating the impact of UAS research.
The second part of this investigation was to understand what is currently happening in the field of UAS research practice and compare it to the newly understood recommendations. To do so, our focus first shifted to include examining the roles and functions of research groups within the Knowledge Triangle (KT).
The KT of Education, Research, and Innovation was developed to capture the dynamic interactions among Higher Educational Institutions, the business sector, and society. This framework aims to go beyond a one-way flow of knowledge by promoting continuous and systematic connections across these spheres (Etzkowitz and Leydesdorff 2000). These interactions are essential for maximizing the impact of investments and addressing societal challenges (Sjoer et al. 2012).
Based on the work of Kyvik (2012), we selected primary roles in which (UAS) academics participate: teaching; research; internal organization; and external networking. Demographics such as age, gender, and education level were considered as they could potentially influence how the researcher groups functioned. Principle Component, and Regression Analysis was conducted on questionnaire results.
The analysis revealed two primary dimensions of Connectivity and Content. Respondents were asked to indicate what function they contractually fulfil within the research group; Professor, Associate Professor, and Researcher. They were asked to indicate which function they felt their tasks fulfilled; Professor, Associate Professor, Researcher, and Other. They were also asked to indicate the number of contractual hours, as well as the number of hours they felt they work.
The clear delineations between roles and functions within the research group, and consequent contribution of research to education and practice, can result in the knowledge transfer within the KT being vulnerable. It is important for research teams, and those who support them, to be aware of the various roles and functions in the team as well as each of the projects in which that team participates. Each role and function contribute to the impact created as well as the knowledge transfer that occurs. Ensuring the right person with the right role and function is involved in the right project is necessary to maximize impact.
To prevent the KT from becoming fragile, it may be advisable for institutional policy makers and the NAUAS to consider strengthening policy around the KT. This ensures that responsibilities for the KT are not dependent on single members of the research group. A formal description of how research is coupled with Practice and Education, and the processes it goes through, should also be part of the discourse.
A deeper exploration into the impacts of Universities of Applied Sciences was then carried out. At the beginning, ten themes had been discussed in the Strategic Agenda of 2016-2021: Health Care and Vitality; Education and Talent Development; Resilient Society: In Community, City and Region; Smart Technology and Materials; The Built Environment: Sustainable and Liveable; Sustainable Transport and Intelligent Logistics; Sustainable Agriculture, Water and Food Supply; Energy and Energy Supply; Art and Creative Industries; and Business: Responsible and Innovative. (NAUAS 2015). Both a questionnaire and workshops were utilised to facilitate a comprehensive understanding of the impact’s researchers wish to create; the outputs realised to facilitate this impact; and the themes in which they strive to do it. Our study reveals that researchers find it challenging to align their work with a single theme. They often engage in multidisciplinary research with a wide variety of stakeholders which requires flexibility and multiple themes. Project-level evaluations may better capture theme-specific impacts. This is especially true as stakeholders play a crucial role in impact creation, and the choice of theme often depends on project-specific stakeholders.
This part of the study revealed a disconnect between the intended impacts and actual outputs. This was particularly true for educational impact and outputs, and economic impacts and outputs. Echoing the results of our exploration of the research group, this emphasizes the importance of personal engagement in translating research findings into tangible impacts.
The concept of ‘impact’ for Dutch UASs has evolved over time. Currently, the term ‘doorwerking,’ translated as ‘effect’ in English, is widely used in UAS policy discussions. The current definition used by UASs describes ‘effect’ as the influence of the research process, and its results in education, professional practice, and the research domain, encompassing implicit and explicit changes during research and dissemination (NAUAS 2022). According to Brouns et al (2023), this is fostered by ongoing interactions among individuals and their output. Andriessen (2019) suggests that these interactions and outputs facilitate knowledge transfer across the spheres of the KT.
Doorwerking implies an ecosystem where minute developments lead to a succession of adaptive changes, akin to Sivertsen and Meijer’s (2020) concept of ‘Normal impact’—everyday interactions creating scientifically robust and socially relevant impact. Sivertsen and Meijer differentiate ‘Normal’ impact, which stem from daily interactions, while ‘Extraordinary’ impact, though rare, has a widespread effect on society.
Lykke et al (2023) further build on this differentiation by suggesting that Normal impact is made up of micro impacts that occur throughout the research process, contributing to expected and unexpected, planned, and unplanned effects. Budtz Pederson and Hvidtfeldt (2023) suggest that these micro impacts may lead to macro level impacts and require different tools for evaluating the distinct types of impacts. They suggest that the evaluation of macro impacts is better suited to indicators and a narrative while micro impacts are best made visible using a contribution analysis.
The final stage of this study was a case study aimed at applying and evaluating the effectiveness of contribution analysis as a micro impact, or doorwerking, evaluation tool in the context of UASs. Structured around Kok and Schuit’s (2012) Contribution Mapping framework, we evaluated a transdisciplinary project, GoNoord Nederland, based in the Netherlands. While many forms of contribution analysis are based on a Realistic perspective, Contribution Mapping was selected because of its Performative assumption. It aligns with other recommendations for evaluating UAS research impact in that it is formative and stresses the importance of stakeholders throughout the evaluation process in co-production.
In line with co-production, semi-structured interviews with each member of the research team including stakeholders were conducted. A focus group with the research team was also held following the completion of the project, making use of the Contribution Mapping framework in its original ex-post form.
Contribution Mapping itself presented challenges, such as; time intensity, the necessity of insider involvement, and results that often-lacked depth and contextual richness. These findings led to recommendations for enriching contextual details in the mapping, and adopting an iterative, non-linear evaluation approach conducted in real time, suggesting that Contribution Mapping’s effectiveness in the UAS context depends on addressing its limitations and refining the approach.
Conclusion
While there is certainly more to uncover, I hope this work will assist both myself and others in making informed complex decisions about what we mean in the words we use, the policies we write, and the tools we offer. These are impactful decisions about who we involve in our research, the output we generate, and the research process itself. These are weighty decisions about our expectations for researchers, institutions, and the future of Practice Oriented research. By understanding and implementing these rules, my aspiration is not only to contribute to impacting society through the research UASs do but also to make this impact visible.
Sarah Coombs
Download: Visible – discovering the impact of research conducted by universities of applied sciences.
