How do you demonstrate that your research has made a difference? There are many different methods you can use to gather evidence in order to clarify and demonstrate continuous effects. Pedersen, Grønvad & Hvidtfeldt (2020) describe the most commonly used methods. Based on a literature review of impact research, the authors come to the conclusions and insights below. The description begins with the most popular method of impact evaluation.
Interviews
Impact interviews are cited as one of the most useful sources of information and are the most commonly used method (over 40 per cent) to demonstrate research impact. Interviews are used to gain first-hand insight into collaboration within the research project or programme and how knowledge was developed, shared and applied. Often, non-researchers and end users are also interviewed and asked to reflect on what continuous effects have been achieved and how.
Case studies
Another popular method for making impact visible involves case studies. These can be used to describe the, sometimes complex, coherence in a specific case of activities and collaboration to demonstrate what contributed to the research’s continuous effects in practice. It allows researchers and other stakeholders to produce a coherent narrative that describes who or what has benefited or been affected by the research. An advantage of this method is that a wide range of impact pathways (e.g. policy or culture) can be described and substantiated, which would be less evident in a more quantitative approach.
Surveys
Questionnaires are used in 40 per cent of the evaluations studied. Surveys are mainly used to collect data on issues such as motivation, perceived barriers and relational experiences. Questionnaires allow comparative analysis over time and among different target groups during the research process.
Peer review
Peer review is the umbrella term for an expertise-based assessment practice; experts or peers assess the research or parts of it, e.g. for academic excellence or social impact. This method of providing insight into continuous effects can be used during each research phase – from the allocation of research funding to interim and final evaluation.
Statistics
More than a third of the impact studies surveyed employ statistics to describe research infrastructures, such as income, researchers’ education levels and academic awards. Statistics allow for tracking developments over time and between research units.
Data on commercialisation
These data are often used to substantiate the economic effects of research. The standard indicators used are: patents, licences, joint research and development, partner funding, spin-offs and spin-outs. Commercialisation data can also reflect formal and contractual relationships between researchers and stakeholders.
Bibliometrics
Bibliometrics is a statistical method to assess and rank individual academic achievements and substantiate the impact of research by the number of publications, citations and co-authorships. Bibliometric data can also provide information on research outputs, such as datasets, software, funding and networks of researchers. These analyses contribute to the objectivity and transparency of the research evaluation process and provide an overview of publication patterns and academic networks, which would otherwise not readily come to light from an individual researcher’s perspective.
Impact plans and Theory of Change
Impact plans can be developed retroactively to describe the context, activities and results of a research project and its social impact. It is also possible to identify in advance which activities and interventions can contribute to the intended goal. A method often used for this is a theory of change, which acts as a roadmap to guide those involved in the project and demonstrate the continuous effects of research in and on society. During the research process, the following is regularly assessed; ‘is what needs to be researched actually being researched?’ For example, assumptions and indicators are filtered and – where necessary – reformulated.
Workshops and focus groups
These methods are an organised form of facilitating discussions between researchers and stakeholders and are used at different stages of an assessment and research process. Workshops can be given early on in the process to bring researchers and stakeholders together so that they can produce ideas and establish shared objectives. They are also employed retroactively to clarify research findings and how they are valued and used, and to demonstrate continuous effects.
Evaluation of stakeholders/end users
In this method, stakeholders and/or end users play a role in developing, conducting and/or assessing the research. The aim is greater participation and involvement in the research process and more substantial embedding of research findings in professional practice. Involving stakeholders throughout the research process can lead to co-production of knowledge and evaluative indicators.
Impact repositories
Databases and repositories are part of the Open Science agenda of European governments. Databases make it possible to find individuals (colleagues within research universities and universities of applied sciences and partnerships outside the research world). They are also used to share research results and data to promote wider dissemination across projects and fields. Repositories facilitate studies on impact data and help in understanding continuous effects in different fields.
Altmetrics
With altmetrics, communication about and sharing of (research) results via social media and digital platforms can be mapped. These include messages and posts on, for example, Twitter, Facebook, blogs, Mendeley, CiteULike or Impact Story. Policy reports, white papers or manuals can also be found through altmetrics. Various forms of data can be collected from these sources, ranging from quotes, views, downloads, clicks, tweets, shares, likes, bookmarks and comments that go further than traditional bibliometric data. Altmetrics offers the ability to collect big data, which enables research results to be shared and effects on academic conversations to be quantified in the digital domain.
Impact tracking
This method is closely related to ‘impact planning’ but sets fewer conditions for planning. Impact tracking involves tracing ‘impact pathways’: thinking ahead to activities and results from a research objective or thinking back from a particular impact to identify which results and activities led to it.
Review and analysis of documents
In their literature review, Pedersen, Grønvad & Hvidtfeldt found that document analysis was used in 10 per cent of cases. These analyses relate to documents (policy plans, project plans, vision documents, evaluation reports, etc.) in order to establish the research context and thus better interpret the results of the research. This can also be fully or partly automated by means of text mining.
Field/location visits
Visiting locations (field labs, studios, workshops, labs, research spaces) can add an extra dimension to evaluations through direct observation of working conditions, operations and ways of working and communicating – especially when experiences, wishes and strategies can be discussed with the people at the location.
Esther Tielen
