Impact? What impact?

Impact plays an increasingly important role in research funding and evaluation, for example in considering what the research contributes to social challenges. Impact, then, is the litmus test of the research in terms of ‘added value’, as the last step in a chain of research activities. However, characterising the latter step is not always straightforward. And that is inconvenient, because what are we ultimately striving for with impact? What impact are we actually talking about?

At least four types of characterisations of impact can be found in the literature (see the Table below). These characterisations are based on either time dimension (short term, medium term, long term); or on scope/scale (project participants, network of participating parties, everyone – or at least everyone within the application domain); or on the object of change (individual, organisation, system), or on the type of ‘product’ being delivered (project deliverables, knowledge gained by participants, behavioural change among participants).

Because the characterisations are not mutually exclusive, a variety of combinations are also possible. Examples are impact as system changes (object characterisation) through behavioural changes (product characterisation) in a time span of 5-10 years (time characterisation). Although this may seem logical – after all, changes take time – this is not self-evident; the COVID-19 pandemic made it clear that behavioural changes, for instance, can also be ‘effected’ relatively quickly. In common parlance, we talk about the impact of the pandemic on society.

Table: different characterisations of the concept of impact

The impact model presented by NWO clearly shows that the different characterisations are sometimes, unconsciously, used interchangeably. In this model, impact initially represents changes that “only materialise many years after a research project has ended” (NWO, 2019, p. 2). This is a characterisation based on the time dimension. Elsewhere, it states: “By ultimate social impact envisaged, we mean changes in society” (p. 3). This is a characterisation based on the object (society/system). A link is also made in terms of connecting the research with many knowledge users for impact. This is a characterisation based on reach. It remains unclear whether listing these different types of impact is a deliberate choice or an oversight. One argument for the latter is that, ultimately, the premise is that research should report on outputs and outcomes because impact often occurs after the research programme (time characterisation).

Other differentiations also exist in the impact concept. Sivertsen & Meijer (2020) propose a distinction between ‘normal’ versus ‘extraordinary’ social impact. ‘Normal’ impact is defined as “the results of active, productive, and responsible interactions between (units of) research organizations and other organizations according to their purposes and aims in society. Within the research organisations, such interactions often occur informally at the individual researcher or research group level, but they may also follow formalized agreements or well-established traditions for collaboration” (p. 67). This encapsulates the concept of productive interactions.

Productive interactions involve contacts between researchers and other stakeholders in which knowledge is generated and used (see SIAMPI). Sivertsen & Meijer follow this line but aim for a broader characterisation of interactions, namely that they need not be exclusively productive but can also be ‘active’ or ‘responsible’. This is consistent with van Vliet’s (2021) criticism of the characterisation of ‘productive’ in the concept of productive interactions, where he proposes using the term ‘valuable’ interactions. Van Vliet goes even further than Sivertsen & Meijer by pointing out that such interactions are not only confined to researchers among themselves and in relation to stakeholders but that they also apply to stakeholders among themselves.

In contrast to this ‘normal’ impact, ‘extraordinary’ impact is positioned as follows: “more rare incidences where traditional and typical or new and untypical interactions between science and society have unexpected widespread positive or negative implications for society” (p. 67).

It is these compelling examples of (individual) researchers making unexpected breakthroughs that are often cited for how science contributes to society. Yet this overshadows the ‘nuts-and-bolts’ work that is done on a daily basis in research projects and has great impact on a smaller scale. An example cited in the study is the Dutch Heart Foundation’s approach. To foster interactions between research and potential users of research outputs, the Heart Foundation employs two strategies: 1) joint agenda-setting: for example, 11,000 citizens helped prioritise the research agenda, resulting in five main topics; 2) research evaluation: an end-user committee evaluates research proposals based on criteria such as relevance, participation, activities and interactions with users, in parallel with an academic advisory board. These interactions ensure shared responsibility for the intended impact.

The corollary of ‘normal’ impact is a focus on everyday interactions between research and practice, the conversation about what the shared interests, responsibilities and goals are and how to coordinate this properly to achieve change. Described like this, ‘normal’ impact closely resembles what is meant by continuous effects, and the method in the Heart Foundation example is already prevalent in a lot of applied research.

Another example is Katrien Termeer’s ‘small wins’ approach (2019, 2020). This approach was developed not so much from the discussion around impact but from the discussion around social transitions and how to make progress in this area. Small wins are small but far-reaching changes with tangible results for those directly involved, such as a policy tool, a technology, an earnings model or a chain collaboration. These are radical changes that seek to break down barriers and are also disruptive. It is not about the low-hanging fruit of easily obtained results within existing frameworks. Small wins are presented as powerful building blocks for transformation: “Because the focus is on the small, a small win stops people becoming overwhelmed by the complexity of an issue which inhibits their ability to think freely and precisely, which in turn leads to thinking in abstractions. Moreover, focusing on the small helps avoid procrastination. And since it is small, the parties need not wait for all the information and can start straight away to see what it yields. This provides incentives for people not to spend too long talking about the issue but to move on to action. Small steps are also faster to implement because they evoke less resistance and there is less competition in terms of claiming success. Small wins are therefore vital seeds for social transitions.” (2019, p. 5).

To boost these small wins further, there are various incentive mechanisms such as making the result visible so that people are energised and inspired by it (energising) and it also attracts more people and resources (logic of attraction), finding connections between the small win and other elements in the ‘system’ (connecting), and also connecting it with other smalls wins to create critical mass and internalise the benefits (robustness). These kinds of mechanisms can help in disseminating the small win (scaling up, rolling out), expanding the small win by applying it to another field or connecting it to other issues, and achieving a deepening through further radicalisation of the small win.

While small wins undeniably have impact, they are not so much characterised as being long-term and of an ‘extraordinary’ nature as being manageable and tangible small-scale interventions with demonstrable results yet, at the same time, are substantial and can have far-reaching effects, i.e. are ‘extraordinary’ in their ‘smallness’.

These other perspectives on impact can be linked to specific methods. Sivertsen & Meijer (2020) mention methods such as  SIAMPI, ASIRPA and Contribution Mapping, which focus on a variety of interactions in the research process as a key mechanism for impact. The small wins approach seems to fit in well with the Theory of Change method, which centres on a (logical) series of interventions that produce tangible results (outcomes) that need to be reflected on in the light of one’s own assumptions about the intended change and what the next steps are. These ‘interim’ outcomes can be seen as small wins which can be built on further.

Harry van Vliet

(This is a pre-publication of a section of text from the inaugural speech entitled De Voorbeschouwing. Over impact, valorisatie en doorwerking (‘Preview. On impact, valorisation and continuous effects’), delivered on 20 September 2022 in Amsterdam).

Sources

NWO. (2019). Maatschappelijke impact via kennisbenutting. Den Haag.

Sivertsen, G., & Meijer, I. (2020). Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society? Research Evaluation, 29(1), 66-70.

Termeer, K. (2019). Het bewerkstelligen van een transitie naar kringlooplandbouw. Wageningen University & Research.

Van Vliet, H.  (2021). Kritische reflecties: NWO-model van impact. Utrecht: Regieorgaan SIA.