Contextual Response Analysis

What is Contextual Response Analysis?

Contextual Response Analysis (CRA) provides insight into where, how and by whom results of research are used. More precisely, it maps which societal actors, in what ways refer to research results. CRA was developed by Ad Prins, who applied it in social impact evaluations of research institutes and policy assessment agencies (planbureaus).

A CRA is ideally suited to gain insight into research impact. The premise is that it is possible to find traces of use of research and that those traces, or responses, matter. Because such a response means that an actor has made an effort to refer to the results of the research.

A CRA offers insight into the amount of response. But even more interestingly, it also offers insight into the type of response and use: positive or negative, just a general reference, also a reflection, or perhaps an emphasis on a particular aspect. Moreover, a CRA also shows who or what does it.

CRA is partly in line with traditional citation analysis, as both methods attach importance to a citation, or response, by a user. That is an important indication of further use, because there is that actor making an effort. But CRA is also a reaction to those traditional citation analyses and really makes different choices. A CRA can start with very different outcomes of research such as a book or report, a tool or term, while traditional citation analyses start mainly with scientific publications. In addition, a CRA focuses precisely on societal users and societal sources, such as search engines and newspaper databases, rather than scientific users and scientific journals and conferences. Finally, a CRA is also about the type of response, the way the research is referred to. And that provides insight into societal appreciation of research.

How to use Contextual Response Analysis?

A CRA starts by deciding what results of research the analysis is about. This can be all reports from a given period, as in the case of the Education Council’s CRA. The analysis was in the context of an evaluation, and so the analysis covered all reports from the period under evaluation. It could also be a book written (in part) for a social audience, such as ‘De depressie epidemie’ (The depression epidemic) by Trudy Dehue or ‘Winnaars en verliezers. Een Nuchtere Balans Van Vijfhonderd Jaar Immigratie’ (Winners and Losers. A Sober Review of Five Hundred Years of Immigration) by Leo & Jan Lucassen. Or it is a long-term panel or monitor, such as at the CRA of the Amsterdam Institute for Social Science Research, which contributes to the ‘Continuous Survey of Citizen Perspectives’, or at the CRA of the Netherlands Environmental Assessment Agency (Planbureau voor de Leefomgeving), which conducts the ‘Infrastructure and Space Monitor’.

That search is done through systematic queries in various sources. For example, in general internet search engines such as Google and Bing. The search engine selects, for example, websites of interest groups responding to a report, or the website of the publisher of a book, or for contributions on social media. Systematically checking the results is necessary, for instance to rule out false positives. The results can then be classified in different ways. For instance, according to site function, such as a knowledge platform, a repository or library, or a weblog. In addition, it is also possible to classify the sector the reference represents, such as education, government, politics, media, or the cultural or social sector. This kind of further classification provides insight into the type of use.

More specific sources can also be researched for responses. Consider, for example, committee reports of meetings in the House of Representatives or a municipal council. The analysis can then delve deeper into who refers to the results of research in what ways. Which party refers to the research how often, in which debates do they do so, and what do they say? Do they cite the research as valuable, refer to a specific aspect that fits into their political vision, or report that they disagree with the insights?

Another specific source is from daily newspapers. An analysis of responses in the daily newspapers published in the Netherlands can offer insights into a variety of aspects. For instance, to the type of newspaper that refers, such as a national daily or a regional one. Or to the type of reference. Is it an opinion piece by the researchers themselves, which the newspaper does not otherwise have to endorse, or an interview that editors have conducted with the researchers, or is the research cited as a solid source of information (‘according to the report of…’).

In all such analyses, the timing of the response can be interesting. Is the response at the time of publication of the results, when the researchers hope for or expect a response? Or is it much later, for instance when there are developments to which the research is again relevant. As happened, for example, when a Member of Parliament said, ‘Just last year, the Education Council warned….’

It may sometimes be practically necessary to limit the number of research findings. For example, the Education Council CRA used all 24 reports from the period of the review for the parliamentary response. For the other response analyses, 6 reports representative of the period were used.

Finally, it may be useful to use additional search terms to find more specific responses. One of the reports to the Education Council’s CRA deals with the increasing influence of tutoring institutes. For the analysis of newspaper responses, the terms ‘bijlesindustrie’ (tutoring industry) and ‘schaduwonderwijs’ (shadow education) were also used, terms that appear specifically in the report.

What is the origin of  Contextual Response Analysis?

Ad Prins has been developing Contextual Response Analysis over an extended period of time. Starting point is the SIAMPI project, also known for introducing the concept of productive interactions. In the years following that initial start, Ad Prins conducts analyses for a range of research institutes and departments interested in the social response, the social use, of the results of their research. This provided an opportunity to develop the method further.

The examples used come from Contextual Response Analyses conducted by Ad Prins over the past decade.