Official measures of research “influence” do not keep pace with scientists from social networks

A survey of how researchers use social media to encourage people to interact with their research suggests that much of the societal value of their work is likely to be ignored in official “impact” assessments.

A study published in Media and technology traininganalyzed more than 200 examples of how academics discuss and encourage their scholarships on social media. Based on the identified usage patterns, he suggests that the current approach to assessing the public “impact” of universities enshrined in the Research Framework of Excellence (REF) should be updated as scientists now have more social networks than they were when the model was developed. .

REF is the official system for measuring the quality of university research in the UK and informs about the distribution of research funding. The results of the last round of evaluation were published last week (May 12).

As part of the exercise, university departments are asked to demonstrate the impact of their work: to effectively show how it has enriched society. While the new study supports the requirement of case studies of impact, it calls into question how they are assessed. It argues that a gap is opened between how impact is measured in REF and the true scope and range of scientific engagement on social media platforms, some of which did not even exist when they were first developed.

In particular, REF focuses on the extent to which the final results of completed research projects are perceived by the public audience. On the contrary, research has shown that contemporary scholars are often involved in continuous feedback with organizations, community groups, policy makers and other members of the public throughout the life of the project. This provides an opportunity to collaborate and share experiences while research is still ongoing, often in a way that REF can hardly cover.

The study’s author, Dr Katie Jordan of Cambridge University School of Education, said: “Official language represents influence as a top-down stream from universities to the public waiting, but it’s an outdated feature- if it was ever true. Ask researchers about their most effective interactions on social media, and you’ll get a much wider range of examples than REF covers ”.

“You can argue that this means that too many researchers misunderstand what the impact is, but it is also potentially evidence that times have changed. There is much to be said for universities to demonstrate their value to the general public, but perhaps the time has come. to reconsider how we measure it. ”

REF measures impact on two main dimensions: significance (significant difference made by the project) and to achieve (the quantitative degree to which it does). The definition of impact beyond this is very open, differs across disciplines and is often considered ambiguous.

Research shows that REF also offers some confusing advice public participation, encouraging this in general but hindering it in the evaluation indicators. The official guide states: “Involving the public in research is not considered an impact. Influence is what happens when people interact with, accept, respond to or respond to research. Public interaction does not occur until the research is completed.”

Jordan’s survey asked scientists to cite examples of the strong influence they have achieved through social media. She received responses from 107 researchers from 15 different countries, but most participants, from graduate students to renowned professors, were in the UK. Her research analyzed 209 examples presented by them.

It is noteworthy that less than half are related to cases where research has been disseminated to the public as a product, according to REF. In such cases, researchers typically used social platforms to share their findings with a larger audience, stimulate discussion with colleagues, or create evidence of positive interaction with research.

About 56% of the respondents, however, spoke about the consequences of the exchanges, which were not just one-sided. Specifically, participants used social media to test research ideas, report intermediate findings, information and crowdsourcing data, or advertise study participants.

It seems that these discussions have caused not only influence in the official sense. As a result of the exchange, researchers were invited to give public lectures, participate in panel discussions, give evidence and advice to organizations, or conduct training sessions.

Importantly, these opportunities did not always focus on research that stimulated initial interaction. In many cases, researchers who published information about their project were then asked to share their broader experiences – often with human rights organizations or political actors who were interested in learning more about their research as a whole. For example, in one case, a post on social media led a senior civil servant from the Cabinet to visit a group of research colleagues to explore how their work as a whole could shape and shape policy.

Jordan claims that social networks erases the difference between exposure and social activism. As information becomes available to academic projects – from people, companies and organizations who contribute their ideas, questions and feedback through social platforms – they create both formal and informal opportunities for external exchange. It seems that this chain of interaction influences and benefits society in many ways that are not tracked by the REF.

However, evaluators find it difficult to monitor or measure these more nuanced impacts. “One solution may be to make changes to the assessment so that universities are asked not only to provide evidence of research results, but also to explain research process throughout the life of the project, “said Jordan.” This is not a call for even greater ambiguity in the fact that impact there is, but for greater openness in what researchers are achieving. In an increasingly complex culture associated with social networking, this would help ensure that the wider effects of their work are not forgotten. ”