The state-of-the-art in CoARA and the implications on UAS

Publish or perish is a saying closely associated with the scientific community in Europe and beyond. The epitome of scientific excellence is often the written word, published in prestigious journals at high costs for either the author or the reader. Scientists who do not publish frequently may see their reputation tarnished, while those who are more present tend to be considered excellent. This is nourished by quantitative indicators such as the h-index, citation figures or the Journal Impact Factor. In the worst scenario, the individual numerical measurement of scientific excellence outweighs the quality of the research or the researcher.

The scientific community is aware of this problem and is developing a bottom-up approach to reform how research, researchers and research organisations are being assessed. Since the summer of 2022, research performing and research funding organisations have been invited to sign the agreement on reforming research assessment, which is centred on four core commitments: recognise the diversity of research contributions, assess research primarily on qualitative means and abandon inappropriate uses of quantitative metrics and rankings (CoARA, 2023b).

These commitments, forming the cornerstone of the reform, pave the way for a collective transformation in assessment practices, guided by the overarching goal of amplifying research quality and its societal impact.

UAS are Pivotal Players in the Coalition for Advancing Research Assessment (CoARA)

Upon signing the agreement, research entities gain the opportunity to join the Coalition for Advancing Research Assessment (CoARA), the implementing forum of these reforms, structured working groups and national chapters led by coalition members. With an increased focus on qualitative research assessment criteria, this development could more strongly reflect the nature of universities of applied sciences (UAS), ranging from the diverse professional backgrounds of UAS lecturers to practice-based research and research-based education in close concertation with local stakeholders.

In October 2023, the coalition is comprised of 559 CoARA members (CoARA, 2023a). The members are mostly universities and their associations, followed by research infrastructures, learned societies and research funders. Following the constitutive assembly in December 2022 and the first General Assembly in June 2023, CoARA working groups will be established by year-end 2023 in two calls for proposals. Figure 1 below illustrates the composition of CoARA members, as presented in the General Assembly in June 2023.

It is crucial to recognize that the success of this reform initiative hinges on broad participation from various institutions. Without a critical mass, the transformation away from a quantitative-dominated research assessment framework will falter.

Universities of Applied Sciences have unique selling points from which this reform could and should benefit. They conduct practice-based, transdisciplinary research in close collaboration with actors in their regional innovation ecosystem. On the other hand, if this reform goes forward without a clear commitment and participation of UAS, the process will develop to the detriment of applied research institutions.

How to take part in CoARA?

CoARA membership is subject to signing the agreement on reforming research assessment. Institutions around the world are eligible to sign the agreement and join CoARA. While an elected Steering Board is responsible for the general oversight and strategy, the General Assembly of all coalition members is the highest decision-making body. Through different working groups, coalition members are invited to contribute to the reform by inducing their expertise and by working on reforming their assessment practices.

The coalition’s work is based on a trust-based approach, which acknowledges the individual responsibility of each member. Upon signing the agreement, signatories commit to develop an action plan within one year, which outlines “how their organisation has started the process of reviewing or developing criteria, tools and processes […] with defined milestones”. A first touch point will be reached within five years of signing the agreement, by which time signatories “will have worked through at least one cycle of review and development of their assessment criteria, tools and processes”.

In the first call for proposals, ten working groups were selected to begin their work as part of CoARA, which are presented in Table 1. Among the selected ones is a great variety of relevant topics for the reform, including one with a special focus on applied research: Towards Transformations: Transdisciplinarity, Applied/Practice-Based Research, and Impacts. CoARA members can express their interest in joining the selected working groups through the contact details accessible on the CoARA website (CoARA, 2023c).

Table 1: The CoARA working groups of the first call for proposals

Supporting the alignment of research assessment systems with CoARA in biomedical disciplines through administrative reforms and governance

Towards Open Infrastructures for Responsible Research Assessment

Improving practices in the assessment of research proposals

Reforming Academic Career Assessment (ACA)

Early-and-mid-Career Researchers (EMCRs) – Assessment and Research Culture

Experiments in Assessment – Idea generation, co-creation, and piloting

Recognizing and Rewarding Peer Review

Multilingualism and language biases in research assessment

Responsible metrics and indicators

Towards Transformations: Transdisciplinarity, Applied/Practice-Based Research, and Impacts

The key challenges in shifting from quantitative to qualitative research assessment

It is undoubtedly important to move towards more responsible use of quantitative indicators in science and appreciate quality over quantity. While reforming research assessment seems like the right thing to do, some challenges need to be addressed, including through CoARA. With qualitative excellence moving centre stage, high hopes are being placed on peer review. However, this is time-intensive work. So how can we scale peer review? And even if we find a suitable way to do so, it is noteworthy that this is only one approach to qualitative research excellence.

Particularly societal impact is difficult to measure, as the effects of research on societies can often only be seen after several years. In addition, there are a multitude of factors that influence societal impact that go beyond the work of a researcher or a research group.

Furthermore, what exactly is research excellence to start with? Is there a commonly agreed set of principles within and across different types of research institutions and disciplines? It is important for each university to think thoroughly about how to define research excellence, and then link this to the academic career assessment. Finally, something higher education institutions must not forget when discussing research assessment practices is the inclusion of research funders in this process.

It is not a good idea for any higher education institution to wait for CoARA to provide answers to the questions above. Rather, CoARA should be seen as a vehicle for change that needs the active involvement of all actors to succeed.

In sum, there are numerous challenges and questions that need to be addressed when reforming research assessment practices. However, if the higher education landscape gets together to share experiences, best practices and ideas, this challenge can turn into a historic opportunity for research, researchers and research institutions on a global scale.

Conclusions – the benefits of UAS to join CoARA

The starting point for reforming research assessment differs between universities and UAS, primarily stemming from their distinct research assessment criteria. This divergence is typically driven by varying funding schemes, with research holding greater significance in university funding compared to that of UAS in Finland (see Ministry of Education and Culture, 2021a, 2021b for details). The quantified metrics (annual external RDI funding and publications) currently dominate the university and UAS funding in Finland, whereas quality is only assessed for universities in terms of journal ratings.

While the existing funding framework understandably influences the research assessment criteria for both universities and UAS, it is imperative to broaden our perspective when devising CoARA action plans. We need to think beyond the current funding schemes and focus on how the research impact of higher education institutions (HEIs) can be strengthened. When the qualitative aspects get more weight, this will likely lead to a growing role of HEIs in society. UAS must thus embark on the challenging task of measuring the effects of research on societies.

As UAS are still comparatively young institutions within the higher education landscape, the process of reforming research assessment is facilitated by a shorter institutional history and associated path dependencies. For example, while the tenure track systems are widely common at universities, only a few UAS in Finland have developed research career tracks. UAS who are still at the beginning of this journey therefore have a great opportunity to incorporate the CoARA principles into the development of researcher careers.

To conclude: CoARA is a unique opportunity to enhance and ensure the value of applied research to society and calls for collaboration of UAS to lead the change for assessing research quality over quantity. If done properly and with ample participation, the publish or perish conundrum can indeed become a thing of the past.

LISÄÄ AIHEEN YMPÄRILTÄ / RELATED POSTS