The CoARA Assessment Idea Catalogue
The CoARA Assessment Idea Catalogue is designed to be a resource for those developing and/or implementing assessment processes at a variety of different levels, including hiring, promotion, project funding, research programme funding, institutional evaluations or national research system evaluations. A collection of different experimental ideas are represented here, that can be used as inspiration for designing experiments that innovate on current assessment practices.
As the main output of the CoARA Working Group “Experiments in Assessment” (EiA), we aim to foster experimental ideas that advance research assessment towards a more holistic and inclusive definition of quality and excellence. This idea catalogue should help enable those responsible for assessment to experiment and innovate, driving bold and ambitious change to improve the quality of research across institutions and countries.
What is an experiment? Why experiment on assessment?
The goal of experiments is to test open questions, new ideas, and observations in a structured manner using the scientific method. While the process of experimentation has been the cornerstone of research for a long time, driving forward much of scientific knowledge, experiments to the processes of research are much more recent.
We are aware that both good practice exchanges as well as a growing body of literature in the area of metascience exist, and these are immensely valuable for everyone in the research ecosystem. However, there are very few centralized resources for ideas that haven’t been tested as well as evidence databases. We know there is exchange of good practice exchange happening, but this idea catalogue aims to be a resource that can help centralize both this exchange and the evidence arising from experimentation.
Research assessment experiments have many different goals, mostly based on tackling the different issues and overarching questions present in current assessment processes as well as reform initiatives (such as CoARA). The EiA Working Group identified 6 important questions that align with CoARA principles to help guide the development of the idea catalogue:
- How do we create an assessment process based upon what the different players in research value?
- How can we make research more inclusive and equitable through assessment?
- How can we mitigate the negative effects of bias on assessment?
- How can we better recognize process and culture through assessment?
- How can we better foster diversity through assessment?
- How can we better foster collaboration through assessment?
- How can we ask assessment questions differently?
The ideas in this catalogue are seeds to develop questions in order to try and address these questions. This catalogue consists of experiments that are currently hypothetical as well as experiments that are currently implemented at different scales. Through experimenting on research assessment in different types of institutions, gathering evidence, and sharing them openly, we can improve processes determining how research and researchers get funded. By applying the principles of the scientific method to our own ecosystem, we will improve it for current and future generations.
For further reading on experiments in research assessment, we recommend:
The “Experiments in Assessment” Working Group
This idea catalogue is the main output of the CoARA Working Group “Experiments in Assessment”, which was formed with the mission: To form an incubator for experimental ideas that advance research assessment towards a more holistic and inclusive definition of quality and excellence. The Working Group members were a diverse group, covering research institutions, researchers, and research funding organizations from across Europe.
Principles of the WG
Before starting any other activities, the Working Group developed a set of shared principles and values for how we plan on working together. The goal was to foster an open, collaborative, and creative space, in line with the mission and objectives of the Working Group.
These principles are shared here, not as a prescriptive “this is the right way forward”, but rather to highlight our process and how we implemented the Working Group’s activities. Everything written in this document informed our work, and we are proud of both what we have done as well as how we have done it.
Development of the platform - Where did these ideas come from?
Since 2024, the “Experiments in Assessment” Working Group has been collecting ideas and fostering discussion on potential experiments that could be part of the idea catalog. We had three main activities that have contributed to the content of this platform.
Open Survey
From March 2024 until January 2025, an online one-question survey was open for responses from anyone who had ideas that they wanted to contribute. Over 90 responses were submitted, with a good number of interesting and innovative ideas for potential experiments.
Workshops and (grey) literature review
In the second half of 2024, we organized two open workshops to discuss potential experiments that could be included in the idea catalogue. These workshops had participants from diverse roles and institutions, and the discussions led to both potential experiments to include as well as important themes that the Working Group could focus on in later brainstorming for potential experimental ideas.
Discussion and development in the Working Group
Over the course of the Working Group’s activities, the members utilized their diverse expertise and experience to develop multiple experimental ideas and populate the idea catalogue with ideas arising through our work. This was an organic process, working through structured discussions on both established experiments as well as novel ones, in an open and collaborative atmosphere to foster creativity in line with our principles.
Disclaimer
This idea catalogue platform is not a comprehensive list of experiments for innovating research assessment. There are many potential experimental ideas that we recognize as fitting into the platform, and we welcome any constructive support in adding additional experiments.
Due to the breadth of experimental ideas, the different experiments in the catalogue may be conflicting with one another. The goal of this platform is to value the breadth and diversity of experiments, and there is no inherent judgement of the quality of experiments, especially if they conflict with others.
The goal of this platform is to provide inspiration and resources for you to develop assessment experiments in your own context, but we have no guarantee on the outcomes of any experiments that you implement based on the ideas found here. We are not legally liable or responsible for your experiments, but we are happy for your feedback on your experiences.