How to use the Idea Catalogue

The idea catalogue platform has been built to make it as simple as possible to find ideas for experiments in research assessment that can be tested and/or implemented in your setting and context. We have created categories that allow for filtering around different target groups, challenges in assessment reform, or CoARA commitment, showing all of the potential experiments that could fit in each category.

The platform also contains experimental ideas at different levels of detail - from experiments that just have a concept and an abstract all the way to those that have been tested multiple times across different institutions and countries. This should encourage users of the idea catalogue to add more detail based on their knowledge and experience, including detailing the implementation of their experiments to share with others.

How to find an idea:

The search bar can be used to look for keywords. This will search through both the titles and the full text of each experimental idea, showing each time the search term comes up on each idea page.

Show picture of main window with search bar

The tags on the right side can be used for filtering ideas by different categories. If a tag is clicked, the main page will only show the ideas that are related to that category. Only one tag can be picked at a time. To show all of the ideas again, either click on the “all” tag at the top of the list or refresh the page.

You can find ideas using the following categories:

CoARA Commitments aligned with the idea

CoARA Commitment 1 = Recognise the diversity of contributions to, and careers in, research in accordance with the needs and nature of the research
CoARA Commitment 2 = Base research assessment primarily on qualitative evaluation for which peer review is central, supported by responsible use of quantitative indicators
CoARA Commitment 3 = Abandon inappropriate uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index
CoARA Commitment 4 = Avoid the use of rankings of research organisations in research assessment
CoARA Commitment 5 = Commit resources to reforming research assessment as is needed to achieve the organisational changes committed to
CoARA Commitment 6 = Review and develop research assessment criteria, tools and processes
CoARA Commitment 7 = Raise awareness of research assessment reform and provide transparent communication, guidance, and training on assessment criteria and processes as well as their use
CoARA Commitment 8 = Exchange practices and experiences to enable mutual learning within and beyond the Coalition
CoARA Commitment 9 = Communicate progress made on adherence to the Principles and implementation of the Commitments
CoARA Commitment 10 = Evaluate practices, criteria and tools based on solid evidence and the state-of-the-art in research on research, and make data openly available for evidence gathering and research

Which target group should implement the idea

User - Funder = User could involve Funders and regional authorities
User - Institutes = User could involve Research institutions
User - Academies = User could involve Academies and learned societies
User - Research Groups = User could involve Research groups
User - Meta-Researchers = User could involve Meta-researchers
User - Scientific editors and publishers = User could involve Scientific editors and publishers

Specific challenges in research assessment reform

Challenge - Process = How can we create an assessment process based upon what the different players in research value?
Challenge - Inclusivity = How can we make research more inclusive and equitable through assessment?
Challenge - Bias_Mitigation = How can we mitigate the negative effects of bias on assessment?
Challenge - Process_Culture = How can we recognize process and culture through assessment?
Challenge - Diversity = How can we foster diversity through assessment?
Challenge - Collaboration = How can we foster collaboration through assessment?
Challenge - DifferentQuestions = How can we ask assessment questions differently?

Completeness of the idea and status of experimentation

Level 0 = Level of completeness: 0 – The experiment only contains the description and minimal details. This level is meant to provide inspiration for experiments that can be developed further by those experimenting.
Level 1 = Level of completeness: 1 – The experiment contains minimal details. Organisations will need to build the experimentation plan before they can implement the idea.
Level 2 = Level of completeness: 2 – The experiment contains minimal details that may help with the design and implementation, but organisations will need to build most of the experimentation plan before they can fully implement the idea.
Level 3 = Level of completeness: 3 – The experiment contains some details that may help with the design and implementation, but organisations will need to finalise the experimentation plan before they can fully implement the idea.
Level 4 = Level of completeness: 4 – The experiment contains details that can provide useful guidance for implementation, but additional research will need to be done by experimenting organisation. Few real life cases exist.
Level 5 = Level of completeness: 5 – The experiment contains sufficient details to be implemented, and it has already been experimented in some organisations providing cases, additional suggestions, and often learned lessons.

When you would like to go back to the main page, just click on the title “Assessment Idea Catalogue” to return.

How to implement an idea

Once you have chosen an idea to implement or test, you are free to apply it in your context as you see fit. Each idea has as much detail as possible about potential outcomes, how to evaluate success, possible challenges (and mitigations to those challenges), and other contextual detail that should help you prepare your experiment.

As you proceed with the implementation, please feel free to return to the idea catalogue and add your learnings and evidence to the idea’s page, so that others can learn from your experience, and the catalogue becomes more complete.

How to comment on an idea or add evidence

TO BE ADDED

How to develop or add an idea

If you have an idea that would fit the idea catalogue, we would be happy to hear from you. Please fill out the following template so that we can add the idea in the same format as others on the platform.

Click here to download the idea template file

The template can be used as a set of questions that frame discussions on experimental ideas - if you have a concept for an idea, the questions help to develop that idea towards more structured potential experiments that can be implemented and evaluated.

When filling out the template, please respect the headings and sections that are in there. Not all sections need to be filled out, but the more content present the more helpful the idea will be to others. When it’s ready, to send us the completed file in “.txt” format (if possible),