CoARA Assessment Idea Catalogue – Assessment Idea Catalogue

Tags

All (48)

Challenge - Bias Mitigation (6)

Challenge - Collaboration (4)

Challenge - Different Questions (8)

Challenge - Diversity (8)

Challenge - Inclusivity (9)

Challenge - Process (7)

Challenge - Process Culture (9)

CHALLENGE Inclusivity (1)

CoARA Commitment 1 (12)

CoARA Commitment 10 (3)

CoARA Commitment 2 (11)

CoARA Commitment 3 (3)

CoARA Commitment 5 (1)

CoARA Commitment 6 (12)

CoARA Commitment 7 (1)

CoARA Commitment 8 (1)

Level 0 (41)

Level 1 (3)

Level 2 (1)

Level 3 (1)

Level 4 (2)

User - Academies (8)

User - Funder (8)

User - Funders (6)

User - Institutes (13)

User - Meta-Researchers (10)

User - Research Groups (6)

User - Scientific editors and publishers (4)

The CoARA Assessment Idea Catalogue is designed to be a resource for those developing and/or implementing assessment processes at a variety of different levels, including:

hiring and promotion
project funding
research programme funding
institutional evaluations
national research system evaluations.

A collection of different experimental ideas are represented here, that can be used as inspiration for designing experiments that innovate on current assessment practices.

For information on how to use this catalogue, please look at the Idea Catalogue How-To Guide

We welcome all feedback on the platform! To send us any feedback or ideas, please use the Idea Catalogue Feedback Form.

If you would like to submit a new idea to the catalogue, or suggest edits and content to existing ideas, please use the Idea Submission Form

Fully open peer-review

Peer review in which revieewers and applicants know each other’s identities. This can extend to public sharing of peer review reports.

Experiments in Assessment WG

Team Tenure

Challenge - Process

Challenge - Inclusivity

Challenge - Process Culture

Challenge - Diversity

Challenge - Collaboration

Challenge - Different Questions

CoARA Commitment 1

CoARA Commitment 6

User - Institutes

User - Research Groups

User - Meta-Researchers

Team tenure is a way to provide stability of funding (i.e., through tenure as a permanent funding streams) for a research group rather than for individual researchers…

Experiments in Assessment WG

Purpose-driven assessment

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Reflective evaluation – self-driven criteria creation

Allow the person/group being evaluated to set criteria for their own evaluation, and give a self-reflection on their progress towards those criteria. This can be used in…

Experiments in Assessment WG

Double blind peer-review

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Academic support training

User - Institutes

Academic support training equips staff to guide researchers through assessment processes by providing information, training, and tools to monitor evaluation of their work…

Experiments in Assessment WG

Constructive feedback

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Reducing frequency of assessment

Challenge - Process

Challenge - Process Culture

Challenge - Different Questions

CoARA Commitment 2

CoARA Commitment 6

User - Institutes

Reducing how frequently researchers are assessed, for example by requiring fewer assessment during their career or by increasing the funding periods.

Experiments in Assessment WG

Discussion before scoring

Having interviews as first stage of the assessment process, rather than as a last stage. This could shift the elements that are valued in the assessment process and promote…

Experiments in Assessment WG

Removing assessment in general

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Qualitative assessor training

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Hiring as a site of experimentation

Challenge - Process

Challenge - Inclusivity

Challenge - Process Culture

Challenge - Diversity

Challenge - Different Questions

CoARA Commitment 1

CoARA Commitment 5

CoARA Commitment 6

CoARA Commitment 8

CoARA Commitment 10

User - Institutes

User - Meta-Researchers

Using the institutional hiring process to diversify staff profiles and expertise in line with the goals and needs of the institution.

Experiments in Assessment WG

Acknowledging bias, making it transparent

Challenge - Process

Challenge - Bias Mitigation

Challenge - Different Questions

CoARA Commitment 2

CoARA Commitment 6

User - Institutes

User - Academies

User - Research Groups

User - Meta-Researchers

User - Scientific editors and publishers

Allowing evaluators to declare potential biases to the panel makes bias visible and fosters a fairer evaluation process. It encourages open, transparent discussion, even…

Experiments in Assessment WG

Neutral observer in the review panel

Challenge - Bias Mitigation

Challenge - Process Culture

CoARA Commitment 2

CoARA Commitment 6

CoARA Commitment 7

CoARA Commitment 10

User - Institutes

The role of a neutral observer in research assessment panels is to monitor fairness, transparency, and adherence to procedures. This role enhances trust, mitigates bias, and…

Experiments in Assessment WG

Assessment lotteries

Challenge - Bias Mitigation

Challenge - Inclusivity

Challenge - Diversity

Challenge - Collaboration

CoARA Commitment 1

CoARA Commitment 3

In Assessment lotteries, applications that meet basic criteria (eligibility criteria, quality criteria, ot other) are not further evaluated, ranked etc. by evaluators, but a…

Experiments in Assessment WG

Research users impacted as evaluators

Challenge - Inclusivity

Challenge - Bias Mitigation

Challenge - Process Culture

Challenge - Diversity

Challenge - Different Questions

CoARA Commitment 1

CoARA Commitment 2

CoARA Commitment 6

User - Academies

Individuals or their representatives directly impacted by the research activities can be involved as reviewers, as one specific category of stakeholders. Depending on the…

Experiments in Assessment WG

Fostering co-creation of research programmes

Engage with community while designing funding programmes to ensure the programmes designed are as diverse and includsive as possible and are relevant and speaking to the…

Experiments in Assessment WG

Broadening the meaning of ‘the academic’

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Evaluating the idea only

User - Meta-Researchers

Focusing the evaluation solely on the research proposal itself to remove the influence of the researcher’s reputation, track record, and institutional affiliation.

Experiments in Assessment WG

Making all science anonymous

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Mentee evaluation

CoARA Commitment 1

CoARA Commitment 2

CoARA Commitment 6

User - Institutes

User - Academies

User - Research Groups

User - Meta-Researchers

Evaluating mentorship and making mentorship a part of the assessment. This is especially relevant for PIs, and aims to look at the mentorship and culture created with the…

Experiments in Assessment WG

Distributed Peer-Review

Challenge - Inclusivity

Challenge - Process Culture

Challenge - Collaboration

CoARA Commitment 2

CoARA Commitment 6

CoARA Commitment 10

User - Institutes

User - Meta-Researchers

Distributed peer review is a method in which applicants evaluate each other’s proposals. It promotes equity, transparency, and inclusion, reduces the burden on traditional…

Experiments in Assessment WG

Application burden proportional to money/reward

The length and complexity of the application process should be modified based on the award/result of the process. For larger grants/more senior positions there should be a…

Experiments in Assessment WG

Recognition of peer Review

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Adjusting evaluation weighting

Adapting the weighting of evaluation criteria in an assessment to better reflect the goals of what should be supported and the priorities of the evaluating body.

Experiments in Assessment WG

Evaluation of EDI action plan

In this idea, an EDI action plan must be submitted with an application, and this action plan is then evaluated as part of the process. This aims to ensure that EDI aspects…

Experiments in Assessment WG

Value-based assessment

Identifying fundamental values for a research funding or hiring programme, and centering the assessment on those values (e.g. as evaluation criteria).

Experiments in Assessment WG

Skills based assessment/development

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Primary vs. Secondary evaluation criteria

A secondary set of evaluation criteria that would be used in the event of proposals (or whatever is evaluated) that are of the same quality. Potential tiebreakers could be…

Experiments in Assessment WG

Ignore CV/publication list

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Wildcard in the review panel

CoARA Commitment 1

CoARA Commitment 6

CHALLENGE Inclusivity

User - Institutes

User - Academies

User - Meta-Researchers

Reviewers can use their wildcard or golden ticket (one per call) to directly fund a proposal they consider exceptionally promising but fear may be overlooked—often because…

Thomas Feliciani, Experiments in Assessment WG

Criteria developed by those being assessed

Involving the applicant pool in developing the evaluation criteria for an assessment process. This can be done potentially in a phase before the call is launched, and serves…

Experiments in Assessment WG

EDI tiebreak

An EDI ‘tiebreak’ is a secondary criteria used in the case of identically ranked or scored proposals/applications, for which an EDI element is used as a tie-breaking measure.

Experiments in Assessment WG

Criteria with space for creativity

Adding an evaluation criteria that is focused on creativity, or adjusting multiple existing criteria to include aspects of creativity in research.

Experiments in Assessment WG

Process-based assessment

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Team assessment

Team assessment enables the evaluation of collective research performance, emphasizing shared outputs, interdisciplinary collaboration, and distributed contributions.

Experiments in Assessment WG

Valuing unfunded applications

Valuing unfunded applications acknowledges the intellectual merit and innovative potential of research proposals regardless of funding outcome.

Experiments in Assessment WG

SCOPE framework

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Forward looking assessment

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Including non-academics in evaluation

Challenge - Process

Challenge - Inclusivity

Challenge - Different Questions

CoARA Commitment 1

CoARA Commitment 2

User - Meta-Researchers

Including non-academics in evaluation broadens perspectives by integrating stakeholders from industry, policy, and society. It enhances relevance, transparency, and societal…

Experiments in Assessment WG

Co-design of programmes with specific communities

Challenge - Process

Challenge - Inclusivity

Challenge - Diversity

Challenge - Collaboration

Challenge - Different Questions

CoARA Commitment 1

CoARA Commitment 2

CoARA Commitment 6

User - Institutes

User - Academies

User - Research Groups

Certain funding programmes, research activities or evaluations can be co-designed with communities that are targets of / impacted by the research activities desired to be…

Experiments in Assessment WG

Evaluating negative and null results

CoARA Commitment 1

CoARA Commitment 3

CoARA Commitment 6

Challenge - Process

Challenge - Bias Mitigation

Challenge - Process Culture

Challenge - Diversity

User - Institutes

User - Meta-Researchers

User - Scientific editors and publishers

Evaluating negative and null results is a way to give visibility to the important learnings that come with experiments that did not yiels the expected findings. Allowing…

Experiments in Assessment WG

Multidimensional “diversity” in evaluation committees

Challenge - Inclusivity

Challenge - Bias Mitigation

Challenge - Diversity

CoARA Commitment 1

CoARA Commitment 2

CoARA Commitment 6

User - Academies

User - Scientific editors and publishers

Based on the objective and scope of evaluation, define as many diversity dimensions as possible to be considered and include them as far as possible in the reviewers team …

Experiments in Assessment WG

Non-expert panels

Replacing or supplementing existing panels with non-researchers, to evaluate aspects of the proposal (or whatever is evaluated) that are not directly research related.

Experiments in Assessment WG

Pick your own reviewers

Challenge - Process Culture

CoARA Commitment 1

CoARA Commitment 2

User - Institutes

User - Academies

User - Research Groups

User - Scientific editors and publishers

Applicants recommend peer reviewers for their projects, who are then included in the evaluation process. This approach reduces the burden on evaluation bodies and…

Experiments in Assessment WG

Experimental teaching

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Community impact evaluation

This is where the summary of the experiment goes - Describe the experiment in 50 words or less; Focus on why it’s important, what it is, what it does

Experiments in Assessment WG

Narrative CV

Challenge - Inclusivity

Challenge - Process Culture

Challenge - Diversity

Challenge - Different Questions

CoARA Commitment 1

CoARA Commitment 2

CoARA Commitment 3

User - Institutes

User - Academies

User - Research Groups

User - Meta-Researchers

Narrative CVs change the format of Curriculum Vitaes (CVs) submitted alongside funding or employment applications from an itemised format to a structured narrative style…

Experiments in Assessment WG