Research domains
Assessment lotteries may apply to all domains, but their use is generally limited to the evaluation of research proposals (funding); they are not used for evaluating researcher career advancement (e.g., hiring, promotion,…) since this area implies much more complex elements and challenges.
Context and considerations
Assessment lotteries can take different shapes: - Applications that meet basic criteria (eligibility criteria, quality standards, etc.) are not further evaluated, ranked etc. by evaluators, but a lottery is organised and the ‘winning’ lots get the funding.
- A modified lottery can also be organised: In a first round the proposals are evaluated by peer reviewers, and only the proposals worthy of funding enter the lottery phase. - Partial randomization can also take place: Have larger groups (e.g. Must be funded, should be funded, should not be funded), and then randomize where the limiting factor (budget) is applied - i.e. fund all of the “musts”, and randomly select the “shoulds”.
- There is another kind of lottery, where randomization is used to select who is eligible to submit a funding proposal (see Luebber et al., 2023). The main advantage of “pre-lotteries” relative to the other forms of lotteries is that they reduce the workload on applicants: applicants who do not win the lottery do not have to waste resources in writing their grant application.
Costs to consider for institutions for implementing the experiment (framed as questions for the user): - How accepting is your community (applicants and evaluators) for trialing randomization/lotteries?
Suggestions on how to implement (including research culture aspects) - Start small: small funding calls (limited total budget), small projects (small amount of money per project).
- Foster and grow the buy-in and trust from the applicants and from the evaluators.
- Communicate and be transparent about the lottery, how it works etc.
- Publish the randomization process/algorithm (to build trust in the process).
Challenges and mitigations
Challenges: - Lotteries are not a quick fix to solve issues related to bias or equity, because they do not fix problems at the beginning of the pipeline: some people simply don’t apply, thus making the pool of applicants less diverse
- In a (small) experiment the end result might be a less inclusive, equitable etc. outcome… - some basic knowledge of statistics is necessary to fully understand the pros and cons of lotteries
- Applicants and evaluators might be sceptical about a lottery, but they accept it as a valid method to distribute research funding if the method is well explained to them. In addition, evaluators might find a lottery difficult because for them it is hard to acknowledge that they are not able to identify ‘the best’ proposals (feels like failing). This needs to be taken into consideration in the communication with the evaluators.
- Research culture needs to be ready for this kind of experiment, e.g., a degree of openness is required (with evaluators being open about what they can achieve and what not).
Mitigation discussion: (Questions to address or discuss to enable resolution on the possible challenges identified) - Open communication about why a lottery is being implemented, about what it will achieve and what not, about the statistics behind the model (due to the lottery for instance all money could go to one discipline, to only groups with a male PI etc.), about the outcomes (and the monitoring) of the lottery
- Open communication about disadvantages of peer review procedures (e.g., the role of chance in peer review)
- Start small, with smaller calls, budgets etc. so that all stakeholders can get used to the idea of a lottery
- Align with larger goals, i.e. “increasing diversity of what is funded”, “eliminating panel/evaluator bias for difficult comparisons”
Evaluating success
To assess success of the experiment, it is may be needed to have a predetermined number of funding calls that are trialled with lotteries. Once a determined number is achieved, it may be possible to compare diversity of lottery results to previous similar calls without randomization?
On a smaller scale, it may be more realistic to look at satisfaction in using lotteries. These may include interviews, surveys to find out if reviewers are more satisfied, less overburdened etc. due to the lotteries, or interviews, surveys etc. with applicants to find out how accepting they are of the lottery system and to understand whether they find fairness in the process.
Relevant resources and literature
Although there are some experiments going on, the effect and impact of lotteries remains largely unknown. Research on lotteries is needed to grow our understanding. A suggestion on how this research could be organized: Stafford, Tom; Rombach, Ines; Hind, Dan; Mateen, Bilal; Woods, Hellen Buckley; Dimario, Munya; Wilsdon, James (2024). Where next for partial randomisation of research funding? The feasibility of RCTs and alternatives, Wellcome Open Res., 8:309. https://doi.org/10.12688/wellcomeopenres.19565.1
NRIN Happy Hour Webinar - Funding Through Lottery, 1 May 2025. https://www.youtube.com/watch?v=0oJFMBSkk-M has different types of lotteries explained in the first presentation.
Using lotteries to increase fairness and efficiency in research assessment, DORA, https://sfdora.org/2025/03/27/using-lotteries-to-increase-fairness-and-efficiency-in-research-assessment/
Bendiscioli, Sandra; Firpo, Teo; Bravo-Biosca, Albert; Czibor, Eszter; Garfinkel, Michele; Stafford, Tom; et al. (2022). The experimental research funder’s handbook (2nd edition, ISBN 978-1-7397102-0-0). Research on Research Institute. Report. https://doi.org/10.6084/m9.figshare.19459328.v5
Fang, F. C., & Casadevall, A. (2016). Research funding: The case for a modified lottery. MBio, 7(2). https://doi.org/10.1128/mBio.00422-16
Golberg, A. (2022, November 15). The (partial) rise of (partial) randomisation. Research Professional News. https://www.researchprofessionalnews.com/rr-news-uk-views-of-the-uk-2022-11-the-partial-rise-of-partial-randomisation/
Kolarz et el. 2023. Review of Peer Review Final report. Technopolis https://www.ukri.org/wp-content/uploads/2023/07/UKRI-060723-Review-of-peer-review-Final-report-revs-v2.pdf (Includes a section on use of partial randomisation (page 44- 45 referencing the funders schemes discussed elsewhere in this document)
Woods, Helen Buckley; Wilsdon, James (2021). Experiments with randomisation in research funding: scoping and workshop report (RoRI Working Paper No.4). Research on Research Institute. Report. https://doi.org/10.6084/m9.figshare.16553067.v2
Woods, Helen Buckley; Wilsdon, James (2021). Why draw lots? Funder motivations for using partial randomisation to allocate research grants (RoRI Working Paper No.7). Research on Research Institute. Report. https://doi.org/10.6084/m9.figshare.17102495.v2
Liu, Mengyao; Choy, Vernon; Clarke, Philip et al. (2020). The acceptability of using a lottery to allocate research funding: a survey of applicants, Research Integrity and Peer Review 5,3. https://doi.org/10.1186/s41073-019-0089-z
Roumbanis, L. (2019). Blind Luck – Could lotteries be a more efficient mechanism for allocating research funds than peer review? LSE Impact Blog. • 1 survey response • 6 interviews Review of Peer Review 46 https://blogs.lse.ac.uk/impactofsocialsciences/2019/12/11/blind-luck-could-lotteries-be-amore-efficient-mechanism-for-allocating-research-funds-than-peer-review/
UK Metascience Unit (2025) A Year in Metascience. UK Department for Science, Innovation and Technology and UK Research and Innovation. https://doi.org/10.6084/m9.figshare.29210066
Templates from funders and institutions
Case examples and literature
Volkswagen Stiftung:
- Partially Randomized Procedure - Lottery and Peer Review. https://www.volkswagenstiftung.de/en/funding/peer-review/partially-randomized-procedure-lottery-and-peer-review
- This is how it works: The partially randomized selection process within the initiative “Experiment!” https://www.volkswagenstiftung.de/en/how-it-works-partially-randomized-selection-process-within-initiative-experiment
- Give Chance a Chance https://www.volkswagenstiftung.de/en/news/story/give-chance-chance
Swiss National Science Foundation:
- Drawing lots as a tie-breaker (2021). https://www.snf.ch/en/JyifP2I9SUo8CPxI/news/news-210331-drawing-lots-as-a-tie-breaker - Rachel Heyard, Manuela Ott, Georgia Salanti, Matthias Egger (2022). Rethinking the Funding Line at the Swiss National Science Foundation: Bayesian Ranking and Lottery. Statistics and Public Policy. Volume 9, Issue 1, 110-121. https://doi.org/10.1080/2330443X.2022.2086190
Wellcome Trust:
- Lewis-Wilson, Shomari; Towers, Sonya; Wykeham, Harriet (2023). The luck of draw: Wellcom’s Institutional Fund for Research Culture. Wellcome Open Research. https://doi.org/10.12688/wellcomeopenres.20057.1
- Stafford T, Rombach I, Hind D, Mateen B, Woods HB, Dimario M, Wilsdon J. Where next for partial randomisation of research funding? The feasibility of RCTs and alternatives. Wellcome Open Res. 2024 May https://doi.org/10.12688/wellcomeopenres.19565.2>
British Academy:
- Emond, Ken (2025). How randomisation has changed the British Academy’s approach to research funding. https://blogs.lse.ac.uk/impactofsocialsciences/2025/04/16/how-randomisation-has-changed-the-british-academys-approach-to-research-funding/
Austrian Science Fund:
- Bendiscioli, Sandra; Firpo, Teo; Bravo-Biosca, Albert; Czibor, Eszter; Garfinkel, Michele; Stafford, Tom; et al. (2022). The experimental research funder’s handbook (2nd edition, ISBN 978-1-7397102-0-0). Research on Research Institute. Report. https://doi.org/10.6084/m9.figshare.19459328.v5
Health Research Council of New Zealand:
- New paper shows support for lottery funding allocation. https://www.hrc.govt.nz/news-and-events/new-paper-shows-support-lottery-funding-allocation
- Liu, Mengyao; Choy, Vernon; Clarke, Philip et al. (2020). The acceptability of using a lottery to allocate research funding: a survey of applicants, Research Integrity and Peer Review 5,3. https://doi.org/10.1186/s41073-019-0089-z
Research England’s Enhancing Research Culture fund at University of Leeds:
- Davies, Catherine; Ingram, Holly (2023). A partially randomised approach to internal funding allocation A pilot by the University of Leeds’ Research Culture team. https://researchculture.leeds.ac.uk/wp-content/uploads/sites/116/2023/12/UoL-RC-PRA-case-study-2024-1.pdf
- Davies, Catherine; Ingram, Holly (2025). Sceptics and champions: participant insights on the use of partial randomization to allocate research culture funding, Research Evaluation, Volume 34, https://doi.org/10.1093/reseval/rvaf006
Comments/lived examples