Research domains
This idea is relevant in a variety of contextx (hiring, promotion, funding, etc.) but is mostly applicable for hypothesis-driven research and empirical research (where n > 1).
Context and considerations
Investing in the importance of negative results in assessments is a cultural change and it will often require a systemic experiment - not just on the evaluation itself, but on the whole practice of research. As such, a holistic approach may be needed, with everyone working together to induce the change.
It may be important to define the story/strategy/route behind this (and entry point) - what are the steps (and in what order) to push towards this. It may also be important to consider how this is done in practice, and what is considered in the evaluation. For example, should researchers include all null/negative findings or only the most relevant?
Experiments will generally need:
- Researchers doing the null work/having the data
- Platform to publish the results
- People to evaluate the negative results (and guidance/training on how to evaluate them)
- Funders to recognize it in pre- and post-project evaluation
Targeting the evaluation or recognition of null and negative results in an experiment may take a variety of forms in a variety of contexts. For examples, experimentation may be interested in:
- Looking at whether the ratio of negative results published to positive results published in a certain context/centre may be used as a metric for creativity or risk taking
- Looking at the impact of recognition of pre-prints with negative/null results in assessment of researchers or research outputs
- Looking at the impact of evaluators asking for negative/null results published as an output (i.e., formally recognize these as a part of output)
- Looking at the demands from funders with regards to publication and appreciation of negative results (e.g., In the final reports - accept that only null results may be published and consider this to be satisfactory to provide the last chunk of grant money)
- Looking at the impact of publishing in Open Research repository - Open Research Europe, Wellcome Open Research, pre-print servers (not peer-reviewed), on the outputs reported
- Moving assessments away from novelty and impact in peer-review process of research outputs - peer review tailored towards goals so that not everything needs to aim for novelty
- Looking at the recognition of pre-registration in the evaluation process or implementation of a requirement of pre-registration by the research(ers) once funded (e.g. before contract signature)
- Understanding how negative results reported in “Additional material/negative results” sections in papers/outputs may be used in research assessment
Challenges and mitigations
Challenge: Culture of prestige – There is a huge culture of recognition around high-impact factor journals. This may be difficult to go around, especially the importance of those outputs in university rankings which bring a higher level of resistance.
Mitigation: Change the culture at all levels
Mitigation: Get funders and seniors in institutions on board
Challenge: Acceptance from funders – Given the role of funders and the importance of ‘impact’ for research funding (and for obtaining trust and support from society and governments), it may be difficult to ensure that funders are on board with this idea. However, without funders’ support, it may be difficult to obtain real buy-in from institutions.
Mitigation: Raise awareness on the crucial role of negative findings in building robust knowledge, and in turn, in the role that this may have in helping to avoid harm (e.g., unpublished side effects and ineffective drugs).
Challenge: Lack of finalisation of projects that yield negative results – Risk of abandonment of studies that start showing null results
Mitigation: Ensure protected project time and funding to complete projects regardless or results
Challenge: Metrics impact – Risk of not being cited
Mitigation: Using “Additional material/negative results” sections in papers/outputs to make sure negative results are accessible with the publication
Challenge: Culture – Not much of a “story” around negative/null results
Mitigation: Again, bring more recognition by funders/institutions to shift culture to value negative results
Evaluating success
Relevant resources and literature
This section includes resources, literature, and reports relevant to this specific experimental idea.
Comments/lived examples