In Support of Reproducibility
By Iva Fedorka
Testing and data are foundations of the scientific process. As theories are developed and experiments conducted, the new information discovered will be verified as it becomes accepted by other scientists. Consensus and the adoption of new concepts and paradigms occur when experimental results can be reproduced and thereby validated. For this, test results must be accurate and reproducible.
Reproducibility Defined
The National Academy of Sciences defines reproducibility as the ability to produce consistent results when using the same input data, calculations, methods, code, and other analytical processes. Since random elements may impact results, outcomes do not have to be identical to be reproducible. Reproducibility is also different from replication, which is simply repeating the steps in an experiment or process.
In science, statistical methods are used to help answer questions or to make appropriate inferences about reproducibility based on the data. Today’s experiments are increasingly analytically complex, which places a greater importance on statistical review. Controversies about interpretation may threaten the value of some data, so it’s important to understand how inferences may be made.
Barriers to Reproducibility
It is well known that specific published results cannot always be reproduced by other scientists, but the specific reasons for such failures and ways to correct them are not. Along with other issues, common complaints include data and model availability, publication pressures, and industry standards.
Some barriers, like a lack of access to resources or insufficient data storage, may be significant but straightforward. Others may be more nuanced, like professional pressure to publish research about specific topics or the lack of a suitable audience. A poor understanding of multiple and disparate science topics can also unexpectedly and negatively affect outcomes.
Quality and Reproducibility
Different or inconclusive results can be caused by poor quality technique or materials. One example of how quality issues can affect results can be found in the evaluation of erythropoiesis-stimulating agents used to treat anemia in cancer patients. Some believe that these treatments can also stimulate erythropoietin (EPO) receptors on tumor cells (the EPO receptor-cancer hypothesis). Many articles have been published on this topic, but their data and conclusions conflict.
One search1 found 280 relevant articles related to the topic. These were reviewed for potential relationships between quality and reproducibility. Many conflicts existed between and within articles, caused by quality deficits, including:
- Faulty quality parameters like a lack of appropriate controls (90%)
- Inadequate reagent and method validation (87%)
- Choice of inappropriate statistical methods (84%)
- False-positive or -negative test results (81%)
Producing Reproducible Results
Laboratory managers can establish practices and procedures to support the production of high-quality and reproducible technical results in a number of ways.
Set high expectations
Clearly establish the required levels of competency and compliance for all laboratory personnel, including principal investigators.
Validate methods2
Evaluate all test methods to determine the accuracy, precision, replication, and repeatability of specific steps and the viability of the method over time. In addition, determine specificity, detection and quantitation limits, and the linearity and range of the assay.
Understand measurement uncertainty3
Determine whether observed differences in experiments are “significant.” Does the variation represent real differences?
Establish training
Provide on-the-job training for all employees, regardless of previous educational background or work experience. Training can help improve flexibility, innovation, and agility, and is a vital part of most lab audits.
Effectively document
Most labs use Standard Operating Procedures (SOPs) as “how-to” manuals for all technical and operational laboratory processes. Include further explanations to help staff understand the reasons for maintaining important aspects of the operation, which can help with compliance.
Follow good lab practices
Lab cleanliness, equipment calibration, environmental controls, and documented observations and results are typical good lab practices. More specific practices, like triplicate experiments or specific standard deviation ranges, may be needed as demanded by the science.
Share results
Round-robin testing is a common way to compare results when the same methods are used with identical samples in different labs. These results can provide data about reproducibility and provide feedback to the lab about its ability to reproduce results.
Reproducible results are key to good science. They help verify new discoveries, demonstrate competency, and clearly communicate technical outcomes to others. By creating reproducible data, others may build upon findings and advance scientific discovery.
References:
1 Elliott, S. (2020). Impact of Inadequate Methods and Data Analysis on Reproducibility. Journal of Pharmaceutical Sciences, 109(2), 1211-1219. Retrieved from https://www. sciencedirect.com/science/article/abs/pii/S0022354919304381
2 (European Medicines Agency, 1995) ICH Topic Q 2 (R1) Validation of Analytical Procedures: Text and Methodology. Retrieved from https://www.ema.europa.eu/en/ documents/scientific-guideline/ich-q-2-r1-validation-analytical-procedures-text-methodol- ogy-step-5_en.pdf
3 (NIST, 2015, November 19) Measurement Uncertainty. Retrieved from https://www.nist.gov/itl/sed/topic-areas/measurement-uncertainty
Iva Fedorka is a Fisher Scientific staff writer.
