Published On January 15, 2014
WHEN A RESPECTED MEDICAL JOURNAL PUBLISHES A PAPER suggesting a promising new treatment for Alzheimer’s disease or a better way to predict cancer, it tends to attract a lot of attention. Publication in a top peer-reviewed journal is the currency of biomedical achievement. Completing a scientific manuscript requires months or years of data acquisition, analysis and writing. Getting the manuscript into print requires the additional step of passing peer review, the rigorous scrutiny of experts in the field.
It would seem that after so much effort, the findings of such studies might be irrefutable. Yet the reliability of a startlingly high percentage of research has been called into question because results often can’t be reproduced by other labs. In this issue of Proto, we look at research replication and the disturbing suggestion that much of what biomedical researchers conclude in published studies may be exaggerated or just plain wrong.
Replicating complex studies isn’t easy. Even tiny variations in methods can skew results, and it may be virtually impossible to duplicate each variable exactly—how patients are selected, measurements taken, instruments calibrated, data analyzed and results presented. Moreover, human biology is so complex that no study, regardless of how well it was done, should be taken as gospel, and even the most painstaking efforts to replicate results may not succeed. Also in play could be a research team’s unintended or overt bias toward a particular outcome, or even toward the group that published the original work. Very rarely, research fraud may also play a role. Furthermore, in the world of competitive funding there is limited incentive to repeat studies done by others. Researchers know it is fresh ideas and innovation—not affirming or casting doubt on someone else’s work—that tend to attract funding and get published in major journals.
Every facility doing research has a stake in improving the credibility and reproducibility of published scientific evidence. With support and guidance from the National Institutes of Health and the involvement of high-profile national societies and leading journals, the research community should facilitate replication efforts and provide incentives for researchers to undertake such studies. Even negative studies can provide valuable information. If we can’t replicate our own research or if findings in subsequent studies are not as strong as in the original work, there should be avenues readily available to report such information. Science is a complex undertaking, and good research will inevitably produce failures and disappointment as well as success and progress. We need to understand all these results to advance medicine, and important new results must be confirmed or challenged.
Peter L. Slavin, M.D.
Massachusetts General Hospital
David F. Torchiana, M.D.
CEO and Chairman
Massachusetts General Physicians Organization
- The Problem of Replication
It’s disappointing when seemingly groundbreaking studied can’t be repeated. But it’s happening a lot.