The COVID-19 pandemic has led to widespread questioning of federal public health policies and of the scientific evidence supporting these. This resulted, at least in part, from the lack of data transparency in the federal health journals shaping public policy. It’s past time for these journals to adopt the transparency standards used by top private-sector journals, or continue losing public and scientific credibility.
The federal Department of Health and Human Services (HHS) makes health policy using peer-reviewed research, including publications in four scientific health journals that it controls and manages: Emerging Infectious Diseases, Mortality and Morbidity Weekly Report, Environmental Health Perspectives, and the recently federalized Journal of Health and Pollution.
None of these journals meets the most modest transparency policies identified by Oxford University Press — simply encouraging authors where ethically possible to publicly release all data underlying a published paper. The strongest pro-transparency policy identified by Oxford requires “all authors, where ethically possible, to publicly release all data underlying any published paper as a condition of publication,” and subjects data to peer review.
This lack of attention to independent reproducibility is striking because a defining characteristic of science — one arguably necessary for public trust — is that independent researchers using the same analytic methods reach the same conclusion. Moreover, a 2019 study on reproducibility and replicability by the National Academies of Science concluded that “reproducibility is strongly associated with transparency; a study’s data and code have to be available in order for others to reproduce and confirm results.”
Policies requiring access to data and computer code as a condition of publication exist at highly regarded journals associated with Science, Nature, and the American Economic Association, as well as the Proceedings of the National Academies of Science (PNAS), Environmental Science & Technology, etc. Science, Nature and PNAS are multi-disciplinary journals so their procedures, which are designed to protect privacy of human subjects and recognize use of proprietary data, are intended to be applicable to a wide variety of research areas.
At least one federal journal has already adopted better data access policies. The federal Journal of Fish and Wildlife Management now requires public disclosure of underlying data.
Independent reproducibility of HHS research is not simply an academic concern. In 2020, the U.S. Government Accountability Office studied HHS modeling of infectious disease, finding that guidelines and policy decisions from the Centers for Disease Control and Prevention (CDC) did not address reproducibility of models or code — a flaw that may have jeopardized the reliability of CDC research into COVID-19. It recommended that CDC establish guidelines to ensure full reproducibility of its research, such as sharing with the public all permissible and appropriate information needed to reproduce research results, including, but not limited to, model code.
HHS, or perhaps the White House or Congress, should require that all federal health journals adopt policies for public access to researchers’ data and code of the top science journals. If these journals are unable to adopt such policies, it is unclear why they might merit taxpayer support.
Randall Lutter, Ph.D., is a senior fellow at the Manhattan Institute, where his research focuses on pharmaceutical markets and policy, medical innovation and regulation. He was a senior science and regulatory adviser in the Office of the Commissioner at the U.S. Food and Drug Administration from 2017 to 2020.