Comparative Effectiveness Research: Challenges for Medical Journals

In order to optimize health outcomes within the constraints of inevitably limited resources, low- and high-income countries alike require unbiased means of assessing healthcare interventions for their relative effectiveness. Such interventions include diagnostic tests and treatments (both established and newly developed) and implementation of health policy.¹ Likewise, healthcare professionals and patients need better information to inform healthcare decisions that require weighing benefits and risks in light of the patient’s medical history and personal preferences.

Some countries and international organizations have recognized the need for such evidence and are already allocating funds for research to provide it.² The WHO Ministerial Summit in Mexico called for the establishment of support for a substantive and sustainable program of health systems research aligned with countries’ priority needs and aimed at achieving internationally agreed-upon health-related development goals, including those contained in the United Nations Millennium Declaration.³ The UK has established the National Institute for Health Research to commission and disseminate research that supports decision making by professionals, policy makers and patients and to ensure that the UK’s health system, the National Health Service, has access to the best possible evidence to inform decisions and choices.⁴

The US is now addressing similar goals with an initiative known as comparative effectiveness research (CER). In 2008, a report by the US Institute of Medicine (IOM) noted that patient care “should be based on the conscientious, explicit, and judicious use of current best evidence.”¹ In legislation that allocated US $1.1 billion in the US for CER on healthcare practices in 2009, the US Congress mandated that the IOM set national priorities for CER clinical topics. The IOM defined CER as “The generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition, or to improve the delivery of care.”5 The definition further stated that “The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve healthcare at both the individual and population levels.”

To the authors and endorsers of the present Commentary, the potential value of research with these characteristics is self-evident. The challenge will be to realize the full potential of such research to improve health. Doing so will require assessing a heterogeneous body of evidence consisting of prospective randomized trials—including pragmatic trials—and observational research using data obtained in the course of regular practice. Hence, medical journals must use rigorous approaches, including but not limited to peer review by independent experts, to assess the limitations inherent in such research, such as missing data, incomplete follow-up, unmeasured biases, the potential role of chance, competing interests, and selective reporting of results.

Drawing on many years of collective experience in assessing these issues in the course of evaluating health research through peer review, we support the following principles and standards for CER.

Principles of CER

• The principal goal of CER is to allow decision makers (patients, clinicians, purchasers, and policy makers) to make informed decisions on specific health practices. CER aims to identify and fill knowledge gaps that underlie uncertainties in practice.

• CER may provide information about individual and population benefits, harms, costs, and logistics of different policies or treatments.

• CER will apply to a broad range of interventions, tests, and strategies for prevention, care delivery, and quality of care.

• CER should directly compare tests or active treatments—so-called head-to-head comparisons— of viable clinical alternatives within the current standard of practice (which in some cases may be no intervention).

• CER should primarily assess patient-relevant outcomes, but should also compare the economic implications of different approaches to prevention and care.

• CER should assist patients and physicians to choose between effective treatments. To this end, CER should identify patient characteristics that are associated with meaningful differences in outcomes. Researchers should collect and analyze the data necessary to achieve this goal according to a prespecified plan that clearly indicates specific hypotheses and the methods to test them.

• The scope of CER includes new data, old data newly analyzed, and systematic reviews of existing research.

Standards for the Conduct and Reporting of CER

• CER studies should follow the highest scientific standards for design, analysis, and interpretation, and should adhere to reporting guidelines6 that build upon initiatives to improve the quality and transparency of clinical science.

• Every CER study should have a research protocol, written in advance and addressing the key research question(s), methods, and planned analyses. Researchers should record all changes in the protocol. These protocols should be publicly accessible.

• Patients and other decision makers should be involved in selecting and refining topics for CER.

• The study population for CER should be representative of clinical practice or the relevant public health practice.

• To increase transparency about selective publication, researchers should register CER studies before initiation, in a publicly available registry.

• To increase transparency about the practice of presenting post hoc analyses as conclusive results, study registration should include a clear statement of study hypotheses, outcomes, and analysis plan.

• CER studies must undergo rigorous peer review by independent topical, methodological, and statistical experts.

• To ensure accessibility to the affected public and other researchers, journals (or other sites of publication) should make all CER studies freely available and archive them in a public repository, such as PubMed Central.

• Reports of CER must include a frank discussion of each study’s limitations, including biases, confounding, and scope of applicability.

• Given the potential impact of CER on the profitability of the interventions being evaluated, researchers performing CER studies must commit to stringent and enforceable competing interest policies.⁷

• Researchers, funders, and other contributors to a CER study must clearly state all relevant competing interests at the time of peer review, and publicly upon publication in any forum.

Medical journals are the primary evaluators and disseminators of peer-reviewed health research. As such, they must ready themselves to play a crucial role in advocating for CER, advancing CER methods, and facilitating the translation of CER results into practice. Most importantly, journals and peer reviewers must do their part to ensure that CER, like all research with relevance to health, meets the highest scientific and ethical standards. They must, therefore, develop the methodological and statistical expertise to properly evaluate new or unfamiliar methods of healthcare research.

We recognize that CER has the potential to substantially improve decision making about existing and new approaches to healthcare. To fulfill this potential, researchers must adopt stringent methods, and medical journals must hold them to high standards of ethics, scientific rigor, and reporting.

*The PLoS Medicine Editors are Virginia Barbour, Jocalyn Clark, Susan Jones, Larry Peiperl, and Emma Veitch.

Acknowledgments

The authors acknowledge the following editors who have endorsed this statement: Richard L. Kravitz, MD, and Mitchell D. Feldman, MD, MPhil, Co—Editors-in-Chief, Journal of General Internal Medicine; A. Mark Fendrick, MD, and Michael E. Chernew, PhD, Co—Editors-in-Chief, The American Journal of Managed Care; Arthur M. Feldman, MD, PhD, Editor-in-Chief, Clinical and Translational Science; Ana Maruši , MD, PhD, Editor-in-Chief, Croatian Medical Journal.

Provenance: Written by editors of several research publications including PLoS Medicine; not externally peer reviewed. In order to encourage dissemination of this editorial, this article is freely available in PLoS Medicine and will also be published in Medical Decision Making, Croatian Medical Journal, The Cochrane Library, Trials, The American Journal of Managed Care, and Journal of Clinical Epidemiology. The authors jointly hold the copyright of this article.

Author Affiliations: From the Dartmouth Institute (HCS), Dartmouth Medical School, Hanover, NH; Portland VA Medical Center (MH), Portland, OR; Department of Medicine (MH), Oregon Health & Science University, Portland, OR; Clinical Epidemiology Program (JG), Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; Department of Epidemiology (KD), Johns Hopkins Bloomberg School of Public Health, Baltimore, MD; The Cochrane Library (PLoS Medicine Editors, DT), London, United Kingdom; Department of General Practice (JAK), University of Maastricht, Maastricht, the Netherlands; and Departments of Medicine, and Epidemiology and Community Medicine (PT), University of Ottawa, Ottawa, Ontario, Canada.

Funding Source: The PLoS Medicine Editors are each paid a salary by the Public Library of Science, and they contributed to this editorial during their salaried time. The other authors received no specific funding for this article.

Author Disclosures: Dr Sox is Editor Emeritus, Annals of Internal Medicine. Dr Helfand is Editor, Medical Decision Making. Dr Grimshaw is Director of the Canadian Cochrane Center and Co—Editor-in-Chief, Trials. Dr Dickersin is Director, US Cochrane Center. Dr Tovey is Editor-in-Chief, The Cochrane Library. Drs Knottnerus and Tugwell are Co-Editors, Journal of Clinical Epidemiology.

The PLoS Medicine Editors’ competing interests are at http://www.plosmedicine.org/static/editorsInterests.action. PLoS Medicine is funded partly through manuscript publication charges, but the PLoS Medicine Editors are paid a fixed salary (their salary is not linked to the number of papers published in the journal). The other authors have declared no competing interests.

Author Contributions: ICMJE criteria for authorship read and met: HCS, MH, JG, KD, VB, JC, SJ, LP, EV, DT, JAK, PT. Agree with the manuscript’s results and conclusions: HCS, MH, JG, KD, VB, JC, SJ, LP, EV, DT, JAK, PT. Wrote the first draft of the paper: HCS. Contributed to the writing of the paper: HCS, MH, JG, KD, VB, JC, SJ, LP, EV, DT, JAK, PT.

Copyright: © 2010 Sox et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Address correspondence to: Mark Helfand, MD, MPH, Department of Medicine, Oregon Health & Science University, 3181 SW Sam Jackson Park Rd, Portland, OR 97239-3098. E-mail: helfand@ohsu.edu.

1. Eden J, Wheatley B, McNeil B, Sox H, Editors; Committee on Reviewing Evidence to Identify Highly Effective Clinical Services, Institute of Medicine. (2008) Knowing What Works in Health Care: A Roadmap for the Nation. Washington, DC: National Academy Press. http://www.nap.edu/catalog.php?record_id=12038. Accessed March 26, 2010.

2. Moynihan R, Oxman AD, Lavis JN, Paulsen E. (2008) Evidence-Informed Health Policy: Using Research to Make Health Systems Healthier. Rapport Nr 1-2008. Oslo, Norway: Nasjonalt kunnskapssenter for helsetjenesten.

3. WHO. (2005) WHA 58.34 Ministerial Summit on Health Research. http://apps.who.int/gb/ebwha/pdf_files/WHA58/WHA58_34-en.pdf. Accessed November 30, 2009.

4. UK Department of Health. (2006) Best Research for Best Health: A new national health research strategy. Report 272605. http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_4127127. Accessed March 26, 2010.

5. IOM (Institute of Medicine). (2009) Initial National Priorities for Comparative Effectiveness Research. Washington, DC: National Academy Press. http://www.nap.edu/catalog.php?record_id=12648. Accessed March 26, 2010.

6. Simera I, Altman DG, Moher D, Schulz KF, Hoey J. (2008) Guidelines for reporting health research: The EQUATOR network’s survey of guideline authors. PLoS Med 5: e139. doi:10.1371/journal.pmed.0050139.

7. PLoS Medicine Editors. (2009) Ensuring Integrity in Comparative Effectiveness Research: Accentuate the Negative. PLoS Med 6: e1000152. doi:10.1371/journal.pmed.1000152.

Download PDF: Comparative Effectiveness Research: Challenges for Medical Journals