Real-World Impact of Comparative Effectiveness Research Findings on Clinical Practice

Objectives

Unprecedented funding for comparative effectiveness research (CER) to help provide better evidence for decision making as a way to lower costs and improve quality is under way. Yet how research findings are adopted and applied will impact the nation’s return on this investment. We examine the relationship between the publication of findings from 4 seminal CER trials, the release of subsequent clinical practice guidelines (CPGs), and utilization trends for associated surgical interventions, diagnostic interventions, or medications.

Study Design

Retrospective, observational study.

Methods

Using a large national administrative claims database, we examined time series utilization trends before and after publication of findings from 4 CER trials published within the last decade.

Results

We found no clear pattern of utilization in the first 4 quarters after publication. However, we found that results for 2 of the studies were in concert with the release of CPGs and publication of study results. The trend in intensive statin therapy rose rapidly starting at the end of 2007, while the trend in standard therapy remained relatively constant (PROVE-IT). And, 9 months after trial publication, breast magnetic resolution imaging (MRI) utilization rates rose 43.2%, from 0.033 to 0.048 per 100 enrollees (Mammography With MRI).

Conclusions

Our analysis of 4 case studies supports the call others have made to translate and disseminate CER findings to improve application of research findings to clinical practice and the need for continued development and dissemination of CPGs that serve to synthesize research findings and guide practitioners in clinical decision making. Further research is needed to determine whether these findings apply to different medical topics.

Am J Manag Care. 2014;20(6):e208-e220

Time series utilization trends before and after publication of findings from 4 CER trials published within the last decade revealed no clear pattern of utilization in the first 4 quarters after publication. Results for 2 of the studies (PROVE-IT, Mammography With MRI) were in concert with the release of clinical practice guidelines (CPGs) and publication of study results.

• Our findings support a continued effort to translate and disseminate CER results to improve application of research findings to clinical practice.

• Continued development and dissemination of CPGs to synthesize research findings and guide practitioners in clinical decision making is necessary.

With the Affordable Care Act, public funding for comparative effectiveness research (CER) expands beyond the Agency for Healthcare Research and Quality (AHRQ) and the National Institutes of Health to include the Patient-Centered Outcomes Research Institute (PCORI).¹ According to the Institute of Medicine of the National Academies, “The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve healthcare at both the individual and population levels.”² The return on this investment in CER will be measured by if, and how quickly, results from the research are used in decision making and translated into clinical practice.

One of the most notable examples of the relationship between findings and practice is the Estrogen Plus Progestin trial, a component of the Women’s Health Initiative.³ The trial was stopped in May 2002 after investigators found that the associated health risks of combination hormone therapy among postmenopausal women outweighed the benefits. Almost immediately following publication, rates of hormone therapy in postmenopausal women were shown to drop precipitously, from 90 million prescriptions to 57 million—a 36.7% decrease.⁴

The Estrogen Plus Progestin trial is an example of a study where the evidence resulted in near-term changes in clinical practice. However, beyond a few notable studies,^4-6 researchers have found that it can take several years for study findings to translate to practice.^7,8 Additionally, fewer than 1 study in 1000 is reported by the mainstream media.⁹ Historically, one way that research findings have impacted medical practice has been through the development, dissemination, and use of clinical practice guidelines (CPGs).¹⁰

In this study, we used a large national administrative database to examine real-world utilization trends before and after publication of CER findings (and release of relevant CPGs) from 4 high-profile CER studies published within the last decade. We analyzed changes in utilization rates of procedures and treatments associated with widely communicated CER evidence and CPGs for up to 6 years following the evidence of the publication. Our approach was not normative or prescriptive, but was intended to examine historical trends and resulting changes in practice patterns within a large sample of enrollees with employersponsored health insurance and to discuss the implications of our findings for those seeking to increase the translation of research findings into practice.

METHODS

To distinguish case studies and conditions for our areas of focus, we performed an environmental scan via a literature review. We queried MEDLINE/PubMed,¹¹ Congressional Budget Office publications,¹² and AHRQ CER¹³ studies to find CER evidence reported between January 1, 2000, and December 31, 2009.

Table

eAppendix A

Studies (including randomized controlled trials, metaanalyses, or observational studies) which were highly cited in high-impact medical journals () that compared 2 or more treatment options or reviewed the safety of a drug were selected. Based on these criteria, 23 studies were selected, characteristics of each study were abstracted (eg, type of comparators, primary aim as safety vs efficacy, study design), and then each was reviewed by the study team to determine which should be included in the quantitative analysis (, available at www.ajmc.com). The main criteria for the selected case studies were (1) high-profile study of a medication, surgical, or diagnostic intervention (published in a high-impact journal as ranked by ISI Web of Knowledge)¹⁴; and (2) widely cited results (ISI Web of Knowledge¹⁴ and Google Scholar citations). We also focused on quality criteria that included, as reviewed by the study team: (1) study design (adequate sample size with an experimental design); (2) clear findings (lack of ambiguity in the implications); and (3) findings that were not reversed or contradicted by subsequent evidence. Cost-effectiveness studies were excluded.

Of these, the study team selected 4 case studies that represented varying types of trials: 1 study compared medical therapies (PROVE-IT TIMI 22), 2 compared surgical versus nonsurgical treatments (COURAGE and SPORT) and 1 compared diagnostic screening procedures (called Mammography With MRI here for simplicity; see reference 20 for actual title).

eAppendix B

We also examined the first release of related CPGs, following the publication of trial results. To determine when guidelines might have started to influence uptake or discontinuation of a specific practice related to the clinical trial, we conducted an environmental scan via a literature review. The criteria for the scan limited results to articles that focused on clinical guidelines and that specifically cited 1 of the 4 seminal studies in relationship to guideline development. We queried Google Scholar for publications following the release of the trial(s) through the end of our study period in 2010. We then eliminated nonguidelines. We also conducted a keyword search of guidelines published by the AHRQ National Guidelines Clearinghouse for the same time period. From the combined list we retrieved the first clinical guideline (from search results), corresponding to each trial ().

Data Source and Analysis

This analysis was based on data contained in the Truven Health MarketScan Commercial Claims and Encounters Database and the MarketScan Medicare Supplemental and Coordination of Benefits Database for the period of January 1, 2003, to June 30, 2010. The MarketScan Database includes the enrollment, inpatient, outpatient, and outpatient pharmacy claims experience of tens of millions of individuals across the nation with employer-sponsored insurance or employer-sponsored Medicare Supplemental insurance. Health insurance was offered through a variety of capitated and fee-for-service health plan types, and prescription drug coverage was offered in conjunction with the medical benefit. Its sample size is large enough to allow creation of a nationally representative data sample of Americans with employer-provided health insurance. The enrollment characteristics in MarketScan are largely similar to nationally representative data for employer-sponsored insurance in the Medical Expenditure Panel Survey (MEPS), although a higher percentage of MarketScan enrollees reside in the South census region. The data cover all 50 states and the District of Columbia. Used primarily for research, these databases are fully Health Insurance Portability and Accountability Act compliant.¹⁵

To study trends in the utilization rates for each case study, we calculated the quarterly utilization rates of each procedure (by calendar quarter). First, we selected the continuous cohort of firms contributing data to MarketScan each year from 2003 through 2010 Q1, representing approximately 4 million enrollees annually (eAppendix C), so trends would not be impacted by firms entering and leaving the sample. Within these firms, each enrollee selected for inclusion in the study had medical and pharmacy claims data available, was 18 years or older, was continuously enrolled in the 4 calendar quarters prior to the quarter of interest, and was not pregnant (no diagnosis of pregnancy).

eAppendix D

Second, within this cohort of enrollees, we replicated (as was possible using claims data) the inclusion and exclusion criteria used in each clinical trial (based upon publication or online trial-specific materials) for patient selection (See the Table for summarized criteria and for detailed criteria and codes). To determine these sets of criteria, a nosologist (certified clinical coding expert) provided clinical codes (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] diagnosis codes, ICD-9-Clinical Modification [ICD-9-CM] procedure codes, and Current Procedural Terminology, 4th Edition [CPT-4] procedure codes) for each of the inclusion and exclusion criteria and the procedures listed in the clinical trial. As part of an iterative process, these codes were reviewed by a clinician expert in clinical coding and the resulting list was reviewed a second time by the nosologist. Through a consensus-driven process, any differences between the clinician and the nosologist were resolved in consultation with the project manager, and by 2 other clinicians. A similar process was employed for creating a list of pharmaceutical codes, incorporating a review of National Drug Codes by a licensed pharmacist.

eAppendix D

The study clinicians relaxed some of the criteria (see for a description of how these criteria were relaxed) designated in the trial to focus on patients who were similar to those targeted by very specific study criteria for 2 reasons. First, clinical coding in administrative data sets can lack precision.¹⁶ Second, we hypothesized that these results would have spillover effects on clinical practice beyond patients meeting the restrictive clinical inclusion criteria used in the respective clinical trial. Therefore, we focused on patients highly similar to those included in the trial. For example, for the Mammography With MRI study, we did not require women to have a diagnosis on a claim indicating that they carried the BRCA1 or BRCA2 mutation, because test results are infrequently available in claims analyses.¹⁷ For the SPORT case study, we did not exclude enrollees if they had a previous lumbar surgery. The relaxed criteria were then applied to the data set to find the set of enrollees in each calendar quarter meeting the patient selection criteria (the denominator). For the pre-period for each study, we required patients to be continuously enrolled at least 1 year prior to the quarter, to check for a number of inclusion and exclusion criteria. Claims incurred during this year were used to evaluate inclusion and exclusion criteria (eAppendix D). Once meeting study requirements, patients were followed until the end of their continuous enrollment, or until evidence appeared that they met the exclusion criteria, or until completion of 2 years of follow-up (except in the Mammography With MRI study, where patients were followed through the end of their continuous enrollment).

Finally, a list of the treatments or procedures of interest (eg, breast magnetic resonance imaging [MRI]) was created (numerator of each measure). Within the enrollees meeting the selection criteria, enrollees receiving each treatment or procedure were flagged in the database and the number of occurrences of the treatment or procedure of interest (the numerator) was recorded (note: eAppendix D contains the clinical criteria used in each case study). Using this information, we calculated aggregate quarterly trends in utilization rates for each treatment or procedure.

eAppendix E

We present the descriptive findings in 2 formats. First, we display graphical trends before and after the publication of the CER evidence and guideline release. We show the timing of the first national CPG that referenced the study. Second, we display utilization rates in tabular form at 3- and 6-month intervals and calculate the percent change from the prior value at each time interval. As a separate analysis, we replicated the study results using a stricter set of criteria using claims data, described in the clinical trial (without relaxation). These results represent a smaller group of enrollees ().

RESULTS

A consistent cohort of 66 data contributors (large firms and insurers) contributing data throughout the entire study period was selected for analysis, representing approximately 7 million enrollees each year. Between 2003 and 2010, the number of patients meeting the clinical trial inclusion criteria ranged from 86,080 for PROVE-IT to 5,159,253 for Mammography With MRI. Patient characteristics of these subjects are detailed in eAppendix C. All reported results reflect the relaxed version of the original study criteria.

The rest of this paper summarizes the main findings from each case study and the temporal trends in each study, in chronological order of publication. Figures 1 through 4 show the quarterly utilization trends for each case study.

PROVE-IT (2004)

The PROVE-IT trial¹⁸ investigated whether standard therapy with 40 mg of pravastatin daily or intensive therapy with 80 mg of atorvastatin daily decreased a composite outcome, including all-cause mortality, myocardial infarction (MI), unstable angina, revascularization, and stroke among patients who had been recently hospitalized for acute coronary syndrome. Patients who obtained intensive lipid-lowering statin therapy had a lower risk of death or cardiovascular events than those on a standard statin regimen. The literature search yielded 24 CPGs published during the review period (conclusion of the study through 2010). The first guideline was published in 2004 by the National Cholesterol Education Program.¹⁹ The CPG called for more aggressive lowering of low-density lipoprotein-cholesterol, and was published in response to 5 new statin trials, including PROVE-IT. Based upon these results, we hypothesized that after the release of the PROVE-IT trial results and publication of the CPG, prescription utilization rates (days supplied per person per quarter) would increase for patients on intensive statin therapy and decrease with standard statin therapy.

We found the use of intensive statin therapy increased; however, the increase in utilization started prior to the publication of PROVE-IT in 2004. After the initial rise, the trend in intensive therapy flattened for the next 3 years (after publication of results and release of the subsequent CPG) and rose rapidly starting at the end of 2007. In contrast, the trend in standard therapy remained relatively constant prior to publication and through the 3 years after publication of results and release of the CPG. Utilization of standard therapy declined starting at the end of 2007 and then leveled off after early 2009 (Figure 1).

MAMMOGRAPHY WITH MRI (2004)

The Mammography With MRI study²⁰ evaluated the sensitivity and specificity of 4 breast cancer surveillance methods: mammography, ultrasound, magnetic resonance imaging (MRI), and a clinical breast exam among carriers of BRCA1 or BRCA2 mutations. The sensitivity of MRI to detect breast cancer was highest, with 77% of all cancers detected found by MRI compared with 36% by mammography, 33% by ultrasound, and 9.1% by clinical breast exam.

The literature search yielded 15 CPGs published during the review period. The first CPG was published in 2007 by the American Cancer Society.²¹ This updated CPG added annual MRI screening to mammography for women with greater than or equal to a 20% to 25% lifetime risk. Based on the results of this trial, we hypothesized that MRI utilization rates among women would increase.

We found that mammography rates (number of procedures per 100 enrollees) remained relatively consistent (with seasonal variation) and breast ultrasound utilization rates rose steadily over time. Compared with the other imaging technologies of mammography and ultrasound, there was a steady increase in MRI utilization prior to and following publication in 2004 through the end of 2006, and then a sharper increase after the CPG was released in 2007 Q2. In mid-2005, 9 months after publication, breast MRI utilization rates rose 43.2% from 0.033 to 0.048 per 100 enrollees. Despite the increase in utilization, the absolute rate of MRI utilization was far less than the utilization of ultrasound or mammography (0.048 vs 1.26 and 9.91 per 100 enrollees at 9 months after publication of the results and 9 months after release of the CPG) (Figure 2).

SPORT (2006)

The Spine Patient Outcomes Research Trial (SPORT)²² was designed to assess whether a standard open diskectomy was more effective than non-operative treatment at decreasing pain and improving physical function among patients with intervertebral disk herniation. Additional outcome measures were patient self-reported improvement, work status, symptom satisfaction, and care satisfaction. Intent-to-treat analysis did not find statistically significant differences in pain reduction or physical function, but found differences in measures of sciatica severity and self-reported improvement. However, nonadherence to the assigned treatment was high. After 2 years of follow- up, 60% of the patients randomized to the surgical intervention and 45% of the patients randomized to usual care had surgery. Larger beneficial effects from surgery were seen for the as-treated analysis.

The literature search yielded 4 guidelines citing trial evidence. In 2007, the College of Physicians and the American Pain Society published the first CPG.²³ The SPORT trial was one of many back pain studies cited in this guideline release.

Based on the findings of this trial, we hypothesized that standard diskectomy utilization rates (percent of eligible enrollees with the procedure) would increase after publication of the results in 2006 Q4 and release of the CPG in 2007 Q2. However, we found that standard diskectomy rates remained relatively consistent (remaining around 1 diskectomy per 100 enrollees) after release of the SPORT trial results (Figure 3).

COURAGE (2007)

The COURAGE trial,²⁴ implemented among patients with stable coronary artery disease, was designed to determine whether percutaneous coronary intervention (PCI) with optimal medical therapy (OMT) decreased the risk of death, nonfatal MI, or other major cardiovascular events compared with patients solely treated with OMT. COURAGE found that PCI with OMT did not reduce the risk of these study outcomes compared with OMT alone.

The literature search yielded 13 CPGs citing the trial evidence. In 2007, the American College of Cardiology/ American Heart Association published the first CPG that referenced the trial. The guideline update recommendations explicitly reflected “consensus of expert opinion following a thorough review primarily of late-breaking clinical trials…”²⁵ Based upon the trial results, we hypothesized that PCI utilization rates would not change or would decrease after the release of results in 2007 Q2. We do not present results on trends in OMT because not all of the therapies required are observable in claims (ie, aspirin). However, trends in OMT,that are observable in claims reflect no change in use after publication of results. We found the rates of PCI oscillated over time in the 2 years after publication. (Figure 4).

As a separate analysis for each of the 4 case studies, we used strict inclusion criteria as close as possible to the criteria defined in the trial (no relaxation of criteria). The trends are not directly comparable because of the limitations of the codes available in claims for some of these criteria, and the smaller sample sizes (eAppendix E).

DISCUSSION

To examine the relationship between CER evidence and practice we analyzed pre- and post publication utilization trends related to treatments or diagnostic procedures compared in 4 high-quality, high-profile, published CER studies. In all cases, we found no clear pattern in utilization in the first 4 quarters after publication of these case studies. However, in 2 cases, PROVE-IT and Mammography With MRI, there appeared to be an association between evidence and practice: changes in lipid-lowering therapy and MRI trends began prior to publication of study results and persisted after publication of results and release of CPGs. In SPORT, diskectomy utilization rates did not change; in COURAGE, PCI utilization rates varied.

While we studied trends among a very large cohort of enrollees with commercial insurance provided by large and medium-sized firms, a few studies analyzed similar trends in different patient groups. Howard and colleagues used claims data from the US community, the Veterans Administration, and from England to study the impact of the impact of the COURAGE trial on the volume of PCI procedures. Using data from 2001 to 2008, they found a decline in PCI, but the lower rates were not sustained over time based upon inpatient discharge data 21 months after publication of COURAGE.⁶

In addition, Austin and colleagues analyzed claims data for statin prescribing patterns among adults 65 years and older in Ontario from 1997 to 6 months following publication of PROVE-IT in 2004. The authors found a significant, sustained increase in the use of high-dose atorvastatin compared with moderate statin therapies.²⁶ We found a similar increase in the use of intensive therapy in our patient sample in the 6 months following publication, which was a continuation of the pre-publication trend. Use of standard therapy was relatively unchanged. We found no studies assessing trends in breast MRI or SPORT.

Some of the findings in our analysis may be explained by the design or context surrounding the 4 trials. First, all 4 studies that we examined reviewed treatments or procedures that were in widespread use at the time of the trial. For example, COURAGE and SPORT addressed the comparative effectiveness of surgery versus non-operative care using clinical and economic outcomes. Secondly, the trial findings concluded with clinical evidence in support of one therapy over another. While informative, these types of studies may be less influential than studies that report results from an investigation of a new treatment or drug, or studies that show that treatments may be harmful, such as the Women’s Health Initiative, which in 2002 warned against overuse of hormone therapy after menopause.²⁷

The Role of Clinical Guidelines

The results from each of the 4 clinical trials in our study added to the body of knowledge from previously published clinical practice guidelines (eg, cholesterol management); and, in all cases, there were specific CPG revisions associated with each study. We investigated the association between the timing of relevant clinical guideline releases (citing evidence from these trials) and the observed utilization trends to determine whether there appeared to be an impact of guideline release on trends.

For PROVE-IT, the 2004 update to the National Cholesterol Education Program’s²⁸ clinical practice guidelines on cholesterol management advised physicians to consider new, more intensive treatment options for people at high risk (treatment goal shifted from less than 100 mg/dL to less than 70 mg/dL for very high risk patients).¹⁹ This guideline was published shortly following the release of PROVE-IT and built upon prior studies such as MIRACL²⁹ and the Heart Protection Study.³⁰ This preceded the large uptick we observed in intensive statin therapy starting in 2007.

More than 2 years following the publication of the Mammography With MRI study, the American Cancer Society updated guidelines for breast cancer screening to include MRI as an adjunct to mammography for specific risk groups.²¹ This guideline update occurred after the observed increase in MRI in 2007. With respect to low back pain, between the seminal guideline in 199431 and the next major update in 2007,²³ the SPORT trial was one of many back pain studies cited in this guideline release.²³ Finally, in the case of the COURAGE trial, results confirmed a premise that PCI does not reduce the risk of death, MI, or other major cardiovascular events when added to “optimal medical therapy” compared with optimal medical therapy alone.³² The first guideline citing this trial was released in 2007; however, the update to the guidelines on stable angina, incorporating specific results from COURAGE, did not occur until 2011,²⁵ more than 3 years after the publication of the trial.

While experts generally acknowledge that some lag is necessary to ensure the safety and efficacy of new interventions, there is also a desire to optimize lags. Additional research is needed to understand the relationship between research outcomes and the dissemination of clinical findings.

IMPLICATIONS

Our study provides a descriptive, historical review of utilization rates over time and of translation into practice for cohorts of patients that were most likely to be affected by the evidence. We did not test statistical significance and we do not attempt to trace a causal path between the releases of evidence and realized clinical practice. In this section, we examine factors that are salient in a rapidly learning healthcare system where evidence and practice have close ties, including: stakeholder engagement, evidence clarity and maturity, timing, dissemination, knowledge translation, and incentives.

Stakeholder Engagement From Research Question to Development

In evaluating whether the evidence is adopted, we make the assumption that the right questions are posed and assessed. In many cases, the evidence may be based upon outcomes that (1) are not meaningful to the decision makers and other stakeholders, (2) evaluate irrelevant comparators, or (3) lack typical care settings or patients to inform clinical practice^.33 Stakeholder engagement from the early phases of research through conduct and interpretation may aid study design and recruitment.³⁴ For example, new projects have sought to include stakeholders to prioritize research on genomic tests in cancer to improve the relevance of future studies.³⁵

Evidence Clarity and Maturity

Few single studies have resulted in a wholesale change in clinical practice. To effect such a change, evidence generally must be cumulative, requiring multiple confirming studies, particularly when the evidence is conflicting with current practice.³⁶ Clarity and strength of research findings may be a function of the evidence maturity: the cumulative strength and concordance of the evidence. For example, even in the case of COURAGE, where the results were widely disseminated, the analysis continues to evolve. In 2011, four years after the publication of COURAGE, the National Heart, Lung and Blood Institute (NHLBI) released an $84 million grant (ISCHEMIA) to study initial management strategies for patients with coronary artery disease. While the COURAGE trial emphasized the importance of OMT, the progression to new trials such as ISCHEMIA indicates a desire within the medical community for corroborating studies regarding treatment for this patient population.³⁷

Timing

The conduct of research may lead clinical practice, or may actually lag the current clinical practice, as a confirmatory approach. Much has been written about whether research itself influences practice in a linear fashion, or if research confirms expert opinion, the current practices, and the experience of routine practice.³⁸ As an example, intensive statin therapy was already increasing prior to the publication of PROVE-IT, and prior studies had investigated the effects of intensive statin therapy in acute coronary syndrome (MIRACL).²⁹

Dissemination and Policy

The process of disseminating research findings to patients and clinicians is complex.³⁹ Practitioners often look to specialty societies, researchers, or other guideline writing groups for help interpreting evidence that is vast and sometimes confusing.⁴⁰ CPGs are a necessary first step to synthesize the evidence and serve as a catalyst for uptake. In the case of COURAGE, the amount of publicity in the medical and popular press was such that it seemed reasonable to expect a more significant timely increase in the application of study findings to clinical practice.²⁷ Perhaps, however, the lag in development of a CPG that included recommendations based on COURAGE played a role. Specific mechanisms for dissemination, including written reports, professional presentations, training programs, consumer guides, and marketing campaigns, have all been identified as important components of disseminating CER results.⁴¹

Further funding and research is needed to promote innovative approaches for disseminating CER results, and partnering may be needed to enable adoption of evidence.⁴¹

Knowledge Translation

The need to understand what causes the lag in adoption or abandonment of treatment options, as well as which approaches to translation work best, has sparked an area of research referred to as knowledge translation.

In recent years, several knowledge translation studies have identified some explicit factors of translation.^42,43 For example, Lang and colleagues present a model for closing the evidence-to-practice gap moving from research to practice: (1) organization of the evidence, synthesis of research, and guidelines; (2) bedside evidence-based medicine (EBM), including access to decision rules and locally agreed-on clinical practice guidelines (eg, via handheld computers); (3) continuous quality improvement initiatives designed to support adherence to EBM and guidelines; and (4) the use of decision aids, patient education, and compliance aids.⁴⁴

Incentives

Misalignment of financial incentives with CER evidence has been suggested as one of the primary root causes of incomplete translation of results,⁴⁵ and this misalignment affects many stakeholder groups, including providers, patients, insurers, professional societies, and medical manufacturers. For example, the predominant fee-for-service payment structure rewards the performance more procedures and services, even sometimes when there is not sufficient evidence of clinical value.⁴⁶ By aligning incentives to reimburse providers based on what is known as best practice, we create a more favorable environment for the adoption of CER in the health system.⁴⁷ A variety of strategies should be considered to encourage high-value care, including value-based insurance benefit design,⁴⁸ shared savings models; and bundled, severityadjusted payments.⁴⁶

LIMITATIONS

This study is not without limitations. It is possible that factors other than the CER evidence will change the trend in utilization and spending for patients in the sample. Or, if more contemporary data had been included, different trends may have emerged. For example, over time, practice might have been informed by the singular study. In other cases, a study may be part of a growing body of evidence that translates to slower changes, and others still require synthesis of the evidence (through CPGs or others) to ultimately result in knowledge translation. And, although we selected seminal trials for review, we only reviewed 4 studies.

As with any data source, claims data have limitations in coding and the amount of clinical information available. In addition, the MarketScan sample population is based on a large convenience sample from large employers. In addition, since the study was not conducted on the patient level, factors that may be associated with the various outcomes cannot be adjusted for in analysis.

Using administrative claims, it is difficult to exactly replicate the study-specific inclusion and exclusion criteria as defined in the original trials. For example, we found that in the case of COURAGE, some criteria are not available on claims (eg, vessel inclusion classified as too severe vs less severe). In other cases, the criteria may overlap (eg, PCI is in the inclusion criteria and the exclusion criteria for COURAGE—exclude patients if a PCI within 6 months). Other studies have attempted to replicate the study-specific criteria as defined in the COURAGE trial, and we have used the codes in those analyses as a starting point for coding criteria.49 The goal of our study was to use administrative claims to perform an observational analysis of utilization rates in clinically relevant patient populations. Details for the modifications to the inclusion and exclusion criteria for each of the 4 studies in our analysis are included in eAppendix D.

Finally, these findings may not be generalizable to the general US population, as persons who are uninsured, employed by small firms, or on Medicaid are not included in our sample.

CONCLUSION

CER evidence can increase the understanding and utilization of the most effective healthcare treatments, interventions, and procedures likely to result in improvements in outcomes and quality of care. Differences exist between evidence-based findings and the timing and adoption of the 4 studies examined. With increasing funding there is great potential in CER to address these differences, with an opportunity to fund areas that have gaps in scientific evidence. Specifically, future research could explore the potential heterogeneity in the adoption of CER findings. For example, if there are differences in trends in urban areas versus rural areas, Southeast versus Northwest regions of the United States, integrated health systems versus vertical care systems, or other stratifications, these could be used as a proxy for practice patterns and for lessons on knowledge translation. This study demonstrates that evaluating the impact on clinical practice, based on results of published CER trials and CPGs, is complex. Given the ill-defined nature of clinical practice and decision making, other nonevidence factors that influence clinician and patient uptake of effective medical services (eg, generic availability, payment patterns, public awareness campaigns) are worth exploring. As new mechanisms and policies are developed to increase stakeholder engagement, prioritize research, and proactively plan for dissemination, change may occur.

The greatest value of CER is the development of evidence that is translated and implemented in practice.50 In the future, more of this type of analysis will occur, since by legislative statute, a review will occur examining the “extent to which research findings are used by healthcare decisionmakers, the effect of the dissemination of such findings on reducing practice variation and disparities in healthcare…”¹ Similar evaluations should continue to be conducted to assess the impact of CER on utilization and practice.Author Affiliations: Truven Health Analytics, Ann Arbor, MI (TBG, EDE, AMF); National Pharmaceutical Council, Washington, DC (JG); Harvard Medical School, Boston, MA (MEC); The University of Michigan (AMF).

Source of Funding: This study was funded by the National Pharmaceutical Council.

Author Disclosures: Dr Gibson, Ms Ehrlich, and Ms Farr report employment with Truven Health Analytics, which has received consulting fees from the National Pharmaceutical Council. Dr Dubois reports employment with the National Pharmaceutical Council. The other authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (TBG, RD, AMF, MEC, AMF); acquisition of data (TBG, AMF); analysis and interpretation of data (TBG, EDE, RD, AMF, AMF); drafting of the manuscript (TBG, EDE, RD, AMF); critical revision of the manuscript for important intellectual content (TBG, RD, AMF, MEC, AMF); statistical analysis (TBG, MEC); provision of study materials or patients (TBG); obtaining funding (TBG, RD); administrative, technical, or logistic support (TBG, EDE); supervision (TBG, RD).

Address correspondence to: Teresa B. Gibson, PhD, Truven Health Analytics, 777 E Eisenhower Parkway, Ann Arbor, MI 48108. E-mail: teresa. gibson@truvenhealth.com.1. Subtitle D - Patient-Centered Outcomes Research. Pub L No. 111-148, 124 Stat 727. http://www.gao.gov/about/hcac/pcor_sec_6301.pdf. Accessed July 25, 2012.

2. Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. 2009.

3. Writing Group for the Women’s Health Initiative Investigators. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women’s Health Initiative randomized controlled trial. JAMA. 2002;288(3):321-333.

4. Hersh AL, Stefanick ML, Stafford RS. National use of postmenopausal hormone therapy. JAMA. 2004;291(1):47-53.

5. Stafford RS, Bartholomew LK, Cushman WC, et al. Impact of the ALLHAT/JNC7 Dissemination Project on thiazide-type diuretic use. Arch Intern Med. 2010;170(10):851-858.

6. Howard DH, Shen YC. Comparative effectiveness research, COURAGE, and technological abandonment. NBER Work Pap Ser. August 2011.

7. Green LW, Ottoson JM, Garcia C, Hiatt RA. Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annu Rev Public Health. 2009;30:151-174.

8. Lenfant C. Clinical research to clinical practice—lost in translation? N Engl J Med. 2003;349(9):868-874.

9. Suleski J, Ibaraki M. Scientists are talking, but mostly to each other: a quantitative analysis of research represented in mass media. Public Underst Sci. 2010;19(1):115-125.

10. Davis DA, Taylor-Vaisey A. Translating guidelines into practice: a systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ. 1997;157:408-416.

11. PubMed Research Center. United States National Library of Medicine. www.ncbi.nlm.nih.gov/pubmed/. Published January 1996. Accessed May 21, 2012.

12. Congress of the United States - Congressional Budget Office. Research on the Comparative Effectiveness of Medical Treatments. Congressional Budget Office website. http://www.cbo.gov/sites/default/ files/cbofiles/ftpdocs/88xx/doc8891/12-18-comparativeeffectiveness. pdf. December, 2007.

13. Agency for Healthcare Research & Quality website. http://www. effectivehealthcare.ahrq.gov/. Accessed January 2, 2014.

14. ISI Web of Knowledge Database. http://apps.webofknowledge.com. Accessed May 21, 2012.

15. Hansen LG, Chang S. Health research data for the real world: the MarketScan databases. Truven Health Analytics. Published July 2012.

16. Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems: implications for outcomes research. Ann Intern Med. 1993;119(8):844-850.

17. Schneeweiss S, Avorn J. Using healthcare utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005;58(4): 323-337.

18. Cannon CP, Braunwald E, McCabe CH, et al. Intensive versus moderate lipid lowering with statins after acute coronary syndromes. N Engl J Med. 2004;350(15):1495-1504.

19. Grundy SM, Cleeman JI, Merz CNB, et al. Implications of recent clinical trials for the National Cholesterol Education Program Adult Treatment Panel III Guidelines. J Am Coll Cardiol. 2004;44(3):720-732.

20. Warner E, Plewes DB, Hill KA, et al. Surveillance of BRCA1 and BRCA2 mutation carriers with magnetic resonance imaging, ultrasound, mammography, and clinical breast examination. JAMA. 2004;292(11): 1317-1325.

21. Saslow D, Boetes C, Burke W, et al. American Cancer Society guidelines for breast screening with MRI as an adjunct to mammography. CA Cancer J Clin. 2007;57(2):75-89.

22. Weinstein JN, Lurie JD, Tosteson TD, et al. Surgical vs. nonoperative treatment for lumbar disk herniation: the Spine Patient Outcomes Research Trial (SPORT): a randomized trial. JAMA. 2006;296(20): 2441-2450.

23. Chou R, Qaseem A, Snow V, et al; Clinical Efficacy Assessment Subcommittee of the American College of Physicians; American College of Physicians; American Pain Society Low Back Pain Guidelines Panel. Diagnosis and treatment of low back pain: a joint clinical practice guideline from the American College of Physicians and the American Pain Society. Ann Intern Med. 2007;147(7):478-91.

24. Boden WE, O’Rourke RA, Teo KK, et al. Optimal medical therapy with or without PCI for stable coronary disease. N Engl J Med. 2007; 356(15):1503-1516.

25. King, S. B., Smith, S. C., Hirshfeld, J. W., et al. 2007 Focused Update of the ACC/AHA/SCAI 2005 Guideline Update for Percutaneous Coronary Intervention A Report of the American College of Cardiology/ American Heart Association Task Force on Practice Guidelines: 2007 Writing Group to Review New Evidence and Update the ACC/AHA/ SCAI 2005 Guideline Update for Percutaneous Coronary Intervention, Writing on Behalf of the 2005 Writing Committee. Circulation, 2008;117(2):261-295.

26. Austin PC, Mamdani MM. Impact of the pravastatin or atorvastatin evaluation and infection therapy-thrombolysis in myocardial infarction 22/reversal of atherosclerosis with aggressive lipid lowering trials on trends in intensive versus moderate statin therapy in Ontario, Canada. Circulation. 2005;112(9):1296-1300.

27. Winstein KJ. A simple health-care fix fizzles out. The Wall Street Journal. February 11, 2010. http://online.wsj.com/article/SB1000142405 2748703652104574652401818092212.html.

28. ATP III Update 2004: implications of recent clinical trials for the ATP III guidelines. http://www.nhlbi.nih.gov/guidelines/cholesterol/ atp3upd04.htm. Accessed July 25, 2012.

29. Schwartz GG, Olsson AG, Ezekowitz MD, et al. Effects of atorvastatin on early recurrent ischemic events in acute coronary syndromes: the MIRACL study: a randomized controlled trial. JAMA. 2001;285(13):1711-1718.

30. Heart Protection Study Collaborative Group. MRC/BHF Heart Protection Study of cholesterol lowering with simvastatin in 20 536 high-risk individuals: a randomised placebo-controlled trial. Lancet. 2002;360:7-22.

31. Agency for Healthcare Policy and Research (AHCPR). Acute Low Back Problems in Adults - Clinical Practice Guideline. In: US Department of Health and Human Services. 1994:160.

32. Gibbons RJ, Abrams J, Chatterjee K, et al. ACC/AHA 2002 Guideline Update for the Management of Patients with Chronic Stable Angina - Summary Article: A Report of the American College of Cardiology/ American Heart Association Task Force on Practice Guidelines (Committee on the Management of Patients with Chronic Stable Angina). J Am Coll Cardiol. 2003;41(1):159-168.

33. Green LW. Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence? Fam Pract. 2008; 25(suppl 1):i20-i24.

34. Lomas J, Fulop N, Gagnon D, Allen P. On being a good listener: setting priorities for applied health services research. Milbank Q. 2003;81:363-388.

35. Thariani R, Wong W, Carlson JJ, et al. Prioritization in comparative effectiveness research: the CANCERGEN experience. Med Care. 2012;50(5):388-393.

36. Haas JS, Kaplan CP, Gerstenberger EP, Kerlikowske K. Changes in the use of postmenopausal hormone therapy after the publication of clinical trial results. Ann Intern Med. 2004 3;140(3):184-188.

37. Ischemia. Ischemia Trial website. https://www.ischemiatrial.org/. Accessed February 2013.

38. Heidenreich P, McClellan M. Biomedical research and then some: the causes of technological change in heart attack treatment. In: Murphy K, Topel R, eds. Measuring the Gains from Medical Research: An Economic Approach. Chicago, IL: University of Chicago Press; 2003:163-205.

39. Glasgow RE, Vinson C, Chambers D, Khoury MJ, Kaplan RM, Hunter C. National Institutes of Health approaches to dissemination and implementation science: current and future directions. Am J Public Health. 2012;102:1274-1281.

40. Hubbard T, Daimyo S, Desai K. From evidence to practice: a national strategy for CER dissemination; NEHI 2011: February.

41. Government Accounting Office (GAO). Comparitive effectiveness. Agency for Healthcare Research and Quality’s Process for Awarding Recovery Act Funds and Disseminating Results. GAO-12-332. February 2012.

42. Davis D, Evans M, Jadad A, et al. The case for knowledge translation: shortening the journey from evidence to effect. BMJ. 2003;327: 33-35.

43. Jacobson N, Butterill D, Goering P. Development of a framework for knowledge translation: understanding user context. J Health Serv Res Policy. 2003;8(2):94-99.

44. Lang ES, Wyer PC, Haynes RB. Knowledge translation: closing the evidence-to-practice gap. Ann Emerg Med. 2007;49(3):355-363.

45. Timbie J, Schneider E. Dissemination and Adoption of CER Findings that Challenge Current Practices Academy Health Annual Research Meeting. June 24, 2012.

46. O’Kane M, Corrigan J, Foote S, et al. Crossroads in quality. Health Aff (Millwood). 2008;27(3):749-758.

47. Garber AM. Evidence-based coverage policy. Health Aff (Millwood). 2001;20:62-82.

48. Chernew ME, Fendrick AM. The business case for comparative effectiveness research: learning what works best: the nation’s need for evidence on comparative effectiveness in healthcare. Institute of Medicine. March 2007. Appendix 7:1-13.

49. Weintraub WS, Spertus JA, Kolm P, et al. Effect of PCI on quality of life in patients with stable coronary disease. N Engl J Med. 2008;359(7): 677-687.

50. Sox HC, Greenfield S. Comparative effectiveness research: a report from the Institute of Medicine. Ann Intern Med. 2009;151:203-205.

Download PDF: Real-World Impact of Comparative Effectiveness Research Findings on Clinical Practice