Pragmatic Clinical Trials: US Payers' Views on Their Value

Background:

Randomized controlled trials (RCTs) reflect priorities established by regulators. Recently, pragmatic clinical trials (PCTs) have begun to attract interest. Unlike RCTs, PCTs aim to better inform post-regulatory decision making by using head-to-head comparisons of alternative treatments, diverse patient populations, and outcomes meaningful to patients, prescribers, and payers.

Objectives:

To describe how US insurers and public payers perceive the value of PCTs for assessment of new prescription drugs.

Study Design:

Criterion-based sample of US insurers and public payers.

Methods:

We gathered qualitative evidence from intensive interviews with formulary decision makers at 15 payers, representing 10 major types of US payers. Prior literature and exploratory interviews informed our question selection.

Results:

Payers viewed PCTs favorably despite wariness of drug company—sponsored trials. Payers would accept results from PCTs as part of payers’ synthesis of multiple sources of evidence. Payers were enthusiastic about 2 PCT features—a diverse population (compared with the more homogeneous populations typical of RCTs) and an active comparator drug (not placebo). Payers did not anticipate that PCTs would displace their own analyses of internal data. Pharmaceutical companies’ financial interest in obtaining trial results that favor their own drugs reduces PCTs’ perceived value and dampens their appeal to payers; nonetheless, payers would seek PCT results and review them carefully, as they do other evidence.

Conclusions:

Recommendations to trial designers based on payers’ views include tailoring different types of PCTs to different disease conditions, building in head-to-head comparisons in phase IIIb PCTs, and designing phase IV PCTs to include broader populations.

Am J Manag Care. 2013;19(5):e158-e165US payers desire both scientific rigor in clinical trial design and real-world applicability of trial results. Payers find pragmatic clinical trials (PCTs) appealing because PCTs provide:

• Information that allows evaluation of the comparative value of drugs against active comparators.

• Opportunities to triangulate PCTs’ evidence of effectiveness from real-world settings with data from regulatory RCTs and internal claims data sets.

• Health outcomes for patients who more closely reflect the diversity of patients they enroll (eg, patients with comorbidities, minorities), as well as meaningful data on subpopulationsfor which a drug works.

Interest in comparative-effectiveness research (CER) suggests a need to address both regulatory requirements and requests from post-regulatory decision makers (eg, payers, patients) for evidence of the relative value of comparator treatments. Pragmatic clinical trials (PCTs) have been proposed as potential means for satisfying both regulators and post-regulatory decision makers.¹ Although the definition of PCTs is emerging, a recent definition is indicative:

The RCTs whose explicit purpose is to be most informative to decision makers are called pragmatic or practical clinical trials (PCTs). . . . Common elements of such trials include clinicallyeffective comparators, study patients with common comorbid conditions and diverse demographic characteristics, and providers from community settings. Primary and secondary outcomes are patient-centered, chosen to reflect what matters most to patients and clinicians.^1(p208)

Examples of PCTs include head-to-head comparisons of different pharmacologic agents for treatment of various conditions such as hypercholesterolemia² and acute bipolar mania.³

Randomized controlled trials (RCTs) have long been considered the highest level of evidence on clinical efficacy.⁴ However, regulatory trials historically have focused on narrow patient populations and used placebo comparisons. In contrast, PCTs aim to answer meaningful scientific questions about investigational drugs via study designs with (1) broad patient populations, (2) active comparators, or both. Traits of PCTs may also include (3) outcomes of interest to patients, prescribers, and payers⁵; (4) usual care (in contrast to a strict clinical protocol); and (5) fewer data collection elements than RCTs have (to reduce burden of data collection). Our study defines a PCT as a randomized trial with at least 2 of these 5 “pragmatic” traits; we used this working definition to create a shared frame of reference with our interviewees. Others, including the pragmatic-explanatory continuum indicator summary (PRECIS) group, have further discussed the extent of pragmatism in PCTs.⁶

Table 1

Pragmatic clinical trials are seen as a bridge between RCTs (randomized) and observational studies (using real-world data).7 However, according to some observers, payer decision makers often perceive several factors as barriers to their accepting PCTs ().^8,9

Moreover, because payers’ awareness and acceptance of PCTs have been understudied, uncertainty exists about whether and how US payers, key stakeholders for trials, would use results from PCTs to guide coverage and formulary decision making.

METHODS

Appendix

We gathered qualitative evidence to address our primary study question: Assuming that PCTs are randomized, how would payers view a PCT’s strength of evidence, compared with an RCT on the one hand and an observational study on the other? We pursued specific questions that explored which facets of PCTs payers find more compelling or more problematic than others (See for interview questions). We conducted structured, in-depth interviews of formulary decision makers and senior staff members at private health insurers and public payers. This study was funded by Eli Lilly and Company.

Sample

Table 2

We selected a criterion-based (purposive) sample¹⁰ of 15 payers and related organizations drawn from 10 categories of payers located throughout the United States (). In selecting payers, our main criterion was that the sample reflect the diversity of US payer types—from managed care organizations offering comprehensive medical services and prescription drugs to pharmacy benefit managers; from national insurers to regional insurers; and from commercial insurers with nonelderly enrollees to public payers serving seniors.These organizations reflect most major types of US healthcare payers.

Our use of a nonprobability sample restricts the results’ generalizability. But unlike a conventional survey, the study’s purposive, interview-based approach offers a feasible method of obtaining reliable, credible, and detailed information about the use, perceived merits, and perceived difficulties of PCT evidence within US payers. Our method ensured that respondents were formulary decision makers (not junior staff assigned to fill out a website survey). Moreover, we believe our interviewing method reduced miscommunication.

Interview Format and Protocol Development

The 1-hour interviews typically included 1 to 3 formularydecision makers, drawn from the following: vice president for pharmacy, director of pharmacy, chief medical officer, other physician, vice president for drug evaluation and pharmacoeconomics, and pharmacy and therapeutics committee chair. The interviews followed a protocol of questions, although interviewees’ own logic dictated occasional changes in the order of topics. The protocol permitted interviewees to convey both interest in and concerns about PCTs. After exploratory interviews at 5 payers, we sharpened the protocol’s focus on important, under-researched topics. Moreover, early in each interview, we defined a PCT as a randomized, prospective controlled trial with the traits listed above. This established a common terminology for interviewees and interviewers.

Protocol Features

Figure

The study protocol posed questions that led interviewees to explore a decision tree (). To elicit preferences between complex alternatives, we generally separated each alternative into constituent elements. Decision makers were asked to compare element A with element B, and then to explain the considerations that informed their choice. This process was repeated for various features of a PCT and for comparisons between a PCT and other types of studies. Our approach overcame 1 difficulty decision makers displayed in the initial interviews: describing their preferences regarding alternatives, each of which could involve multiple features of a trial or other type of evidence.

In addition, the protocol elicited each payer’s global assessment of PCTs as a type of evidence. This question guarded against misinterpreting a payer’s whole view of PCTs as being the sum of responses to piecemeal questions. Payers answered our questions in terms of clinical trials that met our definition of a PCT and typically did not refer to a particular PCT that they had reviewed, because their own experience with PCTs was limited.

Data Analysis/Synthesis

To develop findings, the 2 investigators who conducted the interviews independently reviewed their own notes, identified interviewees’ main points, and classified them as “consensus” or as representing 2 or more interviewee groupings. These investigators (who were not pharmaceutical company employees) then reviewed each other’s readings of the main points and their classification, resolved differences by reexamination of their notes, and produced findings that synthesized the interviews.

RESULTS

Theme 1: Payers’ Views of Place of Pragmatic Clinical Trials in Chain of Evidence

Exposure to Pragmatic Clinical Trials. All formulary decision makers consider PCTs to be rare (to date). Some decision makers did cite a head-to-head trial funded by a drugcompany as an example of a PCT, but most struggled to recall PCTs they had used in decision making. Even “classic” PCTs, the Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT) and Clinical Antipsychotic Trials in Intervention Effectiveness (CATIE),^11,12 did not come to mind for many. We could not determine whether the issue was faulty memory (decision makers could not recall these trials at all) or imperfect labeling (decision makers did not link these trials to the term “PCT” because their practical features did not stand out). Segal¹¹ explains why these 2 trials can be considered pragmatic. Payers’ impression that PCTs have been rare is consistent with a review that found only 22 PCTs out of the many clinical trials with reports published from January 1996 through September 2010.⁹

Evidentiary Chain. Once comfortable with the PCT concept as we defined it, payer decision makers stated that PCTs would be part of the continuum of evidence they consider in assessing new drugs. As our interviewees explained, when gathering evidence about a new drug, their default mode is triangulation. Rather than focus on a single type of evidence, decision makers assemble several types—for example, RCTs, analyses of their own claims database, and observational studies of postlaunch data on populations other than their own enrollees. After weighing the different sources’ conclusions, limitations, and relevance to their own enrollees, decision makers triangulate (ie, draw inferences from multiple evidence sources about the relative effectiveness of a new drug). Accordingly, decision makers stated that PCTs would become one more link in the chain of evidence they consider.

Overall, among interviewees, the concept of triangulation across RCTs, PCTs, registry data, internal claims-based studies, and observational studies was widely shared. In an alternative analogy, evidence forms a multidimensional crossword puzzle: each dimension (type of evidence) provides a clue to the answer.

External Validity. Formulary decision makers find PCTs appealing because they improve the external validity of the evidence base. Payers generally view explanatory RCTs as exemplars of internal validity. Nonetheless, decision makers stated they want evidence of external validity: that the results apply not just to the RCT’s population but also to the payer’s own enrollees. Decision makers noted that patients excluded from many RCTs often resemble payers’ own patient populations more than the RCTs’ restricted populations. Consequently, payer decision makers generally favor PCTs over RCTs because PCTs’ broad population holds greater relevance to payers. Furthermore, payers recognize that more external validity (a wider array of subgroups studied) often entails less internal validity (small subgroup samples and less statistical power to detect a treatment effect). Among decision makers interviewed, the consensus was that external validity trumps internal validity—perhaps with caveats; for example, when a drug is the first in a therapeutic class or when evidence of effectiveness of comparators is weak.

Ranking Randomized Controlled Trials and Pragmatic Clinical Trials. The decision makers interviewed differed in their views of where in their hierarchy of evidence PCTs would fit, especially when the evidence available included traditional RCTs. Some decision makers rejected the idea of ranking because RCTs and PCTs are incommensurate: “RCTs are the gold standard for demonstrating whether a drug works” while “PCTs provide evidence on the types of patients for which a drug works.” Other insurers expressed a related perspective: RCTs demonstrate whether a drug works under ideal conditions; in contrast, PCTs demonstrate whether a drug that works in an RCT also provides benefit under real-world conditions (imperfect patient adherence as well as physicians’ uneven conformity with practice guidelines). Most payers stated that traditional RCTs should be conducted to establish efficacy against a placebo in most (not all) therapeutic areas. In contrast, 1 payer wanted PCTs to supplant all RCTs. Nonetheless, most payers want traditional RCTs as well as PCTs and suggest that traditional RCTs should precede the PCTs.

Theme 2: Payers’ Views of Pragmatic Clinical Trial Design

Formulary decision makers emphasized that they hold nuanced views regarding PCT design. Five aspects stand out:

1. One Size Does Not Fit All. As seen by decision makers generally, different medical conditions call for PCTs with different features. For example, condition X might call for a PCT with a broader patient population than most RCTs, while condition Y might call for a PCT with a head-to-head comparison or with particular outcomes measures.

2. Salient Subgroups. Although decision makers noted that diagnoses and clinical factors often affect which subgroups they want included in a trial, they singled out several subgroups frequently as key for inclusion in PCTs. Although designers of a PCT can select from an array of population subgroups, payers most often mentioned women and seniors. Most decision makers stated they want study populations to include more women. A smaller but sizable segment of interviewees wanted study populations to include seniors.

3. Regions and Preferred Subgroup. Payers typically urge PCT designers to include “their” ethnic group in the study population. Ethnic groups that are most prominent among health plans’ enrollees differ by region. For example, it is possible that African Americans are the salient minority group in Northeast plans, Hispanics in Southwest plans, and Asians and Hispanics in West Coast plans.

4. Coexisting Illnesses. Many payer decision makers recognized that PCTs should sample from the many plan enrollees characterized by multiple comorbidities and polypharmacy. Many decision makers understood that groups that payers care about—their typical populations—might not map into trial designers’ strata, based on an index of multiple biologic and clinicalfactors. As one decision maker noted, “It is at least as important to include patients in a PCT who have 3 or more other diseases as it is to include patients with a particular demographic trait.”

5. Active Comparators. Payer decision makers strongly favored trials with active comparators. Our interviewees stated that comparative effectiveness should be a key part of PCTs: while PCTs are not necessarily head to-head studies of unique agents, insurers and other payers wish they were. Decision makers typically agreed about some points regarding selection of comparators, yet disagreed about what makes some comparators best. For example, decision makers displayed near-unanimity in ranking types of comparators: theleast preferred was a placebo; somewhat more preferred was an older drug not commonly prescribed; good was 1 market leader within the therapeutic category; ideal (though perhaps not realistic to expect often in PCTs) was the top 2 market leaders. In contrast, regarding a single, preferred active comparator, payers differed:some favored the market leader; some, the most clinically efficacious drug; and some, a group of drugs consistent with an evidence-based guideline.

Meaningful Outcomes. While agreeing that an active comparator is key, some decision makers stressed the need for trials to focus on the right outcomes. For example, changes in surrogate outcomes and patient-reported outcomes that leave utilization or costs unchanged affect formulary decisions less.

Patient Heterogeneity and Timing of Pragmatic Clinical Trials. Formulary decision makers wanted a broader population that includes subgroups of interest. For decision makers, the payoff of getting trials with this heterogeneity was gauging whether a particular subgroup would be an appropriate target for a new drug: would the improvement in health outcomes be especially high for this subgroup? In addition, decision makers generally agreed that heterogeneity should be included in phase IV trials. In contrast, they see RCTs in phases II and III as laying a foundation of evidence that a drug works, at least for the RCT’s sample and population.

Should patient diversity be introduced earlier, in phase III? Most decision makers said no: patient heterogeneity, while important, would be worth delaying until phase IV if that meant phase III trials were to include active comparators. For these decision makers, the gain from a head-to-head comparison dominated the gain from knowing more about effects for subgroups. But a minority of decision makers favored heterogeneity in phase III, because they valued heterogeneity more than active comparators.

For a few decision makers, heterogeneity in a phase III trial was desirable if evidence of efficacy from prior trials was strong. These decision makers were sensitive to the trade-off between internal validity and external validity in budget-constrained PCTs. If conducted with “too much” heterogeneity, PCTs would lose some allure. Heterogeneity is desirable if the sample size is large enough to power tests of effect size for a PCT’s subgroups. Otherwise, heterogeneity introduces, as one interviewee put it, “too many confounders.”

Theme 3: Payers’ Anticipated Use of Pragmatic Clinical Trials

Pragmatic Clinical Trials and Decision Models. In the payers’ view, most company-supplied models were not believable, in part because they relied on data from explanatory RCTs, not real-world studies. Nonetheless, payer decision makers differed on whether PCT-based results would improve decision models that pharmaceutical companies provide them. Some would consider suspect even evidence from a (randomized) PCT if sponsored by a manufacturer. However, payers typically thought better data from PCTs would improve pharmacoeconomic models.

Internal Studies. Decision makers did not see PCTs as replacing their own internal database studies. Even if more PCTs become available with broader populations than most traditional RCTs, PCTs’ study populations are likely to differ from an individual payer’s enrollee population. Furthermore, payers’ internal database studies can pursue any of several objectives, such as postlaunch surveillance of adverse events. Consequently, decision makers said that PCTs were no substitute

for an internal database study.

Theme 4: Payers’ Value Assessment Rests on Whether Results of Pragmatic Clinical Trials Affect Sponsors’ InterestsThe Dark Side of Pragmatic Clinical Trials. To some payers, the value of PCTs is tainted when they are conducted by self-interested organizations (pharmaceutical companies). Asis well known, formulary decision makers typically are skeptical of information about new drugs supplied by pharmaceutical companies. Some decision makers interviewed said they see PCTs as having a dark side: PCTs could be a new vehicle for pharmaceutical manufacturers to manipulate evidence about their new drugs (or their competitors’ drugs). To some interviewees, the additional dimensions of a trial that a PCT’s sponsors can control—selection of the active comparator, choice of subgroups to include, and the definition of usual care—are additional opportunities to tilt a trial in hard-to-detect ways. One decision maker cited a game that a drug company could play: conduct a PCT for 1 indication and intimate that the results apply to other indications. This marketing twist could be used for an RCT, too. We lack evidence on the accuracy of payers’ suspicions.

Payers’ Bottom Line. Interviewed payer decision makers believed PCTs would make their formulary decision making more informed. Asked at the end of the interview whether PCTs deserve a thumbs-up or thumbs-down, the overwhelming reply was thumbs-up. Nonetheless, regarding drugs’ comparative effectiveness, payers generally wanted to see the full spectrum of types of evidence, including but not limited to PCTs.

DISCUSSION

Trust but Verify

Given the incentives facing pharmaceutical companies to demonstrate better effectiveness, payer decision makers lack confidence that even peer-reviewed studies are immune frombias and believe most studies with negative results about a new drug never get published. (This hurdle also applies to RCTs.) Knowing that pharmaceutical companies will conductmost trials including PCTs,¹³ payers expect to scour PCT results for solid, unbiased evidence of a drug’s effectiveness to include in their assessment.

Payers Willing to Talk About Traits They Prefer

Almost all payers expressed genuine interest in talking with those who design trials during the development phase and would commit time to improving the trial design to make the results more meaningful for payers’ coverage decisions. Payers generally recognized that trial designers may have competing objectives and face time and resource constraints that would prevent them from following all recommendations that payers suggest. Payers appear pragmatic; getting some of what they want is superior to the current environment.

Payers’ hidden message regarding PCTs is “maybe someone is listening for once.” Real-world information on effectiveness is compelling, in the payers’ view, almost regardless of specifics such as the type of head-to-head comparison. Payers are strongly motivated to participate more in the design of PCTs, because they view PCTs as a response to their prior requests for trials to answer questions most meaningful to payers and their enrollees. Our interviewees conveyed a sense that, for the first time, the pharmaceutical industry and others who design clinical trials are listening to payers, not just regulators. In contrast, payers view industry’s previous focus to have been on drug approval.

Payers’ Recommendations for Improving Studies

Table 3

Payers state that they want to provide value for money, both to their enrollees and to the businesses and government agencies that purchase insurance for more than 85% of insured Americans. To assess the value of new drugs, payers seek innovative approaches to generating evidence of pharmaceuticals’ real-world effectiveness. For US payers, PCTs hold the promise of answering salient questions, if payers’ recommendations are followed ().

The 2 authors who conducted the interviews distilled the recommendations from the statements of decision makers at the 15 payers interviewed. Not every payer would support every recommendation, in part because each payer has unique CER review processes.

CONCLUDING OBSERVATIONS

Given payers’ interest in PCTs as a source of valuable CER evidence, drug manufacturers and payers should communicate more to help foster these new types of trials. However, the payer decision makers interviewed maintained that manufacturers often spend more time delivering their own messages than engaging payers in a dialogue about trial design. This should change in order to make PCTs as informative as possible for payers and other decision makers. Furthermore, dialogue could enhance other types of studies that produce CER evidence. Likewise, better methods and dissemination are needed for all types of CER studies and for synthesizing or triangulating across different types of studies, including regulatory RCTs, PCTs, and observational studies. Lastly, while recent work^1,9 has explored ways to improve PCTs, further research is needed to determine how best to make them more transparent and objective.Author Affiliations: From Westat, Rockville (JR), MD; University of Maryland (DM), Baltimore, MD; Eli Lilly and Company (DPB, RAC), Indianapolis,IN.

Funding Source: This study was funded by Eli Lilly and Company.

Author Disclosures: Dr Mullins reports that he has received consulting fees from Amgen, Bayer, Bristol-Myers Squibb, Celgene, GlaxoSmithKline, Novartis, and Pfizer, honoraria from Bayer and Pfizer, and payment for involvement in the preparation of this manuscript from Eli Lilly and Company. Dr Buesching reports employment with Eli Lilly and Company as well as stock options in the company. Dr Cantrell reports former employment with Eli Lilly and Company as well as stock options in the company at the time this manuscript was written. Dr Ratner reports no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (JR, DM, DPB); acquisition of data (JR, DM); analysis and interpretation of data (JR, DM); drafting of the manuscript (JR, DM, RAC); critical revision of the manuscript for important intellectual content (JR, DM, DPB, RAC); statistical analysis (JR, CDA); provision of study materials or patients (JR, CDA); obtaining funding (JR, DPB); administrative, technical, or logistic support (DPB, RAC); and supervision (JR).

Address correspondence to: Jonathan Ratner, PhD, Westat, 1600 Research Blvd, Rockville, MD 20850. E-mail: jonratner@westat.com.1. Luce BR, Kramer JM, Goodman SN, et al. Rethinking randomized clinical trials for comparative effectiveness research: the need for transformational change. Ann Intern Med. 2009;151(3):206-209.

2. Oster G, Borok GM, Menzin J, et al. A randomized trial to assess effectiveness and cost in clinical practice: rationale and design of the Cholesterol Reduction Intervention Study (CRIS). Control Clin Trials. 1995;16(1):3-16.

3. Revicki DA, Hirschfield RM, Ahearn EP, Weisler RH, Palmer C, Keck PE Jr. Effectiveness and medical costs of divalproex versus lithium in the treatment of bipolar disorder: results of a naturalistic clinical trial. J Affect Disord. 2005;86(2-3):183-193.

4. Sackett DL. Evidence-based medicine. Semin Perinatol. 1997;21(1): 3-5.

5. Neumann PJ, Tunis SR. Medicare and medical technology—the growing demand for relevant outcomes. N Engl J Med. 2010;362(5): 377-379.

6. Thorpe KE, Zwarenstein M, Oxman AD, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62(5):464-475.

7. Berger ML, Mamdani M, Atkins D, Johnson ML. Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—Part I. Value Health. 2009;12(8):1044-1052.

8. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290(12):1624-1632.

9. Buesching DP, Luce BR, Berger ML. The role of private industry in pragmatic comparative effectiveness trials. J Comp Eff Res. 2012;1(2): 147-156.

10. Ritchie J, Lewis J, Elam G. Designing and selecting samples. In: Ritchie J, Lewis J, eds. Qualitative Research Practice: A Guide for Social Science Students and Researchers. London: Sage Publications; 2003:78.

11. Segal JB. The Prospect of Early Pharmaceutical Pragmatic Clinical Trials: A Background Paper. http://www.cmtpnet.org/wp-content/uploads/downloads/2012/03/Segal-PCT-Background-Paper.pdf. AccessedJanuary 24, 2012.

12. Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials. 2009;10:37.

13. Kahler KH, Sandy LG, Schneeweiss S. Need for speed—is it possible to efficiently generate and interpret comparative effectiveness research for newly marketed drugs in time to make informed decisions? Presented at: 17th Annual International Meeting of International Society for Pharmacoeconomics and Outcomes Research; June 6, 2012; Washington, DC.

Download PDF: Pragmatic Clinical Trials: US Payers' Views on Their Value