Developing Evidence That Is Fit for Purpose: A Framework for Payer and Research Dialogue

ABSTRACT

Objectives: Matching the supply and demand of evidence requires an understanding of when more evidence is needed, as well as the type of evidence that will meet this need. This article describes efforts to develop and refine a decision-making framework that considers payers’ perspectives on the utility of evidence generated by different types of research methods, including real-world evidence.

Study Design: Conceptual framework development with subsequent testing during a roundtable dialogue.

Methods: The framework development process included a literature scan to identify existing frameworks and relevant articles on payer decision making. The framework was refined during a stand-alone roundtable in December 2013 hosted by the research team, which included representatives from public and private payers, pharmacy benefit management, the life sciences industry, and researchers. The roundtable discussion also included an application of the framework to 3 case studies.

Results: Application of the framework to the clinical scenarios and the resulting discussion provided key insights into when new evidence is needed to inform payer decision making and what questions should be addressed. Payers are not necessarily seeking more evidence about treatment efficacy; rather, they are seeking more evidence for relevant end points that illustrate the differences between treatment alternatives that can justify the resources required to change practice. In addition, payers are interested in obtaining new evidence that goes beyond efficacy, with an emphasis on effectiveness, longer-term safety, and delivery system impact.

Conclusions: We believe that our decision-making framework is a useful tool to increase dialogue between evidence generators and payers, while also allowing for greater efficiency in the research process.

Am J Manag Care. 2015;21(9):e545-e551

Take-Away Points

Key insights into when new evidence is needed to inform payer decision making and what issues should be addressed include:

• Payers are not necessarily seeking more evidence about treatment efficacy. Rather, they are seeking more evidence for relevant end points.
• Payers are interested in obtaining new evidence that goes beyond efficacy with an emphasis on effectiveness, longer-term safety, and delivery system impact.
• In an era focused on care coordination, quality of care, and risk sharing, evidence to understand and improve outcomes associated with real-world use take on increasing importance.

A number of efforts are underway to improve the quantity, quality, and capacity for evidence to inform decision making. These efforts include funding from the Patient Centered Outcomes Research Institute for specific studies and research infrastructure (eg, PCORnet),^1,2 increased electronic health record availability, and research collaborations between the life sciences industry and payers (eg, AstraZeneca and WellPoint’s HealthCore).³ Additionally, tools such as the GRADE framework⁴ and the CER Collaborative⁵ seek to increase the capacity for and acceptance of real-world evidence, while activities like the Green Park Collaborative⁶ seek to clarify the demand for evidence in specific therapeutic areas.

Often, the supply of evidence may not meet the demand. Payers frequently note the gap between the available evidence versus the evidence desired to increase the certainty of coverage decisions,⁷ including evidence to: 1) confirm study results among typical patients in routine care; 2) assess outcomes associated with care delivery, such as adherence and total cost of care, and 3) compare existing care alternatives. Beyond what is available, less than half of the health care recommended in care guidelines or delivered in practice are supported by existing evidence.^8,9 Subsequently, this raises the question of whether investments in evidence should be made for all decisions, and the extent to which the evidence will be applied in care decisions.

Matching evidence supply and demand requires an understanding of not only when more evidence is needed, but also what types of evidence will meet this need. To raise the likelihood that the evidence generated is “fit for purpose” (ie, it responds to the specific needs of the intended user), we developed and tested a decision-making framework. This framework considers payers’ perspectives on the utility of evidence generated by different types of research methods, including real-world evidence. Our hope is that the framework will help to harmonize the evidence that payers desire for coverage and formulary decisions with the evidence received from researchers, as well as help to guide researchers on what types of evidence need to be developed in the future.

Previous attempts to identify the types of research needed to address particular questions have included translation tables^10,11 and collaborative dialogues. However, these efforts differ because they emphasize the perspectives of multiple stakeholder groups who often have different questions of interest based on the patient population, outcomes, or comparators being targeted. In contrast to other stakeholders, payers often seek effectiveness information comparing treatment alternatives and outcomes associated with clinical and economic efficiency. Our focus on a single stakeholder group allows for a narrowing of the comparators, patient populations, and outcomes of interest. Further, given the increased availability of real-world evidence and changing perceptions regarding its applicability and limitations, we aim to understand payers’ interest and use of such evidence.

Payer decision making has typically focused on evidence from randomized control trials (RCTs), with less utility for real-world evidence.¹² This is due in part to the traditional evaluation of evidence based on hierarchies that often place RCTs at the top, with other methods—such as observational studies and indirect treatment comparisons—at the bottom. Application of the framework also allows us to test the notion brought forth by Sir Michael Rawlins and others, that rather than focusing on evidence hierarchies, a variety of approaches should be accepted and included in the evidential base.¹³

METHODS

Fit for Purpose Framework

To help inform the framework development process, we began with a literature scan to identify existing frameworks, articles, and elements relevant to payer decision making and payer use of different evidence types. Our scan included 48 primary articles comprising payer surveys, case studies, and commentaries, and identified 29 elements of interest to payer decision making. While organizing the elements identified in the literature review, we found that David Eddy’s commentary on clinical decision making provided a useful structure. The commentary describes 2 critical steps for a decision: 1) the scientific-focused analysis of evidence and 2) the personal judgments or preferences that one applies to the corresponding evidence analysis. Similar to Eddy’s “Anatomy of a Decision,” ¹⁴ our original framework began with a general technical assessment; however, we determined the technical assessment and type of new evidence would be contingent upon the clinical question or scenario. This led us to include both a review of available evidence, as well as potential contextual factors as key elements in the framework. Using 2 clinical cases, we then vetted the draft framework with 4 experts (1 state Medicaid director, 1 pharmacy director, and 2 medical directors) via telephone. Based on these interviews, we clarified the key outcomes (eg, differentiating between long-term and short-term outcomes) and added additional elements to the contextual factors (eg, current formulary, current market share, and where and how a therapy fits into clinical management).

eAppendix

Figure

Our framework was subsequently vetted and refined during a full-day Fit for Purpose Roundtable in Washington, DC, hosted by the research team, which included 12 representatives from public and private payers, pharmacy benefit managers, the life sciences industry, and researchers (, available at www.ajmc.com). The discussion focused on clarifying and/or adding new elements within the final framework construct and applying the framework to 3 clinical cases. The framework () includes 3 main components: 1) the “Clinical Scenario,” which establishes the research questions of interest, the potential impact of an intervention or therapy on individual patients, as well as the health system, and a determination of value; 2) the “Assessment,” which comprises a review of the available evidence and a consideration of the contextual factors; and 3) the “Coverage Decision,” which is the ultimate output of the payer decision-making process.

A number of elements are considered within each of the framework components. Together, the 3 components help determine the extent to which evidence is fit for purpose with questions such as: Is the current evidence sufficient for that clinical scenario and coverage decision? If not, what types of new evidence questions need to be answered and which research methods would bring more certainty to or change the decision?

The framework is not intended to be used as a checklist where a determination is made for each element. Rather, for a given clinical scenario, one can determine which elements are most applicable and which should be included in evidence review. In addition, we acknowledge that individual elements can, and often should, be viewed across the different components of the framework. For example, an element such as site of care may have an impact on outcomes, care management processes, and the total cost of care; these elements should be considered collectively as one.

Application of the Framework

Table

During the roundtable, participants applied the framework to 3 clinical cases. A clinical monograph and information relevant to the framework elements () were provided in advance of the meeting. These cases included: 1) determining coverage for a new hepatitis C virus (HCV) therapy (sofosbuvir); 2) confirming coverage for oral anticoagulants (rivaroxaban, dabigatran, apixaban); and 3) limiting utilization for a multiple sclerosis (MS) therapy (natalizumab) with known safety concerns.

The review and discussion of the cases and existing evidence was designed to elicit additional considerations and factors related to the evidence needs of payers rather than reaching consensus on specific coverage decisions. Roundtable participants were asked to identify any missing components, confirm the overall organization of the framework, and identify which elements were of greater weight or importance for their coverage decision. Each case was reviewed and each participant rated via a hand-held voting device (Turning Point 2008 software) if the current evidence was sufficient for the decision, if lingering concerns existed, and which additional type(s) of evidence would be desired and required. Meeting transcripts were coded and analyzed for thematic responses leading to the discussion of key findings below.

The discussion of the new HCV therapy occurred just as the product was approved for use by the FDA. This new treatment represents a new method of administration that is hypothesized to have greater adherence. The long-term adverse effects, clinical practice acceptance, and utilization, however, are unknown in practice. Most participants indicated that the available evidence regarding efficacy and safety was sufficient for them to make a decision regarding coverage; however, additional evidence from prospective observational studies that focused on comparisons to existing treatment alternatives, long-term efficacy, safety, and adherence would help to confirm their coverage decision. The desire for observational studies was driven in part by uncertainty regarding the tolerability compared with existing treatment with interferon and the ability for a new oral agent to translate into better treatment outcomes in practice. In this case example, evidence on different clinical outcomes from the registrational trials was desired. Research designs that could assess these outcomes were viewed as having potentially high value, contributing to coverage decisions.

The class review asked participants to confirm coverage and tier placement of oral anticoagulants (either Factor Xa or direct thrombin inhibitors) for the prevention of stroke in atrial fibrillation. In practice, participants and their organizations had already made coverage decisions (albeit varied decisions) and few had plans to alter these decisions. Compared with the HCV scenario, fewer participants felt additional evidence was required, as their clinical experience with the existing agents limited their uncertainty. In this case example, additional evidence on the net benefit was unlikely to be of sufficient magnitude to overcome the existing contextual factors and to subsequently alter the coverage decision.

For the MS example, participants were asked if the risk-benefit profile for the treatment was acceptable compared with alternative treatments, and how this acceptance may change the coverage or utilization management requirements. In general, most roundtable participants did not feel there was a need for additional efficacy evidence, except to identify patients with rapidly progressing MS for whom the risk—benefit profile may be justified. Limited interest in additional evidence may also be due to existing utilization in practice and management of risk through prior authorization and step therapy. In the final case example, new evidence was desired for a subset of patients, but not for the broader patient population. Application of the framework to the clinical cases and the resulting discussion provided key insights into when new evidence is needed to inform payer decision making and what questions should be addressed.

More impact; not more evidence. Payers are not necessarily seeking more evidence about treatment efficacy. Rather, they seek evidence that illustrates meaningful differences in relevant end points between treatment alternatives. The evidence must be of sufficient value to justify the administrative resources required to change practice. In the HCV example, the all-oral regimen was hypothesized to have better tolerability and adherence than the existing treatment regimens. Although adherence was viewed as an important differentiator for payers’ coverage decisions, adherence without clinical and economic benefits may not justify the administrative costs to shift care patterns toward treatments with greater adherence. Finally, many participants noted the need to consider the costs of generating new evidence (eg, the cost of a comparative anticoagulation study) compared with the likelihood that new evidence is likely to change practice. While additional evidence may be desired, participants emphasized the need to allocate healthcare research dollars most efficiently.

Timing matters. Evidence is most useful to reduce uncertainty. Payers perceived additional evidence was most useful when there is uncertainty in how a treatment will work in the real world or when clinical practice and utilization are shifting. Most plans have requirements to make coverage decisions within a specified period (eg, 90 days for Medicare Part D plans) following the approval of a new treatment. This period has the greatest uncertainty and need for evidence on effectiveness. Additional evidence may also be desired when clinical practice behavior is changing in anticipation of new treatments (eg, HCV patient warehousing) to decrease concerns about unintended clinical or economic implications. In contrast, for areas where clinical practice had already shifted (eg, anticoagulation therapies), new evidence was not prioritized.

A broader view of outcomes. Payers are interested in obtaining new evidence that goes beyond efficacy, with an emphasis on effectiveness, longer-term safety, and delivery system impact. For example, roundtable participants sought information on the effectiveness among subpopulations of patients (eg, which patients with MS may be more appropriate for treatment due to rapid disease progression) and adherence in the real world (eg, tolerability for new HCV agents compared to interferon). Further, to better determine the effectiveness of care management, payers desire additional information from researchers on how patient characteristics are identified and appropriately treated in practice, such as various HCV genotypes.

Real-world evidence is needed. In an era focused on care coordination, quality of care, and risk sharing, evidence to understand and improve outcomes associated with real-world use takes on increasing importance. In this environment, participants noted that real-world evidence can fill information gaps and allow payers to consider additional tiering options and/or more population-specific utilization management programs. Since additional comparisons among treatment alternatives were desired, indirect treatment comparisons were noted as a potential tool for decision makers. In general, observational studies are sought for comparing treatments with similar utilization, when there are few comorbidities, or when there is clinically rich information on subpopulations (eg, HCV genotypes). Still, researchers must ensure that the patient population and inclusion and exclusion criteria are well circumscribed and that the clinical end points are credible.

DISCUSSION

The initial framework sought to better match the evidence supply and demand for payer decision making. Other decision aids or discussions seek to clarify the key outcomes and types of evidence desired by payers and health technology bodies for particular therapeutic areas (eg, the Green Park Collaborative), improve the transparent assessment of value (eg, the ICER Value Framework), or increase the consistency of evidence assessment (eg, ICER’s Evidence Based Rating Matrix and the GRADE framework). Our framework is meant to initiate dialogue between researchers, industry, and payers regarding when and what additional evidence is needed by payers. Our framework builds upon other decision aids described above, but expands to include the importance of contextual factors.

Limitations

Despite the framework being substantiated by 3 clinical case studies, there are limitations to its broader application. First, the discussions regarding the framework were limited to a small set of roundtable participants (n = 12). The generalizability and utility of the framework requires discussions with a larger and more diverse set of payer representatives from integrated health plans, regional and national payers, state Medicaid directors, and pharmacy benefit managers. The importance of different elements outlined in our framework may not apply to specific payer groups—for example, care disruption on delivery system may be more critical for integrated health plans than for other payers. Second, our framework was limited to a small subset of clinical cases. Application of the framework to a larger set of clinical cases would provide a more robust understanding of payer decision making, particularly due to the infinite number of potential variations across different cases or therapeutic areas. Finally, although we examined contemporary clinical and coverage decisions, the cases were considered in an artificial decision-making environment and may be dependent upon the type of payer (eg, health plan vs pharmacy benefit provider), line of business (eg, commercial vs Medicaid), or organization’s reach (eg, national, regional, or local). Further research should test the framework in the context of actual clinical decisions within several payer organizations.

CONCLUSIONS

Current efforts to increase the quantity, quality, and capacity for evidence will be insufficient without better alignment between evidence supply and demand. Based upon payers’ perspectives, we sought to better understand when additional evidence is needed, which questions to answer, and what types of data are acceptable. We found that payers did not always need all information for all treatments; rather, the need for more evidence is based on the clinical questions and therapy, the available evidence, and the need to overcome contextual factors such as the ability for evidence to change practice. In some cases, the existing data is sufficient; in other cases, payers might need additional data on long-term safety from either registrational trials or from real-world studies. In other circumstances, there is a need for studies with different comparators or populations; In these cases, the evidence should be “fit for purpose” rather than any single type of study. Having a framework that enables this nuanced discussion between payers and generators of evidence can increase the likelihood that the most important evidence will be made available. Ideally the current framework can be used to increase dialogue between evidence generators and payers, and provide for greater efficiency in the research process. Moving forward, we hope to further refine the utility of the framework by applying it to a broader set of clinical cases and participants, and test the value of a structured assessment of the contextual factors in real-life practice settings.

Acknowledgments

The authors would like to thank the Fit for Purpose Roundtable participants (eAppendix) for their insights and input regarding the refinement and application of the framework.

Author Affiliations: AcademyHealth (RKS, EH), Washington, DC; National Pharmaceutical Council (JSG, RWD), Washington, DC.

Source of Funding: This project was funded by the National Pharmaceutical Council.

Author Disclosures: Drs Graff and Dubois are employees of the National Pharmaceutical Council, which is a policy research organization supported by the nation’s major research-based pharmaceutical companies. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (RKS, JSG, EH, RWD); analysis and interpretation of data (RKS, JSG, EH); drafting of the manuscript (RKS, JSG, EH); critical revision of the manuscript for important intellectual content (RKS, EH, RWD); obtaining funding (RKS, JSG, RWD); administrative, technical, or logistic support (RKS); and supervision (RWD).

Address correspondence to: Rajeev K. Sabharwal, MPH, AcademyHealth, 1150 17th St NW, Ste 600, Washington, DC, 20036. E-mail: Raj.Sabharwal@academyhealth.org.

REFERENCES

1. Research we support. Patient Centered Outcomes Research Institute website. http://www.pcori.org/research-results/research-we-support. Published September 10, 2014. Accessed January 28, 2015.

2. Fleurence RL, Curtis LH, Califf RM, Platt R, Selby JV, Brown JS. Launching PCORnet, a national patient-centered clinical research network. J Am Med Inform Assoc. 2014;21(4):578-582.

3. Delaware to participate in real-world evidence collaboration with Astrazeneca and Healthcore [news release]. Wilmington, DE: Office of the Governor; February 23, 2012. http://news.delaware.gov/2012/02/23/delaware-to-participate-in-real-world-evidence-collaboration-with-astrazeneca-and-healthcore/. Accessed September 2, 2015.

4. Guyatt GH, Oxman AD, Vist GE, et al; GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926.

5. CER Collaborative. http://www.cercollaborative.org. Accessed July 1, 2014.

6. Green Park Collaborative. http://www.greenparkcollaborative.org/gpc-usa/. Accessed June 30, 2014.

7. Wang A, Halbert RJ, Baerwaldt T, Nordyke RJ. US payer perspectives on evidence for formulary decision making. Am J Manag Care. 2012;18(5, spec no. 2):SP71-SP76.

8. Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC Jr. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA. 2009;301(8); 831-841.

9. Infrastructure required for comparative effectiveness research: workshop summary. Institute of Medicine website. http://www.nap.edu/catalog/12214/learning-what-works-infrastructure-required-for-comparative-effectiveness-research-workshop Published 2011. Accessed September 2, 2015.

10. PCORI Methodology Committee. The PCORI methodology report. Patient-Centered Outcomes Research Institute website. http://www.pcori.org/assets/2013/11/PCORI-Methodology-Report.pdf. Published November 2013. Accessed July 3, 2014.

11. Gliklich RE, Leavy MB, Velentgas P, et al. Incorporating stakeholder perspectives in developing a translation table framework for comparative effectiveness research. J Comp Eff Res. 2012;1(3):281-292.

12. Leung MY, Halpern MT, West ND. Pharmaceutical technology assessment: perspectives from payers. J Manag Care Pharm. 2012;18(3):256-264.

13. Rawlins M. De testimonio: on the evidence for decisions about the use of therapeutic interventions. Lancet. 2008;372(9656):2152-2161.

14. Eddy DM. Clinical decision making: from theory to practice: anatomy of a decision. JAMA. 1990;263(3):441-443. 

Download PDF: Developing Evidence That Is Fit for Purpose: A Framework for Payer and Research Dialogue