Evaluation of Medical Technologies With Uncertain Benefits

This publication was supported by Pharmaceutical Research and Manufacturers of America Foundation.

The Opportunities and Challenges of Rethinking Our Approach to Value Assessment in Healthcare

As America’s healthcare system continues to evolve, it is critical that our perception of care and its value to patients evolve with it. In the past, value assessments have marginalized patients’ perspectives in favor of other, more easily quantifiable variables. Unfortunately, this approach to value assessment hasn’t been able to capture individual health states or preferences because it fails to engage with the most important stakeholders: the individuals receiving the care.

Take, for example, the quality-adjusted life-year, also known as the QALY. Explained at the most basic level, the QALY is a measurement of how an intervention improves a patient’s quality and quantity of life. The QALY aims to encapsulate the value of healthcare interventions in a single index number, where 1 equates to 1  year of perfect health and 0 is associated with death.

From the patient perspective, assessing the value and impact of care through a summary metric is akin to summarizing a 200-page novel in a single word. Although many experts acknowledge the limitations of the QALY metric, they often throw their hands up and assert that patient perspectives are just too difficult to quantify as a practicable metric.

But things are beginning to change.

Last year, health economists and health services researchers rolled up their sleeves to offer alternative approaches to measure value as part of the Pharmaceutical Research and Manufacturers of America Foundation’s 2019 Challenge Awards. The awards presented researchers with a single prompt:

“What are innovative, patient-centered approaches to contribute to healthcare value assessment that move beyond the inherent limitations of analyses based on the quality-adjusted life-year metric?”

Researchers responded with myriad novel, innovative, and practical approaches to value assessment that enhance or mitigate past the QALY and allow deep engagement with patients. Perhaps more important, the volume of substantive submissions undermined the idea that successfully incorporating the patient voice into healthcare assessments was too difficult.

Of all the approaches offered, 4 winning submissions were selected based on their innovative and pragmatic approaches to value assessment. Although each of the approaches differs in methodology and design, a common theme throughout is the realistic way in which they account for the perspectives of patients. In one selected model, for example, the authors propose to inform value assessment with learnings from patient-focused drug development (PFDD), a program instituted by the FDA to capture the experiences, perspectives, and needs of patients based on their unique symptoms and medical histories. Another approach argues for using patient stakeholder groups and existing patient registries to better incorporate patient perspectives.

One of the submissions takes value assessment a step further, arguing that conventional approaches to value measurement that emphasize the average (mean) of healthcare outcomes are flawed. According to the authors, because patient care is about more than just average outcomes, we should incorporate the statistical variance and skewness of treatment outcome distributions to more effectively capture the wide spectrum of patient experiences.

Interestingly, many of the selected winners offer methodologies that aren’t constructed from scratch. Instead, they expand on existing infrastructure, such as the PFDD, which is already purposed to capture patient perspectives on a medicine’s effectiveness. Another winning submission discusses the advantages of applying multiple-criteria decision analysis—a long-standing decision-making tool used in other industries—given its ability to capture multiple dimensions of value that often go unmeasured.

But beyond just the winning approaches, many, if not most, of the submissions present realistic methods that expand the scope of value assessment, engage with patients, and innovate beyond the QALY.

It is clear that ongoing efforts such as the Challenge Awards that seek to move beyond the QALY are making meaningful headway in improving value assessments for patients. We should applaud the worthy goals of these efforts to fundamentally rethink what constitutes value.

Eileen Cannon

President

Pharmaceutical Research and Manufacturers of America Foundation

Evaluation of Medical Technologies With Uncertain Benefits

Darius N. Lakdawalla, PhD, and Charles E. Phelps, PhD

Standard cost-effectiveness analyses (CEAs) compute incremental health gains outcomes and costs from new treatments (vs existing therapy), then calculate incremental cost-effectiveness ratios (ICERs) to measure how efficiently new treatments improve health outcomes. This method ignores variability in outcomes that also affects patients’ valuations. Just as people dislike financial risk (and buy insurance to protect against it), they dislike health outcome risks. Our approach to value includes measures of risk that can significantly alter estimated treatment values compared with simple (and misleading) average outcome measures.¹

Uncertainty in health outcomes has multiple dimensions. Most direct (and important) is the actual variance of outcomes, measuring changes in dispersion of treatment outcomes—“the value of insurance.” It measures gains (or losses) in value associated with lower (or greater) uncertainty in health outcomes of medical interventions.

Uncertainty can also create “the value of hope.” For any given average (mean) and dispersion (variance) in health outcomes, treatments that shift distributions of outcomes more positively (“positive skewness”) add value because they increase the chances of beneficial outcomes. We concentrate our discussion here on changes in dispersion (value of insurance) and skewness (value of hope), although our full model can add other effects.

Our model incorporates measures of how uncertainty affects value from standard economic literature. For half a century, economists have been measuring the value of reducing financial risk.^2,3 This work measures “risk aversion”—the rate at which income’s incremental value (“marginal utility of income”) falls as income increases. An additional $10,000 for people with a $1 million income adds less value than an additional $10,000 for people with a $50,000 income. Marginal utilities of income fall as income rises. Risk aversion measures how fast this happens. Where σ² measures the variance of a distribution of a risky income (M), the risk premium

where r* is the relative risk aversion measure. The economic literature centers estimates of r* at about 0.7 to 1.2.

Newer models of risk behavior also estimate how fast r* changes with income. The measure, called relative prudence (denoted by π*), has that name because more “prudent” persons will have higher precautionary savings as their financial situation becomes riskier. The economic literature gives estimates of 1.8 < π* < 2.2, centering around 2.0.

Our model provides a complete way to incorporate relative risk aversion and relative prudence into measures of value of healthcare interventions, thus extending CEA to include uncertainty and average outcomes. Our model yields the following conclusions:

• Reductions in uncertainty of treatment outcomes add value (and conversely).
• Increases in positive skewness of treatment outcomes add value (and conversely).
• These effects become increasingly important as average differences in outcomes become smaller. For small differences in average value, differences in variance and skewness can dominate proper measures of differences in value.
• Medical treatments provide more value when provided to people with greater health burdens. Gains of some specific amount of health (equivalent, say, to 0.2 life-years) matter more to very sick people than to those with less serious health conditions. This simply reflects standard models of diminishing marginal utility—fixed gains in outcomes matter more to those who have less to begin with.

We simulated values of various treatments (T) versus controls (C) to demonstrate how and when incorporating uncertainty (risk and skewness) matters in measuring health gains. Figure 1 compares 3 different medical interventions (T) with a standard C. The vertical axis shows the ratio of value using our measure compared with the standard CEA measure that includes just average outcomes, which we call the risk adjustment factor (RAF). In Figure 1A, T and C have the same variances and skewness, so the RAF equals 1. Figures 1B and 1C show how the RAF grows as the variance of the new treatment (T) shrinks, whereas the variance of C remains the same.

Figure 2

Although these show the direction of effect, they are not all that impressive because the difference in mean outcomes is large. shows what happens as average differences in outcomes between T and C shrink. As the distribution of outcomes for T moves to the left (closer to the average for C), the RAF rises rapidly. This illustrates our third bullet point from above—the importance of incorporating uncertainty into value measures rises as the average differences shrink.

Figure 3

also demonstrates the effects of changing the skewness of outcomes (the “value of hope”) with similar simulations. Unfortunately, there is no way to change the skewness of a distribution without also changing its variance, so each successive panel in Figure 3 shows a distribution of T that has both increasing variance (a “bad”) and increasing positive skewness (a “good”). The key point is that in the final panel, the variance of T equals the variance of C (thus canceling out the effects of variance) and T remains superior to C (RAF = 1.07) because of the strong positive skewness in the distribution of T’s outcomes compared with C’s.

Including our new measures is not burdensome to producers of new therapeutic innovations. They already must measure variances of outcomes in T and C to estimate the precision by which differences in means are known. Randomized controlled trials already capture this information, although few of them report it directly. Our model shows how to incorporate measures of variance directly. Measuring skewness with sufficient precision may require larger sample sizes, but it is clearly feasible.

We cannot fully know the importance of incorporating these measures until numerous clinical studies measure and report variances and skewness. However, our simulations strongly suggest that these measures can importantly alter measures of value, particularly when average differences in outcomes are small. We can anticipate many small incremental gains in average outcomes over time, as much of the low-hanging fruit in medical advances has already been picked. The time has come to incorporate uncertainty of outcomes into health technology assessment, and our model provides a precise and economically grounded way to do this.Darius N. Lakdawalla, PhD, is the Quintiles Chair in Pharmaceutical Development and Regulatory Innovation at the University of Southern California School of Pharmacy and professor at the University of Southern California Sol Price School of Public Policy in Los Angeles, California.

Charles E. Phelps, PhD, is a university professor and provost emeritus of the University of Rochester in Rochester, New York.

1. Lakdawalla DN, Phelps CE. Evaluation of medical technologies with uncertain benefits. nber.org/papers/w26058. National Bureau of Economic Research working paper 26058. Published July 2019. Accessed January 13, 2020.

2. Arrow KJ. Liquidity preference. Lecture VI in Lecture Notes for Economics 285: The Economics of Uncertainty. Stanford, CA: Stanford University; 1963:33-53.

3. Pratt JW. Risk aversion in the small and in the large. Econometrica. 1964;32(1-2):122-136. doi: 10.2307/1913738.

Download PDF: Evaluation of Medical Technologies With Uncertain Benefits