Developing a Composite Weighted Quality Metric to Reflect the Total Benefit Conferred by a Health Plan

ABSTRACTObjectives: To improve individual health quality measures, which are associated with varying degrees of health benefit, and composite quality metrics, which weight individual measures identically.

Study Design: We developed a health-weighted composite quality measure reflecting the total health benefit conferred by a health plan annually, using preventive care as a test case.

Methods: Using national disease prevalence, we simulated a hypothetical insurance panel of individuals aged 25 to 84 years. For each individual, we estimated the gain in life expectancy associated with 1 year of health system exposure to encourage adherence to major preventive care guidelines, controlling for patient characteristics (age, race, gender, comorbidity) and variation in individual adherence rates. This personalized gain in life expectancy was used to proxy for the amount of health benefit conferred by a health plan annually to its members, and formed weights in our health-weighted composite quality measure. We aggregated health benefits across the health insurance membership panel to analyze total health system performance.

Results: Our composite quality metric gave the highest weights to health plans that succeeded in implementing tobacco cessation and weight loss. One year of compliance with these goals was associated with 2 to 10 times as much health benefit as compliance with easier-to-follow preventive care services, such as mammography, aspirin, and antihypertensives. For example, for women aged 55 to 64 years, successful interventions to encourage weight loss were associated with 2.1 times the health benefit of blood pressure reduction and 3.9 times the health benefit of increasing adherence with screening mammography.

Conclusions: A single health-weighted quality metric may inform measurement of total health system performance.

Am J Manag Care. 2015;21(3):221-227

We developed a weighted quality measure to reflect the total health benefit conferred by a health plan annually to its members.  

 

• Quality was weighted by the gain in life expectancy associated with 1 year of member adherence.
• Health plans that helped members quit smoking or lose weight for 1 year contributed 2 to 10 times as much health benefit as those that improved member compliance with easier-to follow preventive care services, such as mammography, aspirin, and antihypertensives.
• Weighted metrics may inform overall health plan evaluation—for example, by reporting the total insurance plan quality next to price on health insurance exchanges.

Measurement of health system performance is difficult. Quality measures are proliferating^1,2 with >350 suggested metrics in 2014,³ and these measures are often criticized because they focus on narrowbased interventions (eg, use of angiotensin-converting enzyme inhibitors for patients with congestive heart failure) that fail to provide a global measure of benefit.^4-6 Existing metrics also have varying correlations with health outcomes of interest to patients and patient representatives;^4-8 therefore, it is unknown whether a health plan that exceeds quality measures confers greater health benefit than other health plans.^9,10 Moreover, composite evaluation metrics such as the Healthcare Effectiveness and Data Information Set typically assign the same weight to each item. For example, the percentage of women aged 40 to 69 years who had a screening mammogram in the past 2 years counts as much as the percentage of diabetics aged 18 to 75 years with controlled A1C,¹¹ even though their contributions to health may differ.

A more appropriate weighting scheme for composite metrics remains subject to debate. Some researchers have suggested all-or-none or 70% standards, in which patients must meet all or ≥70% of identified criteria, respectively, to be included in the numerator.^12-14 Such criteria are used in the evaluation of accountable care organizations for diabetes and coronary artery disease, such as the percent of adults with ischemic vascular disease who received ≥1 lipid profile within 12 months and whose most recent low-density lipoprotein cholesterol (LDL-C) level was <100 mg/dL.¹ Others have argued for development of composite weights based on measurement reliability,¹⁵ post surgical quality of life,16 or population health.¹² No consensus on optimal quality weights has been reached.

More generally, the National Quality Forum requires 4 criteria for endorsement of a quality measure: 1) a focus on a priority area that improves healthcare quality, 2) scientific acceptability (reliability and validity), 3) usability and relevance, and 4) feasibility^.17 To improve their usefulness, some researchers suggest that quality measures must transition to broader, more meaningful metrics, which readily lend themselves to improvement.⁴ Yet, the development of appropriate measures and their implementation remain at early stages; for example, while electronic medical records (EMRs) may enable better measurement of patient care and outcomes, they may under- or overestimate appropriate care utilization. A recent study in a federally qualified health center found that EMRs estimated 38% appropriate asthma medication use, compared with 77% when measured manually^18,19; similar inaccuracies have been found using claims data.²⁰

To consider the potential to condense multiple quality metrics into more actionable, useful information, we aimed to develop a health-weighted composite quality metric, which quantified the amount of preventive health benefit conferred by a health system over 1 year to a typical health insurance system membership panel. Such a measure may complement existing quality metrics by giving a more holistic view of the health benefit imparted by health systems, and may be useful for patients and patient representatives (employers) seeking to compare different health systems or to evaluate whether a health system is improving over time.

METHODS

We adapted a previously published and validated computer simulation²⁰ to estimate the increase in life expectancy that would occur from 1 year of exposure to a health system with specified performance on individual quality metrics on a population with specified distributions of characteristics. Briefly, that model estimated life expectancy for an individual patient, personalized for age, gender, race, risk factors, and comorbidity.²¹ The model then simulated the change in life expectancy associated with adherence to each US Preventive Services Task Force (USPSTF) grade A and B recommendation, and rank-ordered results to help identify the most important preventive care guidelines for each patient.²¹

We targeted the question: “How much additional health has a health plan conferred to a particular patient in the past year, through preventive care?” by making 3 main adaptions to this model. First, we teased apart the health benefit attributable to 1 year of health plan intervention (“attributability”), disambiguating it from additional benefit that might occur in successive years if continued intervention made adherence more likely. For example, we only considered the benefit from taking a lipid-lowering medication that arose from 1 year of health plan intervention, assuming that patient’s adherence would decline in succeeding years unless its persistence was reinforced by subsequent care.

Second, because adherence to medications and lifestyle recommendations are important contributors to the success of chronic disease management and prevention, we modeled both intervention-specific adherence rates (for example, assuming that it was easier to take antihypertensive medications than to stop smoking) and variability in adherence rates across health plan members (above- vs below-average member-specific adherence rates). In this context, “adherence” was defined as compliance with each USPSTF recommendation within 1 year of working with a physician: body mass index (BMI) = 24.9; blood pressure (BP) = 120/80 mm Hg; total cholesterol = 190 mg/dL; LDL-C = 70 mg/dL; no tobacco use; no alcohol misuse; for diabetics, A1C = 6% (controlled diabetes); and mammograms, Pap smears, and colonoscopies at recommended frequencies.²¹

eAppendix A

Finally, in addition to a baseline lifetime horizon, we considered shorter (5-, 10-, and 20-year) time horizons that may be more relevant to health insurers and employers who expect their members to eventually change insurance plans. provides mathematical details (available at www.ajmc.com).

Hypothetical Insured Membership Panel

We constructed a hypothetical health insurance system membership panel using national prevalence of age, race, gender, and comorbidity as a proxy. We focused on individuals aged 25 to 84 years (approximately 64% of the US population)²² as the core group for preventive healthcare exposure, using the most recent (July 2012) age-genderrace distribution reported by the US Census Bureau.^22,23 We then used data from the National Health Interview Survey on the prevalence of select comorbidities, by age and gender, including obesity (estimated by the proportion of BMIs 25.0-29.9, 30.0-34.9, 35.0-39.9, ≥40.0), hypertension, hyperlipidemia, diabetes, tobacco use, and hazardous or harmful alcohol use (≥5 drinks/day on ≥12 occasions in the past year).²⁴ Using the Surveillance, Epidemiology, and End Results Cancer Statistics Review, we also assumed a family history (≥1 first-degree relative) of colorectal cancer for 5% of individuals (of all ages/genders) and a family history of breast cancer for 12.5% of women.²⁵

Additionally, we recognized that some individuals were more likely to suffer from multiple comorbidities. Therefore, we included prior estimates of the relative risks of diabetes and hypertension in obese (BMI ≥30.0) versus nonobese individuals (relative risk [RR] for diabetes, 5.2 for males and 12.7 for females; RR for hypertension, 2.6 for males and 4.2 for females).²¹ For example, among females aged 45 to 54 years, we estimated that 18.7% were obese hypertensive women, 18.1% were obese nonhypertensive women, 12.5% were nonobese hypertensive women, and 50.6% were nonobese nonhypertensive women (eAppendix A). Naturally, a more complete analysis would include representations of all permutations of comorbidities; however, data on correlations across all major comorbidities are not readily available, so we focused on those that were most prevalent and clinically important. We then simulated a basic medical history for each hypothetical health insurance member by sampling simple distributions for BMI, BP, cholesterol, and diabetes (eAppendix A).

Intervention-Specific Adherence Rates

For the sake of analytical tractability, we made a simplistic assumption, classifying each USPSTF recommendation as intrinsically “easy” or “difficult” to adhere to—a characteristic distinct from, but potentially interacting with, a patient’s propensity to adhere. Reported levels of adherence in published literature are heterogeneous and often dated,^26,27 so we used prior studies in conjunction with our clinical judgment to arrive at these classifications. The set of “difficult” recommendations was composed of weight loss, tobacco cessation, reduction in alcohol use, healthy diet, and lifestyle (nonmedication) components of cholesterol and hypertension management. For difficult recommendations, we assumed adherence rates of 30% during the first year, based on reported adherence rates to tobacco cessation (which ranged from 20% to 40% after 26 weeks).26 All other recommendations were considered “easy,” for which we assumed adherence rates of 70% during the first year. These adherence rates were primarily based on compliance rates with antihypertensive^26,27 and statin^28,29 medications, though prior literature had a wide range of estimates. The first-year rate of adherence was followed by a lower probability of continued adherence after 1 year, reflecting what might happen if the health system did not effectively reinforce adherence in subsequent years. (eAppendix A provides further details.)

RESULTS

Table 1

We developed a composite quality metric with weights that reflected the relative quantities of health benefit conferred across a typical health insurance system membership panel. shows summary statistics: across all ages, the 3 most prevalent conditions were overweight or obesity, hypertension, and tobacco use.

Table 2

eAppendix B

(and eAppendix Table) shows the magnitude of health benefit that would be imparted by an insurer that emphasized adherence to the single USPSTF-based guideline of most importance to each member, as reflected by its personalized benefit to life expectancy. For example, among females aged 55 to 64 years, the 3 greatest-benefit recommendations, or component weights, were weight loss, BP reduction, and mammography. One year of health system intervention to encourage adherence with these recommendations was associated with a mean life expectancy increase per member of 0.70, 0.33, and 0.18 years, respectively. Across all similarly aged women, successful implementation of this strategy would increase mean life expectancy per member, or composite health quality, by 0.47 years (SD = 0.33 years). More generally, in most age/gender categories, lifestyle recommendations such as losing weight or quitting smoking received the highest component weight. For individuals 55 years or older, BP reduction and aspirin chemoprophylaxis also conferred large benefits, as did screening mammography for women aged 35 to 64 years, and decreased alcohol use for men aged 25 to 44 years. Recommendations conferring the greatest health benefit were similar at shorter (5-, 10-, and 20-year) time horizons that may be more relevant to health insurers and employers ().

DISCUSSION

Researchers have been unable to quantify the health improvement provided by health systems in general, and from increasing patients’ adherence to healthcare recommendations in particular.^1-11 As a result, health system performance is often based on multiple individual quality metrics that are either considered separately, or counted equally despite their variation in quantity of health benefit conferred. Previous composite metrics have often been based on an all-or-none approach^1,12-14 or quality weights for outcomes surrounding a single intervention (eg, a single surgery).¹⁶ We have developed a health-weighted composite quality metric that estimates the amount of health benefit attributable to preventive care that is conferred by a particular health plan. It gives higher weights to those processes that have been associated with greater health gains, and varies these weights based on patient characteristics. This personalization allows a health-weighted composite quality metric to be different for each health plan member.

We found that health plans would contribute the greatest health benefit among preventive care interventions by encouraging the adoption of lifestyle recommendations such as tobacco cessation and weight loss, even taking into account the lower likelihood of continued adherence to these difficult recommendations. One year of adherence with these goals was associated with an estimated 0.5 to 1 year of additional life expectancy for men and women in every age group between 25 and 84 years. That is, even though patients were unlikely to maintain adherence to these recommendations over extended periods, benefits were so high that they remained a high priority for the majority of patients. Some may argue that it would be inappropriate to gauge individual physicians based on patient compliance to these metrics because whether a patient chooses to adhere (for example, by quitting smoking) lies beyond the physician’s control, and therefore, may not be a suitable quality metric.³⁰

However, health insurers may potentially be held more accountable, as promotion of coordinated care, lifestyle, and other wellness programs—which often involve non-MD time&mdash;may provide their members with large benefits. Additionally, easier recommendations that are more under the control of individual physicians (eg, mammography, aspirin use, and antihypertensives) also ranked high. We caution that these results are based on hypothetical patients using characteristics of the overall US population and disease prevalence; practical implementation would require more thorough consideration of model assumptions, quality metric validation, and feasibility studies for health systems.

Table 3

The attributable health gains estimated by our healthweighted quality metric might be considered a yardstick for health system performance, with plan-to-plan comparison through adjustment for baseline patient characteristics and differential adherence rates. The simplicity of a single holistic, outcomes-oriented measure also may appeal to patients seeking to evaluate the performance of different health plans before enrollment. For example, should future research validate our proposed metric (or a similar measure), health exchanges might consider reporting the overall health benefit conferred by each health insurance plan alongside its price ().

Limitations

Our approach has several limitations. We considered patient adherence with each recommendation in isolation—for example, losing weight or quitting smoking, but not both. Simultaneous adherence to multiple interventions may result in additional life expectancy that is greater or less than the sum of additional life expectancy for each individual intervention; the direction is unclear and may depend on the specific combination of interventions.^31,32 Second, we did not consider the time it would take a provider to encourage adherence—the inclusion of which would reduce emphasis on time-intensive lifestyle recommendations and increase emphasis on less time-intensive services, such as medications. However, as health insurers may choose to adopt nurses and other nonphysician healthcare providers for lifestyle recommendations, a weighting scheme based on time should also reflect the lower cost of these providers. Third, we made strong simplifying assumptions regarding adherence rates. While data on adherence rates were heterogeneous and dated, our framework was more sensitive to the distinction between difficult versus easy recommendations than to exact adherence rates. Fourth, for analytic tractability, we sometimes modeled adherence to a hard target (eg, 120/80 mm Hg), rather than along a continuous spectrum; more realistic implementation might consider progress in small increments. Fifth, we used national disease prevalence data, which may differ from those of an insured population.^33,34 However, improved access to insurance through healthcare reform may attenuate this limitation. Sixth, we did not consider socioeconomic status because its strong correlation with race would require substantive additional analyses. However, we intend to incorporate this consideration in future work. Finally, we weighted health benefit through the additional life expectancy attributable to a health system, which may differ from additional quality of life.

CONCLUSIONS

Mathematical models may be used to develop healthweighted composite quality metrics, which weight individual quality metrics by their contribution to patient health. Estimates of the amount of health imparted annually by a health system may be used to inform choices by patients, payers, and clinicians.Author Affiliations: Medicine Institute, Cleveland Clinic Foundation (GBT), Cleveland, OH; departments of Population Health and Medicine, New York University School of Medicine (RSB), New York, NY.

Source of Funding: This study was funded by unrestricted funds of the New York University School of Medicine. The sponsor had no role in the design or conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.

Author Disclosures: This work was completed while Dr Taksler was with the Departments of Population Health and Medicine, New York University School of Medicine, New York, NY. The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (GBT, RSB); acquisition of data (GBT); analysis and interpretation of data (GBT, RSB); drafting of the manuscript (GBT, RSB); critical revision of the manuscript for important intellectual content (GBT, RSB); statistical analysis (GBT); administrative, technical, or logistic support (GBT); supervision (RSB).

Address correspondence to: Glen B. Taksler, PhD, Medicine Institute, Cleveland Clinic Foundation, 9500 Euclid Ave, G1-40F, Cleveland, OH 44195. E-mail: taksleg@ccf.orgREFERENCES

1. Accountable care organization 2014 program analysis: quality performance standards, narrative measure specifications. CMS website. http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/ sharedsavingsprogram/Downloads/ACO-NarrativeMeasures-Specs. pdf. Published August 15, 2014. Accessed February 20, 2015.

2. Roper WL, Corrigan JM. Changing Healthcare by the Numbers: 2012 NQF Report to Congress. Washington, DC: National Quality Forum; 2012.

3. Panzer RJ, Gitomer RS, Greene WH, Webster PR, Landry KR, Riccobono CA. Increasing demands for quality measurement. JAMA. 2013; 310(18):1971-1980.

4. Conway PH, Mostashari F, Clancy C. The future of quality measurement for improvement and accountability. JAMA. 2013;309(21): 2215-2216.

5. Berenson RA, Pronovost PJ, Krumholz HM. Achieving the Potential of Health Care Performance Measures. Princeton, NJ: Robert Wood Johnson Foundation; 2013.

6. Porter ME. What is value in health care? N Engl J Med. 2010;363(26): 2477-2481.

7. Higgins A, Stewart K, Dawson K, Bocchino C. Early lessons from accountable care models in the private sector: partnerships between health plans and providers. Health Aff (Millwood). 2011;30(9): 1718-1727.

8. Pronovost PJ, Lilford R. Analysis & commentary: a road map for improving the performance of performance measures. Health Aff (Millwood). 2011;30(4):569-573.

9. Braithwaite S, Stine N. Health-weighted composite quality metrics offer promise to improve health outcomes in a learning health system. eGEMs. 2013;1(2):article 6.

10. Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB; ISPOR-SMDM Modeling Good Research Practices Task Force. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—7. Value Health. 2012;15(6): 843-850.

11. Focus on obesity and on Medicare plan improvement: the state of health care quality 2012. National Committee for Quality Assurance website. http://www.ncqa.org/Portals/0/State%20of%20Health%20 Care/2012/SOHC_Report_Web.pdf Published October 2012. Accessed August 14, 2013.

12. Part II. Introduction to measures of quality—selecting quality and resource use measures: a decision guide for community quality collaboratives. Agency for Healthcare Research and Quality. http://www. ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/ perfmeasguide/perfmeaspt2.html. Published May 2010. Accessed June 11, 2014.

13. Reeves D, Campbell SM, Adams J, Shekelle PG, Kontopantelis E, Roland MO. Combining multiple indicators of clinical quality: an evaluation of different analytic approaches. Med Care. 2007;45(6):489-496.

14. Nolan T, Berwick DM. All-or-none measurement raises the bar on performance. JAMA. 2006;295(10):1168-1170.

15. Zaslavsky AM, Shaul JA, Zaborski LB, Cioffi MJ, Cleary PD. Combining health plan performance indicators into simpler composite measures. Health Care Financ Rev. 2002;23(4):101-115.

16. Timbie JW, Shahian DM, Newhouse JP, Rosenthal MB, Normand SL. Composite measures for hospital quality using quality-adjusted life years. Stat Med. 2009;28(8):1238-1254.

17. The ABCs of measurement. National Quality Forum website. http:// www.qualityforum.org/Measuring_Performance/ABCs/ABCs_of_ Measurement.aspx. Accessed June 10, 2014.

18. Kern LM, Malhotra S, Barrón Y, et al. Accuracy of electronically reported “meaningful use” clinical quality measures: a cross-sectional study. Ann Intern Med. 2013;158(2):77-83.

19. Burstin H. The journey to electronic performance measurement. Ann Intern Med. 2013;158(2):131-132.

20. Kaafarani HM, Borzecki AM, Itani KM, et al. Validity of selected patient safety indicators: opportunities and concerns. J Am Coll Surg. 2011;212(6):924-934.

21. Taksler GB, Keshner M, Fagerlin A, Hajizadeh N, Braithwaite RS. Personalized estimates of benefit from preventive care guidelines: a proof of concept. Ann Intern Med. 2013;159(3):161-168.

22. US Census Bureau, Population Division. Annual Estimates of the Resident Population by Sex, Age, Race, and Hispanic Origin for the United States and States: April 1, 2010 to July 1, 2012. US Census Bureau: Washington, DC; 2013.

23. Yarnall KS, Pollak KI, Østbye T, Krause KM, Michener JL. Primary care: is there enough time for prevention? Am J Public Health. 2003; 93(4):635-641.

24. Publications and information products: health, United States, 2011. CDC website. http://www.cdc.gov/nchs/hus/contents2011.htm. Updated March 14, 2013. Accessed May 15, 2013.

25. Howlader N, Noone AM, Krapcho M, et al (eds); National Cancer Institute. SEER Cancer Statistics Review, 1975-2010. http://seer.cancer .gov/csr/1975_2010/. Based on November 2012 SEER data submission, posted to the SEER web site, April 2013. Accessed February 20, 2015.

26. Sabaté E; World Health Organization. Adherence to long-term therapies: evidence for action. WHO website. http://apps.who.int/medicinedocs/pdf/s4883e/s4883e.pdf. Published 2003. Accessed August 13, 2013.

27. Caro JJ, Salas M, Speckman JL, Raggio G, Jackson JD. Persistence with treatment for hypertension in actual practice. CMAJ. 1999;160(1): 31-37.

28. Shepherd J, Cobbe SM, Ford I, et al; for the West of Scotland Coronary Prevention Study Group. Prevention of coronary heart disease with pravastatin in men with hypercholesterolemia. N Engl J Med. 1995;333(20):1301-1307.

29. Blackburn DF, Dobson RT, Blackburn JL, Wilson TW, Stang MR, Semchuk WM. Adherence to statins, beta-blockers and angiotensinconverting enzyme inhibitors following a first cardiovascular event: a retrospective cohort study. Can J Cardiol. 2005;21(6):485-488.

30. Appendix A: measure evaluation framework—selecting quality and resource use measures: a decision guide for community quality collaboratives. AHRQ website. http://www.ahrq.gov/professionals/qualitypatient-safety/quality-resources/tools/perfmeasguide/perfmeasappa. html. Published October 2014. Accessed February 20, 2015.

31. Owens DK, Goldhaber-Fiebert JD. Prioritizing guideline-recommended interventions. Ann Intern Med. 2013;159(3):223-224.

32. Matthys J. To the editor: personalized estimates of benefit from preventive care guidelines. Ann Intern Med. 2014;160(2):140.

33. Davidoff A, Kenney G; Urban Institute; University of Maryland. Uninsured Americans with chronic health conditions: key findings from the National Health Interview Survey. Urban Institute website. http://www.urban.org/UploadedPDF/411161_uninsured_americans.pdf. Published May 2005. Accessed November 14, 2013.

34. Pfizer facts: a profile of uninsured persons in the United States. Pfizer website. http://www.pfizer.com/files/products/Profile_of_uninsured_persons_in_the_United_States.pdf. Published October 2008. Accessed November 14, 2013.

Download PDF: Developing a Composite Weighted Quality Metric to Reflect the Total Benefit Conferred by a Health Plan