Publication
Article
The American Journal of Managed Care
All-or-Nothing Treatment Targets Make Bad Performance Measures
Author(s):
Journal
Last year, the National Committee for QualityAssurance (NCQA) approved 2 new Health PlanEmployer Data and Information Set (HEDIS)measures for people with diabetes mellitus (DM):to achieve a glycosylated hemoglobin (A1C) level measurementof <7% and a blood pressure (BP) of <130/80 mm Hg.This decision was the culmination of a 5-year battle betweenDM advocates/experts and evidence-based medicine advocates/experts. In this issue of the , Pogach et al*address the importance of considering comorbidities whenimplementing optimal care measures.1 This editorial, however,will discuss the broader question of why this intuitivelyappealing approach–using "optimal" treatment goals as performancemeasures–will almost always require more sophisticatedmeasurement approaches (like those proposed byPogach and others) or else risk generating performance measuresthat are inaccurate, promote waste, and perhaps causesubstantial patient harm.1-7
The newly adopted performance measures are outcomemeasures, not processes, and like most outcomes, patients' riskof having the outcome (A1C <7% or BP <130/80) will varydramatically depending on patient attributes (disease severity,personal choices, human capital, and response to treatment).Even when risk adjustment is superb, outcomemeasures can be heavily prone to inaccuracy, statisticalinefficiency, and vulnerability to gaming,3-4,6-8 which is whymodern performance measurement efforts usually focus onprocesses (the care people receive). Therefore, readers may beperplexed as to why 2 new outcome measures lacking any riskadjustment were adopted. In truth, these new measures werea compromise between advocates of optimal goals (diseaseadvocates) and advocates of simple, inexpensive performancemeasures (health plan leadership). Experts in medical evidencewere not included in the compromise, which is part ofthe problem. It would be wonderful if measuring optimal carewas simple and easy (and on rare occasions it is). In general,however, simple and easy measures are restricted to measuringpoor care. Measuring more optimal care almost alwaysrequires added complexity and detail and particular attentionto patient preferences and the risks and costs of the interventionsneeded to achieve idealized treatment goals.
Payers, disease advocates, consumer groups, and politicalleaders are often dismissive of the complex reality of measuringcare, suggesting that performance measures may haveproblems, but medical-evidence experts have unrealistic standardsand are supplying the enemies of accountability withammunition. I am not unsympathetic to such concerns.There are enemies of performance measurement who nitpickat every little imperfection in performance measurements. Ifirmly believe that worrying about modest problems withmeasures is counterproductive. However, it is also true thatwishful thinking will not transform poor performance measuresinto useful ones, and that well-meaning people have aprofound aptitude for letting their desires and ideology blindthem to unwanted facts and complexities that are so vexinglycommon in the real world. In particular, many leaders in performancemeasurement have little appreciation for, or interestin, how unimportant small deviations from "optimal" goalscan be, or how treatment benefits and safety can vary widelyacross a patient population in complex ways, or that optimalcare by definition must consider patient and societal preferences.2,9-13 Most of all, promoting optimal care using performancemeasures requires considering the very real tensionsamong treatment-related benefits and treatment-related burdens,risks, and costs. HL Mencken once said, "For every problem,there is a solution that is simple, neat, and wrong,"14and using unadjusted "all-or-nothing" optimal treatment targetsas performance measures is such an example.
Almost everyone who voted for the 2 new HEDIS measuresdid not know that the Technical Expert Panel (TEP) of theDiabetes Alliance14 unanimously rejected the proposedA1C <7% and BP <130/80 measures even after being asked toreconsider their decision by political leaders of the Alliance.The TEP, citing the evidence-based guidelines of the AmericanCollege of Physicians and Veterans' Affairs/Departmentof Defense, noted that the proposed performance measureswere inconsistent with the medical evidence and that they hadmajor concerns regarding perverse incentives inherent in theproposed measures. However, the TEP offered multiple compromiseapproaches to measuring the quality of glycemic andBP control, with each proposal rejected either because it wasnot strict enough (opposed by diabetes advocates) or was toocomplex (opposed by the health plans).
Blood Pressure Control
Often, the majority of patients who achieve an "optimal"treatment goal, like BP <130/80, are those with no or mild disease,meaning that the measure largely captures the diseaseseverity of the patient population, and even worse, is oftenineffective in capturing true variations in quality, especiallythe variations in the care of high-risk patients where most preventablemorbidity and mortality resides.9-13 Dichotomizingthe intermediate outcome measure makes this problem muchworse. For example, using BP <130/80 as a thin, bright lineresults in rating the care of a patient with a naturally low BPas good care, but does a poor job of distinguishing truly goodcare from truly bad care. Even after prescribing 3 or 4 antihypertensivemedications and paying careful attention to medicationadherence, those with severe hypertension will usuallyhave persistent elevations of systolic BP.13 The irony is thatthe BP <130/80 measure provides greater rewards for speculativelytreating patients with mild disease (no clinical trial hasdemonstrated aggressively treating people with DM with mildBP elevations is either beneficial or safe) and does a poor jobof rewarding the treatments shown in clinical trials to producedramatic reductions in disability and mortality, because only asmall minority of the severely hypertensive patients studiedin the clinical trials achieved a systolic blood pressure (SBP)<130.13 Not only is there no evidence that using more than 3or 4 medications in pursuit of the <130/80 goal is beneficial,but there is consistent grade B evidence that such treatmentmay increase cardiovascular mortality in those who havealready achieved a diastolic blood pressure (DBP) <70.13-16This is not a rare event, a quick look at National Healthand Nutrition Examination Survey data reveals that ofpatients 65 years of age and older, about a third of people withDM with SBP >130 already have a DBP <70, meaning thatthe new HEDIS measure will frequently be promoting carethat the best available evidence suggests will increase cardiovascularmortality.
Kerr et al have proposed measures that focus on how cliniciansand systems respond to clinical indications (such as elevationsin BP, A1C, and lipids) and how this improvesaccuracy and reduces perverse incentives for overtreatmentand gaming.3,4 Such an approach would also result in avoidingthe safety concerns outlined above. However, nonclinicianstend to argue that prescribing the appropriate treatment is lessimportant than whether the treatment target is achieved. Thispoint of view shows a lack of understanding of epidemiologicalevidence and has been discussed fully elsewhere.9-13,17 Much ofthis literature, however, can be summed up as follows: (1) clinicaltrials assess treatments, not treatment targets; (2) in mostinstances, the risks associated with deviations from optimalgoals follow an exponential pattern (so that small deviationsfrom treatment targets are often of little importance); and (3)because most medical treatments have side effects, risks, andcosts, pursuing minimal deviations from idealized goals willoften involve speculative care that is not worth the cost andmay expose patients to undue risks and burden.
A1C Control
The A1C <7% measure produces the same problems asthose discussed above. In addition, the A1C <7% measureallows the manufacturers of several new and expensive hypoglycemicmedications to promote their products using theA1C <7% measure as "evidence" that achieving this goal isvery important. This point was not lost on the pharmaceuticalindustry, which funded a national campaign to get theA1C <7% measure adopted by the NCQA. This was a verysmall investment compared with the costs of funding clinicaltrials to evaluate whether these expensive new medicationshave benefits that are worth their risks (forget about cost-effectiveness;we do not even know if they are safe in pursuitof tight glycemic control). Although diabetes experts oftenpoint to the United Kingdom Prospective Diabetes Study(UKPDS) as evidence that achieving an A1C <7% is worththe unknown long-term risks of new medications, experts inepidemiological evidence will point out that the UKPDSshows no such thing.17,18 The UKPDS found that combinationoral hypoglycemic therapy (metformin plus sulfonylurea)was associated with increased diabetes-relatedmortality, metformin monotherapy resulted in substantialpatient benefit, and monotherapy with sulfonylureas orinsulin had no significant effects on any of the prespecifiedpatient outcomes, although the control achieved for A1Cwas very similar between groups.18 Further, although modelingstudies strongly suggest that those with early-onset diabeteswill get moderate benefit in the long term (if they livemore than 15-20 years and you assume that the treatmentshave almost zero risk and disutility), older patients are veryunlikely to get benefit.12,17 Once again, focusing on thedichotomized "optimal" treatment goal results in givingmuch more credit for providing speculative treatments forpatients with mild disease than for providing high-prioritycare. For example, a system that targets the new, expensivehypoglycemic treatments to patients who least need them(eg, patients with an A1C of 6.7%-7.7%) will do much betterthan one that targets these treatments to the best candidatesfor these treatments (eg, patients with an A1C of8%-9%). This is why the TEP rejected the A1C <7% measurebut proposed an A1C <8% or a weighted measure (onethat places more importance on large deviations from optimalthan trivial deviations) as alternatives.2-4
It is certainly a reasonable goal to want performance measuresfor optimal care standards to be simple, but optimal careis almost never simple. Some leaders in performance measurementhave asked me, "Do you really think that thesemeasures will lead clinicians and health systems toovertreat?" I am frankly amazed by this question. Spiralinghealthcare costs and overtreatment are probably the definingfeatures of the US healthcare system. Industry-funded"experts" and disease advocates have been effectively promotingovertreatment for decades, and performance measurementwas supposed to be a tool to bring better value tohealthcare spending. Although performance measurementhas proved to be a very powerful tool,19 like all tools it providesopportunities for both benefit and harm. It is simplymagical thinking to believe that performance measures willdo good regardless of how haphazardly they are constructed,and that they will not do harm even when the measuresadopted provide strong incentives for over-treatment.
Author Affiliation:
From the VA Ann Arbor Health Services Research &Development, Center of Excellence, University of Michigan.
Correspondence Author:
Rodney A. Hayward, MD, Director, VA AnnArbor Health Services Research & Development, Center of Excellence,Professor of Medicine and Public Health, University of Michigan, 6312 MedSci I, Ann Arbor, MI 48109-0604; E-mail: rhayward@umich.edu.The opinions and views presented in this editorial are solely those of theauthor and do not necessarily represent those of the Department of VeteransAffairs or the University of Michigan.
Am J Manag Care.
1. Pogach LM,Tiwari A, Maney M, Rajan M, Miller DR, Aron D. Shouldmitigating comorbidities be considered in assessing healthcare planperformance in achieving optimal glycemic control. 2007;13:133-140.
Diabetes Care.
2. Pogach LM, Rajan M, Aron DC. Comparison of weighted performancemeasurement and dichotomous thresholds for glycemic controlin the Veterans Health Administration. 2006;29:241-246.
Am J Manag Care.
3. Kerr EA, Krein SL,Vijan S, Hofer TP, Hayward RA. Avoiding pitfalls inchronic disease quality measurement: a case for the next generation oftechnical quality measures. 2001;7:1033-1043.
Med Care.
4. Kerr EA, Smith DM, Hogan MH, et al. Building a better quality measure:are some patients with "poor quality" actually getting good care?2003;41:1173-1182.
Jt Comm J Qual Saf.
5. O'Malley AS, Clancy C,Thompson J, Korabathina R, Meyer GS.Clinical practice guidelines and performance indicators as related—butoften misunderstood—tools. 2004;30:163-171.
Diabetes
Care.
6. Hayward RA, Hofer TP, Kerr EA, Krein SL. Quality improvement initiatives:issues in moving from diabetes guidelines to policy. 2004;27(suppl 2):B54-B60.
JAMA.
7. Hofer TP, Hayward RA, Greenfield S,Wagner EH, Kaplan SH,Manning WG.The unreliability of individual physician "report cards"for assessing the costs and quality of care of a chronic disease. 1999;281:2098-2105.
Ann Intern Med.
8. Rosenberg AL, Hofer TP, Strachan C,Watts CM, Hayward RA.Accepting critically ill transfer patients: adverse effect on a referral center'soutcome and benchmark measures. 2003;138:882-890.
Ann Intern Med.
9. Hayward RA, Hofer TP,Vijan S. Narrative review: lack of evidence forrecommended low-density lipoprotein treatment targets: a solvableproblem. 2006;145:520-530.
Health Aff
(Millwood).
10. Hayward RA, Kent DM,Vijan S, Hofer TP. Reporting clinical trialresults to inform clinical providers, payers and consumers. 2005;24:1571-1581.
Am J Manag Care.
11. McMahon LF, Hayward RA, Saint S, Chernew ME, Fendrick AM.Univariate solutions in a multivariate world: can we afford to practiceas in the "good old days"? 2005;11:473-476.
Ann Intern Med.
12.Vijan S, Hofer TP, Hayward RA. Estimated benefits of glycemic controlin microvascular complications in type 2 diabetes. 1997;127:837-839.
Ann Intern Med.
13.Vijan S, Hayward RA.Treatment of hypertension in type 2 diabetesmellitus: blood pressure goals, choice of agents, and setting prioritiesin diabetes care. 2003;138:593-602.
14. H. L. Mencken Quotes. Available at: http://www.brainyquote.com/quotes/h/hlmencke162005.html. Accessed February 20, 2007.
15. National Diabetes Quality Improvement Alliance. Available at:http://www.nationaldiabetesalliance.org/organizations.html. AccessedFebruary 9, 2007.
Ann Intern Med.
16. Messerli FH, Giuseppe Mancia G, Conti CR, et al. Dogma disputed:can aggressively lowering blood pressure in hypertensive patientswith coronary artery disease be dangerous? 2006;144:884-893.
Diabetologia.
17.Vijan S, Kent DM, Hayward RA. Are randomized controlled trialssufficient evidence to guide clinical practice in type II (non-insulindependent) diabetes mellitus? 2000;43:125-130.
BMJ.
18. Shaughnessy AF, Slawson DC. What happened to the valid POEMs?A survey of review articles on the treatment of type 2 diabetes. 2003;327:266.
Ann Intern Med.
19. Asch SM, McGlynn EA, Hogan MM, et al. Comparison of quality ofcare for patients in the Veterans Health Administration and patients ina national sample. 2004;141:938-945.
