Electronic Health Record Problem Lists: Accurate Enough for Risk Adjustment?

ABSTRACT

Objectives: To determine whether comorbidity information derived from electronic health record (EHR) problem lists is accurate.

Study Design: Retrospective cohort study of 1596 men diagnosed with prostate cancer between 1998 and 2004 at 2 Southern California Veterans Affairs Medical Centers with long-term follow-up.

Methods: We compared EHR problem list—based comorbidity assessment with manual review of EHR free-text notes in terms of sensitivity and specificity for identification of major comorbidities and Charlson Comorbidity Index (CCI) scores. We then compared EHR-based CCI scores with free-text–based CCI scores in prediction of long-term mortality.

Results: EHR problem list—based comorbidity assessment had poor sensitivity for detecting major comorbidities: myocardial infarction (8%), cerebrovascular disease (32%), diabetes (46%), chronic obstructive pulmonary disease (42%), peripheral vascular disease (31%), liver disease (1%), and congestive heart failure (23%). Specificity was above 94% for all comorbidities. Free-text–based CCI scores were predictive of long-term other-cause mortality, whereas EHR problem list–based scores were not.

Conclusions: Inaccuracies in EHR problem list—based comorbidity data can lead to incorrect determinations of case mix. Such data should be validated prior to application to risk adjustment.

Am J Manag Care. 2018;24(1):e24-e29Takeaway Points

Electronic health record (EHR)-based data are increasingly being used for purposes of risk adjustment, but our findings show that the Veterans Affairs EHR problem list led to incorrect determinations of case mix and did not predict survival, in contrast to chart-based comorbidity assessment. Given the increasing importance of EHR data, we believe that:

• EHR-based comorbidity data should be validated prior to application to risk adjustment.
• Other sources of comorbidity information, such as patient-reported measures of health status or natural language processing—derived data, may be considered for risk adjustment in comparisons of performance.

Recent study results highlight the inconsistency of different sources of data (eg, registries, claims, the electronic health record [EHR]) for identifying basic health information, such as major comorbidities.^1-5 For individual physicians, these inconsistencies are less relevant because they have the opportunity to confirm this information directly with the patient. However, when used for risk adjustment for purposes of performance assessment,^6-8 incorrect data may lead to misclassification and unfair comparisons. This is a major concern for health systems participating in alternative payment models, which base some portion of reimbursement on risk-adjusted quality outcomes.^9-13 Comorbidity is a key component of the risk adjustment needed for fair comparisons of measures of quality, as comorbid disease burden affects readmissions,^14-16 complications,^17-19 quality of life,^20,21 and mortality.^22-24 Accurately quantifying comorbidity requires varying degrees of detail regarding number, severity, or types of conditions, depending on the measure used.²⁵ All of these measures require robust data sources to identify the presence or absence of each of the included comorbid conditions.

Whereas comorbidity data from inpatient medical records are reviewed by trained coders, outpatient records may be less reliable,^1-5 as they often rely on “problem lists” in the EHR to identify the index condition. The problem list is a compilation of patient diagnoses entered by clinicians during patient encounters and updated at varying intervals. With increasing numbers of institutions and office-based practices using EHRs to store patient data, interest has grown in utilizing these lists as a source of comorbidity data.²⁶ However, it is unclear whether the data in these lists are sufficiently accurate to assess patients’ total comorbid disease burden. Recent studies have attempted to validate the accuracy of the problem list by comparing it with other diagnosis lists or with short-term future proximate outcomes, such as glycated hemoglobin.¹ However, the most appropriate metric for assessment of validity of these lists is long-term mortality, especially in an elderly population; a longer list of major comorbidities should be strongly associated with higher mortality.

In this study, we compared the ability of Charlson Comorbidity Index (CCI) scores derived from the Veterans Affairs (VA) problem list to predict mortality in an elderly population with a gold standard for comorbidity assessment, manual abstraction directly from the physician’s free-text notes. We captured mortality over a 10-year period, which was long enough to reveal the impact of even minor comorbidities over time. Because the problem list is not actively maintained, we hypothesized that the problem list would provide poor accuracy in identifying comorbidities and would poorly predict survival compared with free-text—based assessment.

METHODS

Data Sources and Study Participants

We used the California Cancer Registry to identify men newly diagnosed with prostate cancer at the Greater Los Angeles and Long Beach VA Medical Centers between 1998 and 2004 (N = 1915). We reviewed EHRs for sociodemographic, tumor risk, comorbidity, and survival data and identified all men with sufficient data to determine comorbidity and survival (n = 1596). Institutional review board approval was granted by the University of California, Los Angeles, and both VA Medical Centers.

Variables

Comorbidity. We assessed comorbid disease burden at the time of prostate cancer diagnosis using 2 sources of data: 1) the EHR free-text notes record and 2) the EHR problem list. The interdisciplinary EHR free-text notes record contained outpatient and inpatient notes from all clinical encounters. Data from the medical record within 12 months of the diagnosis of prostate cancer were used for free-text—based comorbidity assessment. Comorbidities were coded according to the definitions originally indicated by Charlson et al,²² and age-unadjusted CCI scores were calculated. Because free-text—based comorbidity assessment was conducted primarily by 1 author (TJD), reliability was assessed on a random 5% subset of the sample by a separate author (GA). Interrater agreement in Charlson scores was 77.5% and the associated kappa statistic was 0.67.

The VA problem list was populated and updated by clinicians variably as they deemed appropriate; diagnoses were coded by International Classification of Diseases, Ninth Revision (ICD-9) codes and dates and entered into a retrievable database. Comorbidities added to the problem list up to 12 months after the diagnosis of prostate cancer were used for EHR problem list—based comorbidity assessment. Comorbidities were coded according to the claims-based definitions indicated by Deyo et al,²⁷ and age-unadjusted Deyo-CCI scores were calculated. Neither prostate cancer nor any complications of prostate cancer were included in comorbidity scoring for either comorbidity assessment method.

Survival model covariates. Age at diagnosis was coded as a continuous variable. Race/ethnicity was coded as Caucasian, African American, Hispanic, or other. Tumor characteristics included prostate-specific antigen (PSA), Gleason sum, and clinical tumor (T), node, and metastasis stage at diagnosis. Categories for PSA, Gleason sum, and clinical T stage were defined by the widely accepted D’Amico criteria, which have been shown to predict overall and cancer-specific mortality.^28,29

Mortality. Survival was measured from date of treatment until date of death. We determined date of death using a combination of the medical record and the Social Security Death Index. Cause of death was determined using the medical record with an algorithm that has been previously described.³⁰

Statistical Analysis

We determined the prevalence of 7 major comorbidities using free-text—based and EHR problem list–based comorbidity assessment. Sensitivity and specificity for identification of major comorbidities were calculated, using free-text–based assessment as the gold standard. Agreement between assessment methods was ascertained for each comorbidity using Cohen’s kappa statistic.

We compared mean CCI scores based on the problem list and on free-text notes using a paired t test for difference in means. Continuous and categorical (0, 1, 2, and ≥3) versions of CCI scores for each assessment method were also compared using Pearson’s correlation and Cohen’s kappa, respectively.

Multivariable competing risks regression as described by Fine and Gray³¹ was used to compare prediction of survival by CCI scores derived from each comorbidity assessment method. We calculated subhazard and cumulative incidence of non—prostate cancer mortality by CCI scores for each comorbidity assessment method, treating prostate cancer as a competing risk. All multivariable regression models were adjusted for age, race, VA site, clinical stage, Gleason score, PSA, and treatment type.

We then conducted a sensitivity analysis to determine if year of diagnosis affected our results, as use of the EHR problem list may have changed over time. We subdivided our group into those diagnosed in 2001 or earlier (n = 810) and 2002 or later (n = 786) and repeated our analyses.

We used P <.05 to denote statistical significance, and all tests were 2-sided. All statistical analyses were performed in Stata, version 12.0 (Stata, Inc; College Station, Texas).

RESULTS

The majority of the sample (N = 1596) comprised white (44%) and African American men (37%). Approximately one-half of the sample was 65 years or younger, and most had early-stage prostate cancer (Table 1).

EHR problem list—based comorbidity assessment had poor sensitivity but high specificity for identification of common major comorbidities (Table 2). Sensitivity values for EHR problem list—based assessment (using free-text–based assessment as the gold standard) ranged from 8% for myocardial infarction to 46% for diabetes. Specificity was above 94% for all comorbidities. Agreement between EHR problem list–based and free-text–based comorbidity assessment was poor for all major comorbidities, with kappa ranging from 0.02 to 0.44. Results did not change after subdividing our group by those diagnosed before or after 2001.

Comparison of the CCI scores based on the EHR problem list and on free text showed that EHR problem list—based assessment underestimated comorbidity burden (Table 3). Agreement across all scores was 53% (840/1596). Pearson correlation for continuous scores was 0.3, and kappa for categorical scores was 0.2. Among scores that were discordant, 82% (627/765) were higher using free-text—based compared with EHR problem list–based comorbidity assessment. Mean free-text–based and EHR problem list–based CCI scores were 1.1 (95% CI, 1.04-1.19) and 0.5 (95% CI, 0.46-0.56), respectively. Free-text–based scores were significantly higher by a mean of 0.6 points (95% CI, 0.53-0.67; P <.001).

Competing risks regression analysis showed that free-text—based CCI scores predicted noncancer mortality, whereas EHR problem list–based scores did not. Higher free-text–based CCI scores were associated with a graduated increase in hazard of other-cause mortality (Table 4), and 10-year cumulative incidence of noncancer mortality associated with chart-based scores was 41%, 57%, 64%, and 83% for CCI scores of 0, 1, 2, and ≥3, respectively (Figure, part A). Higher EHR problem list—based CCI scores were not associated with increased mortality risk (Table 4), and EHR problem list–based CCI scores did not discriminate 10-year cumulative incidence of noncancer mortality: 55%, 64%, 62%, and 58% for CCI scores of 0, 1, 2, and ≥3, respectively (Figure, part B). Results did not change after subdividing our group by those diagnosed before or after 2001.

DISCUSSION

Despite interest in capitalizing on readily available problem list data in the EHR for purposes of risk adjustment, our findings suggest that these data should be validated prior to application to performance assessment. The sensitivity of the VA problem list for identifying common major comorbidities was poor, ranging from 1% to 46%, compared with manual free-text note abstraction. This lack of accuracy led to incorrect risk adjustment and would lead to unfair comparisons of clinicians’ quality of care if applied directly to risk adjustment for performance assessment. These data may be generalizable to more contemporary EHRs because comorbidity information in problem lists and outpatient claims still is not actively maintained. Absent a valid method of adjustment for comorbidity, it is not possible to confidently distinguish between physicians or groups who provide poor care and those who disproportionately see patients with greater disease burden. Because measures of quality of care are now being tied to compensation in programs like value-based purchasing,³² the stakes are higher and the consequences of errors in performance assessment are much more substantial. These errors are magnified when the units of comparison are smaller (eg, physician groups or individual physicians).

Previous studies have shown that failure to adjust for case mix can alter quality rankings.^33-37 For example, in a comparison of functional outcomes at 4 hospitals 1 year after total hip replacement, significant differences in institutional outcomes were apparent before adjustment for comorbid disease status but not after.³³ In our study, CCI scores from free-text—based assessment predicted mortality, whereas EHR problem list–based scores did not, likely due to the latter’s poor sensitivity for detecting major comorbid diseases. For clinical purposes, such as risk adjustment of quality scores, versus billing and utilization purposes, comorbidity scores relying on EHR diagnoses could be misleading.

Inaccuracy in identification of major comorbid conditions and its impact on risk adjustment could have significant financial consequences for physicians and physician groups participating in accountable care organizations (ACOs). ACOs, such as those participating in the Medicare Shared Savings Program, require participating organizations to report certain quality measures to CMS.¹³ Only providers who meet the program’s quality performance benchmarks will be eligible to receive a share of cost savings, which can be substantial. For example, in the Physician Group Practice Demonstration, 7 of 10 groups received $108 million in shared savings payments over the course of the demonstration.³⁸ Providers who do not meet quality benchmarks will be subject to penalties. Although early ACOs have started with fairly undemanding process benchmarks for quality, such as the Healthcare Effectiveness Data and Information Set measures for chronic disease management, which are met by a national average of 79% to 80% of groups,¹¹ quality benchmarks will likely become more stringent over time, as evidenced by the inclusion of all-cause readmission in quality scoring in the third year of participation in the Medicare Shared Savings Program.³⁹ There is an almost 2-fold variation in readmissions across different areas of the United States,⁴⁰ and these differences have been found to be sensitive to risk adjustment for comorbidity.^14-16 Although ACOs have shown prudence in their meticulous and transparent statistical methodology for risk adjustment,¹³ there has been little focus on the quality of the comorbidity data.

Performance assessment in the outpatient setting may be particularly susceptible to failures in risk adjustment, as outpatient claims typically used for risk adjustment are closely linked with problem list data. In contrast with current approaches to risk adjustment in hospital quality rankings, in which inpatient claims data are reviewed by trained hospital personnel, outpatient claims data may be directly taken from problem list data. They may not be corroborated by trained coding teams, in contrast with inpatient claims. Furthermore, outpatient claims may be limited to those principal diagnoses for which patients seek care in the outpatient setting and may not represent the total comorbid disease burden of the individual. This could be an explanation for the apparent underestimation of CCI scores by EHR problem list—based assessment in our study. Conversion from ICD-9 to International Classification of Diseases, Tenth Revision coding algorithms is also unlikely to affect these deficiencies, as the new coding platform is more apt to improve the specificity, but not sensitivity, of claims data. Previous work has highlighted the deficiencies of outpatient claims—based approaches to comorbidity assessment.^3-5 Yet despite these concerns, outpatient claims data are frequently used for risk adjustment for numerous applications, including quality assessment.^41,42

Although the accuracy of contemporary EHR diagnosis lists may differ from that of the VA problem list of the early 2000s, emphasis still is not placed on the accuracy of this data element more than a decade later, and it is likely that these deficiencies still persist, even outside of the VA. Clinicians must voluntarily add or subtract diagnoses through a process separate from billing. Contemporary EHR diagnosis lists are not routinely updated by administrative or clinical staff, and there is no compelling incentive for reconciliation of the data. Future diagnoses are also collected cumulatively without expiration dates for time-limited or rule-out diagnoses, and some problem lists are truncated to the most current issues, which can unintentionally omit major chronic diagnoses. These facts suggest that more modern EHRs are subject to many of the drawbacks that led to the extremely poor results observed in our study. A study of the Massachusetts General Hospital Primary Care Practice-Based Research Network database from 2005 to 2011, comparing assessment of patient complexity through comprehensive EHR data (demographic, diagnostic, procedure, medication, laboratory, and utilization) versus outpatient CCI scores versus a commercial risk predictor, showed more than 60% disagreement among methods.² The results of a study of contemporary outpatient claims data have also shown poor accuracy compared with registry data obtained from chart review.¹ We feel that the summation of these findings argues for more active management of EHR diagnosis lists (and outpatient claims) going forward.

Limitations

Our study has several limitations. First, because hospital system EHR training varies by institution, it is possible, although highly unlikely, that the accuracy of the VA problem list in the 2 centers in our study differs from that observed across the entire VA. Second, differences in our VA sample (ie, disproportionately male and African American) may affect generalizability to other populations. Third, there are differences in comorbidity assessment between Deyo-Charlson and CCI scores that make comparisons between the 2 scoring systems dissimilar; however, these slight differences would not explain the large disparity observed in our study. Fourth, free-text—based comorbidity assessment included chart review of the comprehensive medical record within 12 months before and after diagnosis of prostate cancer, whereas EHR problem list–based comorbidity assessment included all diagnoses until 12 months after diagnosis. This intentional difference in methodology accounts for the fact that medical notes routinely document the cumulative chronic disease burden of the individual, whereas the problem list relies on date of diagnosis. This difference would tend to enhance sensitivity of the EHR problem list–based approach, opposing the direction of effect observed in our study.

CONCLUSIONS

Although there is great enthusiasm for using readily available comorbidity data in EHRs for risk adjustment in performance assessment, the findings of this proof-of-concept study suggest that such data need to be corroborated prior to application. If these findings are replicated in modern EHRs, what can be done? One approach is to develop an automated electronic search strategy to review free-text notes for key words or phrases indicating comorbid diseases, but the programming efforts for this strategy are daunting.³ We and others have taken another approach: to gather standardized quantifiable information from patients. This strategy has been shown in numerous studies to be accurate and reliable in the assessment of comorbidity.^23,24 As ACOs and other entities increasingly use achievement of quality benchmarks, such as 30-day readmissions and all-cause mortality, to allot shared savings bonuses and penalties, the quality of risk adjustment data will be increasingly important, especially in the outpatient setting, where claims data are taken from the problem list. The stakes are high for quality indicators to be unbiased and fair in order to avoid unduly penalizing providers and organizations who care for the sickest patients.Author Affiliations: Division of Urology (TJD), and Cedars-Sinai Center for Outcomes Research and Education (BS), Cedars-Sinai Medical Center, Los Angeles, CA; Greater Los Angeles Veterans Affairs Medical Center (TJD), Los Angeles, CA; Department of Urology (GA, DS, TA), and Health Policy Research Institute (SHK, SG), and Department of Medicine (SHK, SG), University of California, Irvine, Irvine, CA; Long Beach Veterans Affairs Medical Center (GA, DS), Long Beach, CA; Department of Urology, David Geffen School of Medicine (MSL), and Department of Health Policy & Management, Fielding School of Public Health (MSL), University of California, Los Angeles, Los Angeles, CA.

Source of Funding: None.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (TJD, BS, MSL, SG); acquisition of data (GA, DS, TA); analysis and interpretation of data (GA, SHK, TA); drafting of the manuscript (TJD); critical revision of the manuscript for important intellectual content (TJD, GA, SHK, DS, TA, BS, MSL, SG); statistical analysis (SHK); administrative, technical, or logistic support (DS); and supervision (BS, MSL, SG).

Address Correspondence to: Timothy J. Daskivich, MD, MSHPM, Cedars-Sinai Medical Center, 8635 W 3rd St, Ste 1070W, Los Angeles, CA 90048. Email: Timothy.Daskivich@csmc.edu.REFERENCES

1. McCoy RG, Tulledge-Scheitel SM, Naessens JM, et al. The method for performance measurement matters: diabetes care quality as measured by administrative claims and institutional registry. Health Serv Res. 2016;51(6):2206-2220. doi: 10.1111/1475-6773.12453.

2. Hong CS, Atlas SJ, Ashburner JM, et al. Evaluating a model to predict primary care physician-defined

complexity in a large academic primary care practice-based research network. J Gen Intern Med. 2015;30(12):1741-1747. doi: 10.1007/s11606-015-3357-8.

3. Singh B, Singh A, Ahmed A, et al. Derivation and validation of automated electronic search strategies to extract Charlson comorbidities from electronic medical records. Mayo Clin Proc. 2012;87(9):817-824. doi: 10.1016/j.mayocp.2012.04.015.

4. Olomu AB, Corser WD, Stommel M, Xie Y, Holmes-Rovner M. Do self-report and medical record comorbidity data predict longitudinal functional capacity and quality of life health outcomes similarly? BMC Health Serv Res. 2012;12:398. doi: 10.1186/1472-6963-12-398.

5. Steinwachs DM, Stuart ME, Scholle S, Starfield B, Fox MH, Weiner JP. A comparison of ambulatory Medicaid claims to medical records: a reliability assessment. Am J Med Qual. 1998;13(2):63-69. doi: 10.1177/106286069801300203.3

6. 2015 Physician Quality Reporting System (PQRS): implementation guide. CMS website. cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/PQRS/Downloads/2015_PQRS_ImplementationGuide.pdf. Updated October 2, 2015. Accessed November 2, 2016.

7. Findlay S. What’s next for Physician Compare? The Health Care Blog website. thehealthcareblog.com/blog/2015/01/23/whats-next-for-physician-compare. Published January 23, 2015. Accessed November 2, 2016.

8. Physician Compare. Medicare.gov website. medicare.gov/physiciancompare/search.html. Accessed November 2, 2016.

9. Colla CH, Wennberg DE, Meara E, et al. Spending differences associated with the Medicare Physician Group Practice Demonstration. JAMA. 2012;308(10):1015-1023. doi: 10.1001/2012.jama.10812.

10. Fisher ES, Shortell SM. Accountable care organizations: accountable for what, to whom, and how. JAMA. 2010;304(15):1715-1716. doi: 10.1001/jama.2010.1513.

11. Song Z, Safran DG, Landon BE, et al. The “Alternative Quality Contract,” based on a global budget, lowered medical spending and improved quality. Health Aff (Millwood). 2012;31(8):1885-1894. doi: 10.1377/hlthaff.2012.0327.

12. Song Z, Rose S, Safran DG, Landon BE, Day MP, Chernew ME. Changes in health care spending and quality 4 years into global payment. N Engl J Med. 2014;371(18):1704-1714. doi: 10.1056/NEJMsa1404026.

13. CMS. Methodology for determining shared savings and losses under the Medicare Shared Savings Program. Bi-State Primary Care Association website. bistatepca.org/uploads/pdf/Health%20Center%20Resources/ACO/ACO_Methodology_Factsheet_ICN907405.pdf. Published April 2014. Accessed November 2, 2016.

14. Librero J, Peiró S, Ordiñana R. Chronic comorbidity and outcomes of hospital care: length of stay, mortality, and readmission at 30 and 365 days. J Clin Epidemiol. 1999;52(3):171-179. doi: 10.1016/S0895-4356(98)00160-7.

15. Marcantonio ER, McKean S, Goldfinger M, Kleefield S, Yurkofsky M, Brennan TA. Factors associated with unplanned hospital readmission among patients 65 years of age and older in a Medicare managed care plan. Am J Med. 1999;107(1):13-17. doi: 10.1016/S0002-9343(99)00159-X.

16. Kassin MT, Owen RM, Perez SD, et al. Risk factors for 30-day hospital readmission among general surgery patients. J Am Coll Surg. 2012;215(3):322-330. doi: 10.1016/j.jamcollsurg.2012.05.024.

17. Greenfield S, Apolone G, McNeil BJ, Cleary PD. The importance of co-existent disease in the occurrence of postoperative complications and one-year recovery in patients undergoing total hip replacement. Comorbidity and outcomes after hip replacement. Med Care. 1993;31(2):141-154.

18. Ferrier MB, Spuesens EB, Le Cessie S, Baatenburg de Jong RJ. Comorbidity as a major risk factor for mortality and complications in head and neck surgery. Arch Otolaryngol Head Neck Surg. 2005;131(1):27-32.

19. Alibhai SM, Leach M, Tomlinson G, et al. 30-day mortality and major complications after radical prostatectomy: influence of age and comorbidity. J Natl Cancer Inst. 2005;97(20):1525-1532. doi: 10.1001/archotol.131.1.27.

20. Fortin M, Lapointe L, Hudon C, Vanasse A, Ntetu AL, Maltais D. Multimorbidity and quality of life in primary care: a systematic review. Health Qual Life Outcomes. 2004;2:51. doi: 10.1186/1477-7525-2-51.

21. Daskivich TJ, van de Poll-Franse LV, Kwan L, Sadetsky N, Stein DM, Litwin MS. From bad to worse: comorbidity severity and quality of life after treatment for early-stage prostate cancer. Prostate Cancer Prostatic Dis. 2010;13(4):320-327. doi: 10.1038/pcan.2010.33.

22. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373-383.

23. Greenfield S, Billimek J, Pellegrini F, et al. Comorbidity affects the relationship between glycemic control and cardiovascular outcomes in diabetes: a cohort study. Ann Intern Med. 2009;151(12):854-860. doi: 10.7326/0003-4819-151-12-200912150-00005.

24. Daskivich T, Sadetsky N, Kaplan SH, Greenfield S, Litwin MS. Severity of comorbidity and non-prostate cancer mortality in men with early-stage prostate cancer. Arch Intern Med. 2010;170(15):1396-1397. doi: 10.1001/archinternmed.2010.251.

25. de Groot V, Beckerman H, Lankhorst GJ, Bouter LM. How to measure comorbidity: a critical review of available methods. J Clin Epidemiol. 2003;56(3):221-229. doi: 10.1016/S0895-4356(02)00585-1.

26. Anderson JE, Chang DC. Using electronic health records for surgical quality improvement in the era of big data. JAMA Surg. 2015;150(1):24-29. doi: 10.1001/jamasurg.2014.947.

27. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45(6):613-619.

28. D’Amico AV, Whittington R, Malkowicz SB, et al. Biochemical outcome after radical prostatectomy, external beam radiation therapy, or interstitial radiation therapy for clinically localized prostate cancer. JAMA. 1998;280(11):969-974. doi: 10.1001/jama.280.11.969.

29. Boorjian SA, Karnes RJ, Rangel LJ, Bergstralh EJ, Blute ML. Mayo Clinic validation of the D’Amico risk group classification for predicting survival following radical prostatectomy. J Urol. 2008;179(4):1354-1360. doi: 10.1016/j.juro.2007.11.061.

30. Daskivich TJ, Chamie K, Kwan L, et al. Overtreatment of men with low-risk prostate cancer and significant comorbidity. Cancer. 2011;117(10):2058-2066. doi: 10.1002/cncr.25751.

31. Fine JP, Gray RJ. A proportional hazards model for the subdistribution of a competing risk. J Am Stat Assoc. 1999;94(446):496-509. doi: 10.2307/2670170.

32. VanLare JM, Conway PH. Value-based purchasing—national programs to move from volume to value. N Engl J Med. 2012;367(4):292-295. doi: 10.1056/NEJMp1204939.

33. Greenfield S, Aronow HU, Elashoff RM, Watanabe D. Flaws in mortality data. the hazards of ignoring comorbid disease. JAMA. 1988;260(15):2253-2255. doi: 10.1001/jama.1988.03410150101039.

34. Hannan EL, Racz MJ, Jollis JG, Peterson ED. Using Medicare claims data to assess provider quality for CABG surgery: does it work well enough? Health Serv Res. 1997;31(6):659-678.

35. Mehta RH, Liang L, Karve AM, et al. Association of patient case-mix adjustment, hospital process performance rankings, and eligibility for financial incentives. JAMA. 2008;300(16):1897-1903. doi: 10.1001/jama.300.16.1897.

36. Ohl ME, Richardson KK, Goto M, Vaughan-Sarrazin M, Schweizer ML, Perencevich EN. HIV quality report cards: impact of case-mix adjustment and statistical methods. Clin Infect Dis. 2014;59(8):1160-1167. doi: 10.1093/cid/ciu551.

37. O’Malley AJ, Zaslavsky AM, Elliott MN, Zaborski L, Cleary PD. Case-mix adjustment of the CAHPS Hospital Survey. Health Serv Res. 2005;40(6 pt 2):2162-2181. doi: 10.1111/j.1475-6773.2005.00470.x.

38. Evaluation of the Medicare Physician Group Practice demonstration. CMS website. downloads.cms.gov/files/cmmi/medicare-demonstration/PhysicianGroupPracticeFinalReport.pdf. Published September 2012. Accessed November 2, 2016.

39. Medicare Shared Savings Program quality measure benchmarks for the 2014 reporting year. CMS website. cms.gov/Medicare/Medicare-Fee-for-Service-Payment/sharedsavingsprogram/Downloads/MSSP-QM-Benchmarks.pdf. Published February 2015. Accessed November 2, 2016.

40. The revolving door: a report on US hospital readmissions. Robert Wood Johnson Foundation website.

rwjf.org/en/library/research/2013/02/the-revolving-door--a-report-on-u-s--hospital-readmissions.html. Published February 2013. Accessed November 2, 2016.

41. Friedberg MW, Schneider EC, Rosenthal MB, Volpp KG, Werner RM. Association between participation in a multipayer medical home intervention and changes in quality, utilization, and costs of care. JAMA. 2014;311(8):815-825. doi: 10.1001/jama.2014.353.

42. Rao GA, Mann JR, Shoaibi A, et al. Azithromycin and levofloxacin use and increased risk of cardiac arrhythmia and death. Ann Fam Med. 2014;12(2):121-127. doi: 10.1370/afm.1601.

Download PDF: Electronic Health Record Problem Lists: Accurate Enough for Risk Adjustment?