Understanding the Social Risk Factor Adjustment’s Effect on Star Ratings

ABSTRACT

Objectives: CMS implemented the Categorical Adjustment Index (CAI) to address measurement bias in the Medicare Advantage (MA) Star Ratings, as unadjusted scores may disadvantage MA contracts serving more enrollees at greater social risk. CAI values are added to a contract’s Star Ratings to adjust for the mean within-contract performance disparity associated with its percentage of enrollees with low socioeconomic status (ie, receipt of a Part D low-income subsidy or dual eligibility for Medicare and Medicaid [LIS/DE]) and who are disabled. We examined the CAI’s effect on Star Ratings and the type of contracts affected.

Study Design: Observational study of MA contracts with health and prescription drug coverage.

Methods: We compared adjusted and unadjusted 2017-2020 Star Ratings overall and by contracts’ proportion of LIS/DE and disabled enrollees. We assessed the CAI’s effect on qualifying for quality bonus payments (QBPs), eligibility for rebate payments, and high-performing and low-performing designations.

Results: The CAI’s impact was modest overall (3.2%-14.9% of contracts experienced one-half Star Rating changes). Upward changes were concentrated among contracts with high percentages of LIS/DE or disabled enrollees (7.7%-32.3% of these contracts saw increased Star Ratings). In 2020, 26.0% of contracts with a high proportion of LIS/DE or disabled enrollees that qualified for a QBP did so because of the CAI.

Conclusions: The CAI primarily affected contracts with high LIS/DE or disabled enrollment, which received higher Star Ratings because of the CAI. The adjustment helps ensure that such contracts’ performance is not understated and reduces incentives for MA contracts to avoid patients at greater social risk.

Am J Manag Care. 2023;29(12):e372-e377. https://doi.org/10.37765/ajmc.2023.89471

Takeaway Points

• CMS applies a Categorical Adjustment Index to Medicare Advantage Star Ratings because unadjusted scores may disadvantage contracts serving more enrollees at higher social risk.
• The adjustment’s impact was modest overall (3.2%-14.9% of contracts affected). Upward changes were concentrated among contracts serving large percentages of enrollees with low socioeconomic status or with disabilities (7.7%-32.3% of these contracts saw increased Star Ratings). Improved Star Ratings for such contracts reduce incentives for contracts to avoid enrolling patients at greater social risk.
• Other value-based payment programs may consider similar adjustments to reduce measurement bias in performance-based payment applications.

CMS implemented the Categorical Adjustment Index (CAI)^1-4 as part of the 2017 Medicare Part C and Part D Star Ratings to address concerns that the Star Ratings may disadvantage contracts that serve a higher percentage of enrollees with low socioeconomic status (SES) or with disabilities. The Star Ratings are calculated annually based on a contract’s quality performance on clinical, patient experience, customer service, and complaint measures. CMS reports the Star Ratings on Medicare Plan Finder⁵ and uses the ratings to determine Medicare Advantage (MA) contract eligibility for quality bonus payments (QBPs), MA contract rebates, and marketing and enrollment benefits.

Ensuring the validity of the measures used to assess provider performance is vital when paying differentially for performance in value-based payment (VBP) programs and alternative payment models. Valid measures account for factors that are exogenous (not themselves modifiable by the entity being assessed) but that influence measure performance. Prior studies have found within-contract disparities in performance on many Star Ratings measures across an array of exogenous enrollee characteristics, including social risk factors, such as receipt of the Part D low-income subsidy (LIS) or dual eligibility (DE) for Medicare and Medicaid (collectively termed LIS/DE) and disability.^2,6 Within-contract disparities^2,6 occur when enrollees with social risk factors receive worse care within the same contract than enrollees without social risk factors. Within-contract disparities suggest that something outside the control of the contract and its providers makes it more difficult to deliver the same level of care to patients with social risk factors as is provided to patients without social risk factors, possibly because patients with social risk factors have more barriers to care and health burdens^7-10 (eg, patients with poorer transportation access are less likely to return for follow-up care, patients with financial limitations are less likely to adhere to medications that require additional costs). Absent accounting for these within-contract disparities, estimates of performance may be biased against contracts that serve a large proportion of enrollees with social risk factors and thereby risk misidentifying such contracts as low performing. Unbiased performance measurement assesses the value added by entities and compares their expected performance on the same set of patients.¹¹

A potential consequence of not adjusting is that contracts that serve a large proportion of enrollees at higher social risk and receive lower scores on their performance ratings in VBP programs because of a failure to account for exogenous risk factors that influence performance will receive lower value-based payments than their performance merits. This is of particular concern because contracts serving enrollees at higher social risk tend to have fewer resources to invest in infrastructure and quality improvement than other contracts due to their payer mix; downwardly biased value-based payments may further constrain the resources of these providers and create additional challenges to delivering high-quality care.¹² Relatedly, not accounting for social risk factors in performance measurement can incentivize high-performing contracts to avoid enrolling patients at higher social risk. It may also lead enrollees to make suboptimal choices in contracts if they believe certain contracts provide worse quality of care based on the contracts’ biased lower performance rating scores. This is of particular concern for enrollees at higher social risk, as contracts that serve more patients with particular social risk factors often provide better care to patients with those social risk factors. For example, results of one study showed that although experiences with receiving care tended to be worse for White patients in MA contracts with larger proportions of patients from racial and ethnic minority groups, contracts with high proportions of a given minority group often scored as well as low-minority contracts for members of the minority group prevalent in that contract.^7-10,13 A number of studies^14-18 have concluded that there is a need for adjustment of clinical measures that are not currently adjusted for SES to address potentially biased performance estimates.

To address these concerns about bias in measurement of performance in the MA Star Ratings and to enable fair comparisons across contracts, CMS implemented an indirect adjustment^1-4 for social risk factors starting with the 2017 Star Ratings. The CAI approximates the effect of direct case-mix adjustment for LIS/DE and disability. The CAI is a positive or negative value that is added to a contract’s overall Star Rating to adjust for the mean within-contract disparity in performance that is associated with that contract’s percentage of enrollees with low SES, as indicated by LIS/DE, and the percentage of enrollees whose original reason for Medicare entitlement was disability. To construct these values, CMS first calculates case mix–adjusted contract-level measure scores for eligible Star Ratings measures using logistic regression models. CMS next uses these scores to calculate case mix–adjusted overall Star Ratings and then calculates the difference between case mix–adjusted and unadjusted overall Star Ratings for each contract. Contracts are assigned CAI values based on their percentage of LIS/DE and disabled enrollees; these values reflect the mean difference in case mix–adjusted and unadjusted overall Star Ratings among contracts with a similar percentage of LIS/DE and disabled enrollees as the given contract. The value of the CAI assigned to each contract is updated annually. CAI values typically range from a small negative value (for contracts with a low proportion of enrollees with social risk factors) to a larger positive value. In the 2020 overall Star Ratings, the CAI values ranged from –0.042454 to 0.167650.¹⁹ Details on the CAI methodology are provided in eAppendix A (eAppendices available at ajmc.com); additional details on the CAI methodology and the development of the CAI can be found elsewhere.^1-4

In this article, we assessed the impact of the CAI on overall Star Ratings of MA contracts with health and prescription drug coverage (MA-PD contracts), particularly for contracts that serve a high proportion of LIS/DE and disabled enrollees, from when the CAI was first implemented in the 2017 Star Ratings through the 2020 Star Ratings (prior to the COVID-19 public health emergency). We also assessed the impact of the CAI on qualifying for QBPs, eligibility for rebate payments, and high-performing and low-performing designations.

METHODS

To assess the impact of the CAI, we compared each MA-PD contract’s 2017-2020 overall Star Ratings, which were adjusted ratings that included the CAI, with unadjusted Star Ratings, which reflected the contract’s rating without the CAI. eAppendix B describes how we constructed the unadjusted Star Ratings; we used the CAI values that CMS published19 in these calculations. We excluded terminated, consolidated, and withdrawn contracts (n = 30, 29, 18, and 6 in 2017-2020, respectively).

We also examined how adjusting for the CAI affects contracts’ receipt of 3.5 or more stars, 4 or more stars, and 5 stars by comparing whether each contract received each respective Star Rating level with and without the CAI. Receiving 3.5 or more stars increases the level of a contract’s rebate payments, which contracts use to fund supplemental benefits (eg, dental coverage, quarterly OTC health product benefit, etc) or buy down enrollee premiums for Part B or prescription drug coverage; contracts that receive 4.5 or more stars (not examined) receive an additional increase in the level of rebate.²⁰ MA contracts with 4 or more stars are eligible for QBPs,²⁰ and contracts with 5 stars are designated high performing and allowed to market and enroll year-round. Lastly, we assessed the impact of adjustment on whether a contract received a low-performing icon (ie, if it had any combination of Part C or Part D summary ratings of 2.5 or lower in the current year and in each of the 2 prior years), which can adversely affect a contract’s enrollment (eg, Medicare does not permit online enrollment for such contracts, so enrollees are required to call to enroll); contracts with consistent Part C low performance or consistent Part D low performance are also eligible for potential termination.^1,20

Additional analyses compared the effects of adjustment as well as the effects of adjustments on each benchmark noted earlier on contracts in the top quartile of percentage LIS/DE enrollees or the top quartile of percentage disabled enrollees (ie, contracts with high percentages of LIS/DE or disabled enrollees, or “high LIS/DE or disabled contracts”) vs all other contracts in the given year. We used the definitions of LIS, DE, and disability that CMS used to assign contracts CAI values in each given Star Ratings year.¹

This study was approved by the institutional review board at the RAND Corporation.

RESULTS

The Table describes contracts receiving an overall Star Rating between 2017 and 2020. The percentage of contracts in the top quartile of LIS/DE (contracts with at least 55.8%-65.6% of enrollees who were LIS/DE, depending on the year) or the top quartile of disability (contracts with at least 34.3%-36.9% of enrollees who were disabled, depending on the year) was 30% to 32% across years.

Figure 1 shows the impact of adjustment by year for contracts with high percentages of LIS/DE or disabled enrollees compared with all other contracts. The impact of the CAI was modest overall, with the percentage of contracts affected ranging from 3.2% (2018) to 14.9% (2020). However, the impact of adjusting Star Ratings was concentrated among contracts with high percentages of LIS/DE or disabled enrollees. Between 7.7% (2018) and 32.3% (2020) of contracts with high percentages of LIS/DE or disabled enrollees gained half a star after adjustment; few other contracts were affected by the adjustment (1.2%-7.0% across years). The largest effects of adjustment on Star Ratings were seen in 2020, when 32.3% of contracts with high percentages of LIS/DE or disabled enrollees gained half a star and 7.0% of other contracts were affected (4.0% gained half a star, 2.9% lost half a star); this was likely a function of the CAI adjustments themselves being somewhat larger that year because the adjusted measure set expanded starting with the 2020 Star Ratings to include all eligible clinical measures rather than the subset with substantive or consistent within-contract LIS/DE disparities (see eAppendix C).²¹

Figure 2 shows the effect of the CAI adjustment on contracts that qualified for a QBP. The CAI’s effect was concentrated among contracts with the highest percentage of LIS/DE or disabled enrollees. For example, in the 2020 Star Ratings, 26.0% (n = 13) of contracts with the highest percentage of LIS/DE or disabled enrollees that received at least 4 stars achieved 4-star status due to application of the CAI compared with 1.9% (n = 3) of all other contracts that received at least 4 stars.

Similarly, the effect of the CAI on eligibility for an initial increase in the level of rebate (receiving 3.5 or more stars) was concentrated among contracts with high percentages of LIS/DE or disabled enrollees (2.6%-11.8% of contracts with high percentages of LIS/DE or disabled enrollees were eligible for larger rebate payments across years because of the CAI vs 0.0%-1.1% of all other contracts). Absent adjustment, very few contracts with high percentages of LIS/DE or disabled enrollees received 5 stars (none in 2017 and 2018, 1 in 2019 and 2020); across years, a small number of contracts (3 across years) gained 5-star status because of the CAI. The CAI also protected a small number of contracts with high percentages of LIS/DE or disabled enrollees from receiving the low-performing designation. A few contracts that served smaller percentages of LIS/DE or disabled enrollees were negatively affected by the CAI. No more than 3 contracts each year out of a total of 363 (2017) to 396 (2020) contracts were prevented from attaining 3.5, 4, or 5 stars because of the CAI. These results are detailed in eAppendix D.

DISCUSSION

Our results show that the CAI increased scores in contracts with high percentages of LIS/DE or disabled enrollees. Most notably, contracts with high percentages of LIS/DE or disabled enrollees qualified for a QBP less often than contracts not serving these populations, and a substantial minority of the contracts serving those at greater social risk qualified for a QBP only as a result of the adjustment.

In contrast, we find the CAI had little impact on contracts that served smaller proportions of LIS/DE or disabled enrollees. This was expected because CAI values are largest for contracts with high percentages of LIS/DE or disabled enrollees, so contracts serving a small proportion of LIS/DE or disabled enrollees were less likely to be affected by the CAI given their smaller CAI values.

The results are similar to the typical effect of case-mix adjustment^22,23 of individual measures, which generally has small negative effects for most providers and a few larger positive effects for providers serving a large proportion of enrollees with a less prevalent characteristic that is associated with a disadvantage. These results suggest that the CAI is achieving its goal of making contract comparisons more equitable.

The CAI does not preclude equity measures or changes in payment policy that more fundamentally work to help contracts better serve those at greater social risk. A variety of complementary strategies^6,12,24 could be used to measure and address disparities in MA. For example, strategies could be implemented to increase contracts’ awareness of disparities and decrease disparities in performance, such as reporting performance stratified by those with and without social risk factors, developing measures specific to those at greater social risk, enhancing data collection to better identify/evaluate disparities and better support reporting quality for patients at greater social risk, and undertaking quality improvement intended to achieve better outcomes for enrollees with social risk factors. Simultaneously, strategies could also be used to strengthen payment incentives to address disparities, such as rewarding contracts that have high performance among enrollees at greater social risk, rewarding performance improvement for contracts serving enrollees with greater social risk, developing and upweighting measures focusing on equity or performance among those with social risk factors, and changing incentive programs to reward high performance and improvement among enrollees at greater social risk.

Limitations

This study has several limitations. First, our analyses did not account for contract actions in response to the CAI that may have affected the contract’s performance or its proportion of LIS/DE or disabled enrollees. Additionally, starting with the 2020 Star Ratings, the adjusted measure set used to compute the CAI expanded to include all eligible clinical measures rather than those with substantive or consistent within-contract LIS/DE disparities (see eAppendix C). This resulted in larger CAI adjustments that translated into larger downward adjustments for low LIS/DE or disabled contracts and larger upward adjustments for high LIS/DE or disabled contracts.²¹ This methodological change contributed to the larger effect of the CAI seen in 2020.

CONCLUSIONS

As expected, the effects of within-contract adjustment for social risk factors in MA Star Ratings were primarily within the subset of contracts with high percentages of LIS/DE or disabled enrollees, which had somewhat higher Star Ratings as a result of the CAI. Thus, the CAI is helping ensure that the performance of contracts with high percentages of LIS/DE or disabled enrollees is not understated, and the CAI is reducing incentives for MA contracts to avoid enrolling patients at greater social risk. Adjustments for factors that are related to within-contract or within-provider disparities in performance and are outside the control of contracts and providers also improve the validity of measures used to assess contract and provider performance and determine quality-based payments. Other VBP programs may consider direct adjustment or similar indirect adjustment approaches to improve measurement accuracy and to reduce incentives to avoid enrolling patients at greater social risk.

Author Affiliations: RAND Corporation, Santa Monica, CA (AT, MD, MNE, CLD), and Pittsburgh, PA (MES).

Source of Funding: This work was supported by contract 75FCMC19F0076 funded by CMS. The content of this article does not necessarily reflect the views or policies of CMS. The authors assume full responsibility for the accuracy and completeness of the ideas presented.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (AT, MES, MD, MNE, CLD); analysis and interpretation of data (AT, MES, MD, MNE, CLD); drafting of the manuscript (AT); critical revision of the manuscript for important intellectual content (AT, MES, MD, MNE, CLD); statistical analysis (AT); obtaining funding (MES, MD, CLD); and supervision (MES).

Address Correspondence to: Anagha Tolpadi, MS, RAND Corporation, 1776 Main St, Santa Monica, CA 90401. Email: atolpadi@rand.org.

REFERENCES

1. Medicare 2022 Part C & D Star Ratings Technical Notes. CMS. October 4, 2021. Accessed April 28, 2022. https://www.cms.gov/files/document/2022-star-ratings-technical-notes-oct-4-2022.pdf

2. Sorbero ME, Paddock SM, Damberg CL, et al. Adjusting Medicare Advantage Star Tatings for socioeconomic status and disability. Am J Manag Care. 2018;24(9):e285-e291.

3. Categorical Adjustment Index (CAI) methodology. CMS. Accessed April 28, 2022. https://www.cms.gov/Medicare/Prescription-Drug-Coverage/PrescriptionDrugCovGenIn/Downloads/Supplement-for-Categorical-Adjustment-Index-.pdf

4. 2023 Categorical Adjustment Index measure supplement. CMS. 2022. Accessed April 28, 2022. https://www.cms.gov/files/document/2023-categorical-adjustment-index-measure-supplement.pdf

5. Find Medicare health & drug plans. Medicare.gov. Accessed April 28, 2022. https://www.medicare.gov/plan-compare/#/?year=2022&lang=en

6. Report to Congress: Social Risk Factors and Performance Under Medicare’s Value-Based Purchasing Programs. Office of the Assistant Secretary for Planning and Evaluation; December 20, 2016. Accessed May 23, 2019. https://aspe.hhs.gov/pdf-report/report-congress-social-risk-factors-and-performance-under-medicares-value-based-purchasing-programs

7. Fung V, Reed M, Price M, et al. Responses to Medicare drug costs among near-poor versus subsidized beneficiaries. Health Serv Res. 2013;48(5):1653-1668. doi:10.1111/1475-6773.12062

8. Hsu J, Fung V, Price M, et al. Medicare beneficiaries’ knowledge of Part D prescription drug program benefits and responses to drug costs. JAMA. 2008;299(16):1929-1936. doi:10.1001/jama.299.16.1929

9. Ngo-Metzger Q, Sorkin DH, Billimek J, Greenfield S, Kaplan SH. The effects of financial pressures on adherence and glucose control among racial/ethnically diverse patients with diabetes. J Gen Intern Med. 2012;27(4):432-437. doi:10.1007/s11606-011-1910-7

10. Phelan JC, Link BG, Tehranifar P. Social conditions as fundamental causes of health inequalities: theory, evidence, and policy implications. J Health Soc Behav. 2010;51(suppl 1):S28-S40. doi:10.1177/0022146510383498

11. Damberg C, Elliott MN. Opportunities to address health disparities in performance-based accountability and payment programs. JAMA Health Forum. 2021;2(6):e211143. doi:10.1001/jamahealthforum.2021.1143

12. Damberg CL, Elliott MN, Ewing BA. Pay-for-performance schemes that use patient and provider categories would reduce payment disparities. Health Aff (Millwood). 2015;34(1):134-142. doi:10.1377/hlthaff.2014.0386

13. Price RA, Haviland AM, Hambarsoomian K, et al. Do experiences with Medicare managed care vary according to the proportion of same-race/ethnicity/language individuals enrolled in one’s contract? Health Serv Res. 2015;50(5):1649-1687. doi:10.1111/1475-6773.12292

14. Joynt KE, De Lew N, Sheingold SH, Conway PH, Goodrich K, Epstein AM. Should Medicare value-based purchasing take social risk into account? N Engl J Med. 2017;376(6):510-513. doi:10.1056/NEJMp1616278

15. Joynt KE, Zuckerman R, Epstein AM. Social risk factors and performance under Medicare’s value-based purchasing programs. Circ Cardiovasc Qual Outcomes. 2017;10(5):e003587. doi:10.1161/CIRCOUTCOMES.117.003587

16. Joynt Maddox KE, Reidhead M, Hu J, et al. Adjusting for social risk factors impacts performance and penalties in the hospital readmissions reduction program. Health Serv Res. 2019;54(2):327-336. doi:10.1111/1475-6773.13133

17. Qi AC, Peacock K, Luke AA, Barker A, Olsen MA, Joynt Maddox KE. Associations between social risk factors and surgical site infections after colectomy and abdominal hysterectomy. JAMA Netw Open. 2019;2(10):e1912339. doi:10.1001/jamanetworkopen.2019.12339

18. Topmiller M, McCann J, Rankin J, Hoang H, Bolton J, Sripipatana A. Exploring the association of social determinants of health and clinical quality measures and performance in HRSA-funded health centres. Fam Med Community Health. 2021;9(3):e000853. doi:10.1136/fmch-2020-000853

19. Medicare 2020 Part C & D Star Ratings Technical Notes. CMS. October 1, 2019. Accessed April 28, 2022. https://www.cms.gov/Medicare/Prescription-Drug-Coverage/PrescriptionDrugCovGenIn/Downloads/Star-Ratings-Technical-Notes-Oct-10-2019.pdf

20. Advance notice of methodological changes for calendar year (CY) 2023 for Medicare Advantage (MA) capitation rates and Part C and Part D payment policies. CMS. February 2, 2022. Accessed April 28, 2022. https://www.cms.gov/files/document/2023-advance-notice.pdf

21. Announcement of calendar year (CY) 2020 Medicare Advantage capitation rates and Medicare Advantage and Part D payment policies and final call letter. CMS. April 1, 2019. Accessed April 28, 2022. https://www.cms.gov/Medicare/Health-Plans/MedicareAdvtgSpecRateStats/Downloads/Announcement2020.pdf

22. Elliott MN, Zaslavsky AM, Goldstein E, et al. Effects of survey mode, patient mix, and nonresponse on CAHPS hospital survey scores. Health Serv Res. 2009;44(2, pt 1):501-518. doi:10.1111/j.1475-6773.2008.00914.x

23. Parast L, Haas A, Tolpadi A, et al. Effects of caregiver and decedent characteristics on CAHPS hospice survey scores. J Pain Symptom Manage. 2018;56(4):519-529.e1. doi:10.1016/j.jpainsymman.2018.07.014

24. Risk adjustment for socioeconomic status or other sociodemographic factors. National Quality Forum. August 2014. Accessed April 28, 2022. https://www.qualityforum.org/Publications/2014/08/Risk_Adjustment_for_Socioeconomic_Status_or_Other_Sociodemographic_Factors.aspx