A Comparison of Administrative Claims–Based Risk Predictors for Pediatric Asthma

ABSTRACT

Objectives: Head-to-head comparisons are needed to determine the most accurate and appropriate administrative claims–based exacerbation risk predictor for emergency department (ED) visits and hospitalizations among children with asthma.

Study Design: Retrospective cohort study.

Methods: We analyzed 2013-2014 MarketScan Medicaid data. Children aged 2 to 17 years were included. Seven risk predictors were compared for accuracy in predicting 3-month subsequent ED visits/hospitalizations for asthma: 3-month rolling asthma medication ratio (AMR), Healthcare Effectiveness Data and Information Set (HEDIS) criteria, revised HEDIS criteria, quarterly short-acting β-agonist (SABA) claims, prior ED visit, prior hospitalization, and prior ED visit or hospitalization. Sensitivity, specificity, positive and negative predictive value (NPV), and percentage of population identified as high risk were compared for each risk predictor utilizing the McNemar test to identify statistically significant differences in risk prediction accuracy.

Results: A total of 214,452 children were included; the mean age was 7.8 years. HEDIS and revised HEDIS identified prohibitively large cohorts as high risk (67% and 48%, respectively). For the remaining measures, the NPV range is narrow (97%-99%), indicating high performance at identifying patients who would not benefit from intervention. The ED visit and ED/hospitalization measures have superior sensitivities (44% and 49%, respectively) compared with pharmacy claims–based measures (AMR [5%] and SABA count [10%]). Pharmacy claims–based measures identify a smaller proportion of patients as high risk and maintain high NPV.

Conclusions: Pharmacy-based asthma exacerbation risk predictors such as the AMR and SABA count can rule out low-risk patients with a high degree of specificity and NPV, which is a primary goal of real-time risk monitoring in pediatric asthma.

Am J Manag Care. 2021;27(12):533-537. https://doi.org/10.37765/ajmc.2021.88792

Takeaway Points

• Administrative claims data can be utilized to identify children at high risk for asthma exacerbation requiring emergency department visits or hospitalizations.
• To be scalable and cost-effective, these risk predictors must be available in near real time and must identify a reasonable proportion of the overall asthma population as high risk.
• This head-to-head comparison of 7 claims-based risk predictors provides decision makers with information vital to risk predictor selection.
• Pharmacy claims–based risk predictors, such as the asthma medication ratio or short-acting β-agonist count, are likely to be the ideal choice for many situations.

Asthma is the most common chronic condition affecting children. Given its prevalence, availability of effective medications, historically poor medication adherence, and costly yet preventable exacerbations, it is an ideal disease for the proactive monitoring of medication adherence.^1-3 Inhaled corticosteroids (ICSs) are known to reduce the incidence of emergency department (ED) visits and hospitalizations for asthma, but they remain chronically underutilized.^4,5 Asthma confers immense burden to children, families, and the health care system.^1,2,6 In 2012, among children younger than 18 years, hospital discharges with a primary diagnosis of asthma accounted for a total annual cost of more than $520 million.⁷ Expensive asthma care, including ED visits and hospitalizations, is not evenly distributed among children with asthma.⁸ The majority of ED visits and hospitalizations for asthma can be attributed to a minority of children with asthma. Determining who these high-risk children are is necessary to maximize the efficiency and cost-effectiveness of asthma population management interventions. Unfortunately, identifying children with asthma at risk for costly ED visits and hospitalizations has previously proved challenging.^9,10 Access to near real-time pharmacy dispensing data may be the key to a pragmatic system for asthma risk prediction. This would allow for intervention prior to symptom exacerbation, ultimately preventing ED visits and hospitalizations.

Risk determination through the monitoring of pharmacy dispensing data for children with asthma is possible through either a quarterly count of short-acting β-agonist (SABA) claims (SABA count)¹¹ or the application of the asthma medication ratio (AMR) (No. of controller medication claims / [No. of controller medication claims + No. of rescue medication claims]). These asthma exacerbation risk predictors can be calculated using pharmacy dispensing or claims data.^12-19 Findings of previous studies have shown that the AMR is an accurate predictor of subsequent severe asthma exacerbations (those resulting in ED visit or hospitalization) when calculated with 12 months of claims data. Our research group has recently demonstrated that a rolling 3-month AMR calculation can identify children at higher risk for severe exacerbation in subsequent 3-month time periods.^20,21 The rolling calculation and shorter risk determination period, as well as reliance on only pharmacy data (not medical visit data), suggest that this asthma exacerbation risk predictor could be superior to previously studied claims-based asthma exacerbation risk predictors. Similarly, quarterly SABA counts can be used to identify high-risk children with asthma.¹¹

Before we can proceed with interventional studies aimed at preventing severe exacerbations by using claims-based exacerbation risk predictors to target high-risk individuals, we must first compare their diagnostic accuracy (their ability to accurately identify children who will have an ED visit or hospitalization for asthma). This information, combined with relative high-risk cohort size and feasibility of implementation based on data source availability, will help providers, payers, and health care administrators select the most appropriate predictor for their population. The objective of this study is to perform a head-to-head comparison of the diagnostic accuracy of claims-based asthma exacerbation risk predictors.

METHODS

Population

This is a retrospective analysis of a large national cohort of publicly insured children with asthma. For this analysis, 2013-2014 Truven MarketScan Medicaid data were used. The Truven Medicaid data are from a deidentified multistate database of children enrolled in Medicaid fee-for-service or capitated plans. Children aged 2 to 17 years with at least 1 claim for an ICS in the first year or “measurement” year of the data set were included. Children with a diagnosis of cystic fibrosis (International Classification of Diseases, Ninth Revision [ICD-9] code 277.xx) or eosinophilic esophagitis (ICD-9 code 530.13) were excluded due to their use of ICSs for nonasthma indications.

Risk Predictors

Pharmacy claims–based risk predictors. The AMR is calculated using 3 months of pharmacy claims data using the following formula: (No. of controller claims / [No. of controller claims + No. of rescue claims]). SABA claims have been proposed as an asthma exacerbation risk predictor, with patients with more than 3 SABA claims per quarter having higher risk for subsequent exacerbation.¹¹

Medical claims–based risk predictors. The number of ED visits, hospitalizations, or a combination of ED visits/hospitalizations with a primary diagnosis of asthma in the previous 12 months has been shown to predict subsequent-year ED visits/hospitalizations with a primary diagnosis of asthma.^22,23

Risk predictors that require both pharmacy and medical claims. The Healthcare Effectiveness Data and Information Set (HEDIS) criteria rely on evaluation of 12 months of administrative pharmacy and medical claims data to determine asthma risk. The HEDIS criteria are 1 asthma inpatient admission or ED visit, 4 asthma medication dispensing events, or 4 outpatient asthma visits and at least 2 asthma medication dispensing events within a 12-month period.²⁴ The HEDIS criteria were developed to identify patients with persistent asthma and to inform quality-of-care measures, not as a risk prediction tool. Similarly, a revised HEDIS definition has been proposed as a way to improve the precision of risk assessment, but it also relies on 12 months of both pharmacy and medical claims to make a risk determination.⁹ The revised HEDIS criteria are at least 1 ED visit or at least 1 hospitalization for asthma, or at least 1 oral steroid prescription for asthma, in a 12-month period.⁹

Outcome Variable

The primary outcome variable is an ED visit or hospitalization with a primary diagnosis of asthma (ICD-9 code 493.xx) in the 3-month outcome period immediately following the risk prediction period.

Statistical Analysis

We first determined the proportion of the population identified as high risk for each risk predictor. We then compared the diagnostic accuracy of the 7 administrative claims–based asthma exacerbation risk predictors: (1) 3-month rolling AMR, (2) quarterly count of SABA claims,¹¹ (3) ED visit, (4) hospitalization, (5) ED visit/hospitalization,^22,23 (6) HEDIS criteria for persistent asthma, and (7) revised HEDIS criteria as developed by Bennett et al.⁹ Each of these asthma exacerbation risk predictors is dichotomous: high risk, yes/no. The outcome variable is also dichotomous: ED visit or hospitalization in outcome window, yes/no. One variation among risk predictors is the calculation period for the AMR (rolling 3 months) and SABA count (3 months) vs all others (12 months). Because of the alternative timing of these measures, there were 10 AMR measurement periods (rolling) and 4 SABA count measurement periods (fixed) in the measurement year. Each time period and its respective outcome periods were included in our final analysis.

We utilized the methods described by Trajman and Luiz²⁵ to compare sensitivities and specificities of the diagnostic examinations in question. We first determined each individual test’s sensitivity, specificity, and positive and negative predictive values for identifying patients with ED visits or hospitalizations in the outcome period. We then looked for statistically significant differences in the test metrics by using the McNemar test among those with an exacerbation and then in a separate 2 × 2 table determining statistically significant differences in specificity among those without an exacerbation.

Analyses were performed using SAS version 9.4 (SAS Institute). Statistical significance was determined at α ≤ 0.05. Research conducted on these data was considered exempt by the Medical University of South Carolina Institutional Review Board.

RESULTS

Study Population

We identified 214,452 children with asthma for inclusion; 41% were female, and they had a mean age of 7.8 years. Regarding racial composition, 41% were White, 41% were Black, and 8% were Hispanic. The cohort utilized in this analysis is the same cohort described in our previously published analysis of a rolling AMR (Table 1).²¹

High-risk Cohort Size

There was substantial variation among risk predictors in the proportion of patients who were identified as high risk. High-risk cohort sizes ranged from 5% to 67%. The HEDIS and revised HEDIS measures identified the largest proportion of patients as high risk (67% and 48%, respectively). All other risk predictors identified 15% or less of the population as high risk (Table 2).

Risk Prediction Accuracy

Sensitivity. Sensitivity measures how often the risk predictor will identify patients as high risk who then go on to have ED visits or hospitalizations. Asthma exacerbation risk predictor sensitivity ranged from 10% to 85% (Table 2).

Specificity. Specificity measures how well the risk predictor correctly identifies those patients who will not have an ED visit or hospitalization in the 3-month outcome period. Asthma exacerbation risk predictor specificity ranged from 34% to 95%. Hospitalization, ED visit, SABA count, and AMR all had specificities higher than 90% (Table 2).

Positive predictive value. Positive predictive value (PPV) measures the percentage of children who are identified as high risk who will go on to have an ED visit or hospitalization. Because the prevalence of ED visits and hospitalizations is relatively low and the PPV calculation is sensitive to prevalence, PPVs of any risk predictor will be low. PPVs in this study ranged from 4% to 12% (Table 2).

Negative predictive value. Negative predictive value (NPV) measures the percentage of children who are identified as low risk and do not go on to have an ED visit or hospitalization. Maximizing the NPV of an asthma exacerbation risk predictor is paramount because of the prevalence of the disease and the relative rarity of ED visits/hospitalizations. A risk predictor with high NPV effectively identifies the patients who are less likely to need or benefit from an intervention. All the asthma exacerbation risk predictors analyzed in this study had a high NPV, ranging from 97% to 99% (Table 2).

Bivariate Analysis of Differences in Accuracy Measures

The McNemar test was used to determine statistically significant differences in sensitivity and specificity, focusing on the AMR and SABA measures based on their favorable proportion of high-risk patients identified and reliance on only pharmacy claims data. We compared AMR from months 4 to 6, 7 to 9, and 10 to 12 with quarters 2, 3, and 4, respectively, of the SABA count. We did not compare AMR from months 1 to 3 with quarter 1 of SABA count because AMR from months 1 to 3 is artificially inflated due to our definition of the index month requiring an ICS claim. All comparisons were found to be statistically significant with P < .0001 (Table 3), showing that the AMR had a higher sensitivity whereas the SABA count had a higher specificity.

DISCUSSION

This retrospective study of a national cohort of children covered by publicly funded insurance describes and compares the diagnostic accuracy of 7 claims-based asthma exacerbation risk predictors. These findings can be utilized by providers, practice managers, payers, or health care system managers to select the most appropriate asthma exacerbation risk predictor. Ultimately, application of these risk predictors to large cohorts of children with asthma can allow for targeted interventions for the highest-risk patients with asthma.

To determine the ideal risk predictor, many factors need to be considered. Although we are treating the risk predictors as screening tests for severe exacerbations, the basic tenet of maximizing sensitivity for screening tests might not apply here, as false positives will contribute significantly to the costs of interventions. The diagnostic accuracy of each individual measure needs to be weighed with considerations of convenience and timeliness to determine the best risk predictor for an individual situation. The first consideration must be an assessment of what data are available. With the exception of closed systems such as Kaiser Permanente, any measure that relies on medical claims (HEDIS, revised HEDIS, ED visit, hospitalization, ED visit/hospitalization) will be challenging due to both availability and completeness of the data and the complexity of the algorithm needed for operationalization of the risk tool. The next consideration likely needs to be high-risk cohort size. If the risk predictor identifies a prohibitively large proportion of patients as high risk, interventions are unlikely to be cost-effective or scalable. The HEDIS and revised HEDIS identify 67% and 48% of children with asthma as being high risk, respectively.

Of the measures with a reasonable cohort size (all measures except for HEDIS and revised HEDIS), the NPV range is narrow (97%-98%), so they are performing well at identifying children who can be excluded from an intervention. There is a broader range of sensitivity (10%-50%), which means that any measure with a reasonable cohort size will miss a fair number of children who experience an ED visit or hospital admission. The ED visit and ED/hospitalization measures have higher sensitivities, but they require medical claims data and a 12-month assessment period. For a closed system with ready access to medical claims, this could be an ideal measure, although it requires a 12-month risk assessment period. After excluding measures that identify a prohibitively large cohort size and those that require ready access to complete medical claims data, the ideal risk predictor will be either the AMR or SABA. These 2 measures have essentially equal NPV; the sensitivity of the AMR is slightly higher, but the specificity of SABA is slightly higher. Unlike the AMR, the SABA measure has never been tested as a rolling measure but rather is calculated quarterly.^20,21 Because the AMR can be calculated each month, the risk value represents “current” risk.

Identifying excessive SABA use, as measured by SABA dispensings in administrative data, has been a strategy employed in previous studies for high-risk patient identification and intervention. Zeiger et al, in a randomized controlled trial of patients aged 12 to 56 years, identified patients with 7 or more SABA dispensings in the previous 12 months. Those randomized to the intervention arm received a letter notification and an electronic message to their physician with suggestions for management including allergy referral. Compared with the control group, patients in the intervention were less likely to fill 7 or more SABAs in the subsequent 12 months, suggesting that SABA count could be an effective way to target interventions to the highest-risk patients with asthma.²⁶ In a subsequently published systematic review, McKibben et al analyzed 4 trials that evaluated computerized alerts identifying excessive SABA prescribing. The authors found evidence that these alerts may reduce subsequent excessive prescribing, but further research is needed to optimize such an alert system.²⁷ The findings of this systematic review in combination with the current analysis suggest that either SABA count or the AMR would be suitable measures to identify children with asthma who are at high risk for an exacerbation. Significantly more work needs to be done to design, test, and implement effective patient-centered interventions to improve disease control and reduce risk for exacerbation once a patient is identified as high risk.

Limitations

There are several limitations to this study. First, we used administrative claims data for this analysis. Although claims data lack the clinical detail that might be found in other data sources such as electronic health records, they remain the most accurate source for determining medication adherence and health care utilization patterns in large populations of patients. Because we are not able to determine prescription writing from these data, we do not know the relative roles of provider prescribing vs patient filling behavior in prescription claims. Similarly, we cannot determine actual patient use of a filled medication; it is possible that prescriptions were filled by patients but not routinely used. Additionally, prescriptions that are not paid by the insurance company (ie, free samples) would not be included in these data. Another limitation is that the data set lacks geographical information; therefore, we are unable to determine trends related to geographic differences. ED visits and hospitalizations were included as an outcome only if the primary diagnosis for the encounter was asthma. It is possible that some of these visits were not primarily for asthma, but the diagnosis codes were reordered by the coders prior to claim submission. Similarly, some visits that were for asthma could have ultimately been submitted with asthma as a nonprimary diagnosis. These visits would not have been included in the determination of our outcome variable. This analytical choice is consistent with existing literature on the AMR and other asthma exacerbation risk predictors. The data used in these analyses are more than 5 years old due to their accessibility to the researchers during the grant funding period. However, because there have been no widespread significant changes to pediatric asthma management in the interval since these data were collected, the results should remain externally valid. Finally, because of the nature of research utilizing administrative claims data, we are unable to measure more proximal and patient-centered outcomes, such as asthma-related quality of life.

CONCLUSIONS

Pharmacy-based asthma exacerbation risk predictors such as the AMR and SABA count are available at the point of care and can be calculated with only 3 months of data. They identify a reasonably sized high-risk cohort. They can rule out low-risk patients with a high degree of specificity (and NPV), which is a primary goal of real-time risk monitoring. This approach, utilizing readily available data, could enable providers to intervene prior to the onset of exacerbated symptoms and reduce the likelihood of associated morbidity and mortality. However, systematic methods to capitalize on this novel information source to improve quality of care and patient outcomes will need to be developed and tested. In order to provide the most accurate, targeted risk prediction, future studies should examine the accuracy of these predictors in patient subgroups. 

Author Affiliations: Department of Pediatrics, College of Medicine (ALA), and Department of Healthcare Leadership and Management, College of Health Professions (DLB, ANS), Medical University of South Carolina, Charleston, SC.

Source of Funding: This research was funded by Agency for Healthcare Research and Quality grant No. 1R03HS026783-01A1.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (ALA, DLB, ANS); acquisition of data (ANS); analysis and interpretation of data (ALA, DLB, ANS); drafting of the manuscript (ALA, DLB, ANS); critical revision of the manuscript for important intellectual content (ALA, DLB, ANS); statistical analysis (ALA, DLB, ANS); and obtaining funding (ALA, ANS).

Address Correspondence to: Annie Lintzenich Andrews, MD, MSCR, Medical University of South Carolina, 135 Rutledge Ave, MSC 561, Charleston, SC 29425. Email: andrewsan@musc.edu.

REFERENCES

1. Akinbami LJ, Simon AE, Rossen LM. Changing trends in asthma prevalence among children. Pediatrics. 2016;137(1):1-7. doi:10.1542/peds.2015-2354

2. Akinbami LJ, Moorman JE, Bailey C, et al. Trends in asthma prevalence, health care use, and mortality in the United States, 2001-2010. NCHS Data Brief. 2012;(94):1-8.

3. Akinbami LJ, Moorman JE, Garbe PL, Sondik EJ. Status of childhood asthma in the United States, 1980-2007. Pediatrics. 2009;123(suppl 3):S131-S145. doi:10.1542/peds.2008-2233C

4. Andrews AL, Bundy DG, Simpson KN, Teufel RJ II, Harvey J, Simpson AN. Inhaled corticosteroid claims and outpatient visits after hospitalization for asthma among commercially insured children. Acad Pediatr. 2017;17(2):212-217. doi:10.1016/j.acap.2016.10.016

5. Kenyon CC, Rubin DM, Zorc JJ, Mohamad Z, Faerber JA, Feudtner C. Childhood asthma hospital discharge medication fills and risk of subsequent readmission. J Pediatr. 2015;166(5):1121-1127. doi:10.1016/j.jpeds.2014.12.019

6. Chen SH, Huang JL, Yeh KW, Tsai YF. The stress of caring for children with asthma: a qualitative study of primary caregivers. J Nurs Res. 2015;23(4):298-307. doi:10.1097/JNR.0000000000000083

7. Fingar K, Washington R. Potentially preventable pediatric hospital inpatient stays for asthma and diabetes, 2003-2012. Agency for Healthcare Research and Quality. June 2015. Accessed November 6, 2021. https://www.hcup-us.ahrq.gov/reports/statbriefs/sb192-Pediatric-Preventable-Hospitalizations-Asthma-Diabetes.jsp

8. Wang LY, Zhong Y, Wheeler L. Direct and indirect costs of asthma in school-age children. Prev Chronic Dis. 2005;2(1):A11.

9. Bennett AV, Lozano P, Richardson LP, McCauley E, Katon WJ. Identifying high-risk asthma with utilization data: a revised HEDIS definition. Am J Manag Care. 2008;14(7):450-456.

10. Zeiger RS, Yegin A, Simons FER, et al; TENOR Study Group. Evaluation of the National Heart, Lung, and Blood Institute guidelines impairment domain for classifying asthma control and predicting asthma exacerbations. Ann Allergy Asthma Immunol. 2012;108(2):81-87. doi:10.1016/j.anai.2011.12.001

11. Silver HS, Blanchette CM, Kamble S, et al. Quarterly assessment of short-acting beta(2)-adrenergic agonist use as a predictor of subsequent health care use for asthmatic patients in the United States. J Asthma. 2010;47(6):660-666. doi:10.3109/02770901003702824

12. Broder MS, Gutierrez B, Chang E, Meddis D, Schatz M. Ratio of controller to total asthma medications: determinants of the measure. Am J Manag Care. 2010;16(3):170-178.

13. Andrews AL, Simpson AN, Basco WT Jr, Teufel RJ II. Asthma medication ratio predicts emergency department visits and hospitalizations in children with asthma. Medicare Medicaid Res Rev. 2013;3(4). doi:10.5600/mmrr.003.04.a05

14. Stanford RH, Shah MB, D’Souza AO, Schatz M. Predicting asthma outcomes in commercially insured and Medicaid populations. Am J Manag Care. 2013;19(1):60-67.

15. Schatz M, Zeiger RS, Vollmer WM, et al. The controller-to-total asthma medication ratio is associated with patient-centered as well as utilization outcomes. Chest. 2006;130(1):43-50. doi:10.1378/chest.130.1.43

16. Schatz M, Zeiger RS, Yang SJT, et al. Relationship of asthma control to asthma exacerbations using surrogate markers within a managed care database. Am J Manag Care. 2010;16(5):327-333.

17. Vernacchio L, Trudell EK, Muto JM. Correlation of care process measures with childhood asthma exacerbations. Pediatrics. 2013;131(1):e136-e143. doi:10.1542/peds.2012-1144

18. Fuhlbrigge A, Carey VJ, Adams RJ, et al. Evaluation of asthma prescription measures and health system performance based on emergency department utilization. Med Care. 2004;42(5):465-471. doi:10.1097/01.mlr.0000124249.84045.d7

19. Schatz M, Nakahiro R, Crawford W, Mendoza G, Mosen D, Stibolt TB. Asthma quality-of-care markers using administrative data. Chest. 2005;128(4):1968-1973. doi:10.1378/chest.128.4.1968

20. Andrews AL, Brinton D, Simpson KN, Simpson AN. A longitudinal examination of the asthma medication ratio in children. Am J Manag Care. 2018;24(6):294-300.

21. Andrews AL, Brinton DL, Simpson KN, Simpson AN. A longitudinal examination of the asthma medication ratio in children with Medicaid. J Asthma. 2020;57(10):1083-1091. doi:10.1080/02770903.2019.1640727

22. Tolomeo C, Savrin C, Heinzer M, Bazzy-Asaad A. Predictors of asthma-related pediatric emergency department visits and hospitalizations. J Asthma. 2009;46(8):829-834.

23. Reznik M, Hailpern SM, Ozuah PO. Predictors of early hospital readmission for asthma among inner-city children. J Asthma. 2006;43(1):37-40. doi:10.1080/02770900500446997

24. HEDIS measures and technical resources. National Committee for Quality Assurance. Accessed November 6, 2021. https://www.ncqa.org/hedis/measures/

25. Trajman A, Luiz RR. McNemar chi2 test revisited: comparing sensitivity and specificity of diagnostic examinations. Scand J Clin Lab Invest. 2008;68(1):77-80. doi:10.1080/00365510701666031

26. Zeiger RS, Schatz M, Li Q, Solari PG, Zazzali JL, Chen W. Real-time asthma outreach reduces excessive short-acting beta2-agonist use: a randomized study. J Allergy Clin Immunol Pract. 2014;2(4):445-456, 456.e1-456.e5. doi:10.1016/j.jaip.2014.01.018

27. McKibben S, Bush A, Thomas M, Griffiths C. The use of electronic alerts in primary care computer systems to identify the over-prescription of short-acting beta2-agonists in people with asthma: a protocol for a systematic review. NPJ Prim Care Respir Med. 2017;27(1):30. doi:10.1038/s41533-017-0033-y