Emergency Department Visit Classification Using the NYU Algorithm

Objectives:

Reliable measures of emergency department (ED) use are important for studying ED utilization and access to care. We assessed the association of emergent classification of an ED visit based on the New York University ED Algorithm (EDA) with hospital mortality and hospital admission.

Study Design:

Using diagnosis codes, we applied the EDA to classify ED visits into emergent, intermediate, and nonemergent categories and studied associations of emergent status with hospital mortality and hospital admissions.

Methods:

We used a nationally representative sample of patients with visits to hospital-based EDs from repeated cross sections of the National Hospital Ambulatory Medical Care Survey from 2006 to 2009. We performed survey-weighted logistic regression analyses, adjusting for year and patient demographic and socioeconomic characteristics, to estimate the association of emergent ED visits with the probability of hospital mortality or hospital admission.

Results:

The EDA measure of emergent visits was significantly and positively associated with mortality (odds ratio [OR]: 3.79, 95% confidence interval [CI]: 2.50-5.75) and hospital admission (OR: 5.28, 95% CI, 4.93-5.66).

Conclusions:

This analysis assessed the NYU algorithm in measuring emergent and nonemergent ED use in the general population. Emergent classification based on the algorithm was strongly and significantly positively associated with hospitalization and death in a nationally representative population. The algorithm can be useful in studying ED utilization and evaluating policies that aim to change it.

Am J Manag Care. 2014;20(4):315-320

The New York University Emergency Department (ED) Algorithm is a powerful tool that can be used to classify ED visits.

• Researchers are in need of methods to categorize ED visits as emergent and nonemergent.

• Public and private decision makers may use the algorithm to evaluate the impact of policies that alter ED utilization when information on diagnosis codes is available.

Emergency departments (EDs) are a critical component of the healthcare system, but face growing demand and often have inadequate capacity.¹ Between 2000 and 2008, hospital-based ED visits grew by 15% to 124 million visits, while the total number of hospital-based EDs declined.^2,3 Most of this increase in ED visits can be attributed to increases in visits by publicly insured patients.⁴ Further, EDs often serve as primary care providers despite the fact that they are not optimally designed to provide this type of care.¹ In response, payers and policy makers have attempted to deter nonemergent ED use and increase efficiency using financial incentives. For example, some health insurance plans and state Medicaid programs have implemented or increased ED copayments in recent years.^5,6 Efforts to improve access to primary care may also reduce nonemergent ED use. Researchers require reliable tools for classifying emergent and nonemergent ED use in order to evaluate policies and initiatives designed to improve access to primary care and reduce nonemergent use. Application of these methods to assess changes in access and reliance on EDs will become even more salient after implementation of the Patient Protection and Affordable Care Act in 2014, when much of the population will gain health insurance coverage and the number of publicly insured individuals will increase dramatically.⁷

To this end, we assessed a tool for categorizing ED visits as emergent, by applying each visit to a nationally representative sample of ED visits and testing its strength as a predictor of hospital admission and mortality. Specifically, we used the New York University ED Algorithm (EDA). The EDA is based on a full chart review, using information on patient complaints, symptoms, vital signs, diagnoses, procedures, and ED resource use in order to determine the emergent nature of ED visits.⁸ The EDA could be a useful tool to evaluate ED use, but has only been tested in settings with more limited generalizability and has not previously been validated using a nationally representative sample of ED visits.

A previous study evaluated the EDA using data on commercial and Medicare patients from an integrated delivery system from 1999 to 2001, and found that patients classified as having an emergent visit under the EDA were more likely to be hospitalized within 1 day or to die within 30 days than were patients with visits classified as nonemergent.⁹ Because the EDA may perform differently among groups of patients with limited access to care who are more likely to use the ED for nonemergent reasons, such as uninsured or Medicaid populations,^10,11 results from that study may not generalize to other populations. Recent research has also shown significant changes in the demographics of ED patients over the past decade, so previous results may be outdated.⁴

This study improves on previous research in a number of ways. We applied the EDA to a national sample of hospital-based ED visits. We used the same criteria to assess the EDA as the previous study, but improve on it by using it in a nationally representative sample of all insurance groups in order to examine its generalizability to the US population, while previous research focused on privately insured and Medicare populations in a managed care context. Finally, we used several years of more recent data—from 2006 to 2009—while the previous study used data from 1999 to 2001. Similar to previous work, we examined the association between EDA categorization and hospital mortality or hospital admission, controlling for observed patient demographic and socioeconomic factors.

METHODSData and Sample Selection

We used a nationally representative sample of visits to hospital-based EDs at noninstitutional, general, and short-stay hospitals (excluding Federal, military, and Veterans Administration hospitals) located in the 50 states and the District of Columbia, from repeated cross sections of the 2006 to 2009 National Hospital Ambulatory Medical Care Survey (NHAMCS). EDs within hospitals were randomly selected using a 4-stage probability design and surveyed over a randomly selected 4-week period each year. The sampling unit was the visit, with a target sample of 100 visits per ED sampled. Visit-level information was reported by hospital staff.¹² Our sample included 117,477 ED visits with nonmissing primary diagnosis codes, representing a population of about 420 million. We used patient age, sex, and race, and socioeconomic indicators for the patient’s zip code, including median household income and urban-rural classification, as controls. We recoded the expected source of payment into private insurance, Medicare, Medicaid/State Children’s Health Insurance Program (SCHIP), self-pay/charity, and other insurance categories.

Classification of Emergent ED Visits Using the EDA

We applied the EDA to the NHAMCS data to categorize ED visits as emergent or nonemergent. The EDA used the primary International Classification of Diseases, Ninth Revision diagnosis code at discharge for each visit and assigned the probability of the visit being: (1) nonemergent (NE); (2) emergent but treatable in a primary care setting (E-PCT); (3) emergent/ED care required, but preventable or avoidable if appropriate ambulatory care had been received (E-PA); and (4) emergent/ED care required and not preventable or avoidable (E-NPA). The EDA also separately assigned primary diagnosis codes to categories for injury, mental health, alcohol, and drug-related diagnoses. For about 16% of diagnosis codes, the EDA was not able to assign a visit to any classification category due to missing or uncommon primary diagnosis codes.¹³

Following Ballard et al, we categorized ED visits into 3 levels: emergent, intermediate, and nonemergent.⁹ We classified a visit as emergent if the sum of the probabilities of E-NPA and E-PA were greater than 50%. We classified a visit as nonemergent if the sum of the probabilities of NE and E-PCT were greater than 50%. Visits where the sum of the probabilities of E-NPA and E-PA or the sum of the probabilities of NE and E-PCT equaled exactly 50% were categorized as intermediate. We also conducted a sensitivity analysis of probability thresholds at 75% and 90%, following other work using the EDA.¹⁴

Statistical Model

To estimate the association of emergent and intermediate ED visit classification based on the EDA with mortality and hospitalization, we estimated logistic regression models, using NHAMCS complex survey weights. The 2 main outcome variables of interest were dichotomous indicators for whether an individual died (either on arrival to the ED, in the ED, or after being hospitalized) and whether the patient was admitted to the hospital from the ED. All models controlled for individual characteristics, including age, gender, race (white, Black, other race); payment source (private, Medicare, Medicaid/SCHIP, self-pay/charity, other); socioeconomic indicators at the patient zip code level, including low median household income and urban location; and survey year. The covariates in the model were selected because they are known to be associated with hospitalization and health outcomes.^15,16

RESULTS

Table 1

presents the breakdown of ED visits as classified by the EDA. We were able to classify 73,076 of 111,970 total ED visits into the emergent, intermediate, and nonemergent categories. This represented a population of about 260 million ED visits. The remaining ED visits (38,894) were classified into injury (32,149), drug (304), alcohol (1718), and mental health (4723). We were not able to classify about 16% of ED visits using the EDA, which is similar to the percentage of diagnosis codes that are uncommon and could not be classified as reported by the developers of the EDA.¹³ We limited our analysis sample to patients with diagnosis codes that could be classified into emergent, intermediate, and nonemergent. We hypothesized that patients with ED visits classified as emergent would have a higher probability of mortality and hospitalization. Like Ballard et al, we excluded visits categorized as drug, alcohol, or mental health, since they may have been emergent but were less clearly associated with mortality or hospitalization.⁹ In addition, we excluded visits categorized as injury since they may have been justifiably emergent, but had a low probability of mortality or hospitalization. Within the sample that could be classified into 1 of the 3 levels, 24.1% of visits were categorized as emergent, 1.9% as intermediate, and 74.0% as nonemergent, using the 50% probability threshold.

Table 2

presents summary statistics for the analysis sample. Patients with ED visits in the sample had an average age of 37 years and were more likely to be female, white, have Medicaid or SCHIP as a primary payer, and live in an urban area. About 15% of patients were either admitted or died on arrival to the ED, in the ED, or in the hospital after an ED-associated visit.

Analysis of Emergent ED Visits Using the EDA Table 3 presents logistic regression results estimating the association between EDA visit classification and the outcome variables, mortality, and hospital admission. We found that emergent ED visit classification was positively and significantly associated with mortality (odds ratio [OR]: 3.79, P <.01) and with ED-related hospitalization (OR: 5.28, P <.01), relative to nonemergent classification. Intermediate classification was also positively associated with mortality (OR: 2.16, P = .12) and with ED-related hospitalization (OR: 4.14, P <.01), relative to nonemergent status. The results on the control variables were as expected. Older patients, males, and patients with public insurance or self-pay/charity as the primary payer (vs private insurance) were associated with significantly higher probabilities of ED-associated death and hospitalization. The findings regarding publicly insured and self-pay ED visits were consistent with other research that has shown these patient populations have poorer health and higher inpatient mortality than the privately insured.^17-19

Specification Checks

We performed 2 specification checks to test the sensitivity of our results (full results available upon request). First, we estimated conditional logit models with year-specific ED fixed effects in order to control for any unobserved time-invariant ED-specific differences (such as staffing and resources) and to examine within-ED differences in hospitalization and mortality. Results from these models were very similar to our baseline models for both mortality (OR: 3.52, 95% confidence interval [CI]: 2.58-4.80) and hospital admission (OR: 5.22, 95% CI, 4.96-5.50). Second, following Ballard et al, we repeated our analyses using different probability thresholds (75% and 90%) for the ED measures in order to check the sensitivity of our results.⁹ In these models, visits were categorized as intermediate if the probability they were emergent was between 25% (10%) and 75% (90%). The results for our emergent variable represent the comparison of visits with a predicted probability of being emergent of 75% (90%) or greater with visits with a predicted probability of being emergent of 25% (10%) or less, thus we expect the ORs to increase, but be in the same direction as our baseline results. We found that this is generally the case, with ORs for our mortality outcome of 6.11 (95% CI, 3.73-10.02) and 4.90 (95% CI, 2.16-11.15) for the 75% and 90% cutoffs, respectively. Similarly, ORs for our admission outcome are 8.35 (95% CI, 7.60-9.17) and 11.26 (95% CI, 9.83-12.92) for the 75% and 90% cutoffs, respectively. All specification checks support our main finding that emergent visit status is significantly associated with hospital admission and death.

DISCUSSION

This analysis evaluated methods for classifying ED visits as emergent, using the EDA in a nationally representative sample of ED visits, and found that measures based on the EDA were significantly associated with mortality and EDassociated hospitalizations. The magnitude of association of emergent classification with hospitalization and mortality were similar to, but larger than, those from Ballard et al, who used data from privately insured individuals in a single integrated delivery system.⁹ The difference may be due to the fact that our sample included both the Medicaid and uninsured populations, and was not restricted to privately insured and Medicare patients in an integrated delivery system, who are likely to have better access to care. In addition, our sample excluded visits categorized as injury, which may also explain our larger result. Further, our results suggest that the EDA can be used to categorize visits as emergent and nonemergent by researchers when information is available on diagnosis codes but not on triage time (eg, claims data) or when a full chart review is not possible.

There were a few limitations to our approach. In categorizing visits using the EDA, there may have been some measurement error in determining if a visit is truly emergent. For example, some diagnoses may have been appropriately categorized as emergent, but were not associated with death or hospital admission (eg, a broken leg). We attempted to address this possible issue by excluding ED admissions for injury. Another limitation to our research was that we were only able to observe mortality and hospital admission as a direct admit from the ED, but not able to observe subsequent mortality or hospital admission within a limited time frame after the ED visit. Further, we used a blunt measure of hospital admission in order to directly compare our results with Ballard et al,⁹ which may not necessarily reflect severity in cases where hospitalizations resulted in shorter lengths of stay or were less acute. Finally, thresholds for hospital admission may have differed by payer type or other subjective patient characteristics. EDs may serve as a gateway through which to admit patients who do not have access to care through other channels due to their insurance status or other socioeconomic factors. This question is beyond the scope of this paper and requires further research.

This study demonstrated that the EDA can be used to identify ED visits associated with mortality and hospitalization. Classifying ED visits as emergent or nonemergent has been a shortcoming of the literature on ED use. The EDA is increasingly being used at the state and local levels^{20- 23} and, despite its limitations, the EDA has the potential to be a useful tool for understanding patterns of use and assessing the effects of policies and programs aimed at reducing nonemergent ED use. While the developers of the EDA have cautioned that the algorithm would not be appropriate to use for making individual reimbursementbased decisions, and recent research has supported this assessment,²⁴ it can be applied to assess overall trends in ED use and to study how interventions and policies may affect these trends. For example, it has been used by researchers studying how new programs providing primary care for the uninsured affected ED use among these particular patient populations.^20,21 In such contexts, where administrative data are available to assess how a program or policy change affected utilization, the EDA can be useful.

With the implementation of health reform, there will be major changes in the number of uninsured, the distribution of insurance coverage types, the payment and organization of healthcare providers, and other aspects of the healthcare system that could affect the way various patient populations utilize the ED. It will be important to have tools to classify ED visits and to test the success of interventions and policies designed to alter ED utilization and improve access to alternative sources of care. We have shown that the conclusions of earlier research validating the EDA in the context of managed care also hold when a nationally representative sample of ED visits is examined, suggesting that the EDA is a useful tool for health services and policy researchers.Author Affiliations: RTI International, Washington, DC (SOG); Department of Healthcare Policy and Research, Virginia Commonwealth University School of Medicine (LS).

Source of Funding: None reported.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (SOG, LS); acquisition of data (SOG, LS); analysis and interpretation of data (SOG, LS); drafting of the manuscript (SOG, LS); critical revision of the manuscript for important intellectual content (SOG, LS); statistical analysis (SOG, LS).

Address correspondence to: Sabina Ohri Gandhi, PhD, RTI International, 701 13th St NW, Ste 750, Washington, DC 20005-3967. E-mail: sgandhi@rti.org.1. Institute of Medicine. Hospital-Based Emergency Care: At the Breaking Point. Washington, DC: National Academies Press; 2007.

2. National Center for Health Statistics; US Department of Health and Human Services. Health, United States, 2010: with special feature on death and dying. http://www.cdc.gov/nchs/data/hus/hus10.pdf#089. Published 2011. Accessed February 21, 2012.

3. Hsia RY, Kellerman AL, Shen YC. Factors associated with closures of emergency departments in the United States. JAMA. 2011;305(19): 1978-1985.

4. Tang N, Stein J, Hsia RY, Maselli JH, Gonzales RJ. Trends and characteristics of US emergency department visits, 1997-2007. JAMA. 2010;304(6):664-670.

5. Hsu J, Price M, Brand R, et al.Cost-sharing for emergency care and unfavorable clinical events: findings from the safety and financial ramifications of ED copayments study. Health Serv Res. 2006;41(5): 1801-1820.

6. Mortensen K. Copayments did not reduce Medicaid enrollees’ nonemergency use of emergency departments. Health Aff (Millwood). 2010;29(9):1643-1650.

7. Congressional Budget Office. Selected CBO publications reltaed to healthcare legislation, 2009-2010. Washington, DC: Congress of the United States; 2010.

8. Billings J, Parikh N, Mijanovich T. Emergency room use: the New York story. Commonwealth Fund issue brief 2000. http://wagner.nyu .edu/chpsr/index.html?p=25. Published 2000. Accessed October 31, 2011.

9. Ballard DW, Price M, Fung V, et al. Validation of an algorithm for categorizing the severity of hospital emergency department visits. Med Care. 2010;48(1):58-63.

10. Cunningham PJ, May J. Insured Americans drive surge in emergency department visits: issue brief for the Center for Studying Health System Change. 2003;(70):1-6.

11. Zuckerman S, Shen YC. Characteristics of occasional and frequent emergency department users: do insurance coverage and access to care matter? Med Care. 2004;42(2):176-182.

12. Inter-University Consortium for Political and Social Research. National Hospital Ambulatory Medical Care Survey series. http://www .icpsr.umich.edu/icpsrweb/ICPSR/series/42. Published 2011. Accessed February 21, 2012.

13. New York University Center for Health and Public Service Research. NYU ED algorithm. http://wagner.nyu.edu/chpsr/index.html?p=25. Published 2011. Accessed July 27, 2011.

14. Wharam JF, Landon BE, Galbraith AA, Kleinman KP, Soumerai SB, Ross-Degnan D. Emergency department use and subsequent hospitalizations among members of a high-deductible health plan. JAMA. 2007;297(10):1093-1102.

15. Billings J, Zeitel L, Lukomnik J, Carey TS, Blank AE, Newman L. Impact of socioeconomic status on hospital use in New York City. Health Aff (Millwood). 1993;12(1):162-173.

16. Agency for Healthcare Research and Quality.US Department of Health and Human Services. Addressing racial and ethnic disparities in healthcare. http://www.ahrq.gov/research/findings/factsheets/minority/ disparit/. Published April 2013. Accessed July 21, 2013.

17. Hasan O, Orav EJ, Hicks LS. Insurance status and hospital care for myocardial infarction, stroke, and pneumonia. J Hosp Med. 2010;5(8): 452-459.

18. Ayanian JZ, Kohler BA, Abe T, Epstein AM.The relation between health insurance coverage and clinical outcomes among women with breast cancer. N Engl J Med. 1993;29(359):326-331.

19. Weissman JS, Gatsonis C, Epstein AM. Rates of avoidable hospitalization by insurance status in Massachusetts and Maryland. JAMA. 1992;286(17):2388-2394.

20. Bradley CJ, Gandhi SO, Neumark D, Garland S, Retchin S. Lessons for coverage expansion: a Virginia primary care program for the uninsured reduced utilization and cut costs. Health Aff (Millwood). 2012; 31(2):350-359.

21. McLaughlin C, Colby M, Bee G, Libersky J. Healthy San Francisco: changes in access to and utilization of healthcare services. Ann Arbor: Mathematica Policy Research; 2011.

22. Utah Office of Healthcare Statistics. Primary care sensitive emergency department visits in Utah, 2001. http://health.utah.gov/hda/ reports/Primary_Care_ERvisits_Utah2001.pdf. Published April 2004. Accessed October 31, 2011.

23. Washington State Hospital Association. Washington emergency room use: safety net or unneeded services? http://wacmhc.org/documents/ WSHA%20Study.pdf. Published 2007. Accessed October 13, 2011.

24. Raven MC, Lowe RA, Maselli J, Hsia RY. Comparison of presenting complaint vs discharge diagnosis for identifying ‘nonemergency’ emergency department visits. JAMA. 2013;309(11):1145-1153.

Download PDF: Emergency Department Visit Classification Using the NYU Algorithm