Publication
Article
Author(s):
The authors examined 2 high-risk classification methods to compare and contrast the patient populations, and to identify the preferred method for predicting subsequent emergency department visits.
ABSTRACTObjectives: To compare 2 methods of identifying patients at high-risk of repeat emergency department (ED) use: high Care Assessment Need (CAN) score (≥90), derived from a model using Veterans Health Administration (VHA) data, and "Super User" status, defined as more than 3 ED visits within 6 months of the index ED visit.
Study Design: Retrospective cohort study.
Methods: Using McNemar’s test, we compared rates of high-risk classification between CAN score and Super User status. We examined differences in patient characteristics and healthcare utilization across 4 levels of risk classification: high CAN and Super User status (n = 198), CAN <90 and non—Super User (n = 622), high CAN and non–Super User (n = 616), or Super User and CAN score <90 (n = 106). We used logistic regression to identify associations between risk classification and any ED visit within 90 days.
Results: Of 1542 veterans, 52.8% (n = 814) had a CAN score ≥90 and 19.7% (n = 304) were Super Users (P <.0001), indicating discrepant rates of high-risk classification. However, we found no differences in patient characteristics. Rates of subsequent ED use were high: 63.1% of patients had 1 or more ED visits. No levels of risk classification were associated with subsequent ED use within 90 days (P = .25).
Conclusions: Among the VHA users with multimorbidity and 3 or more prior ED visits or hospitalizations, subsequent ED use was high. Although CAN scores have demonstrated utility for predicting hospitalizations and deaths, prior utilization and multimorbidity without further risk classification identified a high-risk group for repeat ED use.
Am J Manag Care. 2017;23(8):e275-e279
This article has been corrected in Am J Manag Care. 2019;25(3):140.Takeaway Points
We compared 2 methods of identifying patients at high risk of subsequent emergency department (ED) use: 1) Care Assessment Need (CAN) score and 2) Super User status.
Emergency department (ED) utilization not resulting in hospital admission, referred to as outpatient ED visits, may be avoidable1 and more costly than an outpatient clinic2; thus, it is considered potentially low-value care. To reduce low-value care, risk prediction models have been developed to identify the patients who account for a disproportionately large amount of healthcare utilization; the goal is to target these patients for interventions that can reduce avoidable utilization.3 Researchers often develop and validate risk prediction models for disease-specific populations3; however, the models may not be generalizable to broader and more medically complex populations.
Outpatient ED visits are common in the Veterans Health Administration (VHA), the largest integrated healthcare system in the United States, which serves more than 9 million veterans nationally.4 From 2007 to 2008, 80% of ED visits were outpatient; of these, 15% had a repeat ED visit within 30 days—a higher rate than Medicare beneficiaries.1 The VHA has been at the forefront of predictive analytics in healthcare and has implemented Care Assessment Need (CAN) scores for all VHA users. CAN scores use complex multivariate modeling to generate a validated risk prediction of hospitalization and/or death within 90 days or 1 year, using available electronic health records (EHRs) and administrative data.5 CAN scores are utilized to optimize care coordination and resource allocation for high-risk veterans.5
However, it is unknown whether CAN scores identify patients at higher risk for repeat ED utilization—especially compared with simpler strategies, such as a previous history of high ED utilization.5-11 Thus, in this exploratory study, we examined whether the CAN score provided further information on risk for repeat ED visits for a high-risk cohort of VHA-affiliated patients. First, we compared whether CAN scores and Super User status (ie, having 4 or more ED visits within the last year)12 identified the same patients as high risk. Then, we assessed whether these risk classifications could predict repeat ED visits that occurred within 90 days of an index ED visit.
METHODS
Study Cohort
The study cohort met initial eligibility criteria for an ongoing randomized clinical trial at the Durham Veterans Affairs Medical Center (DVAMC), Discharge Information and Support for Patients Receiving Outpatient Care in the ED (DISPO ED), which took place from March 10 to September 30, 2014.13 DISPO ED examined the effectiveness of a nurse-led intervention to reduce repeat ED visits. In addition to having an index outpatient ED visit, inclusion criteria included: 1 or more visits to a DVAMC-affiliated primary care clinic within the previous 12 months (proxy for engagement with the VA system), 1 or more DVAMC ED visit or hospital admission in the 6 months prior to the index ED visit, and 2 or more chronic conditions.13 By the end of the study time period, 17% of all ED visits met these eligibility criteria. Exclusion criteria included current enrollment or previous refusal to participate in DISPO ED, residence in a nursing home, and death on date of the index ED visit.
Data Sources
We used VHA administrative data files, including the Vital Status Mini File,14 enrollment tables from the VHA’s Assistant Deputy Under Secretary for Health for Policy and Planning,15 Medical SAS datasets,16 and additional domains from the Corporate Data Warehouse.
Measures
Primary outcome: repeat VHA ED visit within 90 days. We determined ED use within 90 days of the index visit through administrative stop codes for ED care at US Veterans Affairs medical centers. We determined outpatient ED encounters by using administrative codes for VHA ED visits and VHA inpatient care administrative datasets.
Key predictors (CAN score). We extracted the CAN score predicting the percentile of risk of hospital admission in the 90 days closest to the index ED visit date and dichotomized CAN scores using the median split (<90 or ≥90). For example, a CAN score of 90 is associated with an average observed hospitalization rate (≤90 days) of 14% compared with an average of 2.7% in the general VHA population.17
ED Super User. Veterans with more than 4 ED visits to the DVAMC within 6 months (including the index ED visit) were categorized as Super Users, based on prior studies and clinical experience.13
Covariates (sociodemographics). Demographics included race, age, marital status, and gender. To indicate economic status, we determined whether the veteran was exempted from co-payments due to limited financial means and had unstable housing within the 12 months prior to the index ED visit.1
Chronic conditions. We used diagnosis codes associated with encounters in the year prior to the index ED visit to identify anemia, congestive heart failure, chronic lung disease, chronic renal failure, diabetes, hypertension, ischemic heart disease (IHD), peripheral vascular disease, and mental health conditions, including anxiety disorder, depressive disorder, posttraumatic stress disorder (PTSD), and substance abuse disorder, in accordance with the VHA definition of chronic conditions per the VHA Support Service Center Chronic Disease Registry Development Rules.13,18
Medical complexity (Quan Charlson Comorbidity Index). The Quan Charlson Comorbidity Index predicts mortality within 12 months using 17 comorbidities based on the original Charlson Comorbidity Index,19 but using updated weights identified by Schneeweiss et al.20
Outpatient utilization in year prior to index ED visit. We counted the number of VHA primary care, outpatient specialty services, and mental health clinic encounters.
Statistical Analysis
We first compared high-risk classification by CAN score of ≥90 and Super User status, using McNemar’s test. Second, we examined differences in demographics, chronic conditions, and utilization in the year prior to the index ED visit across the 4 classification groups: high-risk by both (CAN score ≥90 and identified Super User), high-risk by CAN score only (CAN score ≥90 and non—Super User status), high-risk by Super User status only (CAN Score <90 and identified Super User), or not considered high risk by both (CAN score <90 and non–Super User). For categorical variables, we used χ2 analysis. Analysis of variance was used for continuous variables and Poisson regressions for count variables. Finally, we compared repeat ED visits within 90 days (yes/no) for these 4 groups, examining CAN score and Super User status in logistic models, adjusting for the aforementioned demographic, economic, comorbidity, and prior healthcare use covariates.
RESULTS
Study Cohort Characteristics
Fifty percent of participants were African American, and 46% were white. The majority (81%) were exempt from co-payments due to financial need. Moreover, 80% had hypertension, 50% had diabetes, and 28% had IHD. Additionally, 66% had 1 or more mental health conditions: 31% were diagnosed with PTSD, 41% with depression, and 19% with anxiety. This cohort demonstrated a high level of engagement with the VHA in the year prior to their index ED visit, with 49% having had 1 or more mental health outpatient encounters and 93% having had 1 or more outpatient specialty service encounters. Veterans with mental health, outpatient specialty, or primary care use in the year prior to the index ED visit had an average of 11 mental health, 9 specialty, and 5 primary care outpatient clinic encounters for the year, respectively (eAppendix A [eAppendices available at ajmc.com]).
Risk Classification by CAN Score Versus Super User Status
Using McNemar’s test, the rate of classification between the 2 methods was discrepant (P <.0001) (Table 1). Based on a CAN score of ≥90, the rate of high-risk classification was 52.8% (n = 814) versus 19.7% (n = 304) for ED Super Users. Of those with a CAN score of 90 or above (n = 814), 76% were not identified as Super Users. Of Super Users (n = 304), 34.9% had a CAN score greater than 90. The 2 methods identified different sets of patients as high risk.
Characteristics and Utilization Outcomes by Risk Classification of CAN Score and Super User Status
In general, there were few differences in characteristics across the 4 groups based on cross-classification of CAN score of ≥90 and Super User status (eAppendix A). Significant differences included gender (P <.001), primary care utilization in the year prior to the index ED visit (P <.001), specialty care and mental health care utilization in the year prior to the index ED visit (P <.001), and number of ED visits in the year prior to the index ED visit (P <.05). Patients who were identified as Super Users and had a CAN score below 90 had the highest number of primary care visits in the year prior, and those identified as high risk by both methods had the highest number of mental health visits. There were no significant differences in proportion with chronic conditions, with the exception of anxiety (P <.05).
Repeat ED Visits
Overall, 63% (n = 973) of the cohort had 1 or more repeat ED visits within 90 days after the index ED visit (mean = 1.7 repeat ED visits within the observation period); 90% of which were outpatient ED visits. When examining only repeat outpatient ED visits, 59% (n = 906) of the cohort had more than 1 repeat encounter within 90 days. Veterans with a repeat outpatient ED visit had an average of 1.6 repeat ED visits. Six percent (n = 92) of patients died within 90 days of their index ED visit.
There were no statistically significant differences in the proportion of patients with 1 or more repeat ED visits within 90 days of the index encounter, the number of repeat ED visits, or outpatient ED visits across the 4 risk groups (eAppendix B).
Risk classification of the 4 groups was not associated with repeat ED visits within 90 days of the index encounter (P = .28). Adjustment for covariates yielded similar results. When limiting the outcomes to outpatient ED visits within 90 days, we found similar results (Table 2).
DISCUSSION
In our cohort of VHA ED patients with multimorbidity (defined as 2 or more chronic conditions) and history of a prior ED visit or hospitalization, nearly two-thirds had at least 1 repeat ED visit within 90 days. We found that the CAN score and ED Super User status identified different groups of patients. When we examined 4 risk stratification groups cross classifying by CAN score and Super User status, we found no association between the risk classification groups and repeat ED visits within 90 days of the index date.
Our findings of high repeat ED use in the cohort overall are consistent with previous studies of older veterans, which found that high rates of chronic conditions and prior ED and hospital use were independent predictors of repeat ED use.6,21,22 Compared with the general VHA ED population, of which 15% of patients had a repeat ED visit within 30 days, our cohort had much higher repeat ED utilization.1 Considering these past studies, our cohort was more racially diverse6,23,24 and had more mental health conditions.25
The fact that these 2 methods of risk stratification did not improve prediction of ED returns in this already high-risk population has important clinical and research implications. First, new applications of existing risk prediction tools should be validated before being put into practice. CAN scores have been available to primary care providers throughout the VA system; however, more data about their clinical utility outside of recognizing patients at high risk for hospitalization and mortality are needed before they are repurposed. Second, although the CAN model, along with other EHR-based methods of risk stratification, incorporates diverse information on patient demographics, medical conditions, and previous utilization, it excludes potentially important data, such as socioeconomic, cultural, and other contextual factors that often play significant roles.26 Third, considering the CAN score is a comprehensive model of health status predictors, improvements in prediction may be achieved through the use of alternative models more frequently seen in other disciplines, such as models based on machine learning techniques.27-30
Although we did not identify subgroups of patients at higher risk of ED returns based on CAN scores, ED Super User status, or a combination of these variables, an examination of the cohort characteristics reveals possibilities for future study into clinical populations of interest and potential ways to improve care. First, we observed high engagement with VHA services, with an average of 5 primary care clinic encounters in the year prior to the index ED visit. Previously, lack of access and low engagement with PCPs and specialists have been highlighted as associated with increased ED use.31-35 Our results suggest that high repeat ED visit rates may not be due exclusively to access barriers; other issues, such as inadequate care coordination, may also play a role.36 If this is the case, increased use of strategies, such as telehealth, may be essential to reducing repeat ED visits. Moreover, the prevalence of mental health diagnoses in our cohort (49%; mean = 11 mental health outpatient encounters in the year prior) was much higher than other studies examining recurring ED use25 and may have contributed to the higher rates of repeat ED visits. Future research should consider the high rates of mental health conditions explicitly. It is notable that the VHA has more extensive and available mental health services compared with many non-VHA systems,36 and interventions directed at non-VHA patients may need to surmount additional barriers to access for mental health services.
Limitations
There are several limitations of this study. First, we identified VHA ED utilization and diagnoses of interest to veterans using VHA administrative claims data, which are subject to coding bias, errors in record keeping, and delayed records of utilization. Second, the single-site cohort potentially limited generalization. Third, we had limited information about other potentially relevant variables related to ED use, such as socioeconomic status data. Fourth, no gold-standard definition of the term "Super User" exists. We examined ED visits over a 9-month period, which does not address the issue of seasonality. We also only examined repeat ED visits at a single time point (within 90 days); findings may have been somewhat different with a longer time horizon (ie, 6, 9, and 12 months). To address these limitations, we extracted data after allowing sufficient time for records to be updated and relied on clinical expertise from senior researchers regarding best practices to determine diagnoses using claims data. We also relied on the literature, clinical expertise, and prior work on the distribution of ED utilization to define Super User status.12 As an exploratory analysis of associations using secondary data, there was no power calculation for this study. However, the percentages of repeat ED use across the different categorizations were similar and generally above 60%. Thus, there were no indications of significant differences we were not powered to detect; the narrow widths of the confidence intervals are reasonable from the logistic models.
In conclusion, among DVAMC users with multimorbidity and more than 2 prior ED visits or hospitalizations, repeat ED use within 90 days was very high. Applying 2 methods of risk stratification in this population identified discrepant groups of patients, and classification of risk by these 2 measures was not associated with repeat ED use within 90 days. Identifying clinically relevant subgroups is important for future interventions to improve care and provide high-value services for high-risk groups defined by multimorbidity and utilization.
Acknowledgments
The study was funded by the Department of Veterans Affairs, Health Services Research and Development Service (IIR 12-052) and was also supported by the Durham VA Center for Health Services Research in Primary Care and Geriatrics Research, Education and Clinical Center. Dr Weinberger is supported by the Research Career Scientist Program (RCS 91-408). The VA Office of Academic Affiliations provided fellowship support for Dr Duan-Porter (No. TPP 21-022). The contents of this manuscript are solely the responsibility of the authors and do not necessarily represent the official views of the Department of Veterans Affairs.
This article has been corrected in Am J Manag Care. 2019;25(3):140.Author Affiliations: Health Services Research and Development Service (KEMM, WD-P, KMS, EM, CJC, MW, CHVH, EZO, KM, KES, CCH, SNH), and Geriatric Research, Education, and Clinical Center (KES, CCH, SNH), Durham VA Medical Center, Durham, NC; Department of Medicine, Duke University Medical Center (WD-P, CHVH, EZO, KES, SNH), Durham, NC; Department of Biostatistics and Bioinformatics (CJC), and Center for the Study of Human Aging and Development (KES, CCH, SNH), Duke University, Durham, NC; Department of Health Policy and Management, University of North Carolina at Chapel Hill (MW), Chapel Hill, NC; Duke University School of Nursing (CCH), Durham, NC; Ambulatory Care Service, Durham VA Medical Center (CK), Durham, NC.
Source of Funding: The study was funded by the Department of Veterans Affairs, Health Services Research and Development Service (IIR 12-052) and was also supported by the Durham VA Center for Health Services Research in Primary Care and Geriatrics Research, Education and Clinical Center. Dr Weinberger is supported by the Research Career Scientist Program (RCS 91-408). The VA Office of Academic Affiliations provided fellowship support for Dr Duan-Porter (No. TPP 21-022).
Author Disclosures: Dr Hastings is a VA employee, and has received grants from the VA Health Services Research and Development Service. The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.
Authorship Information: Concept and design (KEMM, WD-P, CJC, MW, EZO, KES, SNH, CH); acquisition of data (KEMM, KM, SNH, EM); analysis and interpretation of data (KEMM, WD-P, KMS, CJC, MW, CHVH, EZO, KES, SNH, CCH); drafting of the manuscript (KEMM, WD-P, CJC, MW, KES, SNH, CCH); critical revision of the manuscript for important intellectual content (KEMM,WD-P, KMS, CJC, MW, CHVH, EZO, KES, CK, SNH, EM, CH); statistical analysis (KEMM, KMS); provision of patients or study materials (KM); obtaining funding (EZO, CK, SNH); administrative, technical, or logistic support (EZO, KM, CK, EM); and supervision (CHVH, EZO).
Address Correspondence to: Katherine E.M. Miller, VA Medical Center (152), 508 Fulton St, Durham, NC 27705. E-mail: Katherine.miller9@va.gov.REFERENCES
1. Hastings SN, Smith VA, Weinberger M, Schmader KE, Olsen MK, Oddone EZ. Emergency department visits in Veterans Affairs medical facilities. Am J Manag Care. 2011;17(spec no 6):e215-e223.
2. Galarraga JE, Mutter R, Pines JM. Costs associated with ambulatory care sensitive conditions across hospital-based settings. Acad Emerg Med. 2015;22(2):172-181. doi: 10.1111/acem.12579.
3. Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306(15):1688-1698. doi: 10.1001/jama.2011.1515.
4. Department of Veterans Affairs statistics at a glance. Department of Veterans Affairs website. https://www.va.gov/vetdata/docs/Quickfacts/Stats_at_a_glance_08_27_15.pdf. Updated June 30, 2015. Accessed January 11, 2016.
5. Wang L, Porter B, Maynard C, et al. Predicting risk of hospitalization or death among patients receiving primary care in the Veterans Health Administration. Med Care. 2013;51(4):368-373. doi: 10.1097/MLR.0b013e31827da95a.
6. Hastings SN, Schmader KE, Sloane RJ, Weinberger M, Goldberg KC, Oddone EZ. Adverse health outcomes after discharge from the emergency department—incidence and risk factors in a veteran population. J Gen Int Med. 2007;22(11):1527-1531.
7. Bodenmann P, Velonaki VS, Ruggeri O, et al. Case management for frequent users of the emergency department: study protocol of a randomised controlled trial. BMC Health Serv Res 2014;14:264. doi: 10.1186/1472-6963-14-264.
8. Doupe MB, Palatnick W, Day S, et al. Frequent users of emergency departments: developing standard definitions and defining prominent risk factors. Ann Emerg Med. 2012;60(1):24-32. doi: 10.1016/j.annemergmed.2011.11.036.
9. Hunt KA, Weber EJ, Showstack JA, Colby DC, Callaham ML. Characteristics of frequent users of emergency departments. Ann Emerg Med. 2006;48(1):1-8.
10. Hansagi H, Olsson M, Sjoberg S, Tomson Y, Göransson S. Frequent use of the hospital emergency department is indicative of high use of other health care services. Ann Emerg Med. 2001;37(6):561-567.
11. Vinton DT, Capp R, Rooks SP, Abbott JT, Ginde AA. Frequent users of US emergency departments: characteristics and opportunities for intervention. Emerg Med J. 2014;31(7):526-532. doi: 10.1136/emermed-2013-202407.
12. Hastings S, Stechuchak K, Oddone E, et al. Older veterans and emergency department discharge information. BMJ Qual Saf. 2012;21(10):835-842.
13. Hastings SN, Betts E, Schmader KE, et al. Discharge information and support for veterans receiving outpatient care in the emergency department: study design and methods. Contemp Clin Trials. 2014;39(2):342-350. doi: 10.1016/j.cct.2014.10.008.
14. Sohn MW, Arnold N, Maynard C, Hynes DM. Accuracy and completeness of mortality data in the Department of Veterans Affairs. Popul Health Metr. 2006;4:2.
15. VIReC research user guides: VHA Assistant Deputy Under Secretary for Health enrollment files. US Department of Veterans Affairs Information Resource Center website. https://www.virec.research.va.gov/Resources/RUGs.asp. Published September 2013. Accessed 2017.
16. VIReC research user guides: fiscal year 2014 VHA Medical SAS outpatient datasets and inpatient encounters dataset. US Department of Veterans Affairs Information Resource Center website. https://www.virec.research.va.gov/Resources/RUGs.asp. Published 2015. Accessed 2017.
17. Wang L, Porter B, Maynard C, et al. Predicting risk of hospitalization or death among patients with heart failure in the Veterans Health Administration. Am J Cardiol. 2012;110(9):1342-1349. doi: 10.1016/j.amjcard.2012.06.038.
18. VHA Support Service Center (VSSC). Chronic Disease Registry Development Rules. In: Affairs DoV, ed 2011.
19. Quan H, Li B, Couris CM, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol. 2011;173(6):676-682. doi: 10.1093/aje/kwq433.
20. Schneeweiss S, Wang PS, Avorn J, Glynn RJ. Improved comorbidity adjustment for predicting mortality in Medicare populations. Health Serv Res. 2003;38(4):1103-1120.
21. Horney C, Schmader K, Sanders LL, et al. Health care utilization before and after an outpatient ED visit in older people. Amer J Emerg Med. 2012;30(1):135-142. doi: 10.1016/j.ajem.2010.10.036.
22. McCusker J, Healey E, Bellavance F, Connolly B. Predictors of repeat emergency department visits by elders. Acad Emerg Med. 1997;4(6):581-588.
23. Friedmann PD, Jin L, Karrison TG, et al. Early revisit, hospitalization, or death among older persons discharged from the ED. Am J Emerg Med. 2001;19(2):125-129.
24. Hastings SN, Oddone EZ, Fillenbaum G, Sloane RJ, Schmader KE. Frequency and predictors of adverse health outcomes in older Medicare beneficiaries discharged from the emergency department. Med Care. 2008;46(8):771-777. doi: 10.1097/MLR.0b013e3181791a2d.
25. Trivedi RB, Post EP, Sun H, et al. Prevalence, comorbidity, and prognosis of mental health among US veterans. Am J Public Health. 2015;105(12):2564-2569. doi: 10.2105/AJPH.2015.302836.
26. Zullig LL, Whitson HE, Hastings SN, et al. A systematic review of conceptual frameworks of medical complexity and new model development. J Gen Intern Med. 2016;31(3):329-337. doi: 10.1007/s11606-015-3512-2.
27. Sahoo AJ, Kumar Y. Seminal quality prediction using data mining methods. Techno Health Care. 2014;22(4):531-545. doi: 10.3233/THC-140816.
28. Chen HL, Yang B, Wang G, Wang SJ, Liu J, Liu DY. Support vector machine based diagnostic system for breast cancer using swarm intelligence. J Med Syst. 2012;36(4):2505-2519. doi: 10.1007/s10916-011-9723-0.
29. Kourou K, Exarchos TP, Exarchos KP, Karamouzis MV, Fotiadis DI. Machine learning applications in cancer prognosis and prediction. Comput Struct Biotechnol J. 2015;13:8-17. doi: 10.1016/j.csbj.2014.11.005.
30. Crown WH. Potential application of machine learning in health outcomes research and some statistical cautions. Value Health. 2015;18(2):137-140. doi: 10.1016/j.jval.2014.12.005.
31. Mustard CA, Kozyrskyj AL, Barer ML, Sheps S. Emergency department use as a component of total ambulatory care: a population perspective. CMAJ. 1998;158(1):49-55.
32. Savageau JA, McLoughlin M, Ursan A, Bai Y, Collins M, Cashman SB. Characteristics of frequent attenders at a community health center. J Am Board Family Med. 2006;19(3):265-275.
33. Zuckerman S, Shen YC. Characteristics of occasional and frequent emergency department users: do insurance coverage and access to care matter? Med Care. 2004;42(2):176-182.
34. Huang JA, Weng RH, Lai CS, Hu JS. Exploring medical utilization patterns of emergency department users. J Formos Med Assoc. 2008;107(2):119-128. doi: 10.1016/S0929-6646(08)60125-4.
35. Byrne M, Murphy AW, Plunkett PK, McGee HM, Murray A, Bury G. Frequent attenders to an emergency department: a study of primary health care use, medical profile, and psychosocial characteristics. Ann Emerg Med. 2003;41(3):309-318.
36. Shekelle PG, Asch S, Glassman P, Matula S, Trivedi A, Miake-Lye I. Comparison of quality of care in VA and non-VA settings: a systematic review. Department of Veterans Affairs website. https://www.hsrd.research.va.gov/publications/esp/quality.pdf. Published September, 2010. Access date: February 2, 2016.