Managed Critical Care: Impact of Remote Decision-Making on Patient Outcomes

ABSTRACT

Objectives: Tele–intensive care unit (tele-ICU) use has become increasingly common as an extension of bedside care for critically ill patients. The objective of this work was to illustrate the degree of tele-ICU involvement in critical care processes and evaluate the impact of tele-ICU decision-making authority.

Study Design: Previous studies examining tele-ICU impact on patient outcomes do not sufficiently account for the extent of decision-making authority between remote and bedside providers. In this study, we examine patient outcomes with respect to different levels of remote intervention.

Methods: Analysis and summary statistics were generated to characterize demographics and patient outcomes across different levels of tele-ICU intervention for 82,049 critically ill patients. Multivariate logistic regression was used to evaluate odds of mortality, readmission, and likelihood of patients being assigned to a particular remote intervention category.

Results: Managing (vs consulting) physician type influenced the level of remote intervention (adjusted odds ratio [AOR], 2.42). A higher level of tele-ICU intervention was a significant factor for patient mortality (AOR, 1.25). Female sex (AOR, 1.05), illness severity (AOR, 1.01), and higher tele-ICU intervention level (AOR, 1.13) increased odds of ICU readmission, whereas length of stay in number of days (AOR, 0.93) and consulting (vs managing) physician type (AOR, 0.79) decreased readmission odds.

Conclusions: This study’s findings suggest that higher levels of tele-ICU intervention do not negatively affect patient outcomes. Our results are a step toward understanding tele-ICU impact on patient outcomes by accounting for extent of decision-making authority, and they suggest that the level of remote intervention may reflect patient severity. Further research using more granular data is needed to better understand assignment of intervention category and how variable levels of authority affect clinical decision-making in tele-ICU settings.

Am J Manag Care. 2023;29(7):e208-e214. https://doi.org/10.37765/ajmc.2023.89400

Takeaway Points

• Use of the tele–intensive care unit (tele-ICU) does not negatively affect patient outcomes or care processes.
• This work serves as a step toward better understanding the implications of remote critical care intervention.
• Our results suggest that use of tele-ICU should be analyzed from a systems perspective to better understand the impact of remote intervention on critical care workflow.
• Practitioners should achieve collaborative communication between the bedside and remote ICU teams in combination with focusing on individual patient outcomes.

Critical care provided via telemedicine in the intensive care unit (ICU), or tele-ICU, is increasingly common to extend the reach of intensivists across geographically distinct ICUs and rural critical access hospitals. Studies show varying results of tele-ICU on patient-centered outcomes such as mortality and length of stay (LOS).¹ Some previous studies have demonstrated decreased mortality and LOS, increased adherence to best practices, and fewer preventable complications,^2-4 whereas others have found minimal benefit except when adjusting for severity of illness.^5-7 Among those studies analyzing outcomes before and after tele-ICU implementation, the level of involvement by the tele-ICU ranges widely from consultation only to full decision-making authority. Thus, direct before-and-after analyses are difficult because the impact on patient outcomes varies across decision-making authority level.^8-10

Teamwork, communication, trust, and level of engagement are all proposed components of tele-ICU success but are not sufficiently supported by evidence.^11,12 A major issue in previous analyses of tele-intensivist managed care is the assumption that tele-ICU adoption directly affects patient outcomes. Based on Donabedian’s model of structure-process-outcome,¹³ tele-ICU uptake changes the care system as it redefines the technological context in which care providers are embedded. Consequently, it also changes the process of how care providers conduct their tasks individually or collaboratively to deliver care, which could ultimately influence patient outcomes. Previous analyses of tele-ICU effect on patient outcomes were built on a simplified notion that the tele-ICU structure would directly affect outcomes, but these overlooked the role and impact of care process. Therefore, a complete evaluation of tele-ICU impact on patient outcomes should consider factors from both the structure level (eg, before and after tele-ICU uptake) and process level (eg, different levels of decision-making authority) to account for the varied interpretations of tele-ICU impact.

Objectives

To facilitate effective collaboration between onsite and tele-intensivists, the tele-ICU practice requires specification of decision-making authority. For example, the onsite physicians, either consulting or attending, can assign an intervention category to each patient to designate the level of decision-making involvement the remote staff is afforded for each patient. Although this is not currently done at every institution, there is a need to examine the effects of tele-ICU decision-making to better understand the impact of tele-intensivist managed care on patient outcomes. Our study aims to explore the upstream process changes from tele-ICU that affect downstream patient outcomes by leveraging data from intervention category assignments and examine their association with patient outcomes.

METHODS

Data Source

Data were extracted from a publicly available tele-ICU database, the Philips eICU Collaborative Research Database. The database contains structured data from adult (≥18 years) ICU patients from more than 200 hospitals across the United States during 2014 and 2015 and includes demographics, diagnosis, treatments, vital signs, medications, laboratory values, nursing and respiratory therapy notes, and patient outcomes, among others. Hospitals contributing to the data set are from both academic and nonacademic settings and vary in size up to 500 beds. All major ICU types are included (eg, medical, surgical, cardiothoracic, neurologic), and data contributions from each hospital depend on site-specific policies, procedures, and interfaces. Further details on data availability, quality, and patient characteristics are available in published descriptive studies.^14,15 Importantly, this data set also contains quantitative measures of intervention or decision-making authority for the remote team as designated by bedside physicians for individual patients.

Inclusion Criteria

All adult patients with at least 1 assigned intervention category record were included in the analysis. Intervention categories are used to define the amount of oversight or authority given to tele-ICU clinicians by the bedside clinician for each patient. Our data include 3 designations: (1) emergency only (category 1), (2) emergency and best practices (category 2), and (3) full intervention authority (category 3). Best practices may be institution determined or related to illness-specific, documented medical best practices,^16,17 and emergency and full intervention authority may be interpreted differently across institutions.

In addition, implementation of these categories may vary. The remote team overseeing a patient who repeatedly requires intervention due to physiological decompensation may be assigned a higher decision-making authority, allowing the team to address the needs of the patient without consulting the bedside physician first (category 3). However, the remote team overseeing that same patient at a different hospital may be assigned a lower decision-making authority (category 1), such that the tele-ICU clinicians must consult with the bedside team unless an emergency occurs (eg, cardiac arrest).

Patients without any intervention category records and those missing data for the following variables were excluded: Acute Physiology and Chronic Health Evaluation IVa (APACHE) severity score, ICU and hospital mortality, and LOS. Some patients had multiple intervention category records for a single stay, and some remained in the same category and some changed categories during their stay. We included 3 cohorts for analysis corresponding to patients who remained in categories 1, 2, and 3 consistently during their ICU stay. A fourth potential cohort of patients with more than 1 category during their ICU stay was excluded. This mixed group includes patients who moved to a higher or lower category at least once during a single stay and warrants in-depth analysis in a subsequent study. This study was reviewed and approved by the University of Arizona institutional review board.

Statistical Analysis

We developed 3 multivariable logistic regression models to evaluate features within each cohort and their association with outcomes of mortality, levels of remote intervention, and ICU readmission, respectively. The first model evaluated demographics (ie, age and gender), ICU and hospital LOS, ICU readmission, and intervention category relative to hospital mortality as a binary, one-vs-one model. The second model evaluated input features relative to the assigned remote intervention category as a binary, one-vs-any model.¹⁸ Input features included age, gender, severity score, ICU readmission, and either managing or consulting physician type at the bedside. The third model used readmission as a binary dependent variable in a one-vs-one model.

Intervention category in all models was treated as a binary factor to compare the impact of remote intervention on patient outcomes. Emergency and best practices intervention (category 2) and full intervention and interaction (category 3) were combined into a single intervention group to selectively evaluate whether the voluntary decisions made by the remote team were factors influential on patient outcomes. This single group was compared against category 1 (emergency intervention only) as a binary independent variable. Readmissions in the mortality and remote intervention category models were represented as numerical number of ICU visits. In the readmission model, however, a binary readmission factor simply indicating first admissions or readmissions was used (one-vs-one).

Statistical significance and adjusted odds ratios (AORs) are reported for all models. Gender, intervention category, and managing or consulting physician were categorical variables, whereas age, severity score, LOS, and number of ICU visits were continuous variables. Although some statistical tests are appropriate for smaller sample sizes, they do not allow for adequate evaluation of sample sizes greater than 10,000.^19,20 Our approach allows for interpretability of individual features and sufficiently characterizes the decision-making factors in the large patient population well in excess of 10,000 patients. Coefficients comparing characteristics across subgroups illustrate the change in log odds ratio for binary outcomes or the change in log odds ratio with 1-unit change of continuous independent variables.

Variables in all 3 models were selected based on the potential impact they may have on the primary predictor variable of interest. For example, the intervention category assigned to a patient by a physician might be influenced by the severity score at the time of assignment but not by mortality, as the outcome of mortality occurs after the intervention category assignment. Data preprocessing and analyses were performed using Python Language Reference version 2.7.14 (Python Software) and R version 3.4.3 (R Foundation for Statistical Computing).

RESULTS

Summary Statistics

Of the 139,367 patients in the database, a total of 82,049 met the initial inclusion criterion for analysis (ie, had at least 1 intervention category record) (eAppendix Figure 1 [eAppendix available at ajmc.com]). The meannumber of intervention category records per patient was 2.86. Most patients (85.2%) remained in a single category through the entirety of their ICU stay (cohorts 1-3) (Table 1), and 12,127 patients (14.8%) had records in multiple categories and were excluded. Readmissions accounted for 12.5% of the remaining 69,922 on average across all categories. Minimal demographic differences were seen across cohorts. Interestingly, the readmission rate was highest for patients in cohort 3 (full intervention and interaction).

The distribution of primary ICU admission diagnoses across all 3 patient groups was representative of the sample population, with sepsis, heart failure, myocardial infarction, and cerebrovascular stroke among the most common admission diagnoses accounting for more than 21% of admissions in all 3 categories (eAppendix Figure 2).

Logistic Regression Results

In the regression model with mortality as the outcome of interest, all features other than gender were statistically significant (Table 2). There were increased adjusted odds of mortality for higher APACHE score (AOR, 1.04; 95% CI, 1.04-1.05), older age (AOR, 1.01; 95% CI, 1.01-1.02), longer ICU LOS (AOR, 1.07; 95% CI, 1.06-1.07), and readmission to the ICU (AOR, 1.40; 95% CI, 1.34-1.47). Higher intervention category level (AOR, 1.25; 95% CI, 1.17-1.33) also resulted in increased odds of mortality, suggesting that category 3 is used for the sickest of patients, who tend to have higher mortality rates.

In evaluating features related to the level of remote intervention (Table 3), we found that older patients were generally assigned to a lower intervention category (AOR, 0.99; 95% CI, 0.99-0.99). Female sex (AOR, 1.13; 95% CI, 1.09-1.16), increased severity (AOR, 1.01; 95% CI, 1.01-1.01), and ICU readmissions (AOR, 1.05; 95% CI, 1.02-1.08) slightly increased adjusted odds of a patient being assigned to category 2 or 3 rather than category 1 (emergency intervention only). Interestingly, the managing physician type, as opposed to consulting physician type, was also influential, with an AOR of 2.42 (95% CI, 2.32-2.52). If assigned by a consulting physician, the odds were greater of an intervention category level of 2 or 3 rather than category 1.

We found that female sex (AOR, 1.05; 95% CI, 1.01-1.10), higher severity (AOR, 1.01; 95% CI, 1.007-1.008), and higher intervention category (AOR, 1.13; 95% CI, 1.08-1.19) increased the adjusted odds of readmission (Table 4). Higher ICU LOS (AOR, 0.93; 95% CI, 0.93-0.94) slightly decreased odds of readmission. Perhaps patients discharged from the ICU too quickly tend to experience additional ICU visits. Lastly, consulting (as opposed to managing) physician type also decreased the odds of readmission with an AOR of 0.79 (95% CI, 0.75-0.83).

Readmissions and Discharge

As was shown in the intervention model (Table 3), readmissions are more likely to be associated with a higher intervention category, indicating potentially more oversight by the remote team. This leads us to believe that intervention categories, generally, are used to increase oversight for a worse or worsening patient state. We found that patient severity scores upon admission, however, were comparable for patients assigned to a single category, regardless of which category it was (eAppendix Figure 3). Additionally, most patients transitioned from the ICU to the medical-surgical floor; however, cohort 1 had a proportionally higher number of patients discharge to a step-down unit and a lower number of patients discharge to death (Figure 1). Cohorts 2 and 3 had higher proportions of patients who discharged to death as opposed to locations related to an escalation of care due to patient state. This is not to imply that providing care at sites with different costs and staff skill sets is less favorable, although certainly there are underlying decisions as potential confounders behind providing care at various locations. Rather, the spectrum of discharge locations from death to home should be considered.

DISCUSSION

Results from this study provide insight into how intervention category assignments for tele-ICU influence clinical practice and clinical outcomes. Our results show that the degree of intervention authority given to the remote team by the beside physician is, to some extent, a reflection of the trajectory of patient severity of illness (ie, improving or worsening) and physician type (ie, managing or consulting). Although this may not be true for each institution, the tele-ICU broadly was permitted more oversight and treatment authority for readmissions irrespective of severity of illness upon initial admission to the ICU. Alternatively, oversight may be related to external factors such as ICU capacity and resource limitations,²¹ which are not captured in our data.

We observed differences in patient outcomes among the intervention categories and influential factors related to underlying use of tele-ICU. Given that worsening patients were assigned to higher categories (ie, more tele-ICU intervention), differences in mortality and ICU LOS among cohorts may reflect patient severity and illness trajectory rather than a result of tele-ICU intervention throughout an ICU visit as was seen in prior severity-adjusted studies.^5,6 A 2021 systematic review comparing tele-ICU “decision-making authority” vs “tele-consultation” found that tele-ICU implementation with decision-making authority resulted in significant reduction in ICU mortality.²² Patient outcomes of hospital mortality, ICU LOS, and hospital LOS, however, showed minimal advantage with tele-ICU implementation. In addition, tele-ICU functions evolve over time, and changes should be closely monitored to avoid any potential negative impacts on patient outcomes.²³

Our study provides more context to levels of decision-making authority and includes potentially confounding factors related to patient outcomes. However, more granular case-matching and mixed-methods studies are required because severity of illness and subsequent treatment path likely change the trajectory of patient outcomes, although our results suggest that remote oversight by the tele-ICU for those patients requiring continuous monitoring is unlikely to have a negative impact on care processes.

As might be expected, patient age and ICU LOS increase odds of mortality across intervention categories. In addition, the type of physician making the tele-ICU intervention category selection was a factor. Consulting physicians led to increased odds (AOR, 2.42; 95% CI, 2.32-2.52) of patients being assigned to a higher category and decreased the odds (AOR, 0.79; 95% CI, 0.75-0.83) of ICU readmission. It is possible that consulting physicians defer to additional tele-ICU oversight or that consulting physicians are more often involved with more severely ill patients, which also tends to result in higher intervention categories with additional oversight resulting in lower readmission rates.

Alternatively, consulting physicians may not be as invested in the patient as a managing physician and tend to cede control to the tele-ICU to minimize involvement or liability. This is highly institution specific and may depend entirely on prognosis, required treatment, and mode of interprofessional care,²⁴ but it aligns with previous work suggesting that physician-specific factors correlate with patient outcomes.²⁵ Thus, for those institutions in our study that utilized intervention category assignments, the physician type was the most influential factor in the intervention model (Table 3) and the readmission model (Table 4) with AORs of 2.42 (95% CI, 2.32-2.52) and 0.79 (95% CI, 0.75-0.83), respectively.

Overall, patients readmitted to the ICU are assigned their original or higher intervention category. Much work has been done to predict patient readmissions.^26,27 As noted in our readmission model, longer ICU LOS decreased odds (AOR, 0.93; 95% CI, 0.93-0.94) of readmission. Existing readmission prediction models in combination with tele-ICU intervention categories could be used to determine which patients are discharged too quickly and should stay in the ICU.

Sociotechnical Considerations

Although our results did not indicate that higher tele-ICU intervention categories significantly improve patient outcomes, especially among those with higher severity, results still conveyed an optimistic message from a health care technology implementation perspective that the care system sociotechnical balance and care processes were minimally disrupted or affected by tele-ICU uptake. If there were substantial disruptions, patients would likely experience worse outcomes due to communication interruptions resulting in decreased best practice adherence.^28,29 Mixed-methods research from a systems perspective, in combination with this work, is required to specify tele-ICU success mechanisms and best practices for interconnected and collaborative care. This is especially important for preparing health care systems to leverage tele-ICU in response to public health emergencies, such as the COVID-19 pandemic, in which certain clinical needs may be better routed through remote operations.³⁰

To our knowledge, there is not currently a shared, collaborative decision-making framework specific to the tele-ICU beyond high-level implementation models^31-33 and ICU operating guidelines provided by the Leapfrog Group.³⁴ Data and information exchanges between the bedside clinicians and the patient or their guardian are expected and occur regularly,³⁵ but the addition of the tele-ICU changes the dynamics of decision-making and patient monitoring.³⁶ Studying decision-making authority in the context of guideline adherence could provide more evidence of tele-ICU impact on patient outcomes.^37,38 Individual institutions must define and reinforce the level of interaction that the remote team has with both the bedside team and the patient or guardian.³⁹ Most importantly, metadata related to either team and the decisions that are made should be systematically recorded to enable retrospective research and quality improvement studies.

What is not known within our data set is the underlying reason for intervention category assignment. It appears that higher intervention categories are used for sicker patients, and consulting physicians are more likely to assign patients to higher intervention categories. We are unable to validate the reasons for category assignment or any additional communication between bedside and remote teams. Possible reasons for assignment are (1) that a patient is improving, and the managing physician is comfortable releasing decision-making authority to the remote team, (2) that a patient is worsening and requires constant oversight that the managing physician alone cannot provide, or (3) combinations of clinical factors regarding patient status and managing physicians’ practicing tendencies.

Limitations and Future Work

To avoid group-level biases, we did not select group characteristics such as hospital type (teaching or nonteaching) as model inputs.⁴⁰ In addition, we used patients across the United States rather than from a single institution or geographic region. Patients who share group characteristics could otherwise introduce group-level biases and require multilevel regression or other mixed methods to alleviate potential bias.⁴¹ Multilevel regression models could be used in future work at single institutions to evaluate decision-making within the confines of that institution’s protocols and procedures, but they are not necessary given the breadth of our input data.

Future work should include analyzing intervention categories through case matching by diagnosis, further stratified by patient severity, to highlight tele-ICU’s impact on outcomes more granularly. This could be used to identify workflow disruptions related to the tele-ICU within single ICU types (eg, medical-surgical ICUs have a lower mortality post elective surgery) and provide further insight into the direction of change for patients who move to a higher or lower category during a single ICU visit. Regionalization studies and investigations of staffing data and patient census at individual institutions or group of institutions under a single health care system using intervention categories could provide similar insight. Lastly, examining temporality of intervention categories (eg, daytime vs nighttime) and whether patients are improving or worsening could also be insightful.⁴²

CONCLUSIONS

Our results suggest that higher tele-ICU decision-making authority is used for worsening patients and is highly dependent on bedside physician type. Tele-ICU also does not negatively affect the care processes collaboratively carried out by both remote and bedside teams at the expense of patient outcomes. Although there is room for investigation around the impact of tele-ICU on patient outcomes, evaluation of intervention category assignments in the tele-ICU is a step toward better understanding workflow success mechanisms and may guide design of future mixed-methods studies.

Author Affiliations: Department of Systems and Industrial Engineering, College of Engineering, The University of Arizona (PE, TZ, VS), Tucson, AZ; Department of Emergency Medicine and Division of Pulmonary, Allergy, Critical Care, and Sleep, Department of Medicine, The University of Arizona College of Medicine (JM), Tucson, AZ.

Source of Funding: This work was supported in part by the National Science Foundation under grant #1838745 and the National Heart, Lung, and Blood Institute of the National Institutes of Health under award number 5T32HL007955.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (PE, TZ, JM, VS); acquisition of data (PE); analysis and interpretation of data (PE, TZ, VS); drafting of the manuscript (PE, JM); critical revision of the manuscript for important intellectual content (PE, TZ, JM, VS); statistical analysis (PE); obtaining funding (VS); administrative, technical, or logistic support (PE, VS); and supervision (VS).

Address Correspondence to: Patrick Essay, PhD, Department of Systems and Industrial Engineering, College of Engineering, The University of Arizona, 1127 E James E. Rogers Way, Tucson, AZ 85721-0020. Email: pessay@email.arizona.edu.

REFERENCES

1. Chen J, Sun D, Yang W, et al. Clinical and economic outcomes of telemedicine programs in the intensive care unit: a systematic review and meta-analysis. J Intensive Care Med. 2018;33(7):383-393. doi:10.1177/0885066617726942

2. Deisz R, Rademacher S, Gilger K, et al. Additional telemedicine rounds as a successful performance-improvement strategy for sepsis management: observational multicenter study. J Med Internet Res. 2019;21(1):e11161. doi:10.2196/11161

3. Lilly CM, Cody S, Zhao H, et al; University of Massachusetts Memorial Critical Care Operations Group. Hospital mortality, length of stay, and preventable complications among critically ill patients before and after tele-ICU reengineering of critical care processes. JAMA. 2011;305(21):2175-2183. doi:10.1001/jama.2011.697

4. van der Voort PHJ, de Metz J, Wester JPJ, et al. Telemedicine in a Dutch intensive care unit: a descriptive study of the first results. J Telemed Telecare. 2016;22(3):141-147. doi:10.1177/1357633X15590751

5. Thomas EJ, Lucke JF, Wueste L, Weavind L, Patel B. Association of telemedicine for remote monitoring of intensive care patients with mortality, complications, and length of stay. JAMA. 2009;302(24):2671-2678. doi:10.1001/jama.2009.1902

6. Kohl BA, Fortino-Mullen M, Praestgaard A, Hanson CW, DiMartino J, Ochroch EA. The effect of ICU telemedicine on mortality and length of stay. J Telemed Telecare. 2012;18(5):282-286. doi:10.1258/jtt.2012.120208

7. Machado SM, Wilson EH, Elliott JO, Jordan K. Impact of a telemedicine eICU cart on sepsis management in a community hospital emergency department. J Telemed Telecare. 2018;24(3):202-208. doi:10.1177/1357633X17691862

8. Morrison JL, Cai Q, Davis N, et al. Clinical and economic outcomes of the electronic intensive care unit: results from two community hospitals. Crit Care Med. 2010;38(1):2-8. doi:10.1097/CCM.0b013e3181b78fa8

9. Nassar BS, Vaughan-Sarrazin MS, Jiang L, Reisinger HS, Bonello R, Cram P. Impact of an intensive care unit telemedicine program on patient outcomes in an integrated health care system. JAMA Intern Med. 2014;174(7):1160-1167. doi:10.1001/jamainternmed.2014.1503

10. Sadaka F, Palagiri A, Trottier S, et al. Telemedicine intervention improves ICU outcomes. Crit Care Res Pract. 2013;2013:456389. doi:10.1155/2013/456389

11. Kahn JM, Hill NS, Lilly CM, et al. The research agenda in ICU telemedicine: a statement from the Critical Care Societies Collaborative. Chest. 2011;140(1):230-238. doi:10.1378/chest.11-0610

12. Kahn JM, Le TQ, Barnato AE, et al. ICU telemedicine and critical care mortality: a national effectiveness study. Med Care. 2016;54(3):319-325. doi:10.1097/MLR.0000000000000485

13. Berwick D, Fox DM. Evaluating the quality of medical care: Donabedian’s classic article 50 years later. Milbank Q. 2016;94(2):237-241. doi:10.1111/1468-0009.12189

14. Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci Data. 2018;5:180178. doi:10.1038/sdata.2018.178

15. Essay P, Shahin TB, Balkan B, Mosier J, Subbian V. The connected intensive care unit patient: exploratory analyses and cohort discovery from a critical care telemedicine database. JMIR Med Inform. 2019;7(1):e13006. doi:10.2196/13006

16. Alhazzani W, Møller MH, Arabi YM, et al. Surviving Sepsis Campaign: guidelines on the management of critically ill adults with coronavirus disease 2019 (COVID-19). Crit Care Med. 2020;48(6):E440-E469. doi:10.1097/CCM.0000000000004363

17. Lang JK, Paykel MS, Haines KJ, Hodgson CL. Clinical practice guidelines for early mobilization in the ICU: a systematic review. Crit Care Med. 2020;48(11):e1121-e1128. doi:10.1097/CCM.0000000000004574

18. Whitaker T, Beranger B, Sisson SA. Logistic regression models for aggregated data. J Comput Graph Stat. 2021;30(4):1049-1067. doi:10.1080/10618600.2021.1895816

19. Lin M, Lucas HC, Shmueli G. Too big to fail: large samples and the p-value problem. Inf Syst Res. 2013;24(4):906-917. doi:10.1287/isre.2013.0480

20. Kaplan RM, Chambers DA, Glasgow RE. Big data and large sample size: a cautionary note on the potential for bias. Clin Transl Sci. 2014;7(4):342-346. doi:10.1111/cts.12178

21. Collins TA, Robertson MP, Sicoutris CP, et al. Telemedicine coverage for post-operative ICU patients. J Telemed Telecare. 2017;23(2):360-364. doi:10.1177/1357633X16631846

22. Kalvelage C, Rademacher S, Dohmen S, Marx G, Benstoem C. Decision-making authority during tele-ICU care reduces mortality and length of stay – a systematic review and meta-analysis. Crit Care Med. 2021;49(7):1169-1181. doi:10.1097/CCM.0000000000004943

23. Anders SH, Woods DD, Schweikhart S, Ebright P, Patterson E. The effects of health information technology change over time: a study of tele-ICU functions. Appl Clin Inform. 2012;3(2):239-247. doi:10.4338/ACI-2011-12-RA-0073

24. Donovan AL, Matthew Aldrich J, Gross AK, et al; University of California, San Francisco Critical Care Innovations Group. Interprofessional care and teamwork in the ICU. Crit Care Med. 2018;46(6):980-990. doi:10.1097/CCM.0000000000003067

25. Kerlin MP, Epstein A, Kahn JM, et al. Physician-level variation in outcomes of mechanically ventilated patients. Ann Am Thorac Soc. 2018;15(3):371-379. doi:10.1513/AnnalsATS.201711-867OC

26. Rojas JC, Carey KA, Edelson DP, Venable LR, Howell MD, Churpek MM. Predicting intensive care unit readmission with machine learning using electronic health record data. Ann Am Thorac Soc. 2018;15(7):846-853. doi:10.1513/AnnalsATS.201710-787OC

27. Xue Y, Klabjan D, Luo Y. Predicting ICU readmission using grouped physiological and medication trends. Artif Intell Med. 2019;95:27-37. doi:10.1016/j.artmed.2018.08.004

28. Hassan E. Tele-ICU and patient safety considerations. Crit Care Nurs Q. 2018;41(1):47-59. doi:10.1097/CNQ.0000000000000185

29. Moeckli J, Cram P, Cunningham C, Reisinger HS. Staff acceptance of a telemedicine intensive care unit program: a qualitative study. J Crit Care. 2013;28(6):890-901. doi:10.1016/j.jcrc.2013.05.008

30. Grange ES, Neil EJ, Stoffel M, et al. Responding to COVID-19: the UW Medicine information technology services experience. Appl Clin Inform. 2020;11(2):265-275. doi:10.1055/S-0040-1709715

31. Carayon P, Wetterneck TB, Rivera-Rodriguez AJ, et al. Human factors systems approach to healthcare quality and patient safety. Appl Ergon. 2014;45(1):14-25. doi:10.1016/j.apergo.2013.04.023

32. Ramnath VR, Khazeni N. Centralized monitoring and virtual consultant models of tele-ICU care: a side-by-side review. Telemed J E Health. 2014;20(10):962-971. doi:10.1089/tmj.2014.0024

33. Gutsche JT, Raiten JM. Staffing models for the ICU: open, closed, MD, NP, or telemedicine? Curr Anesthesiol Rep. 2013;3(2):65-72. doi:10.1007/s40140-013-0010-0

34. Manthous CA. Leapfrog and critical care: evidence- and reality-based intensive care for the 21st century. Am J Med. 2004;116(3):188-193. doi:10.1016/j.amjmed.2003.08.032

35. Adams AMN, Mannix T, Harrington A. Nurses’ communication with families in the intensive care unit – a literature review. Nurs Crit Care. 2017;22(2):70-80. doi:10.1111/nicc.12141

36. Hoonakker PLT, Carayon P. Work system barriers and strategies reported by tele-intensive care unit nurses: a case study. Crit Care Nurs Clin North Am. 2018;30(2):259-271. doi:10.1016/j.cnc.2018.02.008

37. Angus DC, Shorr AF, White A, Dremsizov TT, Schmitz RJ, Kelley MA. Critical care delivery in the United States: distribution of services and compliance with Leapfrog recommendations. Crit Care Med. 2006;34(4):1016-1024. doi:10.1097/01.CCM.0000206105.05626.15

38. Pronovost PJ, Thompson DA, Holzmueller CG, Dorman T, Rudolph BA. Evaluating the impact of the Leapfrog Group’s standard for intensive care unit physician staffing. Semin Anesth Perioper Med Pain. 2005;24(1):50-58. doi:10.1053/j.sane.2004.11.001

39. Udeh C, Udeh B, Rahman N, Canfield C, Campbell J, Hata JS. Telemedicine/virtual ICU: where are we and where are we going? Methodist Debakey Cardiovasc J. 2018;14(2):126-133. doi:10.14797/mdcj-14-2-126

40. Parker CE, O’Dwyer LM, Irwin CW. The correlates of academic performance for English language learner students in a New England district. National Center for Education Evaluation and Regional Assistance. August 2014. Accessed July 5, 2021. https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/REL_2014020.pdf

41. Tabachnick BG, Fidell LS. Using Multivariate Statistics. 5th ed. Pearson/Allyn & Bacon; 2007.

42. Griffiths CL, Kosmisky DE, Everhart SS. Characterization of dayshift tele-ICU pharmacist activities. J Telemed Telecare. 2022;28(1):77-80. doi:10.1177/1357633X20913712