A Machine Learning Technology for Addressing Medication-Related Risk in Older, Multimorbid Patients

ABSTRACT

Objectives: To evaluate the FeelBetter machine learning system’s ability to accurately identify older patients with multimorbidity at Brigham and Women’s Hospital at highest risk of medication-associated emergency department (ED) visits and hospitalizations, and to assess the system’s ability to provide accurate medication recommendations for these patients.

Study Design: Retrospective cohort study.

Methods: The system uses medications, demographics, diagnoses, laboratory results, health care utilization patterns, and costs to stratify patients’ risk of ED visits and hospitalizations. Patients were assigned 1 of 22 risk levels based on their system-generated risk percentile of either ED visits or hospitalizations. Logistic regression models were used to estimate the odds of ED visits and hospitalizations associated with each successive risk level compared with the 45th to 50th percentiles. After stratification, 100 high-risk (95th-100th percentiles) and 100 medium-risk (45th-55th percentiles) patients were randomly selected for generation of medication recommendations. Two clinical pharmacists reviewed the system-generated medication recommendations for these patients.

Results: Logistic regression models predicting 3-month utilization showed that compared with the 45th to 50th percentiles, patients in the top 1% risk percentile had ORs of 7.9 and 17.3 for ED visits and hospitalizations, respectively. The first 5 high-priority medications on each patient’s medication list were associated with a mean (SD) of 6.65 (4.09) warnings. Of 1290 warnings reviewed, 1151 (89.2%) were assessed as correct.

Conclusions: The FeelBetter system effectively stratifies older patients with multimorbidity at risk of ED use and hospitalizations. Medication recommendations provided by the system are largely accurate and can potentially be beneficial for patient care.

Am J Manag Care. 2024;30(8):e233-e239. https://doi.org/10.37765/ajmc.2024.89592

Takeaway Points

• Adverse drug events contribute to 10% to 30% of hospitalizations in older patients, who are at increased risk of medication-related harm and subsequent health care utilization.
• A novel machine learning system focused on patients with multiple chronic conditions and complex medication regimens effectively risk stratifies older patients with multimorbidity at highest risk for emergency department use and hospitalizations.
• Medication recommendations generated by the system for medium- and high-risk patients are appropriate and potentially beneficial for patient care.

Polypharmacy is common in older adults, and nearly 50% of individuals older than 65 years take 1 or more medications that are not necessary.¹ Medication-related harm and subsequent health care utilization are more common in older adults, with drug-related adverse events contributing to a substantial proportion of hospitalizations in older patients.² Medication-related harm is of particular concern in patients with multimorbidity³ and during care transitions.^4,5

Limiting drug-related harm is also critical for reducing avoidable health care utilization and spending, particularly for organizations with a focus on accountable or value-based care. Suboptimal medication regimens are estimated to cost patients and payers more than $500 billion annually in unnecessary health care expenses.^6,7 Thus, identifying specific subsets of older adults who may benefit from medication-related interventions and tailoring medication-related recommendations and interventions to these patients is of significant interest to health care systems focused on value-based care.

Multiple approaches previously explored to minimize polypharmacy and medication-related harm have not reached their full potential. For example, pharmacist-driven medication reviews and management interventions have been shown to effectively reduce health care utilization.^8-11 However, these approaches are limited in their potential impact because they are time and labor intensive, thus it is difficult to scale them to meet the needs of the broad populations who could potentially benefit from them. Current medication alert systems embedded in the electronic health record (EHR) have multiple limitations as well. First, they often focus on interactions between single medications and alert providers at the point of ordering. Thus, they do not enable a holistic view of opportunities for optimization of patients’ entire medication regimens. Additionally, they are not configured to optimize medication regimens for the highest-risk patients or older adults specifically.^12,13 Finally, current medication alert systems may be repetitive and may not adequately differentiate between severity of alerts,¹⁴ and their effectiveness is limited by the alert fatigue of the clinicians who use them.¹⁵

Hence, there is a continued need for innovation in identifying patients who could most benefit from medication regimen optimization and developing effective interventions to facilitate this optimization. FeelBetter’s machine learning technology has been designed to risk stratify patients with multiple chronic conditions and complex medication regimens and provide medication management recommendations to health care personnel with the goal of preventing pharmacotherapy-associated avoidable health care adverse events. FeelBetter uses a proprietary rule-based algorithm informed by patient-specific demographic, clinical, pharmacologic, and health care utilization data as well as general clinical guidelines and pharmacologic data to identify which interventions and recommendations are relevant for a specific patient. The recommendations provided by the FeelBetter system present the reference for the recommendation and the specific triggers that were used to identify a specific patient. The FeelBetter system also uses the machine learning insights to highlight recommendations that should be prioritized to impact patients’ health outcomes and minimize preventable hospitalizations. In this retrospective study, we evaluated the ability of the FeelBetter technology (hereafter referred to as “the system” or “the algorithm”) to stratify patients by predicted health care utilization. We additionally sought to characterize pharmacist agreement with high-priority medication management recommendations made by the system for patients identified as being at high and medium risk of unnecessary health care utilization.

METHODS

FeelBetter Risk-Stratification Algorithm

In general, FeelBetter uses logistic regression, decision trees, random forests, and graph neural networks to develop its risk-stratification algorithms. These algorithms are executed using Python (PyTorch). The machine learning algorithm chosen and implemented in the research was based on graph neural networks. All patients in the final study cohort were included in algorithm training and ultimately considered for risk stratification by the algorithm. The following variables were identified and summarized for the full cohort and for the patients eligible for risk stratification: sex, age, ethnicity, race, primary insurance, number of problems on the problem list, number of medications, and the modified Charlson Comorbidity Index (CCI) score.¹⁶ Chi-square tests and Kruskal-Wallis tests were used to compare demographic variables between included and excluded patients.

Risk Stratification of Patient Population

The patient population considered for risk stratification included patients of Brigham and Women’s Hospital (BWH) in Boston, Massachusetts, who, on July 1, 2016, were 65 years and older; had at least 1 chronic condition and at least 3 prescribed medications; had at least 1 office visit within the Mass General Brigham health care system between July 1, 2016, and June 30, 2019; and had a designated primary care physician at BWH. Eligible patients were excluded from risk stratification if they met any of the following criteria during the study period (7/1/2016-6/30/2019): were receiving hemodialysis, were oncology patients receiving chemotherapy treatment, had no laboratory results available in the BWH EHR, were inpatients on the last day of the study period, died during the study period, or were reported as deceased with no death date. Figure 1 depicts the flow of patients considered in the study, including the number of patients removed from the cohort due to each exclusion criterion.

Subsequently, the algorithm stratified patients’ risk of emergency department (ED) visits and hospitalizations in the 3 months following July 1, 2019. The algorithm considered the following data variables in its risk stratification: demographics; social history (smoking, alcohol, and recreational drug use); insurance type; problem list; International Statistical Classification of Diseases, Tenth Revision codes from encounters and billing, patient measurements (blood pressure, height, weight, heart rate), laboratory results, procedures, medication orders, allergies, ED visits, observation visits, and hospital admissions; and estimated costs based on average Medicare payments by length of stay. Although the FeelBetter algorithm typically also uses medication fill data, this information was unavailable for the cohort being studied. Patients were assigned 1 of 20 risk levels based on risk percentiles and were then categorized into 5 risk groups: low (0 to < 20th percentiles), low-medium (20th to < 40th percentiles), medium (40th to < 60th percentiles), medium-high (60th to < 80th percentiles), and high (80th-100th percentiles).

Characterizing Risk Strata

Once the risk groups were identified, demographic variables (sex, age, ethnicity, race, primary insurance), as well as the number of problem list problems, number of medications, and modified CCI score as of June 30, 2019, were summarized for each risk group. Additionally, ED visits, including observation visits, and unplanned inpatient admissions up to 1, 3, and 6 months post algorithm prediction (July 1, 2019) for the entire cohort were extracted. Mean (SD) visits for each type of health care utilization by risk group and time frame were quantified, and the percentage of patients with any utilization at each time point was calculated. Chi-square tests and Kruskal-Wallis tests were used to compare demographic and health care utilization variables between the different risk groups.

An interval variable representing the 20 risk levels was used in Poisson regression models to quantify increases in the number of ED visits or hospitalizations uniquely predicted by the system’s risk levels over other covariates commonly used to predict health care utilization (age, sex, race, ethnicity, medication count, and CCI score). Collinearity testing among all the covariates in the models was performed. Additionally, to examine the top 5% of patients at greatest risk in more detail, we expanded the 20 risk levels to
22 to include separate groups for the 95th to less than 98th, 98th to less than 99th, and 99th to 100th percentiles. Logistic regression models were run for 1, 3, and 6 months post prediction for both the inpatient and ED risk levels to evaluate the ORs associated with each risk level compared with the 45th to 50th percentile level. A Bonferroni correction was applied to counteract the multiple comparisons being performed, and a P value less than .004 was required for a result to be considered statistically significant.

Selection of Patients for Generation of Medication Recommendations

One hundred high-risk and 100 medium-risk patients were randomly selected from the 95th to 100th and 45th to 55th risk percentiles for review. Using patient-specific data from the EHR as described earlier, the system then generated medication management recommendations for each of the randomly selected high- and medium-risk patients. Two experienced clinical research pharmacists (D.L.S. and M.G.A.) from BWH independently reviewed the high-priority medication warnings and recommendations provided via weblinks by the system for these 200 patients. Both pharmacists have practiced for more than 40 years in hospital pharmacy and research settings and have experience studying health information technology to improve medication safety.

Pharmacist Assessment of Warnings and Recommendations

Pharmacist reviewers assessed the appropriateness of the high-priority medication warnings and recommendations for potential interventions to consider based on the clinical and medication profiles of patients as of June 30, 2019. All medication warnings were generated retrospectively using only data that would have been available as of June 30, 2019. Pharmacist assessment of the medication warnings was based on whether the warnings were appropriate for the patients at that time. Reviewers had access to patient demographics, allergies, problem lists, medication lists, laboratory results, vital signs, inpatient and outpatient encounters, and diagnoses on the system’s interface. The system also had the capability to display multiple clinical variables for a patient on a time line (eg, when a particular medication was started, relevant vital signs and laboratory values, when a patient was hospitalized, date a diagnosis code was listed) to help with clinical assessment (eAppendix A, part A [eAppendices available at ajmc.com]). For each of the 200 selected patients, study pharmacists reviewed warnings and recommended interventions for up to the first 5 medications with high-priority interventions. A sample listing of warnings is shown in eAppendix A, part B, and a sample recommended intervention is shown in eAppendix A, part C. Because review of the warnings took place several years after June 30, 2019, no interventions were undertaken based on pharmacist reviews.

The following characteristics of each warning were assessed: (1) Was the warning correct? (2) Was the problem type accurate? (3) Does the warning help the clinician make decisions to optimize medication therapy? (4) Is at least 1 intervention correct from the group of possible interventions that were recommended for consideration? (5) Are any interventions in the group not appropriate? (6) Is there an intervention that should be added? Reviewers used clinical practice guidelines and drug information resources to confirm and support their assessments. Disagreements on assessments were discussed and a physician reviewer (L.S.R.) was consulted if consensus was not achieved. If the pharmacist thought the warning was not correct, the other questions about recommended interventions for consideration were marked as not applicable and were not included in the denominator for calculation of percent agreement with the system’s recommendations.

Medication Recommendation Analyses

We described the sex, age, ethnicity, race, and insurance distribution of the overall cohort of high- and medium-risk patients for whom medication recommendations were made, as well as for the high- and medium-risk cohorts separately. The mean (SD) number of medications (among the possible first 5 medications reviewed for each patient) that triggered medication warnings and the mean (SD) number of high-priority medication warnings (for the up to 5 medications reviewed) were quantified for the overall cohort of high- and medium-risk patients and each of the high- and medium-risk cohorts separately. Additionally, we characterized the domains, problem types, problem categories, and therapeutic classes associated with the warnings.

RESULTS

Patient Sample and Risk Stratification

The final study cohort considered for risk stratification comprised 108,817 patients. The study cohort consisted of 59.2% women. The age distribution of the cohort at the time of stratification was as follows: 65 to 69 years (10.1%), 70 to 74 years (33.0%), 75 to 79 years (24.9%), 80 to 84 years (16.0%), and 85 years and older (16.0%). Most participants (86.9%) identified as having non-Hispanic ethnicity, and 88.3% identified as White or Caucasian. Medicare and commercial insurance were the most common primary insurance types in the cohort (74.0% and 23.5%, respectively). Cohort participants had a median (IQR) of 14 (9-20) EHR problem list problems and a median (IQR) of 8 (4-12) medications on their medication list. The median (IQR) CCI score was 0 (0-2). Although χ² and nonparametric tests showed statistically significant differences (P < .0001) when comparing the overall cohort patients with patients included, numerical percentage differences in the distribution of characteristics between the 2 groups were minimal (Table 1).

The demographic characteristics of patients in all risk groups for ED visits and hospitalizations are displayed in eAppendices B and C. Each risk group had steadily increased ED use and hospitalizations at 3 months post prediction. This trend was magnified when analyzing patients in the top 5% risk of ED visits and hospitalizations (Table 2). All Poisson models were statistically significant, and the risk-level variable was statistically significant at the P < .0001 level in all models as well. When assessing for collinearity of the covariates in Poisson models, variance inflation factors were low (all values were < 2, and most were < 1.5), suggesting minimal collinearity. In the model predicting 3-month utilization, each 1 of 20 increase in the patient’s system-assigned risk level predicted a 14.3% increase in ED visits. In the hospitalization model for the same time frame, each 1 of 20 increase in the risk level was associated with an 18.4% increase in hospitalizations.

In the logistic regression models, the variable representing the 22 risk percentiles with the middle 45th to 50th risk percentiles as the reference group was statistically significant at the P < .0001 level in both the ED and hospitalization models. However, when comparing each individual percentile grouping with the 45th to 50th risk percentiles, 8 risk levels closest to the 45th to 50th percentiles did not reach statistical significance in the ED model and 6 did not in the hospitalization model. The top 1% of patients at risk for ED visits at 3 months post prediction had an OR of 7.9 (95% CI, 6.3-10.1) for an ED visit compared with those in the 45th to 50th percentiles (eAppendix D). The OR for unplanned hospitalizations among the top 1% of patients at 3 months post prediction was 17.3 (95% CI, 13.0-22.9) vs those in the 45th to 50th percentiles (Figure 2).

Assessment of Medication Warnings

The demographics of the 200 patients randomly chosen for assessment of medication warnings can be found in eAppendix E. Up to the first 5 medications on each medium- and high-risk patient’s medication list that included high-priority interventions were associated with a mean (SD) of 6.7 (4.1) medication warnings. High-risk patients had more warnings than medium-risk patients (9.1 [3.9] vs 4.0 [2.3]). Of 1290 warnings reviewed, 1151 (89.2%) were assessed as correct and 139 (10.8%) as incorrect. Reviewers agreed that the problem type identified by the warning was correct in 99.3% of cases, that the warning helped the clinician in making decisions to optimize medication therapy in 97.3% of cases, and that suggested interventions were correct in 98.1% of cases. An additional intervention was recommended by reviewers in 2.8% of cases (Table 3 [A]). The pharmacists disagreed on 1 warning, so the physician was brought in to adjudicate. In addition, 6 warnings that the pharmacists were unsure of required input from the physician to help make a final decision.

Of the 1290 high-priority medication warnings reviewed, 15.0% were for cardiovascular medications, 11.1% were for psychotherapeutics, 10.5% were for antiarthritics, 7.8% were for diuretics, and 7.2% were for cardiac medications (eAppendix F). The most common problem-type warnings reviewed were for “undesirable effect” (38.8%), “medication interaction” (21.0%), and “dosing inappropriate” (8.8%). A total of 65.2% of the warnings were under the problem category “adverse medication event,” followed by “dosage inappropriate” (12.8%).

Dosing-related warnings were most represented among the 10.8% of warnings considered incorrect. The most common problem type and category for these warnings were dosage inappropriate (31.7%), followed by “no medication indication—dosage inappropriate” (17.3%) (Table 3 [B]).Antiasthmatic medications accounted for 11 of the 44 (25.0%) dosage inappropriate warnings the reviewers did not agree with, followed by diuretics (8 of 44; 18.2%), electrolytes (5 of 44; 11.4%), and cardiovascular medications (5 of 44; 11.4%). For no medication indication—dosage inappropriate, 10 of 24 (41.7%) were for gastrointestinal medications.

DISCUSSION

In this study of the FeelBetter system’s risk-stratification capabilities and medication management recommendations, the technology successfully stratified patients into risk groupings with distinct demographic and utilization characteristics. Patients in each successive risk grouping had more ED visits and unplanned hospitalizations at 1, 3, and 6 months after the algorithm’s prediction. The system’s risk level variable was significantly and independently associated with risk of both hospitalization and ED visits, even when adjusting for other covariates traditionally used to predict health care utilization. These findings demonstrate that the FeelBetter system can successfully incorporate a wide variety of data from the EHR to accurately stratify the predicted utilization of patients 65 years and older with multiple comorbidities and complex medication regimens. This capability can help prioritization of care management efforts for medically complex, older patients.

Accurate and actionable risk stratification is a fundamental tenet of population health management. Traditional risk prediction approaches have used information about patients’ comorbidities and past utilization to stratify patients for population health interventions, both for general older adult populations^17,18 and for populations with specific diseases.^19,20 More recent literature has described novel methods of risk prediction that incorporate information about patients’ medication management patterns. For example, Kharrazi et al described how consideration of medication adherence indices modestly improved the performance of EHR-derived models for predicting utilization in patients younger than 65 years.²¹ Chang et al described how incorporating medication fill rates into models predicting medical costs improved the performance of all models.²² Building on these studies, our findings demonstrate the additional value of integrating information about polypharmacy and medication-related attributes into risk-stratification methodologies to facilitate incremental accuracy in predicting hospitalizations and ED visits for older adults.

In this study, we additionally demonstrate that medication warnings and recommendations provided by the system were largely accurate and potentially beneficial for patient care. Use of clinical decision support in the EHR has been linked to improved quality of care.²³ However, as EHRs and associated clinical decision support functionality have become more widespread, there is a need to streamline and prioritize the alerts shown to members of the care team to avoid alert fatigue.²⁴ The warnings and potential interventions that were presented by the FeelBetter system in this study were prioritized for high-risk patients who were thought to most likely benefit and additionally prioritized in terms of clinical importance for a particular patient. Given the high prevalence of polypharmacy in older adults²⁵ and the continued elevated rates of medication-related errors and adverse drug events in the postdischarge period⁴ and as a cause of ED visits, technical solutions that help clinicians prioritize opportunities for enhanced medication management can be valuable for enhancing care delivery outcomes for older adults.

Limitations and Strengths

This study has several limitations. First, patients considered for risk stratification and whose medication management recommendations were reviewed derived from a single academic medical center whose patient population may be more complex than the average older patient cared for across health systems. In addition, reviewers did not review high-priority interventions for all medications or any lower-priority interventions provided by the system. Nevertheless, they did review 1290 warnings with high agreement between reviewers. Although the patient-specific alerts presented are prioritized in terms of clinical importance and thus have potential to assist in prioritizing provider workload, the effect of the system on alert fatigue was not measured in this study. Pharmacists evaluated the warnings and potential interventions in this study, and it is unclear whether providers’ approaches to the recommendations would directly align with those of pharmacists. Finally, because there was no comparison group in this study, it is unknown whether this system improves outcomes compared with existing clinical decision support systems. Balancing these limitations are multiple strengths. The algorithm described in this study used information from an EHR (Epic) that is commonly used across the US, particularly in large health care systems. In addition, each warning was reviewed by 2 experienced clinical pharmacists who had good agreement in the adjudication process and a physician who provided support for any questions. Therefore, their assessments should be unbiased and informative to those practicing in similar health care systems.

CONCLUSIONS

This study demonstrated that the FeelBetter machine learning system accurately stratifies older adults taking multiple medications and delivers appropriate medication recommendations that, if utilized as part of clinical care, could potentially facilitate optimization for medication regimens for older patients with multimorbidity. Future directions include updating the medication recommendation interface with feedback derived from the present study and evaluating the ability of the system to prospectively impact patient outcomes and cost-of-care trajectories.

Author Affiliations: Information Systems, Mass General Brigham (DLS, MGA, MF, CI, AM, FC, JF, LAV), Boston, MA; Division of General Internal Medicine, Brigham and Women’s Hospital (DLS, MGA, MF, CI, AM, FC, JF, LAV, LSR), Boston, MA; Harvard Medical School (LSR), Boston, MA.

Source of Funding: FeelBetter Inc. The sponsor had no role in collection of data, its analysis or interpretation, or the right to approve or disapprove publication of the finished manuscript.

Author Disclosures: Ms Seger, Dr Amato, Ms Iannaccone, Ms Mugal, Ms Volk, and Dr Rotenstein report receiving partial salary via grant support from FeelBetter for conducting this study. Ms Frits is employed by Brigham and Women's Hospital, which received funding from FeelBetter for this study. Mr Chang and Ms Fiskio report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (DLS, MGA, MF, CI, LAV, LSR); acquisition of data (DLS, MF, CI, AM, FC, JF, LAV, LSR); analysis and interpretation of data (DLS, MGA, MF, FC, JF, LAV, LSR); drafting of the manuscript (DLS, MF, CI, AM, LAV, LSR); critical revision of the manuscript for important intellectual content (MGA, MF, CI, LAV, LSR); statistical analysis (MF); provision of patients or study materials (AM); obtaining funding (LAV); administrative, technical, or logistic support (AM, FC, JF, LAV, LSR); and supervision (CI, LAV, LSR).

Address Correspondence to: Lisa S. Rotenstein, MD, MBA, Brigham and Women’s Hospital, 75 Francis St, Boston, MA 02215. Email: lrotenstein@bwh.harvard.edu.

REFERENCES

1. Maher RL, Hanlon J, Hajjar ER. Clinical consequences of polypharmacy in elderly. Expert Opin Drug Saf. 2014;13(1):57-65. doi:10.1517/14740338.2013.827660

2. Budnitz DS, Lovegrove MC, Shehab N, Richards CL. Emergency hospitalizations for adverse drug events in older Americans. N Engl J Med. 2011;365(21):2002-2012. doi:10.1056/NEJMsa1103053

3. Khezrian M, McNeil CJ, Murray AD, Myint PK. An overview of prevalence, determinants and health outcomes of polypharmacy. Ther Adv Drug Saf. 2020;11:2042098620933741. doi:10.1177/2042098620933741

4. Alqenae FA, Steinke D, Keers RN. Prevalence and nature of medication errors and medication-related harm following discharge from hospital to community settings: a systematic review. Drug Saf. 2020;43(6):517-537. doi:10.1007/s40264-020-00918-3

5. Parekh N, Ali K, Stevenson JM, et al; PRIME Study Group. Incidence and cost of medication harm in older adults following hospital discharge: a multicentre prospective study in the UK. Br J Clin Pharmacol. 2018;84(8):1789-1797. doi:10.1111/bcp.13613

6. Watanabe JH, McInnis T, Hirsch JD. Cost of prescription drug-related morbidity and mortality.
Ann Pharmacother. 2018;52(9):829-837. doi:10.1177/1060028018765159

7. El Morabet N, Uitvlugt EB, van den Bemt BJF, van den Bemt PMLA, Janssen MJA, Karapinar-Çarkit F. Prevalence and preventability of drug-related hospital readmissions: a systematic review. J Am Geriatr Soc. 2018;66(3):602-608. doi:10.1111/jgs.15244

8. Pellegrin KL, Krenk L, Oakes SJ, et al. Reductions in medication-related hospitalizations in older adults with medication management by hospital and community pharmacists: a quasi-experimental study. J Am Geriatr Soc. 2017;65(1):212-219. doi:10.1111/jgs.14518

9. Sellors C, Dalby DM, Howard M, Kaczorowski J, Sellors J. A pharmacist consultation service in community-based family practices: a randomized, controlled trial in seniors. J Pharm Technol. 2001;17(6):264-269.

10. Altavela JL, Jones MK, Ritter M. A prospective trial of a clinical pharmacy intervention in a primary care practice in a capitated payment system. J Manag Care Pharm. 2008;14(9):831-843. doi:10.18553/jmcp.2008.14.9.831

11. Moczygemba LR, Alshehri AM, Harlow LD III, et al. Comprehensive health management pharmacist-delivered model: impact on healthcare utilization and costs. Am J Manag Care. 2019;25(11):554-560.

12. Gray SL, Hart LA, Perera S, Semla TP, Schmader KE, Hanlon JT. Meta-analysis of interventions to reduce adverse drug reactions in older adults. J Am Geriatr Soc. 2018;66(2):282-288. doi:10.1111/jgs.15195

13. Marasinghe KM. Computerised clinical decision support systems to improve medication safety in long-term care homes: a systematic review. BMJ Open. 2015;5(5):e006539. doi:10.1136/bmjopen-2014-006539

14. Wright A, Aaron S, Seger DL, Samal L, Schiff GD, Bates DW. Reduced effectiveness of interruptive drug-drug interaction alerts after conversion to a commercial electronic health record. J Gen Intern Med. 2018;33(11):1868-1876. doi:10.1007/s11606-018-4415-9

15. Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R; HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017;17(1):36. doi:10.1186/s12911-017-0430-8

16. Glasheen WP, Cordier T, Gumpina R, Haugh G, Davis J, Renda A. Charlson Comorbidity Index: ICD-9 update and ICD-10 translation. Am Health Drug Benefits. 2019;12(4):188-197.

17. Inouye SK, Zhang Y, Jones RN, et al. Risk factors for hospitalization among community-dwelling primary care older patients: development and validation of a predictive model. Med Care. 2008;46(7):726-731. doi:10.1097/MLR.0b013e3181649426

18. Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a Hospital Frailty Risk Score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet. 2018;391(10132):1775-1782. doi:10.1016/S0140-6736(18)30668-8

19. Couchoud CG, Beuscart JB, Aldigier JC, Brunet PJ, Moranne OP; REIN Registry. Development of a risk stratification algorithm to improve patient-centered care and decision making for incident elderly patients with end-stage renal disease. Kidney Int. 2015;88(5):1178-1186. doi:10.1038/ki.2015.245

20. Desai RJ, Wang SV, Vaduganathan M, Evers T, Schneeweiss S. Comparison of machine learning methods with traditional models for use of administrative claims with electronic medical records to predict heart failure outcomes. JAMA Netw Open. 2020;3(1):e1918962. doi:10.1001/jamanetworkopen.2019.18962

21. Kharrazi H, Ma X, Chang HY, Richards TM, Jung C. Comparing the predictive effects of patient medication adherence indices in electronic health record and claims-based risk stratification models. Popul Health Manag. 2021;24(5):601-609. doi:10.1089/pop.2020.0306

22. Chang HY, Richards TM, Shermock KM, et al. Evaluating the impact of prescription fill rates on risk stratification model performance. Med Care. 2017;55(12):1052-1060. doi:10.1097/MLR.0000000000000825

23. Mishuris RG, Linder JA, Bates DW, Bitton A. Using electronic health record clinical decision support is associated with improved quality of care. Am J Manag Care. 2014;20(10):e445-e452.

24. Co Z, Holmgren AJ, Classen DC, et al. The tradeoffs between safety and alert fatigue: data from a national evaluation of hospital medication-related clinical decision support. J Am Med Inform Assoc. 2020;27(8):1252-1258. doi:10.1093/jamia/ocaa098

25. Young EH, Pan S, Yap AG, Reveles KR, Bhakta K. Polypharmacy prevalence in older adults seen in United States physician offices from 2009 to 2016. PLoS One. 2021;16(8):e0255642. doi:10.1371/journal.pone.0255642