Changes in Electronic Notification Volume and Primary Care Provider Burnout

ABSTRACT

Objectives: Electronic health record (EHR) inbox notifications can be burdensome for primary care providers (PCPs), potentially contributing to burnout. We estimated the association between changes in the quantities of EHR inbox notifications and PCP burnout.

Study Design: In this observational study, we tested the association between the percent change in daily inbox notification volumes and PCP burnout after an initiative to reduce low-value notifications at the Veterans Health Administration (VHA).

Methods: The VHA initiative resulted in increases and decreases in notification volumes for PCPs. For each facility, the proportion of PCPs reporting burnout was estimated using VHA All Employee Survey responses before and after the initiative in 2016 and 2018, respectively. Survey responses were aggregated for 6459 PCPs (physicians, nurse practitioners, and physician assistants) at 138 VHA facilities. Fixed effects regression models estimated the association of small and large increases and small and large decreases in notifications on burnout.

Results: Daily inbox notifications per PCP decreased by a mean (SD) of 5.9% (30.1%) across study facilities, from a mean (SD) of 128 (52) notifications to 114 (44) notifications after the initiative. Fifty-one percent of facilities experienced reductions in notifications, 30% experienced no change, and 20% experienced increased notifications. PCP burnout was not significantly associated with any level of increase or decrease in notifications.

Conclusions: Changes in notification volumes alone did not predict PCP burnout. Future research to reduce burnout might still address EHR notification volumes, but as part of a broader set of strategies that consider the other stressors that PCPs experience.

Am J Manag Care. 2023;29(1):57-63. https://doi.org/10.37765/ajmc.2023.89304

Takeaway Points

Automated notifications in the electronic health record (EHR) are a pervasive source of digital workload and fatigue for primary care providers (PCPs). We tested whether provider burnout changed after an initiative was implemented at the Veterans Health Administration to optimize electronic inbox notification volumes.

• We did not find any association between increases or decreases in notification volumes and burnout.
• Reducing notifications alone is likely insufficient to improve burnout.
• Future research should consider multiple aspects of EHR inbox design, in addition to volume, when considering the effects of notification burden on PCPs.

Although electronic health records (EHRs) improve data management and communication within health care, they can be a source of stress and job dissatisfaction for health care providers.^1-6 For primary care providers (PCPs), the EHR inbox is important for communication of test results, referrals, medication refills, and messages.² PCPs spend approximately 1 to 2 hours or more per day managing asynchronous EHR inbox notifications.^7-9 EHR tasks can be time intensive and inefficient and can compete with direct patient care; they may also spill into personal time and could contribute to burnout.^1,4,10-13

Burnout—typified by physical, mental, and emotional responses to stressors at work—is common among PCPs, affecting half of physicians in primary care and up to one-third of nurse practitioners (NPs) and physician assistants (PAs).^4,14-17 Optimizing complex and cumbersome EHR systems could potentially improve the working environment for PCPs, who are among the highest users of EHRs. Decreasing the stress and fatigue related to managing inbox notifications might improve PCP well-being.¹² In the Veterans Health Administration (VHA), 87% of PCPs reported that inbox notification levels were unmanageable, potentially risking patient safety via care delays from missed test result notifications.¹⁸ However, evidence on inbox notifications and burnout is equivocal. The concurrent measurement of notifications and burnout in recent studies also limits inference on the effect of notification volume on burnout.^3,9,19,20 Although notifications are notably burdensome, one study found that notification process time did not predict burnout in physicians.⁹ Other studies correlated high notification volumes with provider burnout²⁰ and emotional exhaustion.¹⁹

It is unclear whether reducing inbox notification burden could reduce burnout. In 2017, the VHA launched a nationwide initiative to address unmanageable notification volumes by optimizing notifications received by PCPs (also called view alerts).²¹ During the initiative, each facility formatted its EHR interface to include a mandatory set of notifications based on VHA and facility priorities (eg, critical laboratory values), trained PCPs in EHR customization of optional notifications, and tracked preinitiative and postinitiative notification levels for samples of PCPs at each facility.^21-23 After the initiative, large shifts in daily notification volume were observed within some VHA facilities (ie, increases or decreases of up to 100 notifications per PCP per day) but not at others.²¹ Assessing the dose-response relationship between notifications and burnout can help determine whether reducing inbox notification volume is sufficient to reduce EHR-related stress. For this study, the initiative served as a natural experiment to test the association between burnout and changes in inbox notification volumes.

METHODS

Study Design and Setting

In this observational study, we linked estimates of inbox notification volumes before and after the VHA’s EHR initiative with serial cross-sectional survey-based estimates of PCP burnout for 138 VHA facilities. The VHA is among the largest health care systems in the United States, and each VHA facility is composed of a local network of a medical center and multiple community clinics.At the time of the inbox notification initiative, the VHA employed the Computerized Patient Record System (CPRS) to manage patient care within the VistA system, the VHA’s integrated EHR. Inbox notifications within the CPRS alert staff to clinical workflow tasks, including laboratory and imaging results, medication refills, messages from colleagues and patients, referral follow-up, and signature requests.²² In some cases, these notifications address critical items needing providers’ attention, such as abnormal imaging results. Other notifications are of lower value for providers, such as notification of a patient no-show for an appointment. These notifications may be better handled by someone other than the provider.²¹ PCPs manage dozens to hundreds of inbox notifications daily.^7,21 The CPRS inbox allows for user customization, such as through enabling or disabling nonmandatory notifications.^5,24 Mandatory notifications generally require action and were further described by Shah and colleagues.²¹

Data and Study Population

Inbox notification data were collected by VHA operations during the inbox notification initiative and were measured before and after the initiative for a subset of PCPs for each facility. A team at the Michael E. DeBakey Veterans Affairs Medical Center in Houston, Texas, performed an evaluation of the initiative; notification volume data were provided by this study team. Although the inbox notification initiative affected providers across specialties, PCPs managed the most inbox notifications⁷ and were the focus of the initiative. The initiative resulted in a mean decrease in notifications,^21,25 and approximately 97% of PCPs were trained on EHR inbox optimization.²¹

For the present study, notification data were limited to VHA facilities with primary medical centers located in the United States. In the original data, some facilities were split into subfacilities. In these few instances, notification volumes for the main medical center were measured independently of other clinics within the facility. These facilities were also omitted because burnout was measured across facilities as a whole. The final sample included 138 facilities.

PCP burnout was estimated for each VHA facility using aggregated responses to the VHA’s annual workforce survey, the All Employee Survey (AES), which is collected each spring.²⁶ This anonymous survey assesses employee attitudes and the workplace environment through questions on topics including job satisfaction, social support, turnover intentions, and burnout. Within each facility, individual AES responses were aggregated for respondents who indicated being a PCP (physician, NP, or PA) and working in primary care (ie, a member of a patient-aligned care team) on the survey. Aggregate responses were reported as proportions (eg, the proportion of PCPs at a facility who were women or who reported burnout). AES response rates for VHA staff were 57% in 2016, 60% in 2017, and 62% in 2018.^26-28

Aggregate AES data were combined into a serial cross-sectional data set from 2016 to 2018 using facility identifiers. Notification volume data were linked to aggregate AES data for 2016 (preinitiative) and 2018 (post initiative). The 2017 AES collection overlapped with the inbox notification initiative, so to ensure a valid exposure timeline, AES data for 2018 were used to estimate postinitiative burnout. The 2017 AES data were used for the description of burnout trends but omitted for inferential analyses. As per data use agreements, we complied with a minimum aggregation requirement of 10 AES respondents for facility-level measures. This resulted in the exclusion of 15.8% of facility-level observations, primarily from smaller VHA facilities. Also, for some facilities, data were used only for either 2016 or 2018 due to the minimum aggregation requirements. The final sample included 6459 PCP survey respondents aggregated for 138 VHA facilities in 2016 and 2018.

Measures

VHA facilities were categorized into 5 groups based on their postinitiative change in notification volume. Using the distribution of the percent change in the mean number of inbox notifications per PCP per day for facilities, we differentiated the facility groups with cut points of ± 0.25 SD and ± 1 SD away from the mean change in notification volume for all VHA facilities. The 5 groups were composed of facilities with (1) a large decrease (> –1 SD of the distribution of percent change in notifications), (2) a small decrease (–0.25 to –1 SD), (3) no change (–0.25 to +0.25 SD), (4) a small increase (+0.25 to +1 SD), and (5) a large increase (> +1 SD) in the percent change in notification volume. Facility group was the primary exposure in this study.

The primary outcome was the prevalence of burnout among PCPs at each facility. The AES includes 3 questions based on each of the 3 dimensions of burnout characterized within the Maslach Burnout Inventory^29,30: emotional exhaustion, depersonalization, and reduced achievement. Consistent with previous literature, we constructed a composite measure of burnout from the emotional exhaustion (“I feel burned out from my work”) and depersonalization (“I worry that this job is hardening me emotionally”) questions.^30-32 These questions were assessed on a 7-point Likert scale of the frequency of burnout symptoms, ranging from never to every day. We defined burnout as answering “once a week,” “a few times a week,” or “every day” to the emotional exhaustion and/or the depersonalization questions.^14,31 Burnout was calculated as the proportion of PCP survey respondents at a facility who screened positively for burnout.

Characteristics of the PCP samples at each facility were determined from aggregate AES measures of PCP respondent gender, age, and VHA tenure. Gender (ie, the proportion of PCP respondents who were women), age (ie, the proportion of PCP respondents younger than 50 years), and short VHA tenure (ie, the proportion of PCP respondents with a VHA tenure of less than 5 years) were used as precision variables. Younger age, female gender, and shorter tenure were previously associated with burnout in providers.^33,34

Statistical Analyses

Characteristics of PCPs in each of the 5 facility groups were described using AES data for the number, age, gender, and tenure of PCP survey respondents. Inbox notification volume was described for each of the 5 facility groups pre- and post initiative. To describe burnout levels pre- and post initiative, mean PCP burnout proportions were plotted from 2016 to 2018 for the facility groups.

Fixed effects (FE) linear regression models were used to estimate the effect of the level and direction of changes in inbox notification volume on the proportion of PCP burnout at the facility level. FE models are useful for limiting bias from confounding in panel data, and they rely on variability in the exposure over time to estimate the effect of the exposure on an outcome.³⁵ For this study, the base FE model (model 1) estimated the effect of the changes in notification volume on PCP burnout for facilities that experienced small and large increases and small and large decreases in notification volume. The base and adjusted models included an indicator for year to assess secular trends in PCP burnout due to unobserved factors over the study period. The adjusted FE model (model 2) of changes in notification volume and burnout included age, gender, and short tenure as precision variables in addition to the year indicator. A fixed effect for VHA facility accounted for time-invariant confounding, or the bias associated with unobserved heterogeneity among VHA facilities. Effectively, each facility served as its own control by estimating the burnout–notification volume relationship in comparison with its own preinitiative PCP burnout prevalence.³⁶ We used robust SEs, clustered by facility, in both models. This study was approved by the VA Puget Sound Health Care System and University of Washington institutional review boards. All analyses were performed using Stata version 17 (StataCorp), and reporting followed STROBE guidelines (eAppendix [available at ajmc.com]).

RESULTS

After the inbox notification initiative, mean (SD) daily inbox notifications for VHA facilities decreased from 128 (52) inbox notifications preinitiative (range, 31-378) to 114 (44) inbox notifications post initiative (range, 40-329). Daily inbox notifications decreased by a mean (SD) of 5.9% (30.1%) post initiative, ranging from a 71% decrease to a 200% increase in notifications (Figure 1). Based on the distribution of percent change in inbox notifications, ± 1 SD equated to a ± 30% change in notifications and ± 0.25 SD equated to a ± 7% change in notifications.

The 5 facility groups were distributed as follows: (1) large decrease facilities, which had a reduction in inbox notifications greater than 30% (14.5% of facilities; n = 20); (2) small decrease facilities, with a 7% to 30% reduction in inbox notifications (37.0% of facilities; n = 51); (3) no change facilities, with a change in inbox notifications within ± 7% (reference category; 29.7% of facilities; n = 41); (4) small increase facilities, with a 7% to 30% increase in inbox notifications (13.8% of facilities; n = 19); and (5) large increase facilities, with an increase in inbox notifications greater than 30% (5.1% of facilities; n = 7).

Table 1 describes PCP sample characteristics and inbox notification volume results for these 5 facility groups. Across the groups, 46% to 56% of PCPs at VHA facilities were women, 35% to 41% were younger than 50 years, and 39% to 44% had VHA tenures shorter than 5 years. Facilities with a large decrease in inbox notifications (> 30%) had both the highest levels of preinitiative notifications (mean [SD], 159 [69]) and the lowest levels of postinitiative notifications (mean [SD], 83 [32]). Facilities with a large increase in inbox notifications (> 30%) had both the lowest levels of preinitiative notifications (mean [SD], 83 [30]) and the highest levels of postinitiative notifications (mean [SD], 142 [40]).

From 2016 to 2018, PCP burnout decreased for all VHA facilities (Figure 2). Burnout in facilities that experienced no change in inbox notifications decreased from 51.1% in 2016 to 43.8% in 2018. PCP burnout was initially highest for facilities that had a large increase in notifications (>30%), at a mean (SD) of 61.3% (11.9%), although burnout in these facilities decreased to 46.3% (14.2%) by 2018.

In both the base and adjusted models (Table 2), neither small nor large increases or decreases in notification volume were significantly associated with facility-level PCP burnout. However, PCP burnout decreased significantly over the study period. In 2018, PCP burnout was 7.3 percentage points lower (95% CI, –11.4 to –3.3) in the base model compared with 2016 and 6.4 percentage points lower (95% CI, –10.4 to –2.4) in the adjusted model compared with 2016.

DISCUSSION

A VHA-wide initiative to improve its EHR-based inbox notification system resulted in large changes in inbox notification volume at VHA facilities. However, neither increases nor decreases in notification volumes were associated with PCP burnout. To our knowledge, this is the first study to go beyond cross-sectional associations and test burnout in response to a change in EHR notification volume in primary care. Although previous findings on notification-related work burden and burnout are mixed,^9,19,20 our finding suggests that reducing inbox notification volume was not sufficient to have a measurable effect on burnout.

Notification volumes observed in this study were comparable with previous estimates at the VHA.^7,21 Although the inbox notification initiative focused on reducing burdensome EHR messages, inbox optimization did not always result in a reduction in notification volume. Almost 20% of facilities in this study saw an increase in their notification volume post initiative. Although compliance with the initiative was high across VHA facilities, notification volumes varied widely among facilities, even with the initiative’s mandatory notification guidelines and PCP training.²¹ The multisite nature of this study demonstrates the wide range in notification volumes that PCPs may experience, depending on personal and organizational preferences.Also, some VHA facilities had already made substantive improvements to their inbox systems, reducing notifications prior to the initiative. These facilities may have increased their notification volumes because they were required to comply with the initiative.²¹

Although our findings contrast with those of recent research relating message volume to increased PCP burnout,^19,20 these prior studies relied on concurrent cross-sectional measurement of message volume and burnout. By assessing change in notification volume within facilities, we controlled for confounding and questions of temporality, which are problematic in cross-sectional research. Our findings support the position that there may not be an optimal number of notifications to minimize PCP burnout, at least at the levels observed in this study.

Although the VHA-wide inbox notification initiative was not specifically directed at reducing burnout, it was developed in recognition that EHR-related workload is a central facet of PCP work life. Patient and job-specific needs may dictate inbox optimization by PCPs. One prior study of missed test results in EHRs found that notification volume was limited in predicting information overload.¹⁸ Another study identified numerous aspects of EHRs that may be associated with burnout, including excessive data entry and slow system response time.⁶ Optimizing how and when notifications are displayed is important to the user’s experience.

In this study, we did not differentiate among notification types. Despite contributing to notification volume, some types of notifications likely contributed little to notification-related workload, whereas others contributed more. For example, the initiative emphasized reducing duplicative notifications and notifications that required reading time but not additional PCP actions. These particular notifications may not have been time intensive. Also, the number of mandatory notifications was reduced by up to one-third for some facilities.²¹ Some decrease in notifications may be attributed to turning off lower-value notifications that PCPs spent little time on or had previously ignored outright. Notification volume alone may be a poor measure of the actual effort required for inbox management. Although substantial increases and decreases in notification volume may reflect improved EHR utilization, changes in notification volume may not have influenced aspects of EHR-related workload that are predictive of burnout.³⁷ Assessment of notification type, value, and cognitive burden may improve measures of EHR notification burden, which may then be associated with burnout. Further work is needed to understand how time spent on low-value notifications and sufficient work time to respond to notifications can influence PCP fatigue and burnout.¹³

Health systems are uniquely positioned to mitigate burdensome aspects of EHRs through improved EHR systems design.²⁰ Ensuring that EHR notifications are relevant and actionable rather than purely informational²; continually monitoring utilization patterns to identify low-value notifications or the efficacy of new and automated notifications^20,38; innovating the design and visual display of notification systems; and ensuring that users can quickly navigate through EHR interfaces may all enhance EHR operability.^6,23,38 PCPs spend more time on inbox messaging than do clinicians in other medical or surgical specialties,³⁹ and interventions reducing EHR work burden should consider PCPs’ specific needs. Collaboration between health system leaders and PCPs on EHR design, with a focus on staff well-being, is essential.^20,40

Limitations

This study relied on annual surveillance of burnout among VHA PCPs. To reduce overlap between the AES collection and inbox notification initiative implementation, we used 2018 rather than 2017 burnout estimates post initiative. Notification levels at facilities may have changed in the time between the initiative and the 2018 AES collection, or the effect of the initiative may have waned in the interceding time. Survey response bias is also a concern and could lead to underreporting of burnout (eg, if PCPs experiencing higher burnout declined survey participation) and potential underestimation of burnout. Also, staffing information on full-time equivalency of PCPs, or the number of PCPs contributing to the mean measurement of daily notifications, was not available. This could affect accuracy of inbox notification measurement.

The VHA uses a unique EHR, and notification volume estimates used in this study were VHA specific. We categorized facilities into 5 facility groups based on the distribution of postinitiative percent change in notifications, although burnout was potentially affected by changes in notification levels for reasons other than the initiative. Facilities where inbox notification counts increased may have conducted previous EHR efficiency work. Also, unmeasured factors influencing PCP burnout occurring simultaneously at the VHA, such as other EHR improvements, could confound the relationship between notification volume changes and burnout. Lastly, the group of facilities in which notifications increased by more than 30% was small (n = 7) and potentially underpowered.

CONCLUSIONS

Asynchronous inbox notifications can be time-consuming and burdensome for PCPs. We hypothesized that decreasing inbox notifications would reduce PCP burnout; however, we did not observe any effect on burnout, even though the VHA’s initiative resulted in substantial decreases in notifications in some facilities but not others. Other characteristics of inbox notifications that we did not measure may be important. Future research on burnout must evaluate characteristics such as notification type and value and time spent managing notifications, especially through use of causal study designs. As EHR systems evolve, health systems should work with PCPs to identify effective and efficient solutions that improve EHR use and support staff well-being.

Acknowledgments

The authors wish to acknowledge the contributions of the VHA National Center for Organization Development for survey administration and thank the VHA employees who took part in the surveys used in this study. The authors would also like to thank Ashley Meyer and Laura Kroupa for their technical assistance and the team at the Michael E. DeBakey Veterans Affairs Medical Center who provided initiative data.

Author Affiliations: Department of Health Systems and Population Health (AWO, CDH, KMN, JMS, ESW), Department of Medicine (KMN), and Department of Environmental and Occupational Health Sciences (JMS), University of Washington, Seattle, WA; Center for Veteran-Centered and Value-Driven Care, VA Puget Sound Health Care System (CDH, KMN, ESW), Seattle, WA; Harborview Injury Prevention and Research Center (JMS), Seattle, WA; Institute for Work and Health (JMS), Toronto, Ontario, Canada; Center for Innovations in Quality, Effectiveness and Safety (IQuESt), Michael E. DeBakey Veterans Affairs Medical Center (HS), Houston, TX; Department of Medicine, Baylor College of Medicine (HS), Houston, TX.

Source of Funding: This study was supported by the Veterans Affairs (VA) Health Services Research and Development (HSR&D) Investigator Initiated Award #15-363. Ms O’Connor’s work on this study was supported by the National Institute for Occupational Safety and Health (NIOSH) under Federal Training Grant T42OH008433. Dr Singh is also funded in part by the Houston VA HSR&D Center for Innovations in Quality, Effectiveness, and Safety (CIN-13-413). The content is solely the responsibility of the authors and does not necessarily represent the official views of NIOSH or the position or policy of the VHA or the US government.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (AWO, CDH, KMN, ESW); acquisition of data (AWO, KMN, ESW); analysis and interpretation of data (AWO, CDH, KMN, JMS, HS, ESW); drafting of the manuscript (AWO, CDH, HS); critical revision of the manuscript for important intellectual content (AWO, CDH, KMN, JMS, HS, ESW); statistical analysis (AWO); provision of patients or study materials (AWO); obtaining funding (AWO, ESW); administrative, technical, or logistic support (AWO); and supervision (CDH, JMS, ESW).

Address Correspondence to: Christian D. Helfrich, PhD, MPH, University of Washington, 3980 15th Ave NE, Fourth Floor, Box 351621, Seattle, WA 98195. Email: helfrich@uw.edu.

REFERENCES

1. West CP, Dyrbye LN, Shanafelt TD. Physician burnout: contributors, consequences and solutions. J Intern Med. 2018;283(6):516-529. doi:10.1111/joim.12752

2. Murphy DR, Satterly T, Giardina TD, Sittig DF, Singh H. Practicing clinicians’ recommendations to reduce burden from the electronic health record inbox: a mixed-methods study. J Gen Intern Med. 2019;34(9):1825-1832. doi:10.1007/s11606-019-05112-5

3. Shanafelt TD, Dyrbye LN, Sinsky C, et al. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin Proc. 2016;91(7):836-848. doi:10.1016/j.mayocp.2016.05.007

4. Shanafelt TD, Boone S, Tan L, et al. Burnout and satisfaction with work-life balance among US physicians relative to the general US population. Arch Intern Med. 2012;172(18):1377-1385. doi:10.1001/archinternmed.2012.3199

5. Hysong SJ, Spitzmuller C, Espadas D, Sittig DF, Singh H. Electronic alerts and clinician turnover: the influence of user acceptance. Am J Manag Care. 2014;20(11) (spec No. 17):SP520-SP530.

6. Kroth PJ, Morioka-Douglas N, Veres S, et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open. 2019;2(8):e199609. doi:10.1001/jamanetworkopen.2019.9609

7. Murphy DR, Meyer AND, Russo E, Sittig DF, Wei L, Singh H. The burden of inbox notifications in commercial electronic health records. JAMA Intern Med. 2016;176(4):559-560. doi:10.1001/jamainternmed.2016.0209

8. Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med. 2012;125(2):209.e1-209.e7. doi:10.1016/j.amjmed.2011.07.029

9. Gregory M, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform. 2017;8(3):686-697. doi:10.4338/aci-2017-01-ra-0003

10. Spinelli WM, Fernstrom KM, Britt H, Pratt R. “Seeing the patient is the joy:” a focus group analysis of burnout in outpatient providers. Fam Med. 2016;48(4):273-278. doi:10.1007/s40596-017-0849-3

11. Singh H, Spitzmueller C, Petersen NJ, et al. Primary care practitioners’ views on test result management in EHR-enabled health systems: a national survey. J Am Med Inform Assoc. 2013;20(4):727-735. doi:10.1136/amiajnl-2012-001267

12. Arndt BG, Beasley JW, Watkinson MD, et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med. 2017;15(5):419-426. doi:10.1370/afm.2121

13. Gardner RL, Cooper E, Haskell J, et al. Physician stress and burnout: the impact of health information technology. J Am Med Inform Assoc. 2019;26(2):106-114. doi:10.1093/jamia/ocy145

14. Rinne ST, Mohr DC, Swamy L, Blok AC, Wong ES, Charns MP. National burnout trends among physicians working in the Department of Veterans Affairs. J Gen Intern Med. 2020;35(5):1382-1388. doi:10.1007/s11606-019-05582-7

15. Helfrich CD, Simonetti JA, Clinton WL, et al. The association of team-specific workload and staffing with odds of burnout among VA primary care team members. J Gen Intern Med. 2017;32(7):760-766. doi:10.1007/s11606-017-4011-4

16. Chan GK, Kuriakose C, Blacker A, et al. An organizational initiative to assess and improve well-being in advanced practice providers. J Interprof Educ Pract. 2021;25:100469. doi:10.1016/j.xjep.2021.100469

17. Abraham CM, Zheng K, Norful AA, Ghaffari A, Liu J, Poghosyan L. Primary care practice environment and burnout among nurse practitioners. J Nurse Pract. 2021;17(2):157-162. doi:10.1016/j.nurpra.2020.11.009

18. Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and missed test results in electronic health record–based settings. JAMA Intern Med. 2013;173(8):702-704. doi:10.1001/2013.jamainternmed.61

19. Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc. 2020;27(4):531-538. doi:10.1093/jamia/ocz220

20. Tai-Seale M, Dillon EC, Yang Y, et al. Physicians’ well-being linked to in-basket messages generated by algorithms in electronic health records. Health Aff (Millwood). 2019;38(7):1073-1078. doi:10.1377/hlthaff.2018.05509

21. Shah T, Patel-Teague S, Kroupa L, Meyer AND, Singh H. Impact of a national QI programme on reducing electronic health record notifications to clinicians. BMJ Qual Saf. 2019;28(1):10-14. doi:10.1136/bmjqs-2017-007447

22. Singh H. A checklist to improve CPRS ‘view alert’ notifications: recommendations for VA facility leadership and staff. ResearchGate. May 2018. Accessed July 26, 2022. https://www.researchgate.net/publication/325439240_A_Checklist_to_Improve_CPRS_‘View_Alert’_Notifications_Recommendations_for_VA_Facility_Leadership_and_Staff

23. Singh H, Wilson L, Reis B, Sawhney MK, Espadas D, Sittig DF. Ten strategies to improve management of abnormal test result alerts in the electronic health record. J Patient Saf. 2010;6(2):121-123. doi:10.1097/PTS.0b013e3181ddf652

24. VistA Computerized Patient Record System (CPRS) setup guide. US Department of Veterans Affairs. June 2021. Accessed May 5, 2022. https://www.va.gov/vdl/documents/clinical/comp_patient_recrd_sys_%28cprs%29/cprssetup.pdf

25. Improving patient safety through better communication. US Department of Veterans Affairs. October 11, 2018. Accessed January 18, 2022. https://www.hsrd.research.va.gov/impacts/patient-safety.cfm

26. 2018 Department of Veterans Affairs All Employee Survey/Federal Employee Viewpoint Survey: technical report. US Department of Veterans Affairs. Accessed July 26, 2022. https://www.va.gov/NCOD/docs/TechnicalReportv1.pdf

27. 2013 VA All Employee Survey Veterans Health Administration Results and Findings. VHA National Center for Organization Development; 2013.

28. 2017 VA All Employee Survey (AES) Veterans Health Administration Summary and Results. VHA National Center for Organization Development; 2017.

29. Schaufeli WB, Bakker AB, Hoogduin K, Schaap C, Kladler A. On the clinical validity of the Maslach burnout inventory and the burnout measure. Psychol Health. 2001;16(5):565-582. doi:10.1080/08870440108405527

30. Dolan ED, Mohr D, Lempa M, et al. Using a single item to measure burnout in primary care staff: a psychometric evaluation. J Gen Intern Med. 2015;30(5):582-587. doi:10.1007/s11606-014-3112-6

31. West CP, Dyrbye LN, Satele DV, Sloan JA, Shanafelt TD. Concurrent validity of single-item measures of emotional exhaustion and depersonalization in burnout assessment. J Gen Intern Med. 2012;27(11):1445-1452. doi:10.1007/s11606-012-2015-7

32. Rotenstein LS, Torre M, Ramos MA, et al. Prevalence of burnout among physicians: a systematic review. JAMA. 2018;320(11):1131-1150. doi:10.1001/jama.2018.12777

33. Cordes CL, Dougherty TW. A review and an integration of research on job burnout. Acad Manage Rev. 1993;18(4):621-656. doi:10.5465/amr.1993.9402210153

34. Breaux DM, Meurs JA, Zellars KL, Perrewe PL. Burnout in health care: when helping hurts. In: Halbesleben JRB, ed. Handbook of Stress and Burnout in Health Care. 3rd ed. Nova Science Publishers; 2008:39-50.

35. Gunasekara FI, Richardson K, Carter K, Blakely T. Fixed effects analysis of repeated measures data. Int J Epidemiol. 2014;43(1):264-269. doi:10.1093/ije/dyt221

36. Strumpf EC, Harper S, Kaufman JS. Fixed effects and difference-in-differences. In: Oakes JM, Kaufman JS, eds. Methods in Social Epidemiology. 2nd ed. Jossey-Bass; 2017:341-368.

37. Muhiyaddin R, Elfadl A, Mohamed E, et al. Electronic health records and physician burnout: a scoping review. Stud Health Technol Inform. 2022;289:481-484. doi:10.3233/SHTI210962

38. McGreevey JD III, Mallozzi CP, Perkins RM, Shelov E, Schreiber R. Reducing alert burden in electronic health records: state of the art recommendations from four health systems. Appl Clin Inform. 2020;11(1):1-12. doi:10.1055/s-0039-3402715

39. Rotenstein LS, Holmgren AJ, Downing NL, Bates DW. Differences in total and after-hours electronic health record time across ambulatory specialties. JAMA Intern Med. 2021;181(6):863-865. doi:10.1001/jamainternmed.2021.0256

40. Montgomery A, Panagopoulou E, Esmail A, Richards T, Maslach C. Burnout in healthcare: the case for organisational change. BMJ. 2019;366:l4774. doi:10.1136/bmj.l4774