Patient Experience After Modifying Visit Delivery During the COVID-19 Pandemic

ABSTRACT

Objectives: To describe real-time changes in medical visits (MVs), visit mode, and patient-reported visit experience associated with rapidly deployed care reorganization during the coronavirus disease 2019 (COVID-19) pandemic.

Study Design: Cross-sectional time series from September 29, 2019, through June 20, 2020.

Methods: Responding to official public health and clinical guidance, team-based systematic structural changes were implemented in a large, integrated health system to reorganize and transition delivery of care from office-based to virtual care platforms. Overall and discipline-specific weekly MVs, visit mode (office-based, telephone, or video), and associated aggregate measures of patient-reported visit experience were reported. A 38-week time-series analysis with March 8, 2020, and May 3, 2020, as the interruption dates was performed.

Results: After the first interruption, there was a decreased weekly visit trend for all visits (β₃ = –388.94; P < .05), an immediate decrease in office-based visits (β₂ = –25,175.16; P < .01), increase in telephone-based visits (β₂ = 17,179.60; P < .01), and increased video-based visit trend (β₃ = 282.02; P < .01). After the second interruption, there was an increased visit trend for all visits (β₅ = 565.76; P < .01), immediate increase in video-based visits (β₄ = 3523.79; P < .05), increased office-based visit trend (β₅ = 998.13; P < .01), and decreased trend in video-based visits (β₅ = –360.22; P < .01). After the second interruption, there were increased weekly long-term visit trends for the proportion of patients reporting “excellent” as to how well their visit needs were met for all visits (β₅ = 0.17; P < .01), telephone-based visits (β₅ = 0.34; P < .01), and video-based visits (β₅ = 0.32; P < .01). Video-based visits had the highest proportion of respondents rating “excellent” as to how well their scheduling and visit needs were met.

Conclusions: COVID-19 required prompt organizational transformation to optimize the patient experience.

Am J Manag Care. 2021;27(2):e54-e63. https://doi.org/10.37765/ajmc.2021.88535

Takeaway Points

Large, integrated group practices have the essential framework and infrastructure to allow for effective coordination of care, whether in person or virtually, to effectively function during crises. During the coronavirus disease 2019 pandemic, the following were observed:

• There were significant immediate decreases in weekly visit trends for all visits, primary care, and specialty care, with a significant increase in weekly visit trend for behavioral health. Indicators of longer-term recovery for overall visits were observed.
• A significant immediate decrease occurred in weekly visits and trends for office-based visits, and immediate increases occurred in telephone-based visits and weekly visit trend for video-based visits.
• Significant increases occurred in long-term trends for the proportion of patients rating “excellent” as to how well their visit needs were met for telephone-based, video-based, and all visits combined.
• Video-based visits had the highest proportion of respondents rating “excellent” as to how well their scheduling and visit needs were met before and throughout the pandemic.

The coronavirus disease 2019 (COVID-19) pandemic has placed unprecedented strain on health systems. Prudent federal and state public health policies that include varying degrees of physical and social isolation and distancing have required rapid changes in the usual delivery of patient care.^1,2 Among CDC recommendations are the rescheduling of nonurgent care and increased use of alternative approaches to in-person visits.² The Coronavirus Aid, Relief, and Economic Security Act accelerated expansion and flexibility of telehealth activities.³ Physical and social distancing will likely continue for the foreseeable future and have a negative impact on what is typically a recession-proof health care economy.⁴

Available Knowledge

Little is known about how the COVID-19 pandemic will ultimately affect practice and patient experience. An April 2020 report noted that 97% of physician practices experienced a direct or indirect negative financial impact, as evidenced by a 60% decrease in volume and corresponding 55% decrease in revenue.⁵ Others have reported a 30% to 70% reduction in independent primary care practice visit volume.⁶ Recently, a news report cited Department of Commerce data estimating an 18% annualized decline in health care spending based on the first 3 months of 2020.⁷ Negative financial impacts are likely exacerbated with the expansion of telehealth technology^5,6 and purchase of personal protective equipment.⁵ One health system reported increasing telemedicine visits from 0 to 70,000 visits in a 1-month period, although it did not describe the delivery mode (ie, telephone or video).⁷ Medical specialties with a track record of delivering virtual care are not exempt from the pandemic’s impact, as another recent report suggested that radiology practices will likely experience a 50% to 70% decrease in volume lasting at least 3 to 4 months.⁸

Rationale

Managing immediate pandemic hazards while continuing to provide routine medical and preventive care that fulfills the patient care experience, population health, and per capita cost of care dimensions of the Triple Aim^9,10 is challenging. Preparing and responding to the medical caseload surge and implementation of necessary public health measures to manage the COVID-19 pandemic in large, integrated health systems requires rapid and carefully choreographed organizational transformation. Health systems and medical practices unable to respond promptly to these demands face enhanced risk of failure and insolvency.

Specific Aims

This report describes real-time changes in medical visits (MVs), visit mode, and patient visit experience prior to and during the COVID-19 pandemic in Kaiser Permanente Mid-Atlantic States (KPMAS), a large integrated health system, in accordance with the Standards for Quality Improvement Reporting Excellence guideline.¹¹

METHODS

Context

Effective January 2020, KPMAS had 767,163 members across Maryland, District of Columbia (DC), and Northern Virginia geographic regions. Membership reflects diverse regional demographics and encompasses public (Medicaid, Medicare, Affordable Care Act health insurance exchange plans), private, individual, and charity care lines of business.¹² Within KPMAS, medical care is provided by the Mid-Atlantic Permanente Medical Group (MAPMG), an integrated medical practice of more than 1700 physicians that includes more than 50 services and specialties. Medical care is provided at 34 medical centers and 12 partner hospitals. Medical centers primarily provide outpatient and observational care. KPMAS contracts with local hospitals to provide inpatient care while using its own providers. These 12 “premier” hospitals, many with dedicated KPMAS units exclusively for KPMAS members, provide medical, surgical, and intensive care and rehabilitation services to KPMAS members, most often by MAPMG physicians and staff.

COVID-19 Caseload and Timeline

COVID-19 case accrual in KPMAS began on March 1, 2020. Between March 1, 2020, and June 20, 2020, KPMAS had 9256 patients with a confirmed COVID-19 infection (ie, clinical diagnosis or positive laboratory test [either within or external to KPMAS]).

Intervention

In response to external public health and clinical guidance, KPMAS made rapid structural changes to effectively reorganize care and transition from office-based to virtual care platforms beginning in late February 2020.

Care reorganization. Medical centers were reorganized for in-person care to include screening for fever, cough, and shortness of breath at every entrance; triaging and testing patients outside of the centers; and colocation of patients at risk for COVID-19. Elective procedures were postponed. Of the 34 medical office buildings (MOBs), only 15 were designated to provide continued clinical care (consistent with Maryland regulations) and 8 provided urgent care. Those MOBs not providing clinical care remained open for pharmacy, laboratory, and radiology services. Six MOBs maintained 24-hour operations, with the remainder having adjusted hours. Drive-up testing for COVID-19 was available during limited times at 6 MOBs in DC, Maryland, and Virginia. Although core contract hospitals were not able to accept all KPMAS patients because of the dramatic increase in COVID-19–related admissions, emergency hospital licensure was obtained, and 163 hospital beds were added in 3 locations to alleviate the burden on community hospital partners providing focused care for critically ill patients with COVID-19.

Training and education. Daily internal emails were generated to update physicians and staff about the rapidly evolving status of the COVID-19 pandemic within KPMAS. Ongoing COVID-19 testing and diagnosis education was provided for physicians and staff via emails and webinars. All operations documents, including workflows, process, and organization, were made available on a SharePoint website accessible to all providers and staff at any time. Weekly COVID-19 lunchtime interactive seminars were provided to address physician and staff questions about testing guidelines, coding, resource management and KPMAS electronic health record (EHR), Epic-based HealthConnect, and any other related COVID-19 issues. Training sessions were also archived for later viewing.

Transition to telemedicine. Appointment rescheduling from in person to telehealth was implemented. All face-to-face appointments, including behavioral health care visits, were given the option to convert to video visits as the goal. Because CMS waived guidelines requiring a license to practice in-state, it became possible to provide care to patients outside of DC, Maryland, and Virginia through telephone and video visits. Existing telemedicine infrastructure made it possible to scale up, including the availability of a telehealth support team for member and physician support for HealthCare Anywhere and integrated video visits (IVV). The HealthCare Anywhere app was used to provide patient care appointments on the fly (ie, not requiring prior appointments or scheduling). Members could be invited via phone number or telephoned by care providers for appointments if the KP.org website was temporarily unavailable or if members were not registered on KP.org. IVVs could be conducted via Enterprise Clinician Connect (eCC) for members already registered on KP.org. Epic-based HealthConnect could be used by providers to switch appointments from face-to-face to video visits. eCC sends alerts to physicians to update them when a member has joined for a video visit. Cortext Out of Network is a Health Insurance Portability and Accountability Act–compliant app that can be used to send text messages from physicians to members to update appointment status if needed. Instructions for conducting video visits can be sent to members via HealthConnect. Member support included interpreter services for the diverse population served by KPMAS.

Measures

Common operational metrics were used to describe medical practice impact, response, and patient experience associated with the organizational changes implemented. Specifically, MVs (ie, visits per week) by medical discipline (ie, primary care, specialty care, and behavioral health care) and visit mode (ie, office-based, telephone, and video) were measured using weekly visit counts.

Patient visit experience was measured by patient response to the following: (1) How well your needs and schedule were taken into consideration when this [office, telephone, video] appointment was scheduled and (2) How well your needs were met in this [office, telephone, or video] appointment/visit. Response sets were “no experience,” “poor,” “fair,” “good,” “very good,” and “excellent.”

Patient experience surveys were administered in accordance with an external vendor contract. Although there are well-recognized challenges with measuring and reporting sampling and response rates for patient experience surveys,^13,14 they are commonly used to ascertain patient perceptions of their health care experience. The 77,403 responses for the first question and 82,762 responses for the second question provide a robust sense of patient experience during the observation period.

Study of the Intervention

A 38-week time-series analysis^15-17 (September 29, 2019, through June 20, 2020) was performed using March 8, 2020, and May 3, 2020, interruption dates to describe real-time changes in both effect and trend of the previously defined operational metrics during the immediate phase (post–COVID-19 period 1) and long-term phase (post–COVID-19 period 2) of the pandemic. The pre–COVID-19 period included 23 weekly time intervals beginning September 29, 2019, followed by 8-week and 7-week sequential post–COVID-19 periods. The first interruption (March 8, 2020) for post–COVID-19 period 1 allowed for sufficient lag after the CDC released official guidance for health care facilities on February 29, 2020,² and followed the week after KPMAS experienced its first COVID-19 case. Post–COVID-19 period 2 began May 3, 2020, coinciding with rapidly evolving federal, state, and local policy changes.^18-20

Analysis

Mean weekly MVs were described overall, by discipline, and by visit mode for the pre–COVID-19 period and post–COVID-19 periods 1 and 2. Patient visit experience was described overall, stratified by visit mode, and by the pre–COVID-19 period and post–COVID-19 periods 1 and 2. For patient visit experience, the mean proportion of respondents reporting an excellent response (ie, top-box analysis) was used. Top-box analysis is commonly used for the Consumer Assessment of Healthcare Providers and Systems²¹ and is consistent with operational reporting practices in KPMAS. Aggregate differences among the pre–COVID-19 period and post–COVID-19 periods 1 and 2 were evaluated using independent-groups, 1-way analysis of variance with Bonferroni correction for multiple comparisons.

A time-series analysis was performed using the itsa^16,17 command (Stata/MP 16.1 for Windows [StataCorp LLC]) and assumed the following form:

Y_t = β₀ + β₁T_t + β₂X_1t + β₃X_1tT_1t + β₄X_2t + β₅X_2tT_2t + ε_t,

where Y_t is the summary outcome (ie, medical visits, patient experience rating) at each time point. T_t is the time since September 29, 2019; X_1t is the dichotomous indicator variable representing the March 8, 2020, interruption; X_1tT_1t is the interaction between time and the indicator for the first interruption; X_2t is the dichotomous indicator variable representing the May 3, 2020, interruption; and X_2tT_2t is the interaction between time and the indicator for the second interruption. β₀ represents the intercept; β₁ is the trend for the pre–COVID-19 period; β₂ is the effect (ie, change in the level of MVs and patient experience rating) immediately after post–COVID-19 period 1; β₃ is the trend difference between the pre–COVID-19 period and post–COVID-19 period 1; β₄ is the effect immediately after post–COVID-19 period 2; and β₅ is the trend difference between post–COVID-19 period 1 and post–COVID-19 period 2. Additional posttrend analyses were performed to estimate the post–COVID-19 period 1 (β₁ + β₃) trend, post–COVID-19 period 2 (β₁ + β₃ + β₅) trend, and trend difference between the pre–COVID-19 period and post–COVID-19 period 2 (β₃ + β₅).

All models used the Newey-West method for estimating standard errors and a 6-lag order model to account for potential autocorrelation and heteroskedasticity. The Cumby-Huizinga post hoc test was used to evaluate the autocorrelation structure. Alpha was set to < .05.

Ethical Considerations

The KPMAS Institutional Review Board determined that this activity was not human subjects research.

RESULTS

More than 2.2 million MVs and more than 77,000 patient visit experience responses were used to describe MVs, visit mode, and patient-reported visit experience during the study period.

MVs by Medical Discipline

For all visits, there was an immediate decrease in mean weekly MVs from the pre–COVID-19 period to post–COVID-19 period 1 (63,415.13 to 49,317.88; F_2,35 = 10.15; P < .01) (Table 1). A decrease in weekly primary care visits was also observed (34,396.87 to 29,948.13; F_2,35 = 3.69; P < .05) for post–COVID-19 period 1. For specialty care, there was a decrease in mean weekly MVs from 25,143.87 in the pre–COVID-19 period to 15,115.38 in post–COVID-19 period 1 and 16,870.43 in post–COVID-19 period 2 (F_2,35 = 27.64; P < .01). An increase in mean weekly MVs for behavioral health care from the pre–COVID-19 period to post–COVID-19 period 2 was also observed (3874.39 to 4606.43; F_2,35 = 5.17; P < .05).

From the pre–COVID-19 period to post–COVID-19 period 1, there was an immediate decrease in specialty care visits (β₂ = –4670.35; P < .05) (Table 2 [part A and part B] and Figure 1). Also, there was a decreased trend for all visits (β₃ = –388.94; P < .05), primary care (β₃ = –180.76; P < .05), and specialty care (β₃ = –215.35; P < .05), and an increased trend in behavioral health care (β₃ = 7.17; P < .01). From post–COVID-19 period 1 to post–COVID-19 period 2, there was an immediate increase in all visits (β₄ = 13,535.18; P < .05) and primary care visits (β₄ = 8993.25; P < .01). In addition, there was an increased trend for all visits (β₅ = 565.76; P < .01) and specialty care visits (β₅ = 405.95; P < .01).

MVs by Visit Mode

For office-based visits, there was a decrease in mean weekly MVs from 54,479.35 in the pre–COVID-19 period to 12,848.25 in post–COVID-19 period 1 and 10,246.86 in post–COVID-19 period 2 (F_2,35 = 89.75; P < .01) (Table 1). In contrast, there was an increase in the mean weekly MVs from 7897.26 to 27,177.25 and 25,018.14 (F_2,35 = 135.10; P < .01) for telephone-based visits and from 1038.52 to 9292.38 and 20,157.86 (F_2,35 = 154.49; P < .01) for video-based visits during post–COVID-19 periods 1 and 2, respectively. Mean weekly video-based visits were also significantly higher in post–COVID-19 period 2 compared with post–COVID-19 period 1 (F_2,35 = 154.49; P < .01).

From the pre–COVID-19 period to post–COVID-19 period 1, there was an immediate decrease in office-based visits (β₂ = –25,175.16; P < .01) and an immediate increase in telephone-based visits (β₂ = 17,179.60; P < .01), along with a decreased trend for office-based visits (β₃ = –736.76; P < .05) and an increased trend for video-based visits (β₃ = 282.02; P < .01) (Table 2 and Figure 2). From post–COVID-19 period 1 to post–COVID-19 period 2, there was an immediate increase in video-based visits (β₄ = 3523.79; P < .01), an increased trend in office-based visits (β₅ = 998.13; P < .01), and a decreased trend in video-based visits (β₅ = –360.22; P < .01).

Patient Visit Experience: Scheduling Needs Met

Overall, the mean proportion of respondents rating how well their scheduling needs were met as “excellent” significantly increased from 57.83% in the pre–COVID-19 period to 64.45% in post–COVID-19 period 1 and 67.71% in post–COVID-19 period 2 (F_2,35 = 89.76; P < .01) (Table 1). When stratified by visit mode, similar significant patterns were observed for office-based visits. Video-based visits consistently had the highest proportion of respondents rating “excellent” as to how well their scheduling needs were met, with 72.14%, 71.57%, and 71.26% for the pre–COVID-19 period, post–COVID-19 period 1, and post–COVID-19 period 2, respectively.

From the pre–COVID-19 period to post–COVID-19 period 1, there was an immediate increase in the proportion of patients reporting “excellent” as to how well their scheduling needs were met (β₂ = 4.67; P < .01) for all visits along with an increased trend (β₃ = 0.08; P < .01) (Table 2). For telephone-based and video-based visits, there was a decreased trend in the proportion of patients reporting “excellent” as to how well their scheduling needs were met (β₃ = –0.17; P < .05; and β₃ = –0.17; P < .01, respectively). In contrast, there was an immediate increase in the proportion of patients reporting “excellent” as to how well their scheduling needs were met between post–COVID-19 period 1 and post–COVID-19 period 2 for video-based visits (β₄ = 2.58; P < .01), as well as increased trends for both telephone and video-based visits (β₅ = 0.39; P < .01; and β₅ = 0.22; P < .01, respectively).

Patient Visit Experience: Visit Needs Met

Overall, there was a significant increase in the mean proportion of respondents rating how well their visit needs were met as “excellent,” from 64.35% in the pre–COVID-19 period to 68.29% in post–COVID-19 period 2 (F_2,35 = 16.33; P < .01) (Table 1). A similar significant pattern was observed for office-based visits. Video-based visits consistently had the highest proportion of respondents rating “excellent” as to how well their visit needs were met, with 72.74%, 71.09%, and 70.60% of respondents for the pre–COVID-19 period, post–COVID-19 period 1, and post–COVID-19 period 2, respectively.

For all visits and telephone-based visits, there was an increased trend in the proportion of patients reporting “excellent” as to how well their visit needs were met from post–COVID-19 period 1 to post–COVID-19 period 2 (β₅ = 0.17; P < .01; and β₅ = .34; P < .01, respectively) (Table 2 and Figure 3). For video-based visits, there was a decreased trend in the proportion of patients reporting “excellent” as to how well their visit needs were met from the pre–COVID-19 period to post–COVID-19 period 1 (β₃ = –0.19; P < .01) with an immediate increase in both effect (β₄ = 2.40; P < .01) and trend (β₅ = 0.32; P < .01) between post–COVID-19 period 1 and post–COVID-19 period 2.

DISCUSSION

Summary

This study describes the replacement of office-based visits with an immediate shift to telephone-based visits, a subsequent transition to video-based visits, and a gradual return to office-based visits during the COVID-19 pandemic. A corresponding positive long-term trend for meeting patient scheduling and visit needs was observed.

Results confirm anecdotal and trade association reports suggesting that the COVID-19 pandemic would have a significant impact on MVs and result in changes to care delivery.^4-8 Observed changes demonstrate prompt organizational action in response to COVID-19 caseload surge and evolving public policy. By rapidly converting from office-based to virtual care environments, KPMAS met patient needs in a timely, patient-centric way²² for a large majority of patients, which optimized the patient experience dimension of the Triple Aim.⁹

Interpretation

Essential infrastructure must be in place to rapidly deploy resources and adjust internal policies to meet a pandemic challenge. Unlike siloed, nonintegrated health care models that may depend on the viability of independent partners, large, integrated group practices have the essential framework that allows for coordination of care, whether in person or virtually, to function during crises effectively. Existing infrastructure and shared experience create an opportunity to rapidly and efficiently engage health care providers in training and education to continually refine care delivery. The results herein suggest a learning curve when abrupt implementation takes place as health care providers and patients acclimate to a new care delivery paradigm. These changes may have a short-term negative impact on patient-reported visit experience, but as time progresses and experience with virtual care delivery increases, patient-reported visit experience tends to improve. Importantly, key quality¹⁹ aspects of the patient visit experience that include safety, effectiveness, efficiency, and equity remain unstudied during this transition of care delivery and should be further explored to better understand changes in reported patient visit experience.

Ensuring population health during pandemic crises is a challenge.^23-25 The adjustments discussed have allowed KPMAS to address essential and urgent care needs. Preventive care and screening, which often cannot be accomplished virtually, may be postponed or partially completed and are not explored herein. These delays pose challenges for integrated delivery systems like KPMAS for which prevention, early detection, and population health are a hallmark. Strategies to meet these needs are ongoing.

Limitations

This ecological study^26,27 does not imply causality. Rather, it describes key changes in a group of logically related operational data preceding and during a pandemic event. The results generate important hypotheses requiring more detailed interrogation at the patient and visit levels of observation to control for potential confounding factors. One may expect decreases in long-term patient visit experience measures if there were major challenges with adopting the technology. Although the results presented herein demonstrate that video-based visits consistently received the highest patient experience scores, there is a need to further evaluate variations in patient experiences with respect to the adoption, sustained use, and subsequent outcomes of the video technology, as has been suggested in the literature.²⁸ It is unlikely that seasonality was a significant confounding factor in this study given the clearly defined, naturally occurring pandemic during the observation period. Although changes in enrollment during the observation period could contribute to changes in the number of visits, KPMAS experienced a 2.59% enrollment increase from December 2019 to January 2020, with an additional 0.9% increase through June 2020, primarily attributed to Medicaid and ACA options. In the spirit of timeliness, the sustainability of observed changes during the postpandemic period cannot be determined at this time. Future studies should extend the analysis to evaluate reemergence from the COVID-19 pandemic and to identify the optimal mix of office-based and virtual care to best meet patient needs and expectations.

CONCLUSIONS

The COVID-19 pandemic required KPMAS to enact rapid and timely organizational transformation to optimize the patient experience dimension of the Triple Aim during a pandemic crisis. This report informs further investigation and dialogue that should assist in formulating best practices in preparation for future emergencies to address all dimensions of the Triple Aim. Additionally, this report should inform future public policy debates involving the structural and financial considerations necessary to manage practice transformation during both pandemic and nonpandemic times.

Acknowledgments

The authors would like to acknowledge Ms Pradnya H. Chahande, Ms Swee K. Chew, Ms Jessica L. Locke, Ms Chandrika Pai, and Dr Kumar Velayuthan for their assistance and insights related to the acquisition and curation of data used in this report.

Author Affiliations: Mid-Atlantic Permanente Research Institute (MJM, ESW, MAH, MB), Rockville, MD; Mid-Atlantic Permanente Medical Group (MAH, BRT), Rockville, MD; The Permanente Medical Group (RJM), Rockville, MD.

Source of Funding: None.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (MJM, ESW, MAH, MB, BRT, RJM); acquisition of data (MJM, ESW, MAH, BRT, RJM); analysis and interpretation of data (MJM, ESW, MAH, BRT, RJM); drafting of the manuscript (MJM, MB, RJM); critical revision of the manuscript for important intellectual content (MJM, ESW, MAH, MB, BRT, RJM); statistical analysis (MJM, RJM); provision of patients or study materials (MJM, RJM); obtaining funding (RJM); administrative, technical, or logistic support (MJM, MAH, MB, BRT, RJM); and supervision (MJM, MAH, RJM).

Address Correspondence to: Michael J. Miller, DrPH, Mid-Atlantic Permanente Research Institute, 2101 E Jefferson St, Rockville, MD 20852. Email: Michael.J1.Miller@kp.org.

REFERENCES

1. Helping private practices navigate non-essential care during COVID-19. American Medical Association. Updated May 21, 2020. Accessed April 27, 2020. https://www.ama-assn.org/delivering-care/public-health/helping-private-practices-navigate-non-essential-care-during-covid-19

2. Healthcare facilities: managing operations during the COVID-19 pandemic. Updated June 28, 2020. Accessed April 27, 2020. https://www.cdc.gov/coronavirus/2019-ncov/hcp/guidance-hcf.html

3. Coronavirus Aid, Relief and Economic Security (CARES) Act, Pub L No. 116-136, 134 Stat 281 (2020).

4. Cutler D. How will COVID-19 affect the health care economy? JAMA Health Forum. April 9, 2020. Accessed June 1, 2020. https://jamanetwork.com/channels/health-forum/fullarticle/2764547

5. COVID-19 financial impact on medical practices. Medical Group Management Association. 2020. Accessed April 27, 2020. https://www.mgma.com/getattachment/9b8be0c2-0744-41bf-864f-04007d6adbd2/2004-G09621D-COVID-Financial-Impact-One-Pager-8-5x11-MW-2.pdf.aspx?lang=en-US&ext=.pdf

6. Phillip RL, Bazemore A, Baum A. The COVID-19 tsunami: the tide goes out before it comes in. Health Affairs. April 17, 2020. Accessed May 4, 2020. https://www.healthaffairs.org/do/10.1377/hblog20200415.293535/full/

7. Frankel TC, Romm T. Historic financial decline hits doctors, dentists and hospitals—despite Covid-19—threatening overall economy. Washington Post. May 4, 2020. Accessed June 2, 2020. https://www.washingtonpost.com/business/2020/05/04/financial-distress-among-doctors-hospitals-despite-covid-19-weighs-heavily-economy/

8. Cavallo JJ, Forman HP. The economic impact of the COVID-19 pandemic on radiology practices. Radiology. 2020;296(3):E141-E144. doi:10.1148/radiol.2020201495

9. Triple Aim for populations. Institute for Health Care Improvement. Accessed May 2, 2020. http://www.ihi.org/Topics/TripleAim/Pages/default.aspx

10. Berwick DM, Nolan TW, Whittington J. The Triple Aim: care, health, and cost. Health Aff (Millwood). 2008;27(3):759-769. doi:10.1377/hlthaff.27.3.759

11. Goodman D, Ogrinc G, Davies L, et al. Explanation and elaboration of the SQUIRE (Standards for Quality Improvement Reporting Excellence) Guidelines, V.2.0: examples of SQUIRE elements in the healthcare improvement literature. BMJ Qual Saf. 2016;25(12):e7. doi:10.1136/bmjqs-2015-004480

12. Kaiser Permanente Mid-Atlantic States. Diversity and inclusion annual report with language analysis supplements: October 1, 2016-September 31, 2017. Kaiser Permanente. Accessed June 2, 2020. https://providers.kaiserpermanente.org/info_assets/cpp_mas/mas_Diversity_Inclusion_Annual_Report_May2018.pdf

13. Tesler R, Sorra J. CAHPS survey administration: what we know and potential research questions. Agency for Healthcare Research and Quality. October 2017. Accessed May 6, 2020. https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/about-cahps/research/survey-administration-literature-review.pdf

14. Fowler FJ Jr, Cosenza C, Cripps LA, Edgman-Levitan S, Cleary PD. The effect of administration mode on CAHPS survey response rates and results: a comparison of mail and web-based approaches. Health Serv Res. 2019;54(3):714-721. doi:10.1111/1475-6773.13109

15. Fretheim A, Tomic O. Statistical process control and interrupted time series: a golden opportunity for impact evaluation in quality improvement. BMJ Qual Saf. 2015;24(12):748-752. doi:10.1136/bmjqs-2014-003756

16. Linden A. Conducting interrupted time-series analysis for single-and multiple-group comparisons. Stata J. 2015;15(2):480-500. doi:10.1177/1536867X1501500208

17. Linden A. A comprehensive set of postestimation measures to enrich interrupted time-series analysis. Stata J. 2017;17(1):73-88. doi:10.1177/1536867X1701700105

18. Opening up America again: Centers for Medicare & Medicaid Services (CMS) recommendations—re-opening facilities to provide non-emergent non-COVID-19 healthcare: phase I. CMS. April 19, 2020. Accessed July 29, 2020. https://www.cms.gov/files/document/covid-flexibility-reopen-essential-non-covid-services.pdf

19. Amended directive and order regarding various healthcare matters. Maryland Department of Health. May 6, 2020. Accessed July 29, 2020. https://www.mbp.state.md.us/forms/05062020ExecOrder.pdf

20. VDH COVID-19 interim guidance for offering inpatient or outpatient elective procedures. Virginia Department of Health. May 4, 2020. Accessed July 29, 2020. https://www.vdh.virginia.gov/content/uploads/sites/182/2020/05/COVID_19_-Elective-Surgery-Guidance_05072020_FINAL.pdf

21. Analyzing CAHPS survey data. Agency for Healthcare Research and Quality. Updated September 2020. Accessed May 6, 2020. https://www.ahrq.gov/cahps/surveys-guidance/helpful-resources/analysis/index.html

22. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academies Press; 2001.

23. Pawlowski A. Are annual doctor visits essential? which exams you can delay, which you shouldn’t. Today. May 6, 2020. Accessed June 2, 2020. https://www.today.com/health/should-i-go-doctor-coronavirus-covid-19-impact-routine-preventive-t180851

24. Preventive cancer screenings during COVID-19 pandemic. Epic Health Research Network. May 1, 2020. Accessed June 2, 2020. https://ehrn.org/wp-content/uploads/Preventive-Cancer-Screenings-during-COVID-19-Pandemic.pdf

25. Santoli JM, Lindley MC, DeSilva MB, et al. Effects of the COVID-19 pandemic on routine pediatric vaccine ordering and administration—United States, 2020. MMWR Morb Mortal Wkly Rep. 2020;69(19):591-593. doi:10.15585/mmwr.mm6919e2

26. Susser M. The logic in ecological: I. the logic of analysis. Am J Public Health. 1994;84(5):825-829. doi:10.2105/ajph.84.5.825

27. Susser M. The logic in ecological: II. the logic of design. Am J Public Health. 1994;84(5):830-835. doi:10.2105/ajph.84.5.830

28. Nouri S, Khoong EC, Lyles CR, Karliner L. Addressing equity in telemedicine for chronic disease management during the Covid-19 pandemic. NEJM Catalyst Innovations in Care Delivery. May 4, 2020. Accessed July 21, 2020. https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0123