Improving Antibiotic Stewardship: A Stepped-Wedge Cluster Randomized Trial

ABSTRACT

Objectives: Antibiotic stewardship is key to optimizing patient outcomes and affordable care. The study objective was to examine the effect of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis among providers of varying experience.

Study Design: A stepped-wedge cluster randomized intervention to evaluate antibiotic use for acute sinusitis encounters at 126 Kaiser Permanente Southern California clinics between September 2014 and April 2015.

Methods: The primary outcome was receipt of an antibiotic prescription. Multivariate analysis adjusted for patient, provider, and medical center characteristics. Secondary analyses described sinusitis and other common upper respiratory infection (URI) diagnoses and antibiotic use during the study period compared with prior years. Chart review of a random sample reported the proportion of encounters receiving guideline-concordant antibiotics.

Results: Analysis of 21,949 encounters (10,491 pre- and 11,458 post intervention) showed CDS reduced the use of antibiotics (adjusted odds ratio [AOR], 0.78; 95% CI, 0.71-0.87), although the pre-post absolute difference was small (85.9% vs 83.9%, respectively). Education had a large initial effect (AOR, 0.51; 95% CI, 0.46-0.57), which did not persist. Increasing years of provider experience raised the rates of antibiotic prescribing, but did not have a significant interaction with CDS (P = .19). The effect of CDS varied by medical center (P <.001). In addition, sinusitis diagnoses decreased post intervention, with no overall increase in antibiotic prescribing for URI diagnoses. Lastly, guideline-concordant antibiotic use increased by 14%.

Conclusions: Provider education and CDS improved antibiotic stewardship and changed diagnosis patterns. The benefits of education were brief, and CDS effectiveness varied by medical center.

Am J Manag Care. 2017;23(11):e360-e365Takeaway Points

It is vital to understand the effectiveness of interventions designed to improve patient care in order to disseminate and replicate successful initiatives and avoid fruitless care improvement efforts. Our study of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis offers the following to inform health systems and future research:

• CDS integrated in an electronic health record can discourage inappropriate antibiotic prescribing and will likely change diagnostic patterns.
• Provider education is unlikely to have a sustained effect on antibiotic prescribing.
• The effectiveness of education and CDS will vary based on inherent differences among locations.

Antimicrobial stewardship is key to optimizing patient outcomes and minimizing the emergence of drug-resistant bacterial organisms.^1-3 Unfortunately, the goal of reducing inappropriate antibiotic prescribing has remained elusive.⁴ This is true despite the existence of a variety of strategies to improve antibiotic stewardship, such as pay-for-performance incentives, documentation change strategies, provider education, and social “nudges.”^5-7 We currently do not understand why these approaches have variable results across different settings. Research is needed to describe the effect of such interventions in general community practice and to understand variations in effect due to patient, provider, or medical center characteristics in order to inform current efforts to improve care.

Acute sinusitis is a condition that affects 31 million annually⁸ and costs billions.⁹ The preponderance of evidence suggests antibiotics are not only ubiquitously overused for acute sinusitis treatment,^10-13 but the correct antibiotic is often not chosen.¹⁴ Despite agreement regarding overuse, there is no universal recommendation regarding the precise group of patients with acute sinusitis who may benefit from antibiotics.^8,15-17 We used the recommendations from our health system, patterned after the Infectious Diseases Society of America (IDSA)’s guideline,¹⁶ and a practical study design¹⁸ to report the effects of a 2-component intervention designed to improve antibiotic stewardship for acute sinusitis encounters. The components were computerized clinical decision support (CDS) within an electronic health record (EHR)^19,20 and provider education.¹

Clinical leaders and policy makers need to understand the variable effectiveness of multifaceted interventions, as well as the importance that individual provider characteristics may have on expected results. Our aim was to enhance the understanding of the effects of clinician education and CDS in a real-life clinical setting and, specifically, to assess any differences in effectiveness on providers with varying years of experience or among medical centers.

METHODS

We performed a pragmatic stepped-wedge cluster randomized study of provider education and CDS for acute sinusitis encounters. The study was performed in outpatient clinics (105 primary care, 21 urgent care) between September 2014 and April 2015 within Kaiser Permanente Southern California (KPSC), a large integrated health system. This system provides healthcare to more than 4 million members representative of the population diversity found in southern California. We used information from the EHR collected during routine care. This study was approved by the KPSC Institutional Review Board.

Each participating primary care and urgent care clinic is clustered within 1 of 6 medical service areas. Each service area was randomly assigned to receive the CDS intervention as part of a staged implementation over the course of 6 months. The total study period was 8 months, allowing for 1 month of pre-data for all areas prior to CDS implementation. CDS was turned on in 1 of the 6 areas each month until all were receiving CDS, then 1 month of post data was collected after all had received the intervention (Figure 1). This staggered implementation allowed for a stepped-wedge analysis.²¹ This design controls for confounders due to secular trends and strong champion effects.

CDS was based on qualitative interviews with practicing clinicians²² and designed using features shown to be effective in previous trials.¹⁹ Automated in the EHR, CDS was initiated at the time of antibiotic prescribing for patients with an acute sinusitis diagnosis. Once triggered, it offered a clear recommendation based on current evidence.¹⁶ The alert informed providers that antibiotics were not generally recommended for patients with symptoms of less than 10 days’ duration. Additionally, CDS included options to justify use of antibiotics in the case of prolonged symptoms, “double worsening” of symptoms, or abnormal physical exam findings. CDS could be bypassed by providers, but required a response or cancellation to proceed with the order. In addition, CDS directed providers to an order set to facilitate the prescribing of amoxicillin or amoxicillin clavulanate if they had ordered a nonrecommended antibiotic for acute sinusitis.

Provider education included a recorded online presentation and 2 webinars delivered in the middle of the staggered rollout (December). The intervention was created after previous research showed the need for improved stewardship and no temporal trends in improvement in the years leading up to our study period.¹³

All acute sinusitis encounters (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] code 461.x) taking place in primary care settings were included. We excluded encounters with specialists, follow-up visits, and patients with chronic diseases compromising their immune response (chronic liver disease, ICD-9-CM code 571; end-stage renal disease, ICD-9-CM code 585.6; congestive heart failure, ICD-9-CM code 428; immune disorders, ICD-9-CM code 279; malignant neoplasms, ICD-9-CM codes 140-165, 170-176, 179-209, and 235-239; and common rheumatologic disorders, ICD-9-CM codes 714, 710, 555.9, and 556). To ensure completeness of data, we limited our sample to Kaiser health plan members.

The primary outcome was prescription of an antibiotic, classified by the generic product identifier. Secondary outcomes included prescription of an antibiotic for diagnoses with similar symptoms as acute sinusitis (acute nasopharyngitis, ICD-9-CM code 460; nonstreptococcal pharyngitis, ICD-9-CM code 462; acute laryngitis, ICD-9-CM code 464; acute laryngopharyngitis, ICD-9 code 465; acute upper respiratory tract infections of other multiple sites, ICD-9-CM code 465.8; acute upper respiratory tract infections not otherwise specified, ICD-9-CM code 465.9; acute bronchitis, ICD-9 code 466; influenza with respiratory manifestations, ICD-9-CM code 487.1; and bronchitis not specified as acute or chronic, ICD-9-CM code 490)⁷ and concordance with current IDSA¹⁶ guidelines.

Concordance estimates were done based on structured chart review from a random sample of 50 charts with some from each of the 6 medical service areas. The structured chart review was performed by a trained member of the research team who followed a standardized protocol to limit bias.^23,24 This random sample was selected from encounters in which antibiotics were prescribed post intervention. We used a standardized data abstraction form and manual previously developed to identify guideline-concordant antibiotic use for similar acute sinusitis encounters, which is proven to have high interrater reliability (93% agreement, kappa = .86).¹³ This previous research provided estimates of concordance rates (32%) for acute sinusitis encounters receiving antibiotics for comparison purposes. We used results from our chart review and applied the sampling fraction to estimate the variance of the proportion of encounters receiving antibiotics concordant with recommendations post intervention. This estimate was calculated via standard formulas routinely used for survey sampling research,²⁵ then applied to the entire sample to construct 95% confidence limits.

We used multiple logistic regression analyses to examine the main effect of CDS on antibiotic prescribing. To account for the staggered implementation of CDS, a dichotomous variable was created to indicate whether the encounter was pre- or post CDS. Another indicator variable was included to assess the impact of education amidst the rollout of CDS. A facility variable was included to identify differences in effect among the service areas. The analysis adjusted for variables that were statistically associated (P <.05) with receipt of antibiotics.

The final model used the provider as a random intercept, the education indicator as a fixed effect, and indicator variables modeled as fixed effects to assess the separate effects of CDS and provider education.

Secondary analyses examined the interaction between CDS and provider groups of varying years of experience. We categorized the years of provider experience into 4 groups (<3 years, 3-9 years, 10-20 years, and >20 years), using the largest group of providers (10-20 years) as the reference for comparison. We similarly examined the interaction between CDS and each of the medical service areas. Within these groups of providers and service areas we also compared the odds of receiving antibiotics pre- and post intervention.

We performed additional analyses to understand changes in sinusitis encounter diagnoses compared with similar upper respiratory infections (URIs) and to evaluate if these encounters resulted in changes to antibiotic prescribing. We report the number of acute sinusitis encounters and other similar URIs, as well as the proportion resulting in antibiotics during the study period (September 2014-April 2015) compared with the same months in the 2 prior years, 2012 to 2013 and 2013 to 2014. Similarly, we report these diagnoses and the percent receiving antibiotics stratified pre-post CDA implementation and pre-post provider education during the study period, as well as the type of antibiotic used pre-post CDS for acute sinusitis encounters.

RESULTS

The study sample included 21,949 initial acute sinusitis encounters for adults (Figure 1). A similar number of encounters were included before and after the implementation of CDS (10,491 pre- and 11,458 post intervention), and more patients were seen at primary care clinics (76%) than in urgent care settings (Table 1).

Primary Outcome

The CDS intervention significantly reduced antibiotic use for acute sinusitis encounters (adjusted odds ratio [AOR], 0.78; 95% CI, 0.71-0.87) (Table 2), although it demonstrated only a 2% absolute reduction. Provider education, assessed as a time effect during December, showed significant decreases in antibiotic prescribing (aOR, 0.51; 95% CI, 0.46-0.57) (Table 2), but these effects were not sustained over the study period (Figure 2).

Secondary Outcomes

The odds of receiving antibiotics increased with additional years of provider experience (Table 2), although there was not a statistically significant interaction between years of provider experience and CDS (P = .19) (Table 3). There were significant differences in the effect of CDS and antibiotic prescribing based on the interaction with each medical service area (P <.001).

The overall number of encounters with acute sinusitis diagnoses decreased compared with the same temporal period in the 2 prior years and represented a smaller proportion of acute URI diagnoses (8.1% vs 11.7% and 11.3%). The proportion of acute sinusitis encounters, compared with all similar URI diagnoses, decreased in the posteducation (7.5% vs 9.3%) and the post-CDS (7.1% vs 9.2%) periods compared with pre-intervention. The overall proportion of acute URI encounters receiving antibiotics decreased during the study period (27.1%) compared with prior years (31.2% and 30.7%), but this does not appear to be attributable to the intervention. Pre-post stratification showed there may have been some diagnosis shifting where antibiotics were still prescribed, as CDS (3.3% increase) and education (2% increase) showed a small increase in antibiotic prescribing for other URIs before and after implementation during the study period (eAppendices 1 and 2 [eAppendices available at ajmc.com]). The types of antibiotics used pre-post intervention did not differ significantly (eAppendix 3).

Lastly, a structured review of acute sinusitis encounters for patients who received antibiotics found that 46% (95% CI, 32%-60%) were guideline concordant. This was a 14% absolute improvement from a structured review of acute sinusitis encounters from the same primary care clinics in the pre-implementation period (32%; 95% CI, 19%-45%).¹³

DISCUSSION

We performed an effectiveness study of the ability of CDS and provider education to decrease the use of antibiotics for acute sinusitis encounters. Adjusted comparisons showed a 22% improvement in the odds of prescribing antibiotics after the intervention, but the absolute reduction was small (2%). After CDS and education, the number of encounters with an acute sinusitis diagnosis decreased substantially. Lastly, we found that CDS had variable effectiveness based on the medical service area.

This study makes several important contributions to the existing body of literature describing the effectiveness of CDS. First, we confirmed that CDS is effective in real-world clinical settings, but showed that the magnitude of benefit may be less than that observed in highly controlled studies of efficacy.²⁶

Second, we observed a pattern of diagnosis-shifting as a result of education and CDS implementation. This may be an example of documented CDS “workarounds,” and future studies and quality-improvement efforts should account for changes in diagnosis patterns.²⁷ It is possible that providers substituted a diagnosis other than acute sinusitis once they were made aware that antibiotics were not indicated for most patients with sinusitis. Understanding how CDS modifies diagnosis patterns warrants future investigation, although, based on our results, these changes did not result in an overall increase in the use of antibiotics for all URI diagnoses.

Third, we observed increasing odds of antibiotic prescribing correlated with increasing years of provider experience, but this did not result in varying CDS impact. CDS did vary in effectiveness, however, based on the medical service area of the encounter. This may be a result of the culture and context within particular settings that promotes success. This finding supports other reports that have shown these less tangible factors to be an important aspect of the success of interventions.^28,29 What is clear is that clinical leaders, policy makers, and researchers should account for the variable effectiveness of CDS depending on the local medical center characteristics.

Despite overall results demonstrating improvements in antibiotic stewardship and guideline-concordant prescribing, we observed a diminution in effect over time. Much of the overall effectiveness of the 2-component intervention may be attributed to the acute drop in prescribing in December associated with provider education, rather than the CDS intervention. This temporal improvement was not sustained, and by the end of the study period rebounded to near the prior baseline. This reinforces that sustained improvement in clinical practice is difficult to achieve.

Our study confirms results from recent efficacy trials showing the benefit of “accountable justification”³⁰ with automated CDS during outpatient encounters, although in a slightly different approach. The limited effectiveness of our results may be partially due to limited “accountability,” as our CDS required a clicked response, but did not require the provider to document in the medical record the reason for use of antibiotics. Additionally, we looked only at acute sinusitis encounters, a condition for which antibiotics are generally not indicated, except under special circumstances, instead of viral diagnoses for which antibiotics are always contraindicated. Based on our relatively large aOR and comparatively small absolute effect of CDS, we hypothesize that patients with shorter symptoms or less severe symptoms were those changed to a different diagnosis and not given antibiotics. This is also based on our chart review, which showed improvements in guideline-concordant use of antibiotics. This may explain the increase in other similar URI diagnoses without a rise in antibiotics for those conditions. Therefore, it is reasonable to predict that the effect of our intervention on overall use of antibiotics may be underestimated by the absolute 2% decrease in the proportion of encounters receiving antibiotics for acute sinusitis.

CONCLUSIONS

Provider education and integrated CDS reduced antibiotic use for acute sinusitis encounters and substantially changed patterns of diagnosis for acute URIs. The benefits of education were brief and did not persist through the study period, and CDS effectiveness varied by medical center.

Acknowledgments

The authors thank Ellen J. Rippberger and Lorena Perez-Reynoso for their administrative assistance and organization throughout this study. They also recognize Jason Doctor, PhD, and Daniella Meeker, PhD, for reviewing and providing feedback about the manuscript. Author Affiliations: Department of Research and Evaluation, Kaiser Permanente Southern California (ALS, YRH, ES, MKG), Pasadena, CA; Department of Emergency Medicine, Los Angeles Medical Center, Kaiser Permanente Southern California (ALS), Los Angeles, CA; Department of Emergency Medicine, San Diego Medical Center, Kaiser Permanente Southern California (RC), San Diego, CA; Department of Emergency Medicine, Kaiser Permanente Northwest (RPR), Portland, OR; Medical Director of Quality and Clinical Analysis, Kaiser Permanente Southern California (MHK), Pasadena, CA.

Source of Funding: Internal funding from the KPSC Care Improvement Research Team (CIRT) supported this project.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (ALS, ES, RC, MHK); acquisition of data (YRH); analysis and interpretation of data (ALS, YRH, ES, RC, RPR, MHK, MKG); drafting of the manuscript (ALS, ES, RPR); critical revision of the manuscript for important intellectual content (ALS, ES, RPR, MHK, MKG); statistical analysis (YRH, ES); administrative, technical, or logistic support (RC); and supervision (ALS).

Address Correspondence to: Adam L. Sharp, MD, MS, Kaiser Permanente Southern California, 100 S Los Robles Ave, 2nd Fl, Pasadena, CA 91101. E-mail: adam.l.sharp@kp.org. REFERENCES

1. Society for Healthcare Epidemiology of America; Infectious Diseases Society of America; Pediatric Infectious Diseases Society. Policy statement on antimicrobial stewardship by the Society for Healthcare Epidemiology of America (SHEA), the Infectious Diseases Society of America (IDSA), and the Pediatric Infectious Diseases Society (PIDS). Infect Control Hosp Epidemiol. 2012;33(4):322-327. doi: 10.1086/665010.

2. Fishman N. Antimicrobial stewardship. Am J Med. 2006;119(6 suppl 1):S53-S61; discussion S62-S70. doi: 10.1016/j.amjmed.2006.04.003.

3. Roberts RR, Hota B, Ahmad I, et al. Hospital and societal costs of antimicrobial-resistant infections in a Chicago teaching hospital: implications for antibiotic stewardship. Clin Infect Dis. 2009;49(8):1175-1184. doi: 10.1086/605630.

4. Carlet J, Collignon P, Goldmann D, et al. Society’s failure to protect a precious resource: antibiotics. Lancet. 2011;378(9788):369-371. doi: 10.1016/S0140-6736(11)60401-7.

5. Wagner B, Filice GA, Drekonja D, et al. Antimicrobial stewardship programs in inpatient hospital settings: a systematic review. Infect Control Hosp Epidemiol. 2014;35(10):1209-1228. doi: 10.1086/678057.

6. Drekonja DM, Filice GA, Greer N, et al. Antimicrobial stewardship programs in outpatient settings: a systematic review. Infect Control Hosp Epidemiol. 2015;36(2):142-152. doi: 10.1017/ice.2014.41.

7. Meeker D, Knight TK, Friedberg MW, et al. Nudging guideline-concordant antibiotic prescribing: a randomized clinical trial. JAMA Intern Med. 2014;174(3):425-431. doi: 10.1001/jamainternmed.2013.14191.

8. Rosenfeld RM, Andes D, Bhattacharyya N, et al. Clinical practice guideline: adult sinusitis. Otolaryngol Head Neck Surg. 2007;137(suppl 3):S1-S31. doi: 10.1016/j.otohns.2007.06.726.

9. Anand VK. Epidemiology and economic impact of rhinosinusitis. Ann Otol Rhinol Laryngol Suppl. 2004;193:3-5.

10. Gill JM, Fleischut P, Haas S, Pellini B, Crawford A, Nash DB. Use of antibiotics for adult upper respiratory infections in outpatient settings: a national ambulatory network study. Fam Med. 2006;38(5):349-354.

11. Jørgensen LC, Friis Christensen S, Cordoba Currea G, Llor C, Bjerrum L. Antibiotic prescribing in patients with acute rhinosinusitis is not in agreement with European recommendations. Scand J Prim Health Care. 2013;31(2):101-105. doi: 10.3109/02813432.2013.788270.

12. Rosenberg A, Agiro A, Gottlieb M, et al. Early trends among seven recommendations from the Choosing Wisely campaign. JAMA Intern Med. 2015;175(12):1913-1920. doi: 10.1001/jamainternmed.2015.5441.

13. Sharp AL, Klau MH, Keschner D, et al. Low-value care for acute sinusitis encounters: who’s choosing wisely? Am J Manag Care. 2015;21(7):479-485.

14. Fairlie T, Shapiro DJ, Hersh AL, Hicks LA. National trends in visit rates and antibiotic prescribing for adults with acute sinusitis. Arch Intern Med. 2012;172(19):1513-1514. doi: 10.1001/archinternmed.2012.4089.

15. Aring AM, Chan MM. Acute rhinosinusitis in adults. Am Fam Physician. 2011;83(9):1057-1063.

16. Chow AW, Benninger MS, Brook I, et al; Infectious Diseases Society of America. IDSA clinical practice guideline for acute bacterial rhinosinusitis in children and adults. Clin Infect Dis. 2012;54(8):e72-e112. doi: 10.1093/cid/cir1043.

17. Lemiengre MB, van Driel ML, Merenstein D, Young J, De Sutter AI. Antibiotics for clinically diagnosed acute rhinosinusitis in adults. Cochrane Database Syst Rev. 2012;10:CD006089. doi: 10.1002/14651858.CD006089.pub4.

18. Glasgow RE, Magid DJ, Beck A, Ritzwoller D, Estabrooks PA. Practical clinical trials for translating research to practice: design and measurement recommendations. Med Care. 2005;43(6):551-557.

19. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. doi: 10.1136/bmj.38398.500764.8F.

20. Westphal JF, Jehl F, Javelot H, Nonnenmacher C. Enhanced physician adherence to antibiotic use guidelines through increased availability of guidelines at the time of drug ordering in hospital setting. Pharmacoepidemiol Drug Saf. 2011;20(2):162-168. doi: 10.1002/pds.2078.

21. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 2006;6:54. doi: 10.1186/1471-2288-6-54.

22. Munoz-Plaza CE, Parry C, Hahn EE, et al. Integrating qualitative research methods into care improvement efforts within a learning health system: addressing antibiotic overuse. Health Res Policy Syst. 2016;14(1):63. doi: 10.1186/s12961-016-0122-3.

23. Vassar M, Holzmann M. The retrospective chart review: important methodological considerations. J Educ Eval Health Prof. 2013;10:12. doi: 10.3352/jeehp.2013.10.12.

24. Worster A, Bledsoe RD, Cleve P, Fernandes CM, Upadhye S, Eva K. Reassessing the methods of medical record review studies in emergency medicine research. Ann Emerg Med. 2005;45(4):448-451. doi: 10.1016/j.annemergmed.2004.11.021.

25. Levy PS, Lemeshow S. Sampling of Populations: Methods and Applications. 4th ed. Hoboken, NJ: Wiley; 2008.

26. Gonzales R, Anderer T, McCulloch CE, et al. A cluster randomized trial of decision support strategies for reducing antibiotic use in acute bronchitis. JAMA Intern Med. 2013;173(4):267-273. doi: 10.1001/jamainternmed.2013.1589.

27. Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc. 2007;14(4):415-423. doi: 10.1197/jamia.M2373.

28. Baker R, Camosso-Stefinovic J, Gillies C, et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2010;(3):CD005470. doi: 10.1002/14651858.CD005470.pub2.

29. Flottorp SA, Oxman AD, Krause J, et al. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35. doi: 10.1186/1748-5908-8-35.

30. Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: a randomized clinical trial. JAMA. 2016;315(6):562-570. doi: 10.1001/jama.2016.0275.

Download PDF: Improving Antibiotic Stewardship: A Stepped-Wedge Cluster Randomized Trial