Choosing Wisely Interventions to Reduce Antibiotic Overuse in the Safety Net

ABSTRACT

Objectives: Physician pay-for-performance (P4P) programs frequently target inappropriate antibiotics. Yet little is known about P4P programs’ effects on antibiotic prescribing among safety-net populations at risk for unintended harms from reducing care. We evaluated effects of P4P-motivated interventions to reduce antibiotic prescriptions for safety-net patients with acute respiratory tract infections (ARTIs).

Study Design: Interrupted time series.

Methods: A nonrandomized intervention (5/28/2015-2/1/2018) was conducted at 2 large academic safety-net hospitals: Los Angeles County+University of Southern California (LAC+USC) and Olive View-UCLA (OV-UCLA). In response to California’s 2016 P4P program to reduce antibiotics for acute bronchitis, 5 staggered Choosing Wisely–based interventions were launched in combination: audit and feedback, clinician education, suggested alternatives, procalcitonin, and public commitment. We also assessed 5 unintended effects: reductions in Healthcare Effectiveness Data and Information Set (HEDIS)–appropriate prescribing, diagnosis shifting, substituting antibiotics with steroids, increasing antibiotics for ARTIs not penalized by the P4P program, and inappropriate withholding of antibiotics.

Results: Among 3583 consecutive patients with ARTIs, mean antibiotic prescribing rates for ARTIs decreased from 35.9% to 22.9% (odds ratio [OR], 0.60; 95% CI, 0.39-0.93) at LAC+USC and from 48.7% to 27.3% (OR, 0.81; 95% CI, 0.70-0.93) at OV-UCLA after the intervention. HEDIS-inappropriate prescribing rates decreased from 28.9% to 19.7% (OR, 0.69; 95% CI, 0.39-1.21) at LAC+USC and from 40.9% to 12.5% (OR, 0.72; 95% CI, 0.59-0.88) at OV-UCLA. There was no evidence of unintended consequences.

Conclusions: These real-world multicomponent interventions responding to P4P incentives were associated with substantial reductions in antibiotic prescriptions for ARTIs in 2 safety-net health systems without unintended harms.

Am J Manag Care. 2023;29(10):488-496. https://doi.org/10.37765/ajmc.2023.89367

Takeaway Points

• Quality improvement initiatives, particularly pay-for-performance models, have a track record of translating poorly to safety-net systems and unintentionally harming the medically underserved populations for which they care.
• In a real-world environment within 2 hospitals in the nation’s second largest public safety-net health care system, we observed a 17.2% absolute reduction in total antibiotic prescribing for viral respiratory infections after multicomponent behavioral interventions were implemented in response to a large pay-for-performance initiative based on Choosing Wisely guidelines.
• Our findings suggest that both pay-for-performance programs and Choosing Wisely–based interventions can successfully and safely reduce antibiotic overuse among disadvantaged patients.

Inappropriate antibiotics account for 25% to 50% of all US antibiotic prescriptions^1-5 and are associated with substantial harms, such as antibiotic-resistant organisms, which account for more than 35,000 excess deaths in the United States annually.⁶ Three-quarters of the 266 million outpatient antibiotic prescriptions given to Americans each year are to treat acute respiratory tract infections (ARTIs), for which antibiotics are indicated in less than half of cases, making ARTIs the single largest area of opportunity for reducing inappropriate outpatient antibiotics and their associated harms.^4,7-9

Many interventions have successfully reduced inappropriate antibiotic prescribing, but few of these initiatives have been evaluated within safety-net systems.^10-19 This is concerning given that efficacy studies often translate poorly to safety-net systems,^20-22 and quality improvement (QI) and pay-for-performance (P4P) programs (eg, 30-day readmissions) have a history of unintentionally penalizing safety-net systems and worsening health care disparities.^23-25 It is therefore crucial to understand the effects of complex QI initiatives within safety-net systems to prevent unintended harms from befalling disadvantaged and medically underserved patients.²⁶

In 2016, California Medicaid (Medi-Cal) launched a 5-year, $7.5 billion statewide P4P QI initiative called Public Hospital Redesign and Incentives in Medi-Cal (PRIME).²⁷ PRIME was one of the largest US statewide P4P programs ever implemented,²⁸ and it included an aim to reduce inappropriate antibiotic prescriptions for acute bronchitis.²⁹ We evaluated the intended and unintended effects of real-world multicomponent interventions to reduce antibiotic prescribing that arose in response to this large P4P program.

METHODS

Study Design

This study compared rates of prescribing antibiotics for ARTIs after behavioral interventions at 2 major academic safety-net medical centers in the nation’s second largest public safety-net health care system: Los Angeles County+University of Southern California (LAC+USC) and Olive View-UCLA (OV-UCLA).³⁰ The original design was a quasi-experiment to study the effects of multicomponent interventions at LAC+USC compared with OV-UCLA (control), where to the authors’ knowledge no interventions were planned at the time of study design. However, after data collection, the authors learned that PRIME had unexpectedly (and independently of the study team) led OV-UCLA leadership to implement its own interventions to reduce antibiotic prescriptions for acute bronchitis. Given that OV-UCLA was no longer a suitable control site, the study protocol was amended to evaluate the effects of these multicomponent interventions within each site rather than between sites.

The study population and analysis protocol (eAppendix I [eAppendices available at ajmc.com]) were preregistered on ClinicalTrials.gov (NCT03464279).

Settings

The intervention occurred in the urgent care centers of LAC+USC (600-bed hospital in a high-density urban area; urgent care with 3 physicians, 10 nurse practitioners, and 2 physician assistants) and OV-UCLA (375-bed hospital with an urban and large rural catchment area; urgent care with ~10 physicians, 1 nurse practitioner, and many rotating trainees). Although LAC+USC and OV-UCLA are both part of the LAC Department of Health Services (LAC-DHS) health system, they function as separate, independent medical centers with their own leadership. Almost half of all LAC-DHS patients speak a primary language other than English, 65% identify as Hispanic/Latinx, 65% are Medi-Cal enrollees, and 23% are uninsured.³⁰

Enrollment

The interventions did not require enrollment (Figure 1); they were applied to all urgent care clinicians without direct incentives (financial or otherwise).

Outcome Measures

The primary study outcome was the rate of inappropriate antibiotic prescriptions during encounters for any ARTI that typically does not require antibiotics (acute bronchitis, nonacute bronchitis, acute bronchiolitis, acute nasopharyngitis, chronic sinusitis, nonstrep pharyngitis, acute upper respiratory infection, influenza, viral pneumonia, and cough; eAppendix Table 1 in eAppendix II). This prespecified outcome was intentionally broader than that of the P4P program, which measured only antibiotics for acute bronchitis (International Statistical Classification of Diseases, Tenth Revision [ICD-10] code J20), given that we reasonably expected the interventions to affect antibiotic prescribing for many other nonbacterial ARTIs (eg, J21, acute bronchiolitis) and to maximize power.

The Healthcare Effectiveness Data and Information Set (HEDIS) criteria were used to stratify antibiotic prescriptions as appropriate (≥ 1 concomitant diagnosis or comorbidity to justify the antibiotic) or inappropriate (no concomitant diagnoses/comorbidities). Concomitant diagnoses that justified antibiotics were active bacterial infections (eg, pneumonia) or comorbid conditions (eg, HIV, chronic obstructive pulmonary disease ) defined by the HEDIS “Competing Diagnosis” and “Comorbid Conditions” value sets.^31,32 In addition to HEDIS-inappropriate prescriptions, we also defined potentially inappropriate antibiotic prescriptions more broadly as any prescriptions for the aforementioned ARTIs, which are typically viral.

These secondary outcomes assessed for unintended consequences: reductions in HEDIS-appropriate prescribing for bacterial respiratory infections (eg, pneumonia), increased coding for antibiotic-appropriate conditions (ie, diagnosis shifting),³³ increased steroids for ARTIs or antibiotics for ARTIs not targeted by PRIME (suggesting a substitution of antibiotics for steroids or shifting of antibiotics to similar conditions), and inappropriate withholding of antibiotics upon medical chart review.

Data

Prescribing data, patient and encounter information, and ICD, Ninth Revision and ICD-10 codes were extracted from the electronic health record (EHR) data repositories of both sites and transferred to UCLA for analysis. Urgent care encounters with a primary ICD code for an eligible ARTI occurring May 28, 2015, to February 1, 2018, were included. Incidentally, a concurrent EHR upgrade led to missing age, gender, race, and ethnicity data at LAC+USC from March 1, 2017, to May 31, 2017. Age data were missing from OV-UCLA, but because the mean age of the chart review sample from LAC+USC was concordant with the mean age of the full LAC+USC population (mean [SD] age, 53 [15] years vs 51 [16] years), the mean age of the chart review sample from OV-UCLA (49 [15] years) was used, which is also consistent with publicly available sources.³⁴

Gender and race/ethnicity information were missing in 6% and 10%, respectively, of the population; these patients were included in the analysis.

Aggregated, deidentified data may be provided upon reasonable request to the corresponding author and with express written permission from LAC-DHS.

Interventions

PRIME was a 5-year, $7.5 billion P4P program launched in California in 2016 consisting of 18 QI initiatives.²⁷ One project aimed to reduce antibiotic prescriptions for acute bronchitis (bronchitis encounters with antibiotic prescriptions/all encounters for acute bronchitis; eAppendix I) in accordance with the Choosing Wisely campaign.³⁵ To receive payments, a site had to perform higher than the 25th percentile of all sites and exhibit at least a 10% year-over-year gap closure between current performance and the 90th percentile of performance (or maintain performance at the 90th percentile or higher if a site had already achieved this benchmark).³⁶ Payments were based upon the performance percentile and the magnitude of gap closure, and they were worth hundreds of millions of dollars in federal/state funds.²⁹

With the exception of the public commitment intervention below (part of a broader Choosing Wisely effort to reduce medical overuse across LAC-DHS^37,38), all interventions were hospital initiated (independently of the study team) in response to the PRIME incentive payments. All interventions remained in place throughout the entire study period once implemented.

Clinician case-audit feedback³⁹ (LAC+USC, 11/21/2016; OV-UCLA, 3/28/2017) was an email and in-person intervention and the crux of the multicomponent interventions. Urgent care directors delivered one-on-one case-specific feedback to clinicians numerous times throughout the intervention when antibiotic prescribing for acute bronchitis was at risk of missing the PRIME benchmark reduction. Clinic-level (not clinician-level) performance reports on PRIME across all of LAC-DHS were also disseminated to all clinicians by email multiple times throughout the intervention.

Clinician education (LAC+USC, 11/21/2016; OV-UCLA, 11/30/2016) was a physical intervention consisting of regular emails from leadership (both sites), 2 journal clubs (LAC+USC only), and posters/screen savers (OV-UCLA only) based on infographics from the Choosing Wisely campaign to reduce prescribing of antibiotics for ARTIs.³⁵

Suggested alternatives (LAC+USC, 10/28/2016; OV-UCLA, 11/30/2016) was a physical intervention delivered at both sites through CDC “viral prescription pads” given to urgent care clinicians; the pads listed nonantibiotic treatments for ARTIs (eg, acetaminophen).⁴⁰ Given that many patients within the LAC-DHS safety-net system present to urgent care seeking antibiotics,⁴¹ clinicians could write these “prescriptions” to assuage this pressure.

Procalcitonin^42-44 (LAC+USC only, 12/5/2016) was a rapid turnaround laboratory-based intervention. Procalcitonin is a blood marker for bacterial infection, and providers were discouraged from using antibiotics if procalcitonin was less than 0.25 μg/L.

Public commitment⁴⁵ (LAC+USC only, 3/20/2017) was an intervention involving all urgent care clinicians; they signed prominently displayed 48 × 36-in posters on which they pledged to not prescribe unnecessary antibiotics, in accordance with Choosing Wisely guidelines (eAppendix Figure 1 in eAppendix II).¹⁶

Appropriateness

HEDIS appropriateness was determined per the guidelines for “Avoidance of Antibiotic Treatment in Adults with Bronchitis”^31,32 using an e-measure that queried EHR data for (1) urgent care encounters for ARTIs, (2) antibiotics prescribed during those encounters, and (3) the presence of competing/comorbid diagnoses that potentially justified an antibiotic.

An independent internal medicine physician and study coauthor (R.K.L.) blindly reviewed a sample of 101 medical charts, of which the majority (~75%) were randomly selected from antibiotic-present encounters in order to focus on measuring overuse (rather than underuse). Objective guidelines of professional medical societies (eg, Infectious Diseases Society of America) were used as the gold standard for determining treatment appropriateness (eAppendix Table 2 in eAppendix II). Concordance between HEDIS criteria and professional society guidelines was used to calculate sensitivity/specificity of the e-measure. A second general internist and study coauthor (J.N.M.) reconciled cases with unclear adjudication (n = 5).

Analysis

We evaluated the percentage of encounters for ARTIs during which an antibiotic was prescribed from 16 months (LAC+USC) or 12 months (OV-UCLA) prior to any intervention components through 15 months after the first intervention components at both sites.

Given that these pragmatic interventions were implemented in rapid succession, the analysis aggregated all components into a single “intervention” defined by 1 site-specific start date. Unstructured qualitative interviews with the physician leads of both urgent care centers (who were blinded to study results at the time of the interviews) revealed that they intuitively felt that the most substantial component of the interventions was one-on-one clinician case-audit feedback, so the initiation of this component was used as the official intervention start date. One-on-one feedback started on November 21, 2016, at LAC+USC and was among the first elements implemented there, whereas it was among the last rolled out at OV-UCLA on March 28, 2017 (Figure 2 [part A and part B]).

We performed 2 interrupted time series (ITS) analyses (between group and within group) consisting of segmented logistic regressions on the dependent variable (antibiotic prescribing) clustered by patient, to estimate the odds ratios (ORs) of each outcome during each time period. Time periods (Figure 2) were defined as follows: T1, before any intervention implementation; T2, after one-on-one feedback implementation at LAC+USC but not OV-UCLA (although OV-UCLA had other interventions implemented during T2); and T3, after all intervention implementation at both sites. Interactions between time terms and site were used to estimate within-site and between-site differences. Changes in outcomes between time periods were expressed as ratios of ORs (rORs; ie, T2 OR / T1 OR). For example, an rOR of 0.60 indicated that the odds of antibiotic prescribing fell by 40% from one time period to another. Estimated probabilities of each outcome event over time were also displayed (Figure 2) for ease of interpretation.

Both analyses also were used to assess appropriate/inappropriate prescribing and the preregistered unintended consequence of diagnosis shifting. Additional unintended consequences (substituting antibiotics with steroids and increasing antibiotic prescriptions for ARTIs not targeted by PRIME) were added post hoc, so we assessed trends visually and did not include them in the formal ITS analysis.

Two-tailed P values less than .05 and ORs with 95% CIs excluding 1 were considered significant. Analyses were performed using R version 3.6.2 (R Foundation for Statistical Computing).

The UCLA Health and USC Health Sciences institutional review boards approved the QI intervention and waived participant informed consent.

RESULTS

We identified 3583 encounters for ARTIs between the 2 sites (LAC+USC, n = 1692; OV-UCLA, n = 1891), 1380 (38.5%) of which resulted in antibiotic prescriptions. The Table depicts patient characteristics.

Total Antibiotic Prescribing

Both sites experienced absolute decreases in unadjusted mean rates of total antibiotic prescribing for ARTIs from their respective intervention starts to study end: –13.0% at LAC+USC and –21.4% at OV-UCLA (Figure 3). Key odds of prescribing all types of antibiotics are summarized in Figure 4.

In the ITS within-group analysis, intervention implementation (starting with one-on-one case-audit feedback) at LAC+USC was associated with a significant decrease in the odds of prescribing antibiotics (rOR, 0.60; 95% CI, 0.39-0.93; P = .022) (Figure 2 [A] T2 vs T1 and Figure 4). At OV-UCLA, where one-on-one feedback was the last intervention component implemented, the reduction in the odds of prescribing antibiotics was significant when compared with the time period before any intervention components (rOR, 0.81; 95% CI, 0.70-0.93; P = .003) (Figure 2 [A] T3 vs T1 and Figure 4) but not when compared with the time period immediately preceding one-on-one feedback when some intervention components were already in place (rOR, 0.89; 95% CI, 0.57-1.38; P = .60) (Figure 2 [A] T3 vs T2 and Figure 4).

After one-on-one feedback had been implemented at LAC+USC but not OV-UCLA (ie, T2 vs T1), antibiotic prescribing rates decreased from 35.9% to 22.9% at LAC+USC and from 48.7% to 45.8% at OV-UCLA. The preregistered between-group ITS analysis revealed that this reduction was not significant between sites (ratio of rORs, 0.66; 95% CI, 0.35-1.24; P = .20). The within-site ORs for each time period and results of all logistic regressions are available in eAppendix Table 3 and eAppendix Table 4 in eAppendix II, respectively.

HEDIS-Inappropriate Prescribing

Within OV-UCLA, the odds of inappropriately prescribing antibiotics decreased compared with the period before any interventions were implemented (rOR, 0.72; 95% CI, 0.59-0.88; P = .001) (Figure 2 [B] T3 vs T1 and Figure 4). There were no statistically significant changes in these odds at LAC+USC (rOR, 0.69; 95% CI, 0.39-1.21; P = .20) (Figure 2 [B] T2 vs T1 and Figure 4).

Unintended Consequences

Overall, analyses did not suggest any measurable unintended consequences.

HEDIS-appropriate prescribing.The estimated probabilities (Figure 2 [C]), unadjusted rates (Figure 3), and adjusted odds (Figure 4 and eAppendix Table 4) of appropriately prescribing antibiotics for ARTIs did not significantly decline at either site after intervention implementation.

Diagnosis shifting/gaming. In the preregistered analysis of unintended consequences (eAppendix Table 4), the probability of coding for competing diagnoses (Figure 2 [D]) increased before the interventions (T1 at LAC+USC, T2 at OV-UCLA) but decreased immediately after intervention implementation at both sites (T2 at LAC+USC, T3 at OV-UCLA). These probabilities began to increase in T3 at LAC+USC but did not exceed preintervention probabilities. The probability of coding for nonacute bronchitis (Figure 2 [E]) persistently declined after the interventions.

Antibiotic substitution and shifting. There was no sustained increase in prednisone prescribing for ARTIs to suggest a substitution effect, or evidence that antibiotic prescribing was shifted to other ARTIs not targeted by PRIME (eAppendix Figure 2 [A and B] in eAppendix II).

Inappropriately withholding antibiotics. Among 101 manually reviewed encounters for ARTIs, 78 resulted in antibiotic prescriptions (77%). All cases of antibiotic withholding (n = 23) were deemed appropriate, suggesting no inappropriate withholding of antibiotics. The electronic measure exhibited a sensitivity of 85% and specificity of 76% comparing HEDIS appropriateness to professional society guidelines (medical chart review). The reasons for discordance between the e-measure and chart review are outlined in eAppendix Table 5 in eAppendix II. Four of 35 (11%) HEDIS-appropriate antibiotic prescriptions conflicted with professional society guidelines for antibiotic use.

DISCUSSION

These multicomponent interventions at 2 safety-net hospitals were associated with large and statistically significant declines in both the rates and the odds of total antibiotic prescribing for ARTIs. Importantly, these reductions were not associated with any of the 5 potential unintended consequences assessed: (1) reductions in HEDIS-appropriate antibiotics for ARTIs, (2) diagnosis shifting/gaming, (3) substituting antibiotics with steroids, (4) increases in antibiotics for ARTIs not targeted by the P4P measure, and (5) inappropriate withholding of antibiotics.

The fact that reductions in HEDIS-inappropriate prescribing were significant only in 1 analysis at OV-UCLA is likely a reflection of limitations in using claims-based quality metrics. Chart review revealed that up to 11% of HEDIS-appropriate antibiotics were discordant with professional society guidelines. For example, prescribing antibiotics for acute bronchitis in someone with well-controlled HIV is generally inappropriate,^46,47 but the HEDIS-inappropriate metric excludes these cases because of the presence of a diagnostic code for HIV. Reductions in these types of inappropriate prescriptions would be captured by the total antibiotic prescribing metric but not the HEDIS-inappropriate metric, so we believe that decreases in total antibiotic prescribing at both sites capture true reductions in inappropriate prescribing. This interpretation is supported by the fact that HEDIS-appropriate prescribing post intervention held steady at both sites. In the absence of chart-review evidence of inappropriate antibiotic withholding or any other unintended consequences (which is a concern of many P4P initiatives),^23-25 our findings suggest that both the P4P program and ensuing Choosing Wisely–based interventions were successful in safely reducing antibiotic prescriptions for ARTIs within real-word safety-net settings.

The results of these interventions that appeared temporally associated with audit and feedback compare favorably with those of other interventions targeting antibiotic prescriptions for ARTIs, of which as few as 35% report clinically relevant reductions in antibiotic prescriptions.⁴⁸ The effect sizes seen in our results (17% absolute reduction) were at the upper limit of those reported in successful multicomponent interventions^{10,13,14,33,48} and exceeded what are generally seen for single-component education/feedback interventions (mean absolute reduction, 7%).⁴⁴ The large effect size exhibited here may be due to a combination of the magnitude of the incentive payments at stake (up to hundreds of millions of dollars over 5 years)29 and the use of multicomponent interventions, which are generally more effective than single-component interventions.^44,48 The costs of implementing such interventions may be offset through P4P financial incentive programs.

These findings directly address the paucity of studies testing effectiveness of initiatives to reduce antibiotic prescribing for ARTIs among socioeconomically disadvantaged safety-net populations,^44,48 which is important given historical problems with the generalizability^20,21 and fairness of applying QI programs to safety-net systems.^22-25

Chart review also revealed that the HEDIS measures of appropriateness exhibited reduced real-world specificity because of providers failing to code correctly (eg, not coding for pneumonia when all diagnostic criteria were met), which highlights the shortcomings of using administrative claims data in QI outcomes within safety-net settings.⁴⁹

Limitations

This study had several limitations. First, the study protocol had to be amended to include a within-group ITS analysis (see Methods). Although this analysis was post hoc, ITS is one of the most well-established tools to evaluate within-site changes after an intervention,⁵⁰ so it would have been the natural prespecified analysis had the authors known about the OV-UCLA intervention at the time of study design. Second, this pragmatic study assessed the aggregate effects of the multicomponent interventions rather than the precise effects of each component and was not designed to report data on intervention fidelity. Third, the HEDIS e-measures did not use clinical data to adjust for illness severity when determining appropriateness and were subject to diagnostic coding errors. Fourth, the unintended consequences analyses did not investigate clinical outcomes such as hospital admissions, which should be studied in future research.

CONCLUSIONS

These real-world multicomponent interventions were associated with substantial reductions in antibiotic prescriptions for ARTIs within a large safety-net system. Although HEDIS-inappropriate prescribing decreased at only 1 site, these interventions did not result in any observed unintended consequences—suggesting that P4P initiatives can motivate interventions to safely reduce antibiotic prescribing within safety-net settings.

Author Affiliations: Division of General Internal Medicine & Health Services Research, Department of Internal Medicine, David Geffen School of Medicine at UCLA (RKL, CAS, JNM), Los Angeles, CA; Geriatric Research Education and Clinical Center, Greater Los Angeles VA Healthcare System (CAS), Los Angeles, CA; University of Southern California Department of Emergency Medicine, LAC+USC Medical Center (RT-S), Los Angeles, CA; NYC Health + Hospitals (EKW), New York, NY; Division of Geriatrics, Department of Internal Medicine, David Geffen School of Medicine at UCLA (CAC), Los Angeles, CA; Department of Medicine Statistics Core, David Geffen School of Medicine at UCLA (SV), Los Angeles, CA; LAC+USC Medical Center (CC, BS), Los Angeles, CA; Olive View-UCLA Medical Center, Department of Internal Medicine, David Geffen School of Medicine at UCLA (OM), Los Angeles, CA; Olive View-UCLA Medical Center, Division of Infectious Diseases, Department of Internal Medicine, David Geffen School of Medicine at UCLA (ACJ), Los Angeles, CA; RAND Health Care, RAND Corporation (JNM), Santa Monica, CA.

Source of Funding: The authors report the following grants from the National Institutes of Health: grant No. R38HL143614 to RKL from the National Heart, Lung, and Blood Institute; grants No. 5K24AG047899-05 and P30AG021684-16 to CAS and 1K76AG064392-01A1 to JNM from the National Institute on Aging; and grants No. UL1TR001881 to CAS and KL2TR001882 to JNM from the National Center for Advancing Translational Sciences.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (RKL, CAS, RT-S, EKW, SV, CC, BS, JNM); acquisition of data (RT-S, EKW, CAC, CC, BS, OM, ACJ); analysis and interpretation of data (RKL, CAS, SV, JNM); drafting of the manuscript (RKL, JNM); critical revision of the manuscript for important intellectual content (RKL, CAS, SV, ACJ, JNM); statistical analysis (RKL, SV); provision of patients or study materials (RT-S, EKW, CAC, CC, BS, OM); obtaining funding (JNM); administrative, technical, or logistic support (RKL, RT-S, EKW, CAC, CC, BS, OM, ACJ, JNM); and supervision (CAS, JNM).

Address Correspondence to: Richard K. Leuchter, MD, Division of General Internal Medicine & Health Services Research, Department of Internal Medicine, David Geffen School of Medicine at UCLA, 1100 Glendon Ave, Ste 726, Los Angeles, CA 90024. Email: rleuchter@mednet.ucla.edu.

REFERENCES

1. Ray MJ, Tallman GB, Bearden DT, Elman MR, McGregor JC. Antibiotic prescribing without documented indication in ambulatory care clinics: national cross sectional study. BMJ. 2019;367:l6461. doi:10.1136/BMJ.L6461

2. Chua KP, Fischer MA, Linder JA. Appropriateness of outpatient antibiotic prescribing among privately insured US patients: ICD-10-CM based cross sectional study. BMJ. 2019;364:k5092. doi:10.1136/BMJ.K5092

3. Hicks LA, Taylor TH Jr, Hunkler RJ. U.S. outpatient antibiotic prescribing, 2010. N Engl J Med. 2013;368(15):1461-1462. doi:10.1056/NEJMc1212055

4. Fleming-Dutra KE, Hersh AL, Shapiro DJ, et al. Prevalence of inappropriate antibiotic prescriptions among US ambulatory care visits, 2010-2011. JAMA. 2016;315(17):1864-1873. doi:10.1001/jama.2016.4151

5. Mafi JN, Wee CC, Davis RB, Landon BE. Comparing use of low-value health care services among U.S. advanced practice clinicians and physicians. Ann Intern Med. 2016;165(4):237-244. doi:10.7326/M15-2152

6. Antibiotic Resistance Threats in the United States, 2019. CDC; 2019. Accessed January 17, 2020. https://www.cdc.gov/drugresistance/pdf/threats-report/2019-ar-threats-report-508.pdf

7. Lee GC, Reveles KR, Attridge RT, et al. Outpatient antibiotic prescribing in the United States: 2000 to 2010. BMC Med. 2014;12:96. doi:10.1186/1741-7015-12-96

8. Shapiro DJ, Hicks LA, Pavia AT, Hersh AL. Antibiotic prescribing for adults in ambulatory care in the USA, 2007-09. J Antimicrob Chemother. 2014;69(1):234-240. doi:10.1093/jac/dkt301

9. Ebell MH, Radke T. Antibiotic use for viral acute respiratory tract infections remains common. Am J Manag Care. 2015;21(10):e567-e575.

10. Persell SD, Doctor JN, Friedberg MW, et al. Behavioral interventions to reduce inappropriate antibiotic prescribing: a randomized pilot trial. BMC Infect Dis. 2016;16(1):373. doi:10.1186/s12879-016-1715-8

11. Linder JA, Schnipper JL, Tsurikova R, et al. Documentation-based clinical decision support to improve antibiotic prescribing for acute respiratory infections in primary care: a cluster randomised controlled trial. Inform Prim Care. 2009;17(4):231-240. doi:10.14236/jhi.v17i4.742

12. McGinn TG, McCullagh L, Kannry J, et al. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Intern Med. 2013;173(17):1584-1591. doi:10.1001/jamainternmed.2013.8980

13. Sharp AL, Hu YR, Shen E, et al. Improving antibiotic stewardship: a stepped-wedge cluster randomized trial. Am J Manag Care. 2017;23(11):e360-e365.

14. Gonzales R, Anderer T, McCulloch CE, et al. A cluster randomized trial of decision support strategies for reducing antibiotic use in acute bronchitis. JAMA Intern Med. 2013;173(4):267-273. doi:10.1001/jamainternmed.2013.1589

15. Linder JA, Schnipper JL, Tsurikova R, et al. Electronic health record feedback to improve antibiotic prescribing for acute respiratory infections. Am J Manag Care. 2010;16(12 suppl HIT):e311-e319.

16. Meeker D, Knight TK, Friedberg MW, et al. Nudging guideline-concordant antibiotic prescribing:
a randomized clinical trial. JAMA Intern Med. 2014;174(3):425-431. doi:10.1001/jamainternmed.2013.14191

17. Samore MH, Bateman K, Alder SC, et al. Clinical decision support and appropriateness of antimicrobial prescribing: a randomized trial. JAMA. 2005;294(18):2305-2314. doi:10.1001/jama.294.18.2305

18. Litvin CB, Ornstein SM, Wessell AM, Nemeth LS, Nietert PJ. Use of an electronic health record clinical decision support tool to improve antibiotic prescribing for acute respiratory infections: the ABX-TRIP study. J Gen Intern Med. 2013;28(6):810-816. doi:10.1007/s11606-012-2267-2

19. Yadav K, Meeker D, Mistry RD, et al. A multifaceted intervention improves prescribing for acute respiratory infection for adults and children in emergency department and urgent care settings. Acad Emerg Med. 2019;26(7):719-731. doi:10.1111/acem.13690

20. Moore SL, Fischer I, Havranek EP. Translating health services research into practice in the safety net. Health Serv Res. 2016;51(1):16-31. doi:10.1111/1475-6773.12341

21. Cristofalo MA. Implementation of health and mental health evidence-based practices in safety net settings. Soc Work Health Care. 2013;52(8):728-740. doi:10.1080/00981389.2013.813003

22. Hefner JL, Hogan TH, Opoku-Agyeman W, Menachemi N. Defining safety net hospitals in the health services research literature: a systematic review and critical appraisal. BMC Health Serv Res. 2021;21(1):278. doi:10.1186/s12913-021-06292-9

23. Chaiyachati KH, Qi M, Werner RM. Changes to racial disparities in readmission rates after Medicare’s Hospital Readmissions Reduction Program within safety-net and non–safety-net hospitals. JAMA Netw Open. 2018;1(7):e184154. doi:10.1001/jamanetworkopen.2018.4154

24. Carey K, Lin MY. Hospital Readmissions Reduction Program: safety-net hospitals show improvement, modifications to penalty formula still needed. Health Aff (Millwood). 2016;35(10):1918-1923.
doi:10.1377/hlthaff.2016.0537

25. Barbash IJ, Kahn JM. Sepsis quality in safety-net hospitals: an analysis of Medicare’s SEP-1 performance measure. J Crit Care. 2019;54:88-93. doi:10.1016/j.jcrc.2019.08.009

26. Mafi JN, Parchman M. Low-value care: an intractable global problem with no quick fix. BMJ Qual Saf. 2018;27(5):333-336. doi:10.1136/BMJQS-2017-007477

27. Public Hospital Redesign & Incentives in Medi-Cal program. California Department of Health Care Services. Updated March 17, 2023. Accessed April 18, 2023. https://www.dhcs.ca.gov/provgovpart/Pages/PRIME.aspx#:~:text=The%20Public%20Hospital%20Redesign%20and,1115%20Bridge%20to%20Reform%20demonstration

28. Heeringa J, Lipson D, Machta R, Lane K, Vogt R. Designing Medicaid Delivery System Reform Incentive Payment demonstrations to reward better performance. Medicaid.gov. January 2018. Accessed June 29, 2021. https://www.medicaid.gov/medicaid/downloads/1115-ib15-508-dsrip-incentive-design.pdf

29. Pourat N, Chen X, Martinez AE, et al. Public Hospital Redesign and Incentive in Medi-Cal (PRIME) final interim evaluation report. Medicare.gov. January 22, 2020. Accessed June 23, 2021. https://www.medicaid.gov/medicaid/section-1115-demonstrations/downloads/ca-medi-cal-2020-final-interim-eval-rpt-20200122.pdf

30. Department of Health Services (DHS) fact sheet. Los Angeles County Department of Health. Accessed June 23, 2021. http://file.lacounty.gov/SDSInter/dhs/1060051_DHSFactSheetFY19-20-FINAL.pdf

31. Quality ID #116 (NQF 0058): avoidance of antibiotic treatment in adults with acute bronchitis. CMS. December 11, 2017. Accessed November 16, 2021. https://qpp.cms.gov/docs/QPP_quality_measure_specifications/Claims-Registry-Measures/2018_Measure_116_Registry.pdf

32. HEDIS measures and technical resources. National Committee for Quality Assurance. Accessed March 23, 2021. https://www.ncqa.org/hedis/measures/

33. Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices a randomized clinical trial. JAMA. 2016;315(6):562-570. doi:10.1001/jama.2016.0275

34. Dedication to the underserved. UCLA-Olive View Internal Medicine Residency. Accessed February 23, 2022. https://www.uclaoliveview.org/underserved-dedication/

35. American Board of Internal Medicine; Infectious Diseases Society of America. Avoid prescribing antibiotics for upper respiratory infections. February 23, 2015. Accessed May 4, 2021. https://www.choosingwisely.org/clinician-lists/infectious-diseases-society-antbiotics-for-upper-respiratory-infections/

36. Evaluation design for the Public Hospital Redesign and Incentives in Medi-Cal (PRIME) program. California Department of Health Care Services. Accessed April 18, 2023. https://www.dhcs.ca.gov/provgovpart/Documents/PRIMEFinalEvalDesign.pdf

37. Mafi JN, Godoy-Travieso P, Wei E, et al. Evaluation of an intervention to reduce low-value preoperative care for patients undergoing cataract surgery at a safety-net health system. JAMA Intern Med. 2019;179(5):648-657. doi:10.1001/jamainternmed.2018.8358

38. Lee B, Mafi J, Patel MK, et al. Quality improvement time-saving intervention to increase use of a clinical decision support tool to reduce low-value diagnostic imaging in a safety net health system. BMJ Open Qual. 2021;10(1):e001076. doi:10.1136/BMJOQ-2020-001076

39. Improving outpatient antibiotic use through audit and feedback. CDC. 2018. Accessed June 16, 2021. https://www.cdc.gov/antibiotic-use/community/pdfs/Audit-feedback-outpatient-antibiotic-use-508.pdf

40. Educational resources for healthcare professionals: prescription pads. CDC. 2021. Accessed June 16, 2021. https://www.cdc.gov/antibiotic-use/training/materials.html#anchor_1560790162782

41. Cuevas MA, Wachter ND, Reyes C, et al. Seeking care for back pain or upper respiratory infections?: survey results to inform a safety net hospital Choosing Wisely® intervention. Healthc (Amst). 2020;8(3):100424. doi:10.1016/J.HJDSI.2020.100424

42. Costales-Cantrell JK, Chung J, Nomura JH, Ironside KR, Salama PW, Sim JJ. Antibiotic practice patterns with procalcitonin levels in patients with acute lower respiratory tract infection. Am J Manag Care. 2022;28(2):e35-e41. doi:10.37765/ajmc.2022.88825

43. Schuetz P, Wirz Y, Sager R, et al. Procalcitonin to initiate or discontinue antibiotics in acute respiratory tract infections. Cochrane Database Syst Rev. 2017;10(10):CD007498. doi:10.1002/14651858.CD007498.pub3

44. McDonagh MS, Peterson K, Winthrop K, Cantor A, Lazur BH, Buckley DI. Interventions to reduce inappropriate prescribing of antibiotics for acute respiratory tract infections: summary and update of a systematic review. J Int Med Res. 2018;46(8):3337-3357. doi:10.1177/0300060518782519

45. De Cremer D, Bakker M. Accountability and cooperation in social dilemmas: the influence of others’ reputational concerns. Curr Psychol. 2003;22(2):155-163. doi:10.1007/s12144-003-1006-6

46. Guidelines for the prevention and treatment of opportunistic infections in adults and adolescents with HIV. HIV.gov. Accessed July 30, 2022. https://clinicalinfo.hiv.gov/en/guidelines/hiv-clinical-guidelines-adult-and-
adolescent-opportunistic-infections/whats-new

47. Keintz MR, Abbas A, Lyden E, et al. Antibiotic prescribing for acute respiratory illnesses in persons with HIV compared with persons without HIV. Open Forum Infect Dis. 2022;9(7):ofac218. doi:10.1093/OFID/OFAC218

48. Köchling A, Löffler C, Reinsch S, et al. Reduction of antibiotic prescriptions for acute respiratory tract infections in primary care: a systematic review. Implement Sci. 2018;13(1):47. doi:10.1186/s13012-018-0732-y

49. Tyree PT, Lind BK, Lafferty WE. Challenges of using medical insurance claims data for utilization analysis. Am J Med Qual. 2006;21(4):269-275. doi:10.1177/1062860606288774

50. Kontopantelis E, Doran T, Springate DA, Buchan I, Reeves D. Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis. BMJ. 2015;350:h2750. doi:10.1136/bmj.h2750