Publication
Article
The American Journal of Managed Care
Variation in Implementation and Use of Computerized Clinical Reminders in an Integrated Healthcare System
Objectives: To identify patterns of use of computerized clinicalreminders (CCRs) across an integrated healthcare system anddescribe institutional factors associated with their implementation.
Study Design: Cross-sectional study.
Methods: At a national electronic health record (EHR) meeting,we surveyed 261 participants from 104 Veterans Health Administration(VHA) healthcare facilities regarding the number and typesof CCRs available at each facility. Potential explanatory measuresincluded perceived utility and ease of use of CCRs, training andpersonnel support for computer use, EHR functionalities, and performancedata feedback to providers at each facility.
P
Results: The number of conditions with CCRs in use at a facilityranged from 1 to 15; most reported implementation of remindersfor 10 of the 15 conditions surveyed. The most commonly implementedCCRs, used in more than 85% of facilities, were for conditionswith VHA national performance measures (eg, tobaccocessation, immunizations, diabetes mellitus). The least commonlyimplemented CCRs were for post-deployment health evaluationand management, medically unexplained symptoms, and erectiledysfunction. Facilities that had implemented greater numbers ofclinical reminders had providers who reported greater ease of useand utility of the reminders ( = .01).
Conclusions: VHA facilities vary markedly in their implementationof CCRs. This effect may be partly explained by greater incorporationof clinical reminders for conditions with performancemeasures. Further study is needed to determine how to best implementclinical reminders and the institutional factors important intheir use.
(Am J Manag Care. 2004;10(part 2):878-885)
Computerized clinical reminders (CCRs) havebeen widely publicized as potential tools forchanging behavior1 and improving quality ofcare.2-4 They have been particularly effective in improvingadherence with preventive care and screeningguidelines,2,5 monitoring diabetes,6 and treating hypertension.7 However, factors such as workload, time, andperceived reduction of the quality of the provider-patientinteraction may be barriers to effective use ofcomputerized reminders.8
The Veterans Health Administration (VHA), theUnited States' largest integrated healthcare delivery system,has invested heavily in the informatics infrastructurenecessary to support CCRs. It has been on theforefront of developing these tools and incorporatingthem into the computerized patient record system(CPRS), which is the VHA's electronic health record(EHR). The VHA developed CPRS, an application usedthroughout VHA facilities that enables clinicians toreview and analyze patient data and supports clinicaldecision making.9
Computerized clinical reminders may be developedand distributed at the national, regional, or local level.Although VHA mandates use of a few clinical reminderssuch as assessing hepatitis C risk and possible sexualtrauma during military service, most CCRs have beenlocally initiated.10 After reminders are created, they aretested and activated. Some reminders are applicable toall patients, while others apply to a particular group ofpatients. VHA facilities can generate clinical reminderreports, which provide information, for example, aboutthe number of patients in a clinic with completed clinicalreminders and the number of patients eligible for thereminder.
Although CCR technology is in widespread use inVHA, data regarding CCR patterns of use in VHA andinstitutional factors associated with their use have notbeen available. A description of the different types ofreminders implemented in VHA may help other largeorganizations interested in promoting this technology.Identifying factors influencing widespread implementationof CCRs is a necessary step to promote their dissemination.Hence, our primary objective was to surveyVHA facilities for variations in the numbers and typesof CCRs used across VHA facilities nationally. Oursecondary objective was to identify institutional factors associated with increased implementation of CCRtechnology.
METHODS
Study Population and Data
We assessed CCR use across the VHA by distributinga survey instrument to participants at the nationalCamp CPRS meeting that took place in Georgia in May2003. This meeting focused on the VHA's EHR and in2003, included 1304 representatives from 136 of the142 VHA medical facilities participating in the VHA'sExternal Peer Review Program (EPRP). The representatives,who are nominated by their facilities, may be clinicalstaff (eg, physician, nurses, others), administrativepersonnel (eg, chief of staff), or informatics experts.Many attendees are opinion leaders at their facility andhave extensive experience with local CPRS capabilities,either as users or developers of clinical applications forthe EHR and its decision support tools.
We received responses from 261 participants (20%)representing 104 VHA facilities (76%). Nonrespondentswere not tracked because only those who volunteeredwere given the survey instrument. Sixty-five of the 104facilities that responded to our survey had more than1 respondent.
The American Hospital Association database providedadditional background information about VHAfacilities. It provided information about geographicdistribution of participating facilities, teaching statusaffiliation, and size of the facilities.
Questionnaire Development
After discussion with the VHA staff that oversee,develop, or perform research on clinical reminders,including members of the National Clinical PracticeGuideline Council and members of its ad hoc ClinicalReminders Committee, we chose to evaluate facilities'use of 15 different types of CCRs. We selected remindersthat represent a broad range of conditions cliniciansmight encounter in various clinic settings. Somereminders were for conditions that have VHA nationalperformance measures (eg, addressing tobacco cessation),whereas others (eg, low back pain) were for conditionswithout VHA national performance measures.
The remainder of the 77 items in the survey instrumentfocused on institutional factors hypothesized fromour ongoing studies to be important in implementationof clinical guidelines11 or CCRs. The instrumentassessed different forms of computer use (provider education,performance feedback, and clinical support suchas the ability to retrieve radiological images), perceivedutility and ease of use of CCRs, adequacy of computertraining and CCR training, organizational support, hospitalculture/climate, and availability of feedback mechanismsfor modifying CCRs. Responses for this portionof the survey instrument were either dichotomous (yesor no) or measured on a 5-point Likert-type scale.Respondents who answered "don't know" were recodedas "missing" or "no." The questionnaire is available,upon request, from Dr. Doebbeling.
The University of Iowa/Iowa City VA institutionalreview boards approved the study protocol. The datamanagement plan underwent subsequent review andapproval from the institutional review board of the VAGreater Los Angeles Healthcare System.
Outcome Measures
Outcome measures were obtained from questionnaireresponses. For these measures, we aggregatedindividual responses at the facility level. Although 39facilities (37.5%) had only a single respondent, the otherfacilities had multiple respondents, including 29 facilitieswith 2 respondents, 23 with 3 respondents, 6 with4 respondents, 1 with 5 respondents, 3 with 6 respondents,and 3 with 10 or more respondents. When morethan 1 response per facility was available, we used meanscores for questions that had Likert-like responsescales. For dichotomous variables, we assumed thatrespondents from any given facility, while knowledgeableabout their home facility, would be unlikely toreport seeing or using a clinical reminder that they havenever seen or used. In contrast, it is possible that theymay not have seen or used a clinical reminder that wasavailable at their facility, but a colleague at their facilitymay have reported seeing or using the clinicalreminder. Because our goal was to determine whether aclinical reminder exists or not at a given facility, weweighted "yes" responses more than "no" responses.We reasoned that the union rather than intersection ofpositive responses would more accurately reflect, forexample, the actual number of conditions with clinicalreminders in place at a particular facility. Therefore, wetreated any positive answer among respondents at thefacility as a true-positive facility response. For example,if there were 3 responses and 2 stated that CPRS wasused for requesting consults and 1 did not (or viceversa), the facility score was based on a "yes" answer.In a sensitivity analysis, we weighted negative responsesmore heavily than positive responses.
t
P
The first outcome measure represents which CCRswere available at the facility. The second outcome measure,a "facility clinical reminder score" (minimum possiblescore = 0, maximum possible score = 15), wascreated by summing "yes" responses to questions surveyingwhether a facility had at least 1 clinical reminderfor a particular health condition. This variable representedthe level of implementation of CCR technology inVHA facilities. Although some conditions, such as diabetes,may have more than 1 clinical reminder (eg, glycosylatedhemoglobin, microalbumin/creatinine ratio),we counted each condition only once in our facility clinicalreminder score. We used tests to determinewhether responses to each question differed significantlyfor facilities with complete versus incomplete responses.There were 72 facilities with complete data and 32facilities with incomplete data (missing data or "don'tknow" response) for the questions used to construct thefacility clinical reminder score. The mean facility clinicalreminder score was 9.1 (95% confidence interval[CI]: 8.6, 9.7) for facilities with complete responses versus7.4 (95% CI: 6.3, 8.5) for those with incompleteresponses (a difference of 1.7; 95% CI: 0.7, 2.8). Table 1presents comparison data for VHA facilities that participatedin Camp CPRS and had respondents to the surveyversus VHA facilities that did not participate in CampCPRS or did not have respondents to the survey. Facilitieswith complete and incomplete responses were similarwith respect to number of acute-care beds, trainee andresident full-time equivalents, academic affiliation, andurban versus rural location ( > .05 for all comparisons).
Potential Explanatory Measures
To identify potential explanatory variables, we developedscales from questionnaire items. The process ofdeveloping scales also served as a method of data reduction.Existing literature and conceptual models, as well asfactor analysis, guided development of the scales.12,13 Werelied on a priori hypotheses developed from literaturereview, our conceptual framework, and clinical experiencein the identification and labeling of a 5-factor solution.The resulting domains included:
- Computer training and personnel support, measuringperceived adequacy of training and personnelsupport (8 items; alpha = .842).
- EHR functionality, measuring the number of featuresavailable online to clinicians at the point ofcare and at the facility in general (12 items; alpha = .86).
- Clinical reminders utility and ease of use (6items; alpha = .75).
- Graphical data feedback, measuring availabilityof graphical display of individual and clinic performance(2 items; alpha = .95).
For each scale, higher scores indicated greater perceivedsupport, ease of use/utility, functionality, oravailability (Appendix).
Statistical Analysis
P
To meet our primary objective, we used simpledescriptive statistics to present data on the type and distributionof CCRs across VHA. For our secondary objective,bivariate relationships were tested and amultivariate linear regressionmodel was estimated to determinewhich American HospitalAssociation facility characteristicsand domains from our factoranalysis were independentpredictors of having a higherfacility clinical reminder score.The number of facilities withcomplete responses for all variablesincluded in the analysis (n= 62) limited the number ofpotential variables to no morethan 6. Because there weremore than 6 potential variables,only variables with a value of≤.25 in our bivariate analyseswere considered for the multivariateanalysis. Then, we usedstepwise regression to build ourmultivariate model. We used analpha of <.10 for inclusion inthe final model (the level of significancethat was used todefine the minimum acceptable F limit for adding orremoving a variable in the model). All statistical testingwas performed with SAS, version 8 (SAS Institute, Inc,Cary, NC).
RESULTS
Descriptive Results
P
P
P
Table 1 demonstrates that participating facilitieswere geographically distributed across the UnitedStates. Approximately one half (48%) of the hospitalswere members of the Council of Teaching Hospitals.One third of the hospitals had fewer than 200 acute-carebeds, one third had 200-399 acute-care beds, andone third had 400 or more acute-care beds.Participating facilities were similar to facilities that didnot participate in the conference or respond to the surveyin terms of geographic region ( = .53), teaching status( = .35), and bed size ( = .14).
Table 2 summarizes the percent of respondents whowere clinicians, information technologists, and clinicalapplication coordinators. Clinical application coordinators,who are liaisons between informatics and clinicalservices, provide services such as implementing softwarepackages and clinical support systems for VHA'sEHR and customizing national-level VHA packages forlocal VHA healthcare facilities.
Figure 1 summarizes the number of conditions withCCRs across 104 facilities (the facility clinical reminderscore). Of the 15 conditions surveyed, the number ofconditions with reminders implemented across facilitiesranged from 1 to 15, with a median of 9 (interquartilerange 7, 10). In a sensitivity analysis, weighting negativeresponses more than positive responses, themedian number of conditions with remindersimplemented was 6 (interquartile range 4, 8),but variation across facilities remainedunchanged. Figure 2 summarizes the types ofclinical reminders reported at each facility. Themost commonly implemented reminders acrossfacilities were for conditions with VHA nationalperformance measures, including tobacco cessation(n = 99), immunizations (n = 97), diabetesmellitus (n = 95), hypertension (n = 91), anddyslipidemia (n = 88). In contrast, conditions forwhich reminders were infrequently implementedincluded low back pain (n = 14), erectile dysfunction(n = 8), medically unexplainedsymptoms (n = 7), and post-deployment healthevaluation and management (n = 5).
Most facilities had implemented basic componentsof an EHR: entering patient notes (n =100), requesting consultations (n = 101), orderingradiological procedures (n = 101), ordering laboratorytests (n = 102), and prescribing medications (n =104). Many facilities also provided customization of thereminders (n = 96) or formats tailored to specificpatient populations (n = 97). Respondents consideredclinical reminders generally useful, with an overallmean response of 4.0 on a scale 1 to 5 (1 = not at alland 5 = very great).
Bivariate and Multivariate Analyses
P
P
P
Table 3 summarizes the institutional structure andprocess variables associated at a bivariate level with thefacility clinical reminder score. Multivariable regression(Table 4) found 2 hospital-level variables associatedwith implementation of clinical reminders; these were"perceived utility and ease of use" and "number of full-timeequivalent trainees and residents" (F statistic =5.78; = .005). Each 5-point increase on the "ease ofuse and utility" scale was associated with a 1-unitincrease in the number of CCRs at the facility (parameterestimate of 0.21; = .01). Conversely, an increasein the number of residents and trainees had a borderlineassociation with a decrease in CCR implementation(parameter estimate of -0.00711; = .069),meaning our model predicts that facilities will have 1less CCR for every 150 full-time equivalent trainees andresidents gained.
DISCUSSION
We found wide variation across VHA facilitiesnationally in the types and number of conditions withCCRs. We observed this variation even though VHAfacilities are part of a single-payer system and nearly allfacilities have common information technology infrastructure,including a basic EHR that is similar acrossfacilities, to develop CCRs. Our study also sought toidentify factors associated with greater implementationof clinical reminders. We found that facilities with higherperceived utility and ease of use of clinical reminderswere more likely to have implemented a larger numberof CCRs at each facility.
Facilities have incentives to implement clinicalreminders for conditions that have VHA national performancemeasures. The VHA's performance measuresare similar to Health Plan Employer Data andInformation Set (HEDIS®) measures, which weredeveloped from evidence-based practice guidelines.14,15 The VHA's EPRP gathers information on thequality of care delivered for conditions such as tobaccocessation, immunizations, and diabetes care, andfeeds back facility-level data on adherence to performancemeasures for these conditions to administrators.Clinical reminders also have been shown to increaseadherence to these types of measures.2,5-7 Post-deploymenthealth evaluation and management, medicallyunexplained symptoms, erectile dysfunction, and lowback pain did not have associated performance measures.In our study, clinical reminders for many of theEPRP target conditions were more commonly implementedthan clinical reminders for non-EPRP targetconditions. This pattern suggests that the audit andfeedback of performance data may be a strong force foradopting and implementing clinical reminders.
The extent of clinicalreminder implementationat the facility level was independentlyrelated to perceived utility andease of use of clinicalreminders. Theseresults are consistentwith prior studies.Clinicians surveyed ina study by Schellhaseet al also reported thatlack of ease of use wasa possible explanationfor the observed underuseof automatedhealth maintenancereminders.16 In arecent study of human-factorbarriers to theeffective use of HIVclinical reminders in VHA, inapplicability to the specificclinical situation was identified as 1 of multiple barriersto their regular use.8 Our results suggest that greaterattention to utility and ease of use of the reminders, perhapsthrough qualitative or formative studies of theiruse, may be important for healthcare systems trying topromote the adoption of clinical reminder technology.
Prior studies have identified organizational factorssuch as hospital size and location (urban vs rural) asinfluences on the diffusion of healthcare informationtechnology.13,17-19 These organizational factors did notpredict the extent of CCR implementation in the VHAfacilities that participated in our survey. However, otherfactors identified in the literature that we were unable totest may be contributing to the observed variation in thenumber of conditions with CCRs. Centralized developmentand mandated adoption may lead to more uniformadoption of CCRs across a system.12 This uniformity maybe desirable to large healthcare systems, especially asmore studies show the benefits of CCRs and how to optimallyimplement them. In contrast, decentralized developmentand optional adoption of reminders may lead toslower uptake by some facilities and, thus, variation inthe number of CCRs across different facilities.12Decentralized development and local modification, however,may be advantageous for a system, because thesefactors may improve the chances that only those facilitieswith adequate clinical and technological capability toeffectively support clinical reminders will push adoptionand implementation. Furthermore, the input of localproviders in adapting reminders may improve acceptance by providersand, perhaps, usability.Central developmentand testing ofhighly selectedreminders with disseminationand localadaptation, a modelthat VHA has recentlyadopted, maystrike the optimalbalance betweenwell-designed, evidence-based clinicalreminders that servea patient populationand the needs of localfacilities.
The strength ofour study is itsnational scope. Manystudies of clinicalreminders are unableto obtain the nationalsampling we achievedin this study, becausefew healthcare systemswith an EHRand CCR system havethe geographic reachachieved by VHA.Similarly, this reportprovides previouslyunavailable dataregarding the implementationand disseminationof clinicalreminders throughouta national healthcaresystem with an already-implemented EHR.
The chief limitation of this study is that we relied on aconvenience sample. Thus, the responses may not berepresentative of all hospitals within the VHA system.Nevertheless, we aggregated individual responses withinfacilities to better reflect experiences at the facility level.Although we excluded facilities with incomplete responses,we noted that facilities with incomplete scores andcomplete scores differed minimally in their meanreminder scores and that overall they had similar organizationalcharacteristics in terms of acute-care beds,trainees, academic affiliation, and location. Facilities withrespondents were similar to VHA facilities withoutrespondents in terms of hospital size and teaching status(membership in Council of Teaching Hospitals).Although verification of the survey answers throughdirect contact with each facility was beyond the scope ofthe current study, we included participants who hadbeen selected to attend a conference on the VHA's EHRand, thus, were likely to be reliable informants abouttheir home systems. Furthermore, this study designenabled us to survey the number of conditions with clinicalreminders that are clinically relevant. Nevertheless,measurement of all possible types of reminders at eachfacility may be informative in future studies. Other locallyimportant clinical reminders may have been inadvertentlyomitted from our questionnaire, therebyunderestimating the number of clinical reminders used ata particular facility. However, although this may havechanged the overall number per facility, it is unlikely tohave lessened the variation noted. Another limitationinvolves our sample size. Because of a limited samplesize and the descriptive nature of this investigation, weused results from our bivariate analyses and stepwisevariable selection for our multivariate analysis. Finally,in our regression model, we did not adjust for clusteringat the hospital level, which may affect our results.
CONCLUSION
This study found many CCRs in widespread useacross VHA and suggested that considerable variationexists in the number of conditions with reminders.Facilities where more CCRs have been implementedhave greater provider support for their utility and easeof use. Other large healthcare systems interested inadopting CCRs into regular practice should consider theassociation we demonstrated between ease of use/utilityand adoption of CCRs. Future studies should explorehuman and systems factors that facilitate increasedadoption and routine use ofCCRs in clinical care.
Acknowledgments
We appreciate the contributions of Drs. Toni Tripp-Reimer, PHD, RN, Ashley Hedeen, MD, Robert Smith, MD,Bonnie BootsMiller, PHD, and Valerie Forman-Hoffman to the design of the instrument.
From the Department of Medicine, Division of General Internal Medicine, VA GreaterLos Angeles Healthcare System, and the David Geffen School of Medicine at UCLA, LosAngeles, Calif (CHF, PG); the Department of Epidemiology, College of Public Health, theUniversity of Iowa, Iowa City, Iowa (JNW); the VA Center for the Study of HealthcareProvider Behavior, Los Angeles, Calif, and the David Geffen School of Medicine at UCLA(SMA); and the HSR&D Center of Excellence in Implementing Evidence-based Practice,Roudebush Veterans Affairs Medical Center, Indianapolis, Ind, the Department of InternalMedicine, Indiana University School of Medicine, and the Regenstrief Institute,Indianapolis, Ind (BND).
This study was supported by a Department of Veterans Affairs Health Services Researchand Development Service Management Consultation Project and by Investigator InitiatedResearch (IIR) Merit Review grant CPI 01-141.
Address correspondence to: Constance H. Fung, MD, MSHS, 11301 Wilshire Blvd,111G, Los Angeles, CA 90073. E-mail: cfung@mednet.ucla.edu.
BMJ.
1. Bero LA, Grilli R, Grimshaw JM,Harvey E, Oxman AD, Thomson MA.Closing the gap between research and practice:an overview of systematic reviews ofinterventions to promote the implementationof research findings. The Cochrane EffectivePractice and Organization of Care ReviewGroup. 1998;317(7156):465-468.
J Am MedInform Assoc.
2. Shea S, DuMouchel W, Bahamonde L.A meta-analysis of 16 randomized controlledtrials to evaluate computer-basedclinical reminder systems for preventivecare in the ambulatory setting. 1996;3:399-409.
Ann Intern Med.
3. Kuperman GJ, Gibson RF. Computerphysician order entry: benefits, costs, andissues. 2003;139:31-39.
Arch Fam Med.
4. Gorton TA, Cranford CO, Golden WE,Walls RC, Pawelak JE. Primary care physicians'response to dissemination of practiceguidelines. 1995;4:135-142.
JAMA.
5. Hunt DL, Haynes RB, Hanna SE, SmithK. Effects of computer-based clinical decisionsupport systems on physician performanceand patient outcomes: asystematic review. 1998;280:1339-1346.
JAMA.
6. Demakis JG, Beauchamp C, Cull WL,et al. Improving residents' compliancewith standards of ambulatory care: resultsfrom the VA Cooperative Study onComputerized Reminders. 2000;284:1411-1416.
Med Care.
7. Barnett GO, Winickoff RN, Morgan MM,Zielstorff RD. A computer-based monitoring system for follow-up of elevatedblood pressure. 1983;21:400-409.
J Am Med Inform Assoc.
8. Patterson ES, Nguyen AD, Halloran JP, Asch SM. Human factors barriers tothe effective use of ten HIV clinical reminders. 2004;11:50-59.
9. Department of Veterans Affairs. The Electronic Record: CPRS. EmployeeOrientation Toolkit. June 2003. Available at:http://www.va.gov/oaa/orientation/admin_cprs.asp. Accessed May 17, 2004.
J Inform Technol Healthc.
10. Glassman PA, Volpp B, Walder D, et al. Developing electronic clinicalreminders for improving hypertension management: the approach of theDepartment of Veterans Affairs. 2003;1:251-265.
Med Care.
11. Ward MM, Yankey JW, Vaughn T, et al. Physician process and patient outcomeperformance measures for diabetes care: relationships to organizationalcharacteristics. 2004;42:840-850.
Diffusion of Innovation.
12. Rogers EM. New York: Free Press; 2003.
J Subst Abuse Treat.
13. Simpson DD. A conceptual framework for transferring research to practice. 2002;22:171-182.
Med Care.
14. Vaughn TE, McCoy KD, BootsMiller BJ, et al. Organizational predictors ofadherence to ambulatory care screening guidelines. 2002;40:1172-1185.
15. National Committee for Quality Assurance. The Health Plan Employer Dataand Information Set (HEDIS®). Available at http://www.ncqa.org/Programs/HEDIS.Accessed March 17, 2004.
J Am Board Fam Pract.
16. Schellhase KG, Koepsell TD, Norris TE. Providers' reactions to an automatedhealth maintenance reminder system incorporated into the patient's electronicmedical record. 2003;16:312-317.
J Am Med Inform Assoc.
17. Southon FC, Sauer C, Grant CN. Information technology in complex healthservices: organizational impediments to successful technology transfer and diffusion. 1997;4:112-124.
Int J Med Inform.
18. Southon G, Sauer C, Dampney K. Lessons from a failed information systemsinitiative: issues for complex organisations. 1999;55:33-46.
J Med Syst.
19. Burke DE, Wang BB, Wan TT, Diana ML. Exploring hospitals' adoption ofinformation technology. 2002;26:349-355.
