The Comparative Effectiveness of 2 Electronic Prescribing Systems

Objectives:

The increasingly widespread adoption of electronic health records (EHRs) is substantially changing the American healthcare delivery system. Differences in the actual effectiveness of EHRs and their component applications, including electronic prescribing (e-prescribing), is not well understood. We compared the effects of 2 types of e-prescribing systems on medication safety as an example of how comparative effectiveness research (CER) can be applied to the study of healthcare delivery.

Study Design and Methods:

We previously conducted 2 non-randomized, prospective studies with pre—post controls comparing prescribing errors among: (1) providers who adopted a standalone e-prescribing system with robust technical and clinical decision support (CDS) and (2) providers who adopted an EHR with integrated e-prescribing with less robust available CDS and technical support. Both studies evaluated small groups of ambulatory care providers in the same New York community using identical methodology including prescription and chart reviews. We undertook this comparative effectiveness study to directly compare prescribing error rates among the 2 groups of e-prescribing adopters.

Results:

The stand-alone system reduced error rates from 42.5 to 6.6 errors per 100 prescriptions (P <.001). The integrated system reduced error rates from 26.0 to 16.0 per 100 prescriptions (P= .07). After adjusting for baseline differences, stand-alone users had a 4-fold lower rate of errors at 1 year (P <.001).

Conclusions:

Despite improved work flow integration, the integrated e-prescribing application performed less well, likely due to differences in available CDS and technical resources. Results from this small study highlight the importance of CER that directly compares components of healthcare delivery.

(Am J Manag Care. 2011;17(12 Spec No.):SP88-SP94)

Comparative effectiveness research (CER) can be expanded beyond its usual focus on treatment or intervention options to evaluate healthcare delivery systems, including electronic health records (EHRs) and electronic prescribing (e-prescribing).

• Our small CER study of ambulatory providers found that use of a stand-alone e-prescribing application led to a greater reduction in prescribing errors than use of an integrated EHR with e-prescribing.

• System features, as well as implementation and training resources, likely contributed to these findings.

• Given the large national investment in EHRs, future CER research should be conducted to understand the actual effects of EHR systems in active use.

The Health Information Technology for Economic and Clinical Health (HITECH) Act is providing up to $30 billion for the use of interoperable electronic health records (EHRs) in meaningful ways that improve healthcare quality, safety, and efficiency.¹ Specific measures will be used to determine if a provider meets criteria for incentives, including electronic prescribing (e-prescribing).² To decrease intersystem variability, providers must use a certified EHR, including eprescribing that meets certain standards. However, even among certified EHRs, differences in performance arise in actual system use.

Comparative effectiveness research (CER) is designed to inform healthcare decisions by providing evidence on the effectiveness, benefits, and harms of different treatment options and interventions.³ Traditionally, CER has directly compared the outcomes of 2 therapeutic interventions, most commonly medications, using randomized controlled trials. To date, there has been little CER evaluating different health information technology (HIT) in actual use.⁴

Previously, using identical methodology, we conducted 2 separate prospective studies comparing the effects of 2 types of e-prescribing systems on medication safety.^5,6 The systems were implemented and used by ambulatory providers in a single community but had important differences. The first was a stand-alone system with advanced clinical decision support (CDS) and extensive technical support. The second was an e-prescribing application integrated within an EHR with less robust available CDS and more limited technical support. These studies provided an ideal opportunity to perform a novel comparative effectiveness study of these 2 system types. Our hypothesis was that despite technical and CDS limitations, the integrated application would better improve medication safety due to improved work flow integration and increased diversity of patient data from the EHR available to the prescriber at the point of care.

METHODS

Study Design

In this study, we compare the effectiveness of a commercially available stand-alone e-prescribing system to an e-prescribing system integrated within an EHR by analyzing paper prescriptions at baseline and e-prescriptions 1 year later for 21 ambulatory care providers in the same New York community. The data were obtained from 2 previous pre—post studies evaluating prescribing safety among e-prescribing adopters compared with control providers who used handwritten prescriptions. ^5,6 The same group of 15 non-adopters served as concurrent controls for both original studies and, in each case, prescribing errors were high at baseline for all study participants but were significantly higher for non-adopters at 1 year. We obtained institutional review board approval from Weill Cornell Medical College, and providers gave written informed consent.

Definitions

The Institute of Medicine classifies medication errors as any error in the medication use process (prescribing, transcribing, dispensing, administering, and monitoring).⁷ We focused only on prescribing errors, such as omitting quantity to be dispensed. Near misses were prescribing errors with potential for harm that were either intercepted or reached the patient but did not cause harm. An example was prescribing penicillin for a patient with a known allergy who did not receive the medication because of pharmacist error detection. Adverse drug events (ADEs) were injuries from a medication, a subset of which was preventable. Rule violations were departures from strict standards of prescribing that were well understood and unlikely to cause harm, such as failure to write “po” for a medication only taken orally. These were not included in error rates but were counted, as they can result in significant rework.

Sites

We studied 11 adult primary care practices in a predominantly rural and suburban region of New York State between September 2005 and July 2008. Fifteen providers from 6 different practices adopted a stand-alone system and 6 providers from 5 practices adopted an integrated system. Physicians were members of the not-for-profit independent practice association. All members were sent a letter in May 2005 detailing incentives for adopting e-prescribing and inviting them to participate in a research study. Discounts on EHR licenses were provided as an incentive. All practices ranged in size from 1 to 7 providers and none were academically affiliated.

Stand-alone E-prescribing System

The stand-alone system was a Web-based, commercially available system. Providers had access to an electronic reference guide for dosing recommendations, medication lists, and allergies, if this information had been entered. The system provided CDS alerts for drug allergies, drug—drug interactions, duplicate drug therapies, incorrect drug frequencies, incorrect dosing, and pregnancy and breast-feeding contraindications. The system checked for insurance eligibility and formulary compliance. Prescriptions could be sent to pharmacies electronically. This system was ultimately not certified, as it was not integrated within an EHR. Providers performed other clinical documentation on paper.

E-prescribing System Integrated Within an EHR

The integrated system was a commercially available system fully integrated within an EHR. Providers had electronic access to all information in the EHR including patient history, medications, allergies, diagnoses, and laboratory and demographic data. The system provided the same types of alerts as the stand-alone system but additionally provided drug—disease interactions and disease-specific drug recommendations. Patient insurance eligibility and formulary compliance were also checked and prescriptions could be sent electronically to pharmacies. Notably, the e-prescribing module was a more immature module due to limited configuration of the CDS at the time of the study. CDS for e-prescribing was fully configured after the study’s completion.

Technical Support

A for-profit Health Information Service Provider (HSP) provided implementation and ongoing technical support, including routine monitoring of e-prescribing compliance to encourage 100% use.8 Because the HSP was new to providing implementation services at the time the integrated practices went live, those providers received far less training initially (on average 1 hour) compared with stand-alone users (on average 40 hours), who went live later.

Data Collection and Review

Prescription Collection. We collected carbon copies at baseline and electronic downloads of all prescriptions written by providers during a 2-week period at 1 year. During both time periods, we obtained a minimum of 75 prescriptions on at least 25 patients per provider, extending data collection if necessary, and limited review to 3 prescriptions per patient to minimize clustering of errors. Non-duplicate prescription pads were removed at baseline to ensure compliance with use of the duplicate prescription pads.

Prescription Review. A research nurse reviewed each prescription in an identical manner guided by extensively utilized, standardized methodology.^9-11 Training included review of error definitions, legibility assessments, and review of test and actual cases. Two data collectors jointly reviewed cases, after which reviewers worked independently. Reviewers classified prescribing errors, and rule violations, and evaluated the use of ADE-trigger drugs. Inappropriate abbreviation errors were from the Joint Commission on Accreditation of Healthcare Organizations’ “Do Not Use” list, established to denote abbreviations with great potential to cause medical errors.¹²

We determined interrater reliability by having 2 reviewers examine the same random sample of 2% of the data. Interrater agreement for overall and type of error was 1.0, indicating excellent agreement.

Chart Review. An ambulatory chart review was performed for suspected near misses or when a drug often used to treat an ADE was prescribed.

Physician Review and Classification. Two physicians blinded to the providers’ prescribing method independently reviewed all suspected near misses and ADEs. Interrater agreement for the presence of prescribing errors and near misses was 0.96 and 0.93, indicating excellent agreement.

Statistical Analysis. We compared error rates per 100 prescriptions of (1) stand-alone and integrated adopters at baseline, (2) stand-alone adopters at baseline and 1 year, (3) integrated adopters at baseline and 1 year, and (4) between stand-alone and integrated adopters at 1 year using mixedeffects Poisson regression models that included e-prescribing system type, study time, and an interaction term system type and study time. We adjusted for clustering at the provider level and assumed an independent correlation structure for these Poisson models. We calculated 95% Poisson confidence intervals (CIs) with cluster robust standard errors for the rates. We used SAS for PC version 9.2 (SAS Institute Inc, Cary, North Carolina) to estimate Kappa statistics, χ², and t tests, and Stata 11 (StataCorp, College Station, Texas) to

estimate mixed-effects Poisson regression models and to calculate 95% Poisson CIs with clustered robust standard errors.

RESULTS

Prescriber Characteristics

Table 1

There was no significant difference between provider groups (). All physicians were board certified in internal or family medicine. The mean number of prescriptions written per provider and patients seen per provider were similar at baseline (0.65 and 0.26, respectively). At 1 year, integrated adopters saw significantly more patients and wrote significantly more prescriptions during the study period (0.002 and 0.002, respectively).

Patient Characteristics

Table 2

For stand-alone adopters, 1273 unique patients seen at baseline and 1598 unique patients seen at 1 year received prescriptions (). For integrated adopters, 481 unique patients seen at baseline and 368 unique patients seen at 1 year received prescriptions. At baseline, patients of stand-alone adopters were significantly older (56 vs 53 years, P <.001) and at 1 year more were female (60% vs 54%, P = .03).

Baseline Error Rates

We reviewed 2550 prescriptions at baseline, 1901 of which were written by stand-alone adopters and 649 of which were from integrated adopters (Tables 3A and 3B).

Rates of baseline prescribing errors for both groups were high, with planned adopters of integrated e-prescribing 1.6- fold less likely to have prescribing errors (P = .03). Rule violations were also high at baseline for both groups, but with no statistical differences. Near misses were detected infrequently and without statistical differences.

Rates of Errors for Stand-alone E-prescribing Adopters at Baseline and 1 Year

We reviewed 2305 prescriptions at 1 year for stand-alone adopters. Stand-alone e-prescribing reduced prescribing error rates by 6.7-fold (P <.001). Rule violations also decreased significantly (P <.001). Near misses remained low at 1 year and unchanged from baseline.

Rates of Errors for Integrated E-prescribing Adopters at Baseline and 1 Year

We reviewed 536 prescriptions at 1 year for integrated e-prescribing adopters. Use of an integrated system reduced prescribing error rates by 1.6-fold, suggesting a trend for decreasing prescribing errors but without achieving statistical significance. Rule violations significantly decreased (P <.001). Rates of near misses remained low and unchanged.

Rates of Errors for Stand-alone Versus Integrated E-prescribing Adopters at 1 Year

Stand-alone adopters were 2.45 times less likely to have prescribing errors than integrated adopters at 1 year (P <.001). After accounting for differences in baseline error rates, standalone users had a 4-fold lower rate of errors at 1 year (P <.001). There was no significant difference in rates of rule violations (P = .16) or near misses (P = .94) at 1 year.

Types of Prescribing Errors at Baseline and 1 Year

4B

Use of the stand-alone system reduced all types of prescribing errors, while use of the integrated system led to a decrease in only some error types (Tables 4A and ).

DISCUSSION

In our small comparative effectiveness study, we found that use of a commercially available stand-alone e-prescribing system with more sophisticated CDS and more rigorous technical support led to a significantly greater reduction in prescribing errors compared with an e-prescribing system integrated within an EHR. To our knowledge, this study is the first to quantitatively compare the effectiveness of 2 different e-prescribing systems on medication safety in a single community using uniform methodology. The results are in contrast to our original hypothesis and highlight the need to approach HIT evaluation in a framework that goes beyond simply presence or absence of HIT and instead considers other factors, such as implementation processes and system functionalities. This study also highlights a novel application of CER to directly compare the effectiveness of HIT applications, an important method of healthcare delivery, in actual use.

Theoretically, an e-prescribing application within an EHR should perform better than a stand-alone application in reducing ambulatory prescribing errors. Safety benefits from e-prescribing are largely a function of incorporated CDS.13 A stand-alone application is more limited in the types and extent of available CDS. For example, a stand-alone application is unable to incorporate laboratory values and therefore perform drug-laboratory checks. However, this study demonstrates just the opposite.

These findings may be the result of the fact that the CDS was not fully configured, and was therefore relatively immature in the integrated application. Shortly after the study’s completion, with the system fully configured, more sophisticated CDS was introduced. Another explanation may be the different levels of training and support for the 2 groups. Although a for-profit HSP supported all providers, those using the integrated system were trained earlier and received less intensive support initially. Previous research has demonstrated the importance of training and support for successful adoption and use of EHRs and e-prescribing.^14,15 For example, a study evaluating e-prescribing implementation found that challenges with implementation and insufficient technical support can lead physicians to abandon e-prescribing or delegate e-prescribing to support staff, reducing the potential of CDS to positively impact point of care prescribing decisions.¹⁴

A third explanation may be that this study was powered to measure prescribing errors rather than near misses or preventable ADEs. An integrated system may be the most effective at targeting these more serious errors due to the incorporation of more sophisticated CDS. There may also be other unmeasured safety benefits from the integrated system that our study design would not capture. For example, an alert based on a laboratory value that led to a lower drug dose or alternate drug would not be captured, although the possibility of patient harm would be reduced.

Nevertheless, this small study highlights the importance of CER research of EHRs and associated HIT applications. Our results also highlight the need to evaluate and compare HIT applications and their quality and safety effects in a framework that includes more than simply presence or absence of that technology,¹⁶ addressing factors such as implementation and technical support resources and the heterogeneity of the applications themselves. Large national investments are being made to promote widespread EHR adoption.¹⁷ Although EHRs are being certified and incentives to providers are dependent on the demonstration of “meaningful use,”² little work is under way comparing which functionalities, implemented and supported in which ways, and used by whom, are most effective in practice.

Additionally, CDS is one of the most effective tools to deliver evidence-guided clinical recommendations to providers in actionable ways. Research suggests that providers have varying levels of satisfaction with CDS which may impact system use.¹⁸ Therefore, in order to assure that providers receive these recommendations in clinically useful ways, CER on CDS structure, content, and delivery will be critical.

Two of the most exciting and novel models of health systems delivery, the patient-centered medical home and accountable care organizations, include EHRs.^19,20 CER of these models, and variation within these models, is another important application.

As stated by Blumenthal, “Information is the lifeblood of medicine.”²¹ Investments are being made in EHRs to provide evidence-based clinical guidelines and patient information to providers at the point of care to improve quality and promote a “learning health system” through the availability of electronic medical information.²² As such, electronic information will greatly facilitate and enable clinical research, including CER.

Our study has several limitations. We studied only 21 providers using a non-randomized design. Our study was conducted in 1 geographic region among community-based, solo, and small-group practitioners, limiting generalizability. The providers have different baseline error rates, which may be a result of our sample size or due to true differences in the types of providers who chose to adopt one type of e-prescribing system over another. We have controlled for the baseline difference in error rates in our analysis, but future studies evaluating this issue should be conducted. Because prescribers were aware that we were reviewing their prescriptions, our error rates may be underestimates due to the Hawthorne effect. We were unable in our analysis to do patient case mix adjustment and thus cannot determine how the benefits and harms associated with particular mistakes by condition may differ. We studied only 2 e-prescribing systems and focused solely on prescribing errors. However, we reviewed several thousand prescriptions and the e-prescribing systems incorporated many features recommended by an expert review panel.²³ In addition, the implementation support was markedly different between the 2 groups and may have led to lower error rates among the standalone adopters who received more support.

CONCLUSIONS

Our study of this small group of providers is the first to our knowledge to quantitatively compare the effects of 2 commercially available e-prescribing systems on ambulatory prescribing errors. We found that even after adjusting for differences in baseline error rates, stand-alone system users had a 4-fold lower rate of errors compared with integrated system users at 1 year. These results highlight the need for CER of healthcare delivery systems, including EHRs, and the importance of examining factors such as system implementation, configuration, and functionality when evaluating the effectiveness of HIT.

Author Affiliations: From Department of Pediatrics (RK, ELA), Department of Public Health (RK, YB, ELA), Department of Medicine (RK), Weill Cornell Medical College, New York, NY; New York-Presbyterian Hospital (RK, ELA), New York, NY; Health Information Technology Evaluation Collaborative (RK, YB, ELA), New York, NY.

Funding Source: This project was supported by the Agency for Healthcare Research and Quality (1 UC1 HS016316), Rockville, MD.

Author Disclosures: The authors (RK, YB, ELA) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (RK, ELA); acquisition of data (RK, ELA); analysis and interpretation of data (RK, YB, ELB); drafting of the manuscript (RK, YB, ELB); critical revision of the manuscript for important intellectual content (RK, YB, ELB); statistical analysis (RK, YB); obtaining funding (RK); and supervision (RK).

Address correspondence to: Erika L. Abramson, MD, MS, Weill Cornell Medical College, 525 E 68th St, Rm M-610A, New York, NY 10065. E-mail: err9009@med.cornell.edu.

1. US Department of Health & Human Services. American Recovery and Reinvestment Act of 2009. http://www.hhs.gov/recovery/. Accessed December 4, 2009.

2. US Department of Health & Human Services. Health Information Technology: Initial Set of Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology; Final Rule. Vol 45: CFR Part 170; 2010.

3. Institute of Medicine. Initial priorities for comparative effectiveness research. http://www.iom.edu/Reports/2009/ComparativeEffectiveness ResearchPriorities.aspx. Published June 30, 2009. Accessed August 14, 2011.

4. Conway PH, Clancy C. Charting a path from comparative effectiveness funding to improved patient-centered health care. JAMA. 2010;303 (10):985-986.

5. Kaushal R, Kern LM, Barrón Y, Quaresimo J, Abramson EL. Electronic prescribing improves medication safety in community-based office practices. J Gen Intern Med. 2010;25(6):530-536.

6. Abramson E, Barrón Y, Quaresimo J, Kaushal R. Electronic prescribing within an electronic health record reduces ambulatory prescribing errors. Jt Comm J Qual Patient Safety. 2011;37(10):470-478.

7. Kohn LT, Corrigan J, Donaldson MS, Institute of Medicine (US). Committee on Quality of Health Care in America. To err is human: building a safer health system. Washington, DC: National Academy Press; 2000.

8. Collaborative NYeC. Health information service provider (HSP). 2008; http://www.nyehealth.org/index.php/resources/glossary#page_h_7. Accessed May 30, 2009.

9. Gandhi TK, Weingart SN, Seger AC, et al. Outpatient prescribing errors and the impact of computerized prescribing. J Gen Intern Med. 2005;20(9):837-841.

10. Bates DW, Kaushal R, Keohane CA, Cook EF. Center of Excellence for Patient Safety Research and Practice Terminology Training Manual. 2005:1-21.

11. Kaushal R. Using chart review to screen for medication errors and adverse drug events. Am J Health Syst Pharm. 2002;59(23):2323-2325.

12. The Joint Commission. The Official “Do Not Use” List of Abbreviations. http://www.jointcommission.org/assets/1/18/Official_Do%20 Not%20Use_List_%206_10.pdf. Published 2005. Accessed April 9, 2010.

13. Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc. 2008;15(5): 585-600.

14. Crosson JC, Isaacson N, Lancaster D, et al. Variation in electronic prescribing implementation among twelve ambulatory practices. J Gen Intern Med. 2008;23(4):364-371.

15. Devine EB, Wilson-Norton JL, Lawless NM, et al. Implementing an Ambulatory e-Prescribing System: Strategies Employed and Lessons Learned to Minimize Unintended Consequences (vol 4: Technology and Medication Safety). August 2008.

16. Ancker JS, Kern LM, Abramson E, Kaushal R. The Triangle Model for evaluating the effect of health information technology on healthcare quality and safety. [published online ahead of print August 20, 2011] J Am Med Inform Assoc.

17. The American Recovery and Reinvestment Act of 2009. http://www.recovery.gov/About/Pages/The_Act.aspx. Accessed April 13, 2011.

18. Menachemi N, Powers T, Au DW, Brooks RG. Predictors of physician satisfaction among electronic health record system users. J Healthc Qual. 2010;32(1):35-41.

19. National Committee for Quality Assurance. Measuring Quality. Patient Centered Medical Home. http://www.ncqa.org/tabid/631/default.aspx. Published 2011. Accessed March 21, 2011.

20. US Department of Health & Human Services. Legislation Co. Medicare “Accountable Care Organizations” Shared Savings Program— New Section 1899 of Title XVIII. https://www.cms.gov/sharedsavingsprogram/. Accessed March 21, 2011.

21. Blumenthal D. Launching HITECH. N Engl J Med. 2010;362(5):382-385.

22. Friedman CP, Wong AK, Blumenthal D. Achieving a nationwide learning health system. Sci Transl Med. 2010;2(57):57cm29.

23. Bell DS, Marken RS, Meili RC, Wang CJ, Rosen M, Brook RH. Recommendations for comparing electronic prescribing systems: results of an expert consensus process. Health Aff (Millwood). 2004;Suppl Web Exclusives:W4-305-317.

Download PDF: The Comparative Effectiveness of 2 Electronic Prescribing Systems