Publication
Article
The American Journal of Managed Care
Author(s):
Objectives: To investigate prescribers' rationales for overriding drug—drug interaction (DDI) alerts and to determine whether these reasons were helpful to pharmacists as a part of prescription order verification.
Study Design: An observational retrospective database analysis was conducted using override reasons derived from a computerized system at 6 Veterans Affairs medical centers.
Methods: Data on DDI alerts (for interactions designated as "critical" and “significant") were obtained from ambulatory care pharmacy records from July 1, 2003, to June 30, 2004. Prescribers' reasons for overriding alerts were organized into 14 categories and were then rated as clinically useful or not to the pharmacist in the assessment of potential patient harm.
Results: Of 291 890 overrides identified, 72% were for critical DDIs. Across the Veterans Affairs medical centers, only 20% of the override reasons for critical DDI alerts were rated as clinically useful for order verification. Despite a mandatory override reason for critical DDI alerts, 53% of the responses were “no reason provided." The top response categories for critical and significant DDI alerts were “no reason provided," “patient has been taking combination," and “patient being monitored."
Conclusions: When given the opportunity to provide a reason for overriding a DDI alert, prescribers rarely enter clinical justifications that are useful to order verification pharmacists. This brings into question how computerized physician order entry systems should be designed.
(Am J Manag Care. 2007;13:573-580)
An observational retrospective database analysis was conducted using override reasons derived from a computerized system at 6 Veterans Affairs medical centers.
Prescribers' reasons for overriding drug—drug interaction (DDI) alerts were organized into 14 categories and were then rated as clinically useful or not to the pharmacist in the assessment of potential patient harm.
Of 291 890 overrides identified, 72% were for critical DDIs; only 20% of these override reasons were rated as clinically useful for order verification. Despite a mandatory override reason for critical DDI alerts, 53% of the responses were “no reason provided."
Preventable adverse drug events comprise a large percentage of reported medical errors.1 An adverse drug event is defined as "an injury resulting from a medical intervention related to a drug."2(p29) Drug—drug interactions (DDIs) are common causes of preventable adverse drug events, with 20% to 30% of adverse drug events attributed to interactions between drugs.3 The risk of DDIs compromising patient safety is substantial, and the economic burden on the healthcare system that occurs when interactions lead to patient morbidity is considerable. These problems are well documented in the literature and warrant serious investigation of possible therapeutic management strategies.1,4,5
The prescription medication use process, which can be described in general phases, includes prescribing the medication, dispensing and administering the medication, and monitoring the patient. In each step of the process, there are opportunities to prevent potentially harmful DDIs or other adverse drug events from reaching the patient. Individuals in each phase can serve as a backup safety net, particularly when information is communicated between the individuals.
Exposure to potential DDIs has been suggested as a proxy measure for medical care outcomes because of their ability to affect patients' health.6 Exposure to clinically important DDIs can be associated with a wide range of outcomes, including lack of medication efficacy, poor tolerability, and serious adverse events.7 Studies and case reports8-11 have demonstrated that negative clinical outcomes are associated with harmful DDIs. In addition, some DDIs have been shown to lead to increased utilization of healthcare services such as higher rates of emergency department visits and hospitalizations.12-14
Delivery of care in the United States has been increasingly moving to ambulatory and outpatient settings, and several studies15-17 have been conducted to investigate the incidence of DDIs within those settings; the estimates have varied substantially among studies, ranging from 9.2% to 70.3% of patients receiving drugs with a potential DDI. When only clinically relevant DDIs are considered, the incidence drops to 1.2% to 23.3%.14,18 Prevention of exposure to potential DDIs is problematic at the prescribing end of the medication use process, as findings have shown that physicians and other prescribers fail to recognize between 37% and 47% of clinically meaningful DDIs.19
A potential solution for reducing the incidence of DDIs at the prescribing phase is the use of computerized prescriber order entry (CPOE) systems that allow prescribers to enter orders electronically.20 These systems can provide an immediate alert to a prescriber who has selected a medication that interacts with another medication the patient is receiving. However, it has been shown that physicians frequently override such alerts and that some believe that it should not be easy to override clinically important interactions.21 Other
research has found that the overrides by prescribers are generally justifiable, pointing to problems with the quality of the alerting systems.22,23 Some CPOE systems require prescribers to enter a reason if a decision is made to override an alert of clinical relevance. This can serve as a quality control aspect, while also providing information for the next steps in the medication use process. If not appropriately managed, there is a danger that CPOE alerts may be overlooked just as such alerts are in community pharmacies because of their frequency and lack of relevance.24,25 Because interacting drugs continue to reach patients, despite the systems set up to prevent them, additional information is needed to understand why prescribers override DDI alerts within existing CPOE systems.
The objective of this study was to evaluate the DDI override reasons provided by prescribers using a CPOE system at 6 Veterans Affairs medical centers (VAMCs). Specifically, the objectives of the study were (1) to determine the frequency at which physicians override DDI alerts, (2) to categorize the override reasons, and (3) to determine whether the override reasons communicated useful information to pharmacists dispensing the prescribed medication.
METHODS
Data related to prescribers' reasons for overriding DDI alerts used in the VAMC system were obtained from ambulatory pharmacy dispensing records at the following 6 VAMCs: VA Ann Arbor Healthcare System, Ann Arbor, Mich; VA Boston Healthcare System, Boston, Mass; Carl T. Hayden VA Medical Center, Phoenix, Ariz; San Francisco VA Medical Center, San Francisco, Calif; VA Puget Sound Health Care System, Seattle, Wash; and Southern Arizona VA Health Care System, Tucson. The study was approved by the Humans Subjects Protection Program at the University of Arizona and by the institutional review boards and research committees at the participating VAMCs.
The override reasons were collected during a 1-year period from July 1, 2003, to June 30, 2004, for all the sites except 1. One VAMC turned off the alerting system before the beginning of the study. Once the alert system was turned on, 6 months of data were captured for this site. The data included medications involved in the DDI and the reasons provided to override the alerts. No patient-level or prescriber-level data were collected in this study.
The VA classifies 2 levels of severity for DDIs, including "critical" and "significant" interactions. Most of the decisions about whether an interaction is critical or significant are made at the national level by the VA Pharmacy Service. Individual VAMCs have the ability to upgrade an interaction to critical but cannot downgrade a critical interaction to significant. The VAMCs may add additional interactions to the local drug file and may make a local determination on the severity. An internal VA committee is responsible for maintaining the list of combinations that are considered to be interacting drug pairs. When the VAMC CPOE system detected that the prescriber was entering a prescription for a medication that could interact with another medication in the patient's current medication profile, a DDI alert message appeared. At this point, the prescriber could cancel the order or could override the alert and complete the prescription. Typically, providers in the VAMC system are not given the option to provide a reason for overriding significant DDIs. In some instances, individual sites can upgrade significant interactions to include an override reason response field. For critical interaction alerts, prescribers are required to document an override reason. The provider could type any message into the override reason field (free text). There were no preset responses available such as drop-down menus. Once a provider verifies that an order is desired, the order and corresponding override reason are sent to the pharmacy to be reviewed and approved.
After excluding duplicate messages for a particular DDI pair, the reasons provided by prescribers for overriding DDI alerts were organized into 14 major categories developed by the authors. These are given in Table 1. Each reason was then evaluated for its utility to the pharmacist who was responsible for evaluating each medication order (referred to as order verification) before submitting the order to the dispensing area for packaging and distribution of the pharmaceutical product to the patient. Reasons for DDI alert overrides were rated by 2 of us (AJG, MHM) as being clinically useful or not to the pharmacist for his or her assessment of the potential for patient harm before dispensing the medication. If the pharmacist rater perceived that the physician had clearly documented awareness of the potential interaction and had indicated that measures had been taken to mitigate harm, the override reason was rated as useful. Given that medical records were not used in this study, the rating was based solely on the reason provided by the prescriber.
Categorization of each override reason and grading of its utility to the pharmacist were conducted by 2 of us (AJG, MHM), who evaluated the reasons separately. The initial classifications and utility assessments were then evaluated by a third individual (YK) to identify discrepancies between results of the 2 raters. Discrepancies were discussed by the raters to reach a consensus. In situations in which no consensus was reached, override reasons were evaluated by the entire research team for classification. The research team was composed of 5 investigators (JEM, EPA, GHS, DCM, and Jacob Abarca) on the study team, all having clinical pharmacy training and experience. Interrater reliability was assessed using a k statistic.
RESULTS
A total of 291 890 DDI overrides occurred during the 1-year study period (Table 2). The DDI override reasons were reviewed, and duplicate responses for a particular DDI pair were combined, leaving 15 848 unique override reasons that were assessed and categorized. The overall agreement rate between the 2 pharmacist raters in categorizing the override reasons and in determining clinical utility was almost 92% before consensus discussion. The ? statistics for the first round and for the overall agreement for clinical usefulness were 0.42 and 0.86, respectively (P < .001).
Seventy-two percent of DDIs were critical interactions (Table 3). Twenty percent of the critical DDI override reasons (including no responses) were rated as useful. An override reason was not provided for 53% of these critical DDIs. When override reasons were documented, approximately 43% of those for critical interactions were rated as useful, and 50% of those for significant DDIs were rated as useful. For significant DDIs, 4% included an override reason, with 50% of those justifications being rated as useful. When including significant and critical interactions, the most common response sent to the pharmacist was nothing (classified as "no reason provided" [Table 2]). The 3 most common categories were identified for each of the study sites. There was consistency in that 4 of 6 sites had the same 3 categories ("no reason provided," "patient has been taking combination," and "patient being monitored") in the same order. The other 2 sites shared the same top 3 categories ("no reason provided," "prescriber aware of interaction," and "patient has been taking combination").
DISCUSSION
Despite the availability of a system to convey to all healthcare professionals useful clinical information about reasons to dispense 2 medications that interact, this study found that 53% of the time no reason was provided by a prescriber, despite the requirement to provide an override reason (for a critical DDI alert). This may represent a substantial flaw in the ability to use automated systems to prevent serious medical errors and to communicate the medical rationale for prescribing products in combinations that may cause harm to the patient. The results of this study bear some similarities to other investigations of DDI alerts. Weingart et al22 studied physicians' decisions to override alerts for allergies and DDIs. They found that 89.4% of clinically significant DDI alerts were overridden. However, subsequent medical record reviews by an expert panel indicated that 95.6% of the time the decision to override alerts was deemed appropriate. Some of the justifications provided by the expert panel for appropriate overrides in that study were similar to those classified in the present study, including the interactions were not clinically significant (21.6%), the patient currently tolerated (21.6%), the patient was no longer taking 1 of the offending agents (8.0%) or had previously tolerated the combination (12.3%), and a short-term course of therapy was planned for 1 of the agents (6.2%). Furthermore, 40.6% of alerts were considered invalid (ie, not appropriate for the situation).22 Although the present study did not evaluate whether the overrides where justifiable or not, it is feasible that the prescribers at the VAMCs in this study were usually appropriately overriding the DDI alerts. Unfortunately, it is difficult to assess because a reason for override was not provided 84% of the time. The prescribers may not have felt the need to provide override reasons, knowing that other health professionals in the VAMC system have full access to patient medical records. Or, the prescribers may have believed that most of the alerts were not significant and that providing a response was an increased burden associated with the prescribing process that could be safely ignored. In addition, prescribers may not view the alert system as a means of communication to the pharmacist but rather as a personal tool to aid in their decision making.
Whatever the rationale, the fact that most of the time prescribers bypassed this opportunity to provide an override reason is problematic. Although the prescriber may know that the patient has tolerated the medication in the past or plans to carefully monitor the patient for adverse events, the absence of any information places additional burden on other healthcare professionals and especially on pharmacists, who must then make a determination about the clinical appropriateness and risk of harm to the patient.
Because the VAMC system does not generally provide an option for documenting an override reason for significant DDIs, it was surprising that 4% were accompanied by an override reason. It is possible that sites have upgraded these significant alerts to critical alerts. Another possibility is that the reason was meant for a critical alert that was prescribed during the same prescribing session. The VAMC symptom prompts the provider for an override comment only once for all checks if at least 1 is critical. That comment is recorded for all the order checks regardless of the level of severity of the check. Therefore, the comment could appear for a critical DDI alert and for a significant DDI alert if they occur in the same prescribing session.
Although a response is mandatory for critical DDI overrides in the VAMC system, 53% of these response fields were left blank. A possible explanation for a blank field is that the system interprets the space bar or the enter key as a response and essentially allows no response from the prescriber, which is coded as no reason provided. Individual sites may also have the ability to turn the alert system off or to permit an individual user to customize which interactions generate an alert. It is worth considering whether additional safeguards could be put in place to ensure appropriate communication between prescribers and pharmacists.
The issue of alert fatigue is problematic with many automated healthcare information systems.26 Over time, prescribers may become desensitized to DDI alerts, similar to findings from studies18,19,25 involving pharmacists. From an information technology viewpoint, incorrect information is worse than no information. If DDI systems were more accurate and clinically relevant, then providing an override reason should be mandatory. Part of the problem may be that the CPOE system displays inappropriate alerts. Other studies21,27 evaluating CPOE documented that pharmacists and prescribers lack confidence in the ability of computer systems to accurately alert about DDIs. Although all factors affecting override reasons were not examined in this study, it is important to investigate methods that could enhance communication between prescribers and pharmacists to improve patient safety via reduction in exposure to potential adverse events. Communication between prescribers and pharmacists is a critical step to ensuring patient safety, and when CPOE systems are used, they should enhance the ability of practitioners to work together toward that end.
Using free-text fields for providing override reasons may contribute to the lack of communication between physicians and pharmacists. Incorporating preformatted responses and drop-down menus to express clinicians' rationales may enhance communication.
The question of whether a DDI will cause an adverse outcome is a reason why many providers are confused about how relevant various DDI alerts are in clinical practice.7 Many factors affect whether a DDI manifests as a potential or an actual interaction. For example, drug characteristics (eg, therapeutic index) can play an important role in determining the significance of a given DDI.28 Drug—drug interactions that involve drugs with a low therapeutic index (such as warfarin sodium, digoxin, and some anticonvulsant drugs) are considered clinically significant by most clinicians.28 The ambiguity of the clinical relevance of some DDIs adds to this confusion because the outcomes are often not clearly defined, and there is considerable variation in how different clinicians interpret the clinical relevance of DDIs.7 Some clinicians tend to focus only on DDIs associated with serious adverse events such as disability, hospitalization, and death, while others consider DDIs that are associated with any unfavorable outcome as clinically relevant
regardless of the severity of that outcome.7 Other authors have suggested that clinically relevant DDIs be defined as cases in which a modification in drug treatment is required.7 This could include discontinuation of 1 or more drugs, addition of another drug, or close monitoring of the patient.
Another observation across the VAMC healthcare system was that the alerts were tied to the generic medication name, regardless of dosage form. Consequently, alerts were generated for medications that are not likely to interact (such as the combination of oral and topical medications). This has subsequently been corrected to eliminate most topical drugs from the VAMC DDI interaction alerts.
A solution to the issue of alert fatigue is to give the prescriber the ability to customize the DDI alert list based on the individual prescriber's experience, patient characteristics, and practice pattern. Theoretically, the number of clinically irrelevant alerts for this prescriber would be substantially reduced, but at some point this could negate the value of the alert system. It is the responsibility of the system to support the local policies and guidelines. Allowing prescribers to modify alerts could be dangerous, placing patients at increased risk for DDIs and perhaps increasing legal liability when patients are harmed by an interaction that was deleted from the system. Modifying alerts could also introduce unacceptable levels of variation into the overall alerting process. At a minimum, repetitive alerts could be eliminated once a prescriber provides a rational explanation for overriding the alert for a particular patient.
The 4 most common DDI override categories were "no reason provided," "patient has been taking combination," "patient being monitored," and "prescriber aware of interaction." "Patient has been taking combination" was consistently in the top 3 across the VAMCs. Although the prescriber may have intended to convey that the patient had tolerated the combination in the past, those override reasons that were not explicit in this regard were rated as not clinically useful. About half of the responses specifically indicated that the patient had tolerated the combination or had no problems. If all responses indicating the patient had taken the medications before were rated as useful (giving the prescriber the benefit of the doubt), the overall usefulness would still only increase to 7.7%.
There are several limitations to consider when interpreting the findings of this research. Because the study was conducted within the VAMC system, results may not be generalizable to other ambulatory practice sites. The raters were limited to the responses provided by prescribers and could not ask for clarification about the intent of a response. This limitation especially came into play when interpreting responses for "patient has been taking combination" and "prescriber aware of interaction." These situations were considered not clinically useful simply because the pharmacist would be forced to contact the prescriber to verify that the patient was not at risk of an adverse event, wasting valuable time for both practitioners. In reality, pharmacists probably let many interactions go through the system for fear of "crying wolf" too often. Other limitations of this study are that we did not examine the response of pharmacists to specific override reasons and that we did not evaluate the clinical outcomes associated with the potential DDIs reaching patients. Because this study did not collect patient-level data, it was not possible to evaluate the clinical consequences of these interactions if they reached the patient. Finally, this study did not collect data regarding how often a DDI alert resulted in an aborted medication order or an order for a drug that did not interact.
Given the results of this study, it is clear that additional attention is needed to provide solutions that will improve the prescriber's ability to communicate with the pharmacist and to ensure optimal patient outcomes with every medication prescribed. The following is a list of suggestions for improving patient outcomes related to exposure to DDIs:
1. A feedback mechanism should be incorporated into the DDI alert process. Reducing the frequency of clinically irrelevant alerts increases the importance of the remaining
alerts. How prescribers and pharmacists respond to these alerts is then increasingly important. Override reasons (or, most important, the lack of response) need to be reviewed and an educational process used to modify practices that compromise patient safety.
2. The patient's medication history should be incorporated into the DDI alerts. For each patient, systems should recognize responses to previous alerts and prescriber responses. This information could be presented to prescribers at the end of the order entry process.
3. Once an acceptable override reason is provided for a particular patient, repeat alert messages on refills are eliminated.
4. Drop-down menus could be used to more clearly and efficiently communicate override reasons.
5. Mandatory override reason responses could be expanded to include more than the most severe DDIs (in this case, requiring override reasons for significant and critical interactions).
6. Alternative management strategies should be available to prescribers when DDI alerts are first issued. This would provide opportunities for timely decisions to make changes in medication selection.
7. When guidelines require patient monitoring, automatic generation of reminders for laboratory tests and office visits should occur.
In conclusion, the VAMC alert override system may provide limited useful clinical information and brings into question how such systems should be designed. For critical interactions (with a mandatory response), physicians provide override reasons only 53% of the time, and 80% are not useful to the order verification pharmacist. It is clear that considerable thought and empirical testing may be required for information systems to provide clinically useful information for all healthcare providers throughout the medication use process to improve patient safety.
Acknowledgments
We acknowledge the assistance of Raymond L. Woosley, PhD, MD; Rick A. Rehfeld, BS; Perry Richmond, BS; and Jacob Abarca, PharmD, MS.
Funding Source: This study was supported by grant 2 U18 HS10385-04 from the Agency for Healthcare Research and Quality Centers for Education and Research on Therapeutics (Arizona CERT).
Author Affiliations: From the Center for Health Outcomes and PharmacoEconomic Research, College of Pharmacy, University of Arizona (AJG, MHM, YK, JEM, EPA, GHS, WNJ, DCM), and Southern Arizona VA Health Care System (WNJ), Tucson; VA Ann Arbor Healthcare System, Ann Arbor, Mich (GPS); VA Puget Sound Health Care System (WPN, CTH), and School of Medicine (PN, CTH) and School of Pharmacy (CTH), University of Washington, Seattle; VA Boston Healthcare System, Boston, Mass (AH); and San Francisco VA Medical Center, San Francisco, Calif (DCD).
Author Disclosure: Dr Grizzle had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
The views expressed are those of the authors and may not necessarily reflect the views of the Department of Veterans Affairs. Dr Murphy served on an advisory board for Epocrates, which produces drug interaction software. However, the research in this article was not discussed during any meeting. The authors (AJG, MHM, YK, EPA, GHS, WNJ, GPS, WPN, AH, DCD, CTH, DCM) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter discussed in this manuscript.
Authorship Information: Concept and design (AJG, JEM, EPA, GHS, WNJ, DCM); acquisition of data (AJG, WNJ, GPS, AH, DCD, CTH, DCM); analysis and interpretation of data (AJG, MHM, YK, JEM, EPA, GHS, WPN,DCD, CTH, DCM); drafting of the manuscript (AJG, MHM, JEM, EPA, WNJ, GPS, DCM); critical revision of the manuscript for important intellectual content (AJG, YK, JEM, EPA, GHS, WNJ, GPS, CTH, DCM); statistical analysis (AJG, MHM, YK, EPA, GHS, DCM); provision of study materials or patients (AJG, AH, DCD, DCM); obtaining funding (AJG, JEM, GHS, DCM); administrative, technical, or logistic support (GHS, WNJ, WPN, AH); supervision (AJG, DCM).
Address correspondence to: Amy J. Grizzle, PharmD, Center for Health Outcomes and PharmacoEconomic Research, College of Pharmacy, University of Arizona, PO Box 210202, Tucson, AZ 85721-0202. E-mail: grizzle@pharmacy.arizona.edu.
1. Kohn LT, Corrigan JM, Donaldson MS, eds. To Err Is Human: Building a Safer Health System.Washington, DC: National Academy Press; 1999.
2. Bates DW, Cullen DJ, Laird N, et al; ADE Prevention Study Group. Incidence of adverse drug events and potential adverse drug events: implications for prevention. JAMA. 1995;274:29-34.
3. Kuhlmann J, Muck W. Clinical-pharmacological strategies to assess drug interaction potential during drug development. Drug Saf. 2001;24: 715-725.
4. Gaddis GM, Holt TR,Woods M. Drug interactions in at-risk emergency department patients. Acad Emerg Med. 2002;9:1162-1167.
5. Baciu A, Stratton K, Burke S, eds. The Future of Drug Safety: Promoting and Protecting the Health of the Public. Washington, DC: National Academy Press; 2006.
6. Becker ML, Kallewaurd M, Caspers PW, Schalekamp T, Stricker BH. Potential determinants of drug—drug interaction associated dispensing in community pharmacies. Drug Saf. 2005;28:371-378.
7. Preskorn SH. Drug–drug interactions: proof of relevance (part I). J Psychiatr Pract. 2005;11:116-122.
8. Albengres E, Le Louet H, Tillement JP. Systemic antifungal agents: drug interactions of clinical significance. Drug Saf. 1998;18:83-97.
9. Finch CK, Chrisman CR, Baciewicz AM, Self TH. Rifampin and rifabutin drug interactions: an update. Arch Intern Med. 2002;162:985-992.
10. Michalets EL, Williams CR. Drug interactions with cisapride: clinical implications. Clin Pharmacokinet. 2000;39:49-75.
11.Weinberger J. Adverse effects and drug interactions of antithrombotic agents used in prevention of ischaemic stroke. Drugs. 2005;65:461-471.
12. Bjerrum L, Andersen M, Petersen G, Kragstrup J. Exposure to potential drug interactions in primary health care. Scand J Prim Health Care. 2003;21:153-156.
13. Goldberg RM, Mabee J, Chan L,Wong S. Drug—drug and drugdisease interactions in the ED: analysis of a high-risk population. Am J Emerg Med. 1996;14:447-450.
14. Jankel C, Speedie S. Detecting drug interactions: a review of the literature. Ann Pharmacother. 1990;24:982-989.
15. Malone DC, Hutchins DS, Haupert H, et al. Assessment of potential drug—drug interactions with a prescription claims database. Am J Health Syst Pharm. 2005;62:1983-1991.
16. Peng CC, Glassman PA, Marks IR, Fowler C, Castiglione B, Good CB. Retrospective drug utilization review: incidence of clinically relevant potential drug—drug interactions in a large ambulatory population. J Manag Care Pharm. 2003;9:513-522.
17. Phillips DP, Christenfeld N, Glynn LM. Increase in US medicationerror deaths between 1983 and 1993. Lancet. 1998;351:643-644.
18. Bergk V, Gasse C, Rothenbacher D, Loew M, Brenner H, Haefeli WE. Drug interactions in primary care: impact of a new algorithm on risk determination. Clin Pharmacol Ther. 2004;76:85-96.
19. Glassman PA, Belperio P, Simon B, Lanto A, Lee M. Exposure to automated drug alerts over time: effects on clinicians' knowledge and perceptions. Med Care. 2006;44:250-256.
20. Cutler DM, Feldman NE, Horwitz JR. U.S. adoption of computerized physician order entry systems. Health Aff (Millwood). 2005;24:1654-1663.
21. Magnus D, Rodgers S, Avery AJ. GPs' views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther. 2002;27:377-382.
22. Weingart SN,Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163:2625-2631.
23. Hsieh TC, Kuperman GJ, Jaggi T, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc. 2004;11:482-491.
24. Chui MA, Rupp MT. Evaluation of online prospective DUR programs in community pharmacy practice. J Manag Care Pharm. 2000;6:27-32.
25. Murphy JE, Forrey RA, Desiraju U. Community pharmacists' responses to drug—drug interaction alerts. Am J Health Syst Pharm. 2004;61:1484-1487.
26. Peterson JF, Bates DW. Preventable medication errors: identifying and eliminating serious drug interactions. J Am Pharm Assoc (Wash). 2001;41:159-160.
27. Abarca J, Malone D, Skrepnek G, et al. Community pharmacy managers' perception of computerized drug—drug interaction alerts. J Am Pharm Assoc (Wash). 2006;46:148-153.
28. Thomas A, Routledge PA. Drug interactions in clinical practice. Focus Pharmacovigilance Bull. 2003:1-7. Available at: http://www.sfm.univr.it/eng/focus/interactions.pdf.