Frequency of and Harm Associated With Primary Care Safety Incidents

Objective: To assess frequency and severity of patient safety incidents in primary care.

Study Design: Cross-sectional survey of health- care professionals in Swiss primary care offices.

Methods: Physicians and nurses in primary care offices were surveyed about the frequency and severity of 23 safety incidents. Differences between professional groups and types of offices were analyzed. Reported incidents were classified in a matrix.

c

2

Results: A total of 630 individuals (50.2% physi- cians, 49.8% nurses) participated. Among them, 30% of physicians (95% confidence interval [CI] 25%-35%) and 16.6% of nurses (95% CI 12%-21%) reported that at least 1 of the incidents occurred daily or weekly in their offices (16.1, P <.001). On average, each responder reported a total of 92 incidents during the preceding 12 months (mean of 117 events for physicians, mean of 66 events for nurses; P <.001). Documentation failure was reported most frequently.The highest fraction of last occurrences with severe injury or death was for diagnostic errors (4.1%). Unadjusted for caseload, staff working in medical centers reported higher frequencies of several incidents. The frequency-harm matrix suggests that triage by nurse at initial contact, diagnostic errors, medication errors, failure to monitor patients after medical procedures, and test or intervention errors should be prioritized for action.

Conclusions:This study presents a supplemental approach to identification of safety threats in primary care. Many incidents occur regularly and are highly relevant for healthcare professionals’ daily work.The results offer guidance on setting priorities for patient safety in primary care.

(Am J Manag Care. 2012;18(9):e323-e337)

Physicians’ and nurses’ assessments of the frequency and harm of safety incidents can be a supplemental method to study patient safety in the primary care office.

• Because incident reporting varies among professional groups and types of offices, it is essential to involve physicians and nurses and to consider different types of office organization when safety incidents in primary care are studied.

• In this sample of primary care offices, triage by nurse, diagnostic errors, failure to monitor patients after medical procedures, test or intervention errors, medication errors, and documentation failure were identified as “hot spots” that warrant further action.

Patient safety is a major concern in healthcare systems worldwide. Although most safety research has been conducted in the inpatient setting, evidence indicates that medical errors and adverse events pose a serious threat for patients in the primary care setting as well.^1-3 Gandhi and Lee note that safety concerns in the outpatient setting differ from those in the hospital setting in obvious and nonobvious ways.⁴ Diagnostic errors and adverse drug events have been identified as frequent safety concerns, whereas less is known about the safety of outpatient procedures and the consequences of coordination and continuity-of-care failures. Hospital and outpatient care also differ in the infrastructure and processes they have available to detect, monitor, and address safety issues. Information about frequency and outcomes of safety incidents in primary care is required to identify “hot spots,” to evaluate these hot spots for priority, and to take the actions needed. Two methods are commonly applied to identify and quantify safety problems in outpatient care: incident reporting and chart review.

Incident reporting has a long tradition in clinical risk management and is increasingly used in outpatient care.^5-7 Indeed, incident reporting has been the dominant method for study of safety incidents in primary care.⁸ It is based on voluntary and usually anonymous reports of physicians and nurses and is being used to describe the types and characteristics of patient safety incidents. These reports may vary considerably with respect to the included information, and the likelihood of “true” incidents being reported is unclear. Studies based on this method describe strong variations in the number of reports submitted.⁹ Moreover, professional groups differ in their frequency of reporting. In contrast to hospital settings, in primary care physicians report considerably more incidents than nurses do.^10,11 O’Beirne et al concluded from a very low report rate (<1 report per person per year) that incident reporting may be a costly but not very effective way to study safety problems in primary care.¹⁰

With chart review, medical records are analyzed by independent experts in order to identify adverse events and to assess potential harm and preventability in each case.¹² This analysis requires complete and correct patient documentation to provide valid results. In many cases relevant information may be unavailable.¹³ As chart review is a time-consuming approach, many resources are needed to analyze a large number of patient records at different primary care offices. This large sample is necessary to cover a broader range of incidents and contexts.

What chart review and incident reporting have in common is that it is impossible to systematically cover information on the entire range of safety events in medical offices. Instead, these methods cover events with a relative high likelihood of being reported or being documented in the medical records (eg, adverse drug events, errant drug prescriptions). The main aim of this study is to apply a supplemental method to gain additional insight into safety hot spots in primary care. Using survey methodology, we assessed the frequency and severity of incidents in the offices based on the experience of both physicians and nurses. We classified events in a frequency-harm matrix that may be useful for identification and prioritization of safety hot spots.

METHODSSurvey

Appendix A

In a cross-sectional study, primary care physicians and nurses working in outpatient offices were surveyed about the occurrence of safety problems in their offices. A preliminary list of safety incidents was derived from empirical studies.^{1-3,5-7,9,14-22} We also analyzed incidents reported to Swiss primary care critical incident reporting systems. After discussion with primary care safety experts, physicians, and nurses, the list was concomitantly adapted and finally included 23 potential incidents. The incidents were organized along the care continuum in chapters titled diagnostic process, medication, other therapeutic and preventive measures including interventional procedures, patient encounter and information, organization and work flow in the office, cooperation with other providers, and storage of drugs and materials. Each incident was accompanied by 2 sensitively selected brief examples to guide respondents, explain the range of problems covered by the class of incident, and support memory. Respondents were asked to respond to 2 questions for each potential incident: (1) How frequently did the incident occur in their office during the past 12 months? Response categories were “daily,” “weekly,” “monthly,” “yearly,” “never,” and “never in the past 12 months but at least once before.” Respondents were instructed to report all incidents they were aware of irrespective of their personal role in the incident. (2) What was the severity of harm associated with the last occurrence of the incident in the office (“last occurrence”)? The response codes were “no harm,” “minor harm,” “moderate harm,” “severe harm,” and “death.” Concise definitions of the severity ratings were provided and respondents were instructed to consult these before answering (). We also assessed the safety climate in the survey (data not reported). Background information on the respondents and their working environment was obtained as well. An iterative pretest was conducted with 8 practitioners, and the survey was subsequently revised.

Sample

The sample consists of all primary care physicians (n = 627) formally organized in 4 large physician networks and included 472 offices. The physician networks were selected because they cover a broad range of office types and regions in the German-speaking part of Switzerland including 2 metropolitan areas, smaller cities, and rural regions. Two networks oblige their members to adhere to guidelines and quality-ofcare standards (eg, participation in quality circles). The others predominantly collaborate on organizational and financial issues. The sampled physicians were mailed the questionnaire by post, together with a cover letter and prepaid envelope. Each physician received 2 complete survey sets. The physician was instructed to pass the set labeled “nurse” to 1 nurse according to the alphabetic position of the first letter of the last names of all nurses working in the office. The study was promoted by representatives of the networks (eg, medical directors, quality managers) before and during data collection. One network offered compensation for participation (approximately US $45), but very few offices (<5) requested it. A few days after the mailing, nurses in each participating office were sent a letter to inform them about the study, announce the survey, and explain how nurses would receive the questionnaire by the physician. The survey was completely anonymous. A reminder including an identical set of questionnaires was sent after 2 weeks to the entire sample. Ethical approval is not necessary for this study in Switzerland.

Data Analysis

Frequency and severity of reported incidents were analyzed descriptively. Chi-square tests and t tests were conducted to test for group differences in frequency and severity of incidents between occupational groups (physicians vs nurses) and types of office (single handed vs joint practice vs medical center). All tests were 2-sided and P <.05 was considered significant. We did not adjust for multiple testing.

To support interpretation and usability of results, we developed a 2-dimensional frequency-harm matrix similar to the Veterans Health Administration Safety Assessment Code Matrix, the matrix used by the UK National Patient Safety Agency, and to other classification systems.^23,24 Such classifications are used to categorize potential safety incidents and to prioritize hazards for risk management. Rather than using judgments of likelihood and consequences of potential events, we used the quantitative data provided by respondents in our study to plot the reported frequency of incidents against the severity of the last occurrence. As each incident was described by 2 response distributions (frequency and severity),² analytical steps were conducted to collapse and reorganize the data.

Frequency. First, the frequency of incidents was transformed to the mean number of occurrences of each incident reported per respondent. The number of responses in the daily category were multiplied by 308 (the mean number of office days per year), weekly occurrences were multiplied by 52, monthly occurrences were multiplied by 12, and yearly occurrences were multiplied by 1. Incidents that never occurred were multiplied by 0. The sum of these events was divided by the number of respondents per incident. This average number of incidents per respondent and year was then classified as “singular” (<1 event per year per observer), “rare” (1-4 events per year per observer), “occasional” (4-6 events per year per observer), “probable” (6-12 events per year per observer), and “frequent” (>12 events per year per observer). Because responders reported incidents in their offices, these were inn cidence rates per observer with the office as the space of observation.

Severity. Second, the distributions of the severity ratings of last occurrences were transformed to 4 categories: “catastrophic” (at least 1 death reported), “critical” (at least 1 episode of severe harm reported, but no death), “moderate” (>25% of respondents reported minor or moderate harm, and none reported death or severe harm), and “marginal” (>75% of respondents reported no harm, and none reported death or severe harm).

RESULTS

Table 1

Table 2

Appendices B

C

Of the 1254 invited individuals (50.2% physicians, 49.8% nurses), 630 returned the questionnaire for a response rate of 50%. Characteristics of responders and the offices are provided in . The frequency of occurrence of the 23 hotspot incidents during the past 12 months is reported in. Among the respondents, 30% of the physicians (95% confidence interval [CI] 25%-35%) and 16.6% of the nurses (95% CI 13%-21%) reported that at least 1 of the hot-spot incidents occurred daily or weekly in their offices (c2 15.8, P <.001). On average, each responder reported a total of 92 incidents during the preceding 12 months (mean of 117 events for physicians, mean of 66 events for nurses; t test, P <.001). The 3 incidents reported with highest frequency (at least yearly occurrence) were patient documentation was incomplete, unavailable, unclear, or incorrect (88.4%); information from external provider was missing, incomplete, or errant when required (81.7%); and required medication was not prescribed, administered, or dispensed (80.6%). The reported frequency of incidents differed considerably between nurses and physicians (see and ). Physicians reported significantly more frequent occurrences of all incidents except failure to adequately monitor patient subsequently after therapeutic procedure in the office, failure to communicate correct test results or diagnosis to patient, administered or dispensed expired medication, and external provider receives incomplete or incorrect information, or receives it too late.

Frequency of incidents was also associated with office type (eAppendices B and C): all except 3 incidents (failure to adequately monitor patient subsequently after therapeutic procedure in the office, administered or dispensed expired medication, use of defective or expired medical materials or supplies) were reported with significantly higher frequency by staff working in medical centers compared with staff working in joint offices and single-handed offices.

Table 3

Severity of harm associated with last occurrences of incidents is reported in . The incidents most likely to be associated with at least minor harm were error in diagnosis or diagnosis delayed (66.5% of responders reported at least minor harm at last occurrence); urgency of patient need not recognized at contact (56.3% of responders reported at least minor harm at last occurrence); and errant or incorrect prescription, administration, or dispensing of medication: known interaction or contraindication or intolerance or allergy not considered (47.9% of responders reported at least minor harm at last occurrence). The highest fraction of last occurrences with severe injury or patient death was reported for diagnostic errors (4.1% of responders reported at least minor harm at last occurrence). Across incidents, 83.8% (95% CI 81%-87%) of responders reported at least 1 incident of minor harm at last occurrence of the events (88.9% physicians [95% CI 85%- 92%], 78.7% nurses [95% CI 74%-83%]; X² 12.2, P <.001); and 5.4% (95% CI 4%-7%) reported at least 1 incident of severe harm or patient death (6.3% physicians [95% CI 4%- 10%], 4.5% nurses [95% CI 2%-7%]; x² 1.1, P = .299).

Severity of last occurrences was significantly associated with professional group for 6 incidents. Physicians were more likely to report at least minor harm for errors in diagnosis (76.4% of physicians [95% CI 71%-81%] vs 54.3% of nurses [95% CI 48%-61%]; x² 29.3, P <.001); medication errors in terms of interactions or drug allergies not being considered (55.6% of physicians [95% CI 50%-61%] vs 38.8% of nurses [95% CI 33%-45%]; x² 14.7, P <.001); insufficient monitoring of drug treatment and side effects (47.7% of physicians [95% CI 42%-54%] vs 29.8% of nurses [95% CI 24%-36%]; x² 16.7, P <.001); indicated therapeutic intervention not performed or delayed (28.1% of physicians [95% CI 22%-34%] vs 17.3% of nurses [95% CI 13%-23%]; x² 7.5, P = .006); and relevant results from internal or external test missed, not worked up, or not followed up in due time (47.9% of physicians [95% CI 42%-54%] vs 38.4% of nurses [95% CI 33%-44%]; x² 5.0, P = .026). By contrast, nurses were more likely to report at least minor harm at last occurrence when an expired medication was administered or dispensed (8.8% of physicians [95% CI 5%-12%] vs 18.4% of nurses [95% CI 14%-23%]; x² 9.3, P = .002). The fraction of incidents with at least minor harm at last occurrence was generally higher when reported by staff working in medical centers compared with single-handed and joint offices. Exceptions were incidents 7, 14, and 22 (required medication not prescribed, administered, or dispensed; tests or treatments performed without patient consent; and administered or dispensed medication was expired, respectively)—for which no significant associations between severity of harm and office type were observed.

Figure

The presents the frequency-harm matrix after reclassification of the responses. Four incidents were observed singularly (numbers 14 and 22) or rarely (numbers 9 and 23), and were usually not associated with harm at last occurrence. Four incidents resulted in catastrophic consequences for at least 1 patient at last occurrence (numbers 4, 6, 12, and 13). Among these incidents, number 13 occurred most frequently (ie, on average 6 times a year). Incident numbers 2, 5, 7, and 8 occurred at least occasionally in the participating offices and resulted in severe, nonreversible harm in at least 1 patient. Of these 4 incidents, 3 related to medication errors.

DISCUSSION

Our study reports the frequency and harm of 23 welldefined patient safety hot spots in Swiss primary care offices. The results show that several incidents happen on a regular basis in the offices and are thus highly relevant for healthcare professionals’ daily work. Many of the incidents investigated have the potential to severely harm patients. About half of the incidents (11 out of 23 incident types) led to severe harm or patient death at last occurrence—at least in 1 of the responding offices. We also found considerable differences between professional groups (physicians vs nurses) and office types (medical centers vs single-handed and joint offices) in terms of reporting incident frequency and severity. This highlights the importance of involving both physicians and nurses and of considering different types of office organization when threats to patient safety in primary care are analyzed and prioritized for action taking.

Our study provides detailed information about frequency and harm as observed by physicians and nurses. However, due to the multidimensionality of the data, identification and prioritization of safety hot spots for future action are complicated. For this purpose, we designed the frequency-harm matrix, which presents results in a consolidated, concentrated form. This matrix can be used as a tool for risk assessment and prioritization (eg, by participating offices or physician networks). The main advantage of the matrix is that self-reported frequencies of the 23 incidents were transformed to a “common currency,” namely, the average number of events per year and observer. Using this metric, incidents were then classified using a specified set of margins and rules. For example, we decided that all incidents that resulted in at least 1 patient death should be classifi ed as catastrophic. Researchers should focus on incidents that might severely harm patients or even lead to death, especially incidents that occur frequently. However, other approaches to collapsing and classifying incidents may also be useful (eg, centiles of response distributions, median reports of frequency and severity). No matter which metrics and margins are used, their effect on the classifi cation table needs to be borne in mind when interpreting the results.

Based on the rules we used for classifi cation of events in the matrix, the following incidents are of high relevance: not recognizing the urgency of patient’s needs at contact, diagnostic errors, lack of monitoring of patients after medical procedures in the office, failures in performance relating to tests and interventions, and medication errors. Medicationrelated incidents cover a broad range of errors (interactions/ contraindications, failures in monitoring, missed prescription, wrong agent/route), all of which have the potential to cause severe harm. The most frequently reported incident in the participating offi ces was patient documentation failure (ie, incomplete, unavailable, unclear, or incorrect patient documentation). Documentation problems are a relevant safety issue due to their very high frequency, even though they were not associated with severe consequences in our study. Participants may have underestimated the relationship between errors in documentation and harmful outcomes.

Because errant, incomplete, or missing patient documentation may operate as a root cause or contributing factor for other incidents that have the potential for severe patient harm, it seems reasonable to make reductions in documentation failure a priority. For example, incomplete records might be the underlying cause for ignoring a known allergy and might culminate in a medication error that results in severe harm. Nurses and physicians are often interrupted in their work by the problem of incomplete documentation and invest much time in getting a workup and gathering missing information. Interruptions in work flow are not only inefficient, but also pose a serious threat to patient safety and increase perceptions of stress and workload.^25,26

We observed differences in the frequency and severity of reported incidents between occupational groups. Although for the majority of incidents, physicians reported higher levels of patient harm at last occurrence, our results also suggest that nurses seem to be more aware of some events and consequences. This finding clearly supports our approach of involving nurses in risk identification. Physicians and nurses working in medical centers reported higher frequencies of incidents than those in single-handed and joint offices. This difference seems natural and reflects the higher case load and number of staff employed in these centers. However, there

are exceptions to the high frequency of incidents in medical centers. These differences may result from existing office rules for specific processes (eg, the management of drugs and materials). Such rules are often introduced and assessed in the process of quality certification, which in turn is more likely to be completed by group offices than single-handed practices.

We were not able to relate the incident frequency to the actual case load and/or case mix in the offices. However, it is unclear which type of caseload would be the appropriate measure for assessments of safety. It could be the mean number of consultations per physician, the total caseload per office, or other metrics. Caseload could be restricted to personal medical consultations, but could also include telephone contacts (with the physician and/or nurses) or office visits without consultation (eg, patients that come to get their prescriptions filled). Future research is clearly needed to relate these different types of caseload to patient safety. Such studies would also help to explain whether and how threats to safety evolve in relation to office type.

We acknowledge the limitations of our study. First, the collected data were subjective and retrospective assessments of incidents in primary care offices. Thus, the data might have been affected by memory and reporting effects such as recall bias and availability heuristics. In particular, less serious events might be kept less clearly in mind. McKay et al reported that increased grading severity was linked to the willingness of general practitioner groups to analyze and report events.²⁷ Memory effects seem to be a problem with other methods as well. In their study using incident reporting, Dovey et al observed that about 50% of the incidents were reported 2 weeks after their occurrence or later.²⁸ The longest delay between occurrence and reporting of an incident was 141 days. However, there is an advantage to surveying professionals about the incidents that occurred over the preceding 12 months rather than on time. The cases are more likely to be completed and potential consequences to be known. For example, it may take considerable time for physicians even to be informed about severe patient harm. As noted by Gandhi and Lee, missed diagnoses in particular may not be identified for months, if ever.⁴ It thus seems likely that diagnostic errors were highly underreported by physicians in our study, particularly with respect to incident reporting. Second, we could not relate the frequencies of incidents in the offices to caseload in the offices. This would be valuable information for the interpretation of differences between office types. Finally, we asked participants to report the severity of harm associated with the last occurrence of each incident. We thus have no information on the sum of all harmful events during the study period. However, our approach enabled us to provide a cross-sectional average of harm associated with the last remembered incident, a particular advantage when comparing the incidents with each other.

Despite these limitations, our study provides important insight into risks to patient safety in primary care and supplements chart review and incident reporting methodologies. Our results highlight hot spots of ambulatory care safety and can be used to discuss risks and target actions needed for prevention.

The following measures may be thought of as a way to begin addressing safety risks in the primary care office. First, medication errors clearly need to be targeted due to their high prevalence and potential for harm. Although a single solution to prevent medication errors does not exist, software applications such as computerized order entry systems may be useful to reduce the high number of medication errors.^17,29,30 Such tools guide the clinician through the medication process, alert the clinician in case of interactions or contraindications, and assist in monitoring pharmacotherapy. Second, obligatory office rules for patient monitoring following medical procedures in the office are warranted to prevent rare but potentially catastrophic events (eg, falls and injuries in the office following procedures under sedation). Third, a main result of our study is that triaging of patients at first contact is perceived as a serious safety problem.

Triage as a threat to patient safety is rarely documented explicitly in adverse event studies in primary care, but has gained increasing attention more recently.^20,31,32 Some investigations conclude that telephone triage is a safe method of managing “nonserious” medical callers.³³ By contrast, research conducted with incognito standardized patients calling Dutch out-of-hours centers reports that triagists achieved the appropriate outcome in only 58% of calls.³⁴ In many calls, too few clinical questions were asked and too little information provided, in particular relating to home management and safety net advice. Triaging patients for appointment is a

demanding and complex task in which safety, efficiency, and physicians’ and patients’ preferences need to be sensitively balanced. In addition, nurses often get no feedback about the quality of their triage performance, thus limiting the opportunities to learn from errors. Regular trainings in triage as well as clear communication between physicians and nurses about triage (eg, asking back) may help to increase safety.

Acknowledgments

The authors thank all physicians and nurses who participated in the study and completed the survey. The advice of several experts on prior versions of the survey is highly appreciated. Three anonymous referees provided thorough and supportive feedback on the initial version of this article.

Author Affiliations: From Swiss Patient Safety Foundation (KG, DLBS), Zuerich, Switzerland; Institute of Social and Preventive Medicine (DLBS), University of Bern, Switzerland; Praxis Bubenberg (MB), Bern, Switzerland; Landquartstr (RB), Arbon, Switzerland; mediX Gruppenpraxis (FH), Zuerich, Switzerland; eastcare AG (PS), St. Gallen-Winkeln, Switzerland; hawadoc AG (MW), Winterthur, Switzerland.

Funding Source: This work was supported by the Swiss Medical Association (Federatio Medicorum Helveticorum). The funding source had no influence on study design; on the collection, analysis, and interpretation of the data; on the writing of the manuscript; or on the decision to submit the manuscript for publication. The views expressed and any errors or omissions are the sole responsibility of the authors.

Author Disclosures: The authors (KG, DLBS, MB, RB, FH, PS, MW) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (KG, DLBS, MB, RB, FH, PS, MW); acquisition of data (KG, DLBS, MB, RB, FH, PS, MW); analysis and interpretation of data (KG, DLBS); drafting of the manuscript (KG, DLBS, MB, RB, FH, PS, MW); critical revision of the manuscript for important intellectual content (KG, DLBS, MB, RB, FH, PS, MW); statistical analysis (KG, DLBS); provision of study materials or patients (MB, RB, FH, PS, MW); obtaining funding (DLBS); administrative, technical, or logistic support (KG, DLBS, MB, RB, FH, PS, MW); and supervision (KG, DLBS).

Address correspondence to: Katrin Gehring, PhD, Swiss Patient Safety Foundation, Asylstr, 77, 8032 Zuerich, Switzerland. E-mail: gehring@patientensicherheit.ch.1. Sandars J, Esmail A. The frequency and nature of medical error in primary care: understanding the diversity across studies. Fam Pract. 2003;20(3):231-236.

2. Thomsen LA, Winterstein AG, Søndergaard B, Haugbølle LS, Melander A. Systematic review of the incidence and characteristics of preventable adverse drug events in ambulatory care. Ann Pharmacother. 2007;41(9):1411-1426.

3. Miller GC, Britth HC, Valenti L. Adverse drug events in general practice patients in Australia. Med J Aust. 2006;184(7):321-324.

4. Gandhi TK, Lee TH. Patient safety beyond the hospital. N Engl J Med. 2010;363(11):1001-1003.

5. Rubin G, George A, Chinn DJ, Richardson C. Errors in general practice: development of an error classification and pilot study of a method for detecting errors. Qual Saf Health Care. 2003;12(6):443-447.

6. Elder NC, Vonder Meulen M, Cassedy A. The identification of medical errors by family physicians during outpatient visits. Ann Fam Med. 2004;2(2):125-129.

7. Makeham MA, Kidd MR, Saltman DC, et al. The Threats to Australian Patient Safety (TAPS) study: incidence of reported errors in general practice. Med J Aust. 2006;185(2):95-98.

8. Makeham MA, Dovey S, Runciman WB, Larizgoitia I. Methods and Measures used in Primary Care Patient Safety Research. Geneva, Switzerland: World Health Organization; 2008.

9. Makeham MA, Dovey SM, County M, Kidd MR. An international taxonomy for errors in general practice: a pilot study. Med J Aust. 2002; 177(2):68-72.

10. O’Beirne M, Sterling PD, Zwicker K, Hebert P, Norton PG. Safety incidents in family medicine. BMJ Qual Saf. 2011;20(12):1005-1010.

11. Rowin EJ, Lucier D, Pauker SG, Kumar S, Chen J, Salem DN. Does

error and adverse event reporting by physicians and nurses differ? Jt

Comm J Qual Patient Saf. 2008;34(9):537-545.

12. Gandhi TK, Weingart SN, Borus J, et al. Adverse drug events in ambulatory care. N Engl J Med. 2003;348(16):1556-1564.

13. Stange KC, Zyzanski SJ, Smith TF, et al. How valid are medical records and patient questionnaires for physician profiling and health services research? a comparison with direct observation of patient visits. Med Care. 1998;36(6):851-867.

14. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003;289(9):1107-1116.

15. Rosser W, Dovey S, Bordman R, White D, Crighton E, Drummond N. Medical errors in primary care: results of an international study of family practice. Can Fam Physician. 2005;51:386-387.

16. Hoffmann B, Beyer M, Rohe J, Gensichen J, Gerlach FM. “Every error counts”: a web-based incident reporting and learning system for general practice. Qual Saf Health Care. 2008;17(4):307-312.

17. Gandhi TK, Weingart SN, Seger AC, et al. Outpatient prescribing errors and the impact of computerized prescribing. J Gen Intern Med. 2005;20(9):837-841.

18. Kennedy AG, Littenberg B, Senders JW. Using nurses and office staff to report prescribing errors in primary care. Int J Qual Health Care. 2008;20(4):238-245.

19. Kostopoulou O, Delaney B. Confidential reporting of patient safety events in primary care: results from a multilevel classification of cognitive and system factors. Qual Saf Health Care. 2007;16(2): 95-100.

20. Smits M, Huibers L, Kerssemeijer B, de Feijter E, Wensing M, Giesen P. Patient safety in out-of-hours primary care: a review of patient records. BMC Health Serv Res. 2010;10:335.

21. Wetzels R, Wolters R, van Weel C, Wensing M. Harm caused by adverse events in primary care: a clinical observational study. J Eval Clin Pract. 2009;15(2):323-327.

22. Hickner J, Graham DG, Elder NC, et al. Testing process errors and their harms and consequences reported from family medicine practices: a study of the American Academy of Family Physicians National Research Network. Qual Saf Health Care. 2008;17(3):194-200.

23. World Alliance for Patient Safety. WHO Draft Guidelines for Adverse Event Reporting and Learning Systems. From Information to Action. Geneva, Switzerland: World Health Organization; 2005.

24. National Patient Safety Agency. A Risk Matrix for Risk Managers. London: National Patient Safety Agency; 2008.

25. Li SY, Magrabi F, Coiera E. A systematic review of the psychological literature on interruption and its patient safety implications. J Am Med Inform Assoc. 2012;19(1):6-12.

26. Westbrook JI, Coiera E, Dunsmuir WTM, et al. The impact of interruptions on clinical task completion. Qual Saf Health Care. 2010;19(4): 284-289.

27. McKay J, Bowie P, Murray L, Lough M. Levels of agreement on the grading, analysis and reporting of significant events by general practitioners: a cross-sectional study [published correction appears in Qual Saf Health Care. 2009;18(1):80]. Qual Saf Health Care. 2008;17(5): 339-345.

28. Dovey SM, Meyers DS, Phillips RL Jr, et al. A preliminary taxonomy of medical errors in family practice. Qual Saf Health Care. 2002;11(3): 233-238.

29. Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc. 2008;15(5): 585-600.

30. Devine EB, Williams E, Martin D, et al. Prescriber and staff perceptions of an electronic prescribing system in primary care: a qualitative assessment. BMC Med Inform Decis Mak. 2010;10:72.

31. Giesen P, Ferwerda R, Tijssen R, et al. Safety of telephone triage in general practitioner cooperatives: do triage nurses correctly estimate urgency? Qual Saf Health Care. 2007;16(3):181-184.

32. Hansen EH, Hunskaar S. Telephone triage by nurses in primary care out-of-hours services in Norway: an evaluation study based on written case scenarios. BMJ Qual Saf. 2011;20(5):390-396.

33. Dale J, Williams S, Foster T, et al. Safety of telephone consultation for “non-serious” emergency ambulance service patients. Qual Saf Health Care. 2004;13(5):363-373.

34. Derkx HP, Rethans JJ, Muijtjens AM, et al. Quality of clinical aspects of call handling at Dutch out of hours centres: cross sectional national study. BMJ. 2008;337:a1264.