Development and Pilot Testing of Guidelines to Monitor High-Risk Medications in the Ambulatory Setting

Objectives: To develop guidelines to monitor high-risk medications and to assess the prevalence of laboratory testing for these medications among a multispecialty group practice.

Study Design: Safety intervention trial.

Methods: We developed guidelines for the laboratory monitoring of high-risk medications as part of a patient safety intervention trial. An advisory committee of national experts and local leaders used a 2-round Internet-based Delphi process to select guideline medications based on the importance of monitoring for efficacy, safety, and drug—drug interactions. Test frequency recommendations were developed by academic pharmacists based on a literature review and local interdisciplinary consensus. To estimate the potential effect of the planned intervention, we determined the prevalence of high-risk drug dispensings and laboratory testing for guideline medications between January 1, 2008, and July 31, 2008.

Results: Consensus on medications to include in the guidelines was achieved in 2 rounds. Final guidelines included 35 drugs or drug classes and 61 laboratory tests. The prevalence of monitoring ranged from less than 50.0% to greater than 90.0%, with infrequently prescribed drugs having a lower prevalence of recommended testing (P <.001 for new dispensings and P <.01 for chronic dispensings, nonparametric test for trend). When more than 1 test was recommended for a selected medication, monitoring within a medication sometimes differed by greater than 50.0%.

Conclusions: Even among drugs for which there is general consensus that laboratory monitoring is important, the prevalence of monitoring is highly variable. Furthermore, infrequently prescribed medications are at higher risk for poor monitoring.

(Am J Manag Care. 2010;16(7):489-496)

This article adds to the existing literature and informs clinical decisions in the following ways:

• By describing a process to develop guidelines for the laboratory monitoring of high-risk medications within a multispecialty group practice.
• By estimating the potential effect of an intervention to improve laboratory testing of high-risk medications by reporting the use frequency of high-risk medications requiring monitoring in 2008 and the prevalence of test completion for high-risk drugs in 2008.
• By demonstrating that infrequently used medications have lower rates of recommended laboratory test monitoring.

Drug-induced injury is common. Independent risk factors for adverse drug events in the ambulatory setting include advanced age, multiple comorbid conditions, and the use of high-risk medications requiring close monitoring. For example, failure to appropriately monitor older patients receiving drug therapy accounts for 36% of preventable adverse drug events in the ambulatory setting.¹

Efforts to improve monitoring within organizations are hampered by a lack of comprehensive and specific guidelines for the laboratory monitoring of high-risk medications.² Monitoring guidelines for selected drugs exist in recommendations of organizations such as the American Heart Association guidelines to manage heart failure,³ the American Geriatrics Society Assessing Care of Vulnerable Elders medication quality indicators,⁴ and the National Quality Forum—endorsed measures.⁵ A nationwide baseline monitoring assessment study⁶ developed more comprehensive guidelines for medications used through 2000, among which a subset of 14 were updated and adopted for an intervention trial between 2002 and 2003 within a single health maintenance organization.^7,8

METHODS

We sought to develop an updated and comprehensive list of drugs requiring laboratory monitoring for an electronic medical record—embedded clinical decision support intervention at a multispecialty group practice for medications in clinical use during 2008. The intent of the updated guidelines was to include drugs introduced to the market since December 2000, the date of the literature review for the original drug and laboratory monitoring recommendations published by Raebel et al⁶ in 2005, as well as to update laboratory test and frequency recommendations based on 2007 changes in monitoring recommendations.⁹ Herein, we describe the development of guidelines to monitor high-risk medications in the ambulatory setting using a 2-step consensus-based approach, including a national expert advisory panel and local leaders to select candidate medications for monitoring and to determine the frequency of laboratory monitoring. To estimate the potential effect of our guidelines on actual practice, we determined the use frequency of the guideline drugs and the prevalence for each of the recommended laboratory tests. The specific objectives of this study were (1) to develop recommendations to guide the monitoring of high-risk medications in the ambulatory setting, (2) to assess the use prevalence of candidate medications for monitoring, and (3) to determine completion of recommended testing for medications dispensed among the patient population.Study Setting and Population

This study was conducted in a large multispecialty group practice that provides most medical care to members of a closely associated health plan based in the New England area of the United States. The group practice employs 250 outpatient clinicians at 30 ambulatory clinic sites. The practice uses the EpicCare Ambulatory (Epic, Verona, WI) electronic medical record system and provides medical care to approximately 180,000 individuals. For this analysis, we included patients if they received care from the multispecialty group practice, were 18 years or older, and obtained insurance coverage from the health plan between January 1, 2007, and July 31, 2008. Patients had to be continuously enrolled during the observation period and not residing in a long-term care facility. Data about medication exposure were derived from the prescription drug claims of the health plan. Data about laboratory test completion were derived from the multispecialty group practice electronic medical record.

Determination of High-Risk Drugs and Laboratory Monitoring Guidelines

Laboratory monitoring guidelines for high-risk drugs were developed using a sequential process adapted from an approach created and tested for a study⁶ conducted within the Health Maintenance Organization Research Network Center for Education and Research on Therapeutics. This approach used an advisory committee of national experts and local health plan leaders, including practicing clinicians and pharmacists and experts in patient safety and geriatric pharmacotherapy. The charge for this group was as follows: (1) to review a comprehensive initial list of medications requiring monitoring; (2) to assess the importance of including monitoring recommendations to evaluate efficacy, safety, and clinically relevant drug—drug interactions; and (3) to determine the need to include infrequently prescribed medications in the guidelines. The initial list of high-risk drugs included those commonly implicated in adverse events among patients in the ambulatory setting¹ and those associated with adverse events leading to emergency department visits,¹⁰ as well as drugs with low monitoring rates,^6,11 drugs included in national quality guidelines,⁴ and drugs with black box warnings.¹²

We asked panel members to participate in an Internet-based questionnaire administered in a 2-round modified Delphi process¹³ between August and October 2008. Panelists were asked whether electronic monitoring alerts should be sent to primary care physicians, specialists, or both and whether monitoring alerts should be generated for infrequently dispensed or effectively obsolete medications. Panelists were also asked to rate the importance of monitoring each candidate medication for efficacy, toxic effects, and drug—drug interactions. Each question was answered based on a 5-point Likert-type scale to evaluate agreement or disagreement with statements concerning the importance of monitoring each medication or medication class for the domain assessed. The scale ranged from 1 (indicating “strongly agree”) to 5 (indicating “strongly disagree”).

After the first round of the survey, we eliminated questions for which there was agreement and readministered questions for which there was lack of consensus. Consistent with other modified Delphi methods, consensus for a question was defined by agreement on categorization by at least a majority (>50.0%) of respondents.¹³ We then administered a second questionnaire to participants. In this round, panelists were reminded of their original responses to individual questions and were given the group’s aggregated response to the questions in the first round. At this stage, each participant was given the opportunity to revise his or her response to increase consensus with the succeeding round. The results of the Delphi process informed the selection of the final high-risk drug list.

Determination of Laboratory Test Monitoring Frequency

After selection of the final high-risk drug list, 2 research pharmacists (JLD and AOK) reviewed the literature to determine the appropriate frequency of laboratory monitoring for each drug. The review included monitoring recommendations provided in manufacturer labeling information, nationally available published guidelines, and clinical guidelines from national organizations and initiatives (eg, American Heart Association guidelines³ and National Committee for Quality Assurance—developed measures and guidelines⁵). Because manufacturers’ warnings often describe a nonspecific frequency for monitoring such as “periodically,” we reviewed other authoritative sources that pharmacists and clinicians commonly use to guide decisions about the frequency of monitoring. These sources included the following: (1) the published literature; (2) the Micromedex Healthcare Series¹⁴ (Thomson Reuters [Healthcare] Inc, Greenwood Village, CO) database, which sources the primary case report and experimental literature; (3) UpToDate Inc¹⁵ (Waltham, MA), an online peer-reviewed reference database; and (4) the Pharmacist’s Letter¹⁶ (Therapeutic Research Center, Stockton, CA). The final list of laboratory monitoring tests and the associated monitoring frequencies were reviewed by a local panel of academic pharmacists, the clinical pharmacist for the multispecialty group practice, and the medical director of the multispecialty group practice to determine local acceptability and congruence with local quality standards.

Statistical Analysis

RESULTS

We used drug dispensing claims between January 1, 2007, and July 31, 2008, to identify the first dispensing of a high-risk medication for a patient after January 1, 2008. New drug use was defined as an initial dispensing on or after the index date of January 1, 2008, and no drug dispensing in the 6 months preceding the index date. Chronic (ongoing) drug use was defined as a dispensing on or after January 1, 2008, with evidence of drug dispensing in the 6 months before that date. Because clinicians might rely on laboratory test results for up to 6 months before initiation of a drug or might order a test within 2 weeks after initiation of a drug, we defined test completion for a new dispensing in 2008 as having occurred if there was at least 1 associated monitoring test completed 180 days before dispensing to 42 days after dispensing (test ordered <14 days after dispensing plus 28 days for test completion). Completion of a test of serum drug levels (eg, serum carbamazepine) for new prescriptions was measured from the day of dispensing to 30 days after initial dispensing. Test completion for chronic medication use was defined as having occurred if there was at least 1 recommended test for the drug test pair that occurred up to 365 days before the index dispensing in 2008 through 42 days after the dispensing if the test was indicated annually (or 180 days before to 42 days after index dispensing if the test was indicated every 6 months). For each drug—laboratory test combination, the proportion of completed recommended tests was determined for all index dispensings in the observation period. We used a nonparametric test for trend across ordered groups by Cuzick17 to examine whether more frequently dispensed drugs had a higher prevalence of recommended test completion. All analyses were conducted using commercially available software (SAS 9.2; SAS Institute Inc, Cary, NC). The study was approved by the institutional review boards of the University of Massachusetts Medical School, Worcester, and the participating group practice.Consensus Panel Survey

A panel of 3 pharmacists and 6 physicians participated in the consensus survey. All physicians were certified by the American Board of Internal Medicine, 2 had an added qualification in geriatric medicine, and 3 had subspecialty board certifications. All respondents completed the 2 rounds of the survey. After the second round of the survey, consensus was achieved for all 40 medications with respect to parameters (ie, efficacy or toxic effects) for laboratory monitoring and for alerts to physicians (ie, primary care physician vs specialist). The consensus panel agreed that alerts for high-risk medications were indicated to monitor efficacy and toxic effects alone or in the presence of significant drug—drug interactions. The panel also agreed that alerts should be sent to specialists and primary care physicians and should be developed for medications that were frequently and infrequently prescribed. Because strict national prescribing policies already exist to guide the use and monitoring of isotretinoin, we did not include this medication in the newly developed guidelines. The resulting list of drugs included 39 drugs or drug classes.

Pilot Testing of Monitoring Recommendations

Once the medication list was established, we examined overall dispensings of the selected medications. The chronic index dispensings ranged from 0 (procainamide and ganciclovir)to 13,351 (statins), and new index dispensings ranged from 0 (procainamide and ganciclovir) to 2463 (statins) during the observation period of January 1, 2008, to July 31, 2008. Based on these results, 4 drugs with fewer than 5 new or chronic users combined (ticlopidine, tolcapone, ganciclovir, and procainamide) were eliminated from the final drug list, resulting in 35 drugs or drug classes in the final list of monitoring guidelines. There were 61 separate drug—laboratory test combinations for monitoring, with 15 requiring more than 1 laboratory test (eg, amiodarone–thyroid-stimulating hormone [TSH] and amiodarone–aspartate aminotransferase [AST]) and 20 requiring a single test; for the purpose of this study, we considered a basic metabolic panel that includes sodium, potassium, chloride, bicarbonate, blood urea nitrogen, and creatinine levels as a single test.

Overall, the rates of test completion ranged from 0.0% for aspartate aminotransferase or alanine aminotransferase [ALT] among new users of nefazodone, phenobarbital, and quinidine at baseline to 96.8% for AST or ALT test among chronic users of niacin (Table 1and Table 2).Drug—laboratory test pair completion differed between new and chronic medication dispensings. For new medication dispensings, less than 20.0% of the recommended 61 drug–laboratory test pairs were monitored in the majority (>75.0%) of patients. In contrast, close to 40.0% of the recommended drug–laboratory test pairs for chronic dispensings were monitored in greater than 75.0% of patients.

DISCUSSION

For drugs with multiple laboratory tests, the percentage test completion often varied within the drug (Table 2). For example, for new amiodarone use, 52.7% had an AST or ALT test, while 74.6% had a TSH test; for azathioprine use,28.0% had a baseline AST or ALT test, while 72.0% had a complete blood count (CBC). We also found that the frequency of medication use was associated with the frequency of recommended test completion (Table 3). Infrequently dispensed medications had a lower prevalence of test completion. Nonparametric test for trend showed a significant trend toward more frequently dispensed medications having a higher prevalence of appropriate testing (P <.001).Herein, we describe the development of drug—laboratory test monitoring guidelines based on manufacturers’ recommendations and published guidelines. Recently published guidelines to monitor drugs in the long-term care setting included a similar list of medications and laboratory tests.¹⁸ To determine the potential effect of our guidelines on actual practice, we determined the use frequency of the guideline drugs and the prevalence of testing for each of the recommended laboratory tests. We found that (1) certain high-risk drugs had very low rates of use among our target population, (2) rates of testing for high-risk drugs varied widely, and (3) infrequently used drugs had lower testing rates. Our study adds to the literature by identifying ways to inexpensively improve patient safety and quality of care (describing a method for formulating monitoring recommendations) and by providing final monitoring recommendations (detailed guidelines are available from the authors) that are useful to practitioners considering similar interventions at their local institutions.

It is important to point out that the level of evidence for each recommended drug test pair varied and that this likely explains some of the variation in testing among our sample. For example, there is increasing controversy about the role of evaluating renal function in patients taking metformin.¹⁹ Among our sample, we found that 45.0% to 60.0% of metformin users had completed a serum creatinine test. Similarly, there is a difference in the role of digoxin-level monitoring depending on indication, with digoxin serum drug levels of less importance when prescribed for heart rate control and of greater importance when used for heart failure.²⁰ Among our sample, we found digoxin levels completed for 30.0% to 60.0% of users. We did not discern the indication for digoxin use in our study, but previous investigations have shown similarly low rates of digoxin-level testing.²¹ Variance in the level of evidence for testing may explain some of the differences we observed in the prevalence of completion among different tests within the same drug.

We are concerned about the low prevalence of testing for drug test pair recommendations that are much less controversial such as evaluation of thyroid function among amiodarone users. Amiodarone-induced thyroid dysfunction, including thyrotoxicosis and hypothyroidism, can occur in up to 20% of users.22 For this drug test pair, we found that 75.0% of new users and 50.0% of chronic users had completed a TSH test. The reason for the lower rate of testing among new drug users is unclear. Although a possible explanation may be the initiation of drugs in the hospital, as may be the case for amiodarone among using patients with arrhythmias, the prevalence of hospitalizations did not differ among those with versus without appropriate monitoring of amiodarone in a prior study.²³ Further investigation is necessary to determine whether hospitalizations explain low testing rates among our study population.

An important finding of our study was that patients using infrequently prescribed drugs were less likely to complete recommended laboratory tests. This is consistent with literature showing an association between patient volume and quality of care²⁴ but is not supported by at least 1 study²⁵ examining patient volume in association with process measures such as laboratory testing in diabetes. The rationale supported by the human factors model of medical errors is that physicians will likely have better familiarity of the literature for drugs that they prescribe more frequently than for drugs that they do not prescribe.²⁶

The implication of this finding can be considered in terms of potential relative and absolute effects of an intervention to improve monitoring. There is potential for significant effect from interventions aimed at improving infrequently used medications because baseline rates of testing are low. This is important for drugs such as phenobarbital, whose laboratory monitoring is a Healthcare Effectiveness Data and Information Set measurement for assessing health plan quality of care.²⁷ In contrast, the number of patients affected by infrequently prescribed drugs is small from a population health perspective. For example, the number of patients who failed to receive monitoring of statins was 2136, more than 20 times greater than the number of patients affected by failure to monitor the 7 infrequently prescribed drugs combined (ie, imidazole antifungals, terbinafine, angiotensin II receptor blockers, isoniazid, rifampin, theophylline,and nefazodone). Health plans considering the implementation of interventions to improve medication monitoring need to weigh the absolute and relative potential effects of their program. Although the marginal cost of targeting many medications in a health information technology—based intervention to improve monitoring is minimal and allows for the inclusion of many medications, programs using high-cost resources such as pharmacist time need to limit target medications by weighing the relative overall effect carefully to optimally select target medications.

Other important drugs for inclusion in monitoring guidelines are those with a high likelihood of a serious adverse outcome. Our guidelines incorporate several drugs commonly implicated in adverse drug events leading to emergency department visits,¹⁰ including hypoglycemics, warfarin, anticonvulsants, digoxin, theophylline, and lithium. Of these, we found high rates for warfarin, which is often monitored through special anticoagulation programs, but low-to-moderate rates for digoxin, anticonvulsants, lithium, and theophylline.

Limitations of our study should be noted. First, our study was conducted in a single multispecialty group practice. Second, laboratory tests may have been ordered for another reason (ie, not for high-risk medication monitoring), so that we may have overestimated the prevalence of recommended testing. Third, we were unable to confirm patient adherence to drugs and were unable to identify patients who did not complete tests because they were no longer using the medication. Fourth, in many cases, despite the panel’s conclusions, limited evidence was found to support testing frequency or importance, and that may explain the low prevalence of test ordering for some drugs. For many drugs, further study is necessary to determine whether laboratory test monitoring improves health outcomes. Furthermore, lower testing for new use may also be explained by drug initiation in the hospital setting for drugs such as amiodarone and antiepileptics. In these cases, it is possible that baseline testing was obtained after initiation in the hospital and was not repeated after discharge to the ambulatory setting. However, similarly low rates of TSH and aminotransferase testing have been reported for chronic users of amiodarone²³ and for determining serum levels of antiepileptics.²¹ Fifth, we did not assess all guideline recommendations such as monitoring after dose changes, drug—drug interactions, changing renal function, and clinical status.

Our study updates the laboratory monitoring recommendations for high-risk drugs to include drugs used in ambulatory practice until 2008. Our findings and guidelines are generalizable to other institutions seeking to improve monitoring of high-risk medications to enhance the safety for their patient population after review and adaptation by local medical and pharmacy leadership to ensure consistency with local standards. Health plans considering similar interventions can use our findings to weigh the potential benefit of targeting drugs of high-frequency and low-frequency use in clinical practice, as our study demonstrates potential for improvement in the laboratory monitoring of drug types. Further research is necessary to understand factors contributing to the low monitoring of high-risk drugs, including physician factors resulting in lack of test ordering and patient factors leading to incomplete performance of tests.

Acknowledgments

We acknowledge the contributions of the national expert advisory panel, including Susan Andrade, ScD; Steven Simon, MD; Jerry H. Gurwitz, MD; and Marsha A. Raebel, PharmD. Local experts included Josie Cambia-Kiely, PharmD; Mojgan Hajji, PharmD; Michael Kelleher, MD; Leslie Harrold, MD; Sarah McGee, MD; and Robert Yood, MD.

Author Affiliations: From the Division of Geriatric Medicine (JT, TSF, SJG, SHF, JHG), University of Massachusetts, Worcester, MA; Meyers Primary Care Institute (JT, TSF, YZ, JCF, JHG), Worcester, MA; Fallon Clinic (LDG), Worcester, MA; Massachusetts College of Pharmacy (JLD, AOK), Worcester, MA; and Kaiser Permanente Colorado (MAR), Denver, CO.

Funding Source: This study was funded by grants R18 HS017203, R18 HS017817, and R18 HS017906 from the Agency for Healthcare Research and Quality.

Author Disclosures: The authors (JT, TSF, LDG, JLD, AOK, MAR, YZ, JCF, SJG, SHF, JHG) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (JT, TSF, LDG, JHG); acquisition of data (JT, TSF, JCF); analysis and interpretation of data (JT, TSF, LDG, JLD, AOK, MAR, YZ, JCF, SJG, SHF, JHG); drafting of the manuscript (JT, TSF, JLD, AOK, MAR, SJG); critical revision of the manuscript for important intellectual content (JT, TSF, JLD, AOK, MAR, SJG, SHF, JHG); statistical analysis (JT, YZ, JCF); obtaining funding (JT, JHG); and administrative, technical, or logistic support (LDG, YZ, JCF, SJG, SHF).

Address correspondence to: Jennifer Tjia, MD, MSCE, Division of Geriatric Medicine, University of Massachusetts, 377 Plantation St, Ste 315, Worcester, MA 01605. E-mail: jennifer.tjia@umassmed.edu.

1. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003;289(9):1107-1116.

2. Matheny ME, Sequist TD, Seger AC, et al. A randomized trial of electronic clinical reminders to improve medication laboratory monitoring. J Am Med Inform Assoc. 2008;15(4):424-429.

3. Hunt SA, Abraham WT, Chin MH, et al; American College of Cardiology, American Heart Association Task Force on Practice Guidelines, American College of Chest Physicians, International Society for Heart and Lung Transplantation, Heart Rhythm Society. ACC/AHA 2005 Guideline Update for the Diagnosis and Management of Chronic Heart Failure in the Adult: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Update the 2001 Guidelines for the Evaluation and Management of Heart Failure): developed in collaboration with the American College of Chest Physicians and the International Society for Heart and Lung Transplantation: endorsed by the Heart Rhythm Society. Circulation. 2005;112(12):e154-e235.

4. Shrank WH, Polinski JM, Avorn J. Quality indicators for medication use in vulnerable elders. J Am Geriatr Soc. 2007;55(suppl 2): S373-S382.

5. National Committee for Quality Assurance. NQF-endorsed measures. 2008. http://www.ncqa.org/tabid/59/Default.aspx. Accessed November 16, 2009.

6. Raebel MA, Lyons EE, Andrade SE, et al. Laboratory monitoring of drugs at initiation of therapy in ambulatory care. J Gen Intern Med. 2005;20(12):1120-1126.

7. Raebel MA, Lyons EE, Chester EA, et al. Improving laboratory monitoring at initiation of drug therapy in ambulatory care: a randomized trial. Arch Intern Med. 2005;165(20):2395-2401.

8. Raebel MA, Chester EA, Newsom EE, et al. Randomized trial to improve laboratory safety monitoring of ongoing drug therapy in ambulatory patients. Pharmacotherapy. 2006;26(5):619-626.

9. Dalbeth N, Stamp L. Allopurinol dosing in renal impairment: walking the tightrope between adequate urate lowering and adverse events. Semin Dial. 2007;20(5):391-395.

10. Budnitz DS, Pollock DA, Weidenbach KN, Mendelsohn AB, Schroeder TJ, Annest JL. National surveillance of emergency department visits for outpatient adverse drug events. JAMA. 2006;296(15):1858-1866.

11. Feldstein AC, Smith DH, Perrin N, et al. Improved therapeutic monitoring with several interventions: a randomized trial. Arch Intern Med. 2006;166(17):1848-1854.

12. Wagner AK, Chan KA, Dashevsky I, et al. FDA drug prescribing warnings: is the black box half empty or half full? Pharmacoepidemiol Drug Saf. 2006;15(6):369-386.

13. Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41(4):376-382.

14. Micromedex Healthcare Series [online database]. Greenwood Village, CO: Thomson Reuters (Healthcare) Inc; 2009.

15. UpToDate Inc [online database]. Waltham, MA: UpToDate Inc; 2009.

16. Pharmacist’s Letter. Stockton, CA: Therapeutic Research Center; 2009.

17. Cuzick J. A Wilcoxon-type test for trend. Stat Med. 1985;4(1):87-90.

18. Handler SM, Shirts BH, Perera S, Becich MJ, Castle NG, Hanlon JT. Frequency of laboratory monitoring of chronic medications administered to nursing facility residents: results of a national Internet-based study. Consult Pharm. 2008;23(5):387-395.

19. Jones GC, Macklin JP, Alexander WD. Contraindications to the use of metformin. BMJ. 2003;326(7379):4-5.

20. Aronson JK, Hardman M. ABC of monitoring drug therapy: digoxin. BMJ. 1992;305(6862):1149-1152.

21. Palen TE, Raebel M, Lyons E, Magid DM. Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders. Am J Manag Care. 2006;12(7):389-395.

22. Trip MD, Wiersinga W, Plomp TA. Incidence, predictability, and pathogenesis of amiodarone-induced thyrotoxicosis and hypothyroidism. Am J Med. 1991;91(5):507-511.

23. Raebel MA, Carroll NM, Simon SR, et al. Liver and thyroid monitoring in ambulatory patients prescribed amiodarone in 10 HMOs. J Manag Care Pharm. 2006;12(8):656-664.

24. Hewitt M, Petitti D, eds. Interpreting the Volume-Outcome Relationship in the Context of Health Care Quality. Washington, DC: National Academy Press; 2001.

25. Turchin A, Shubina M, Pendergrass ML. Relationship of physician volume with process measures and outcomes in diabetes. Diabetes Care. 2007;30(6):1442-1447.

26. Leape LL. A systems analysis approach to medical error. J Eval Clin Pract. 1997;3(3):213-222.

27. National Committee for Quality Assurance. HEDIS 2009: Healthcare Effectiveness Data & Information Set. Vol 1. Washington, DC: National Commit-tee for Quality Assurance; 2008. Narrative.

Download PDF: Development and Pilot Testing of Guidelines to Monitor High-Risk Medications in the Ambulatory Setting