0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Investigation | Health Care Reform

An Empirical Model to Estimate the Potential Impact of Medication Safety Alerts on Patient Safety, Health Care Utilization, and Cost in Ambulatory Care FREE

Saul N. Weingart, MD, PhD; Brett Simchowitz, BA; Harper Padolsky, MD; Thomas Isaac, MD, MBA, MPH; Andrew C. Seger, PharmD; Michael Massagli, PhD; Roger B. Davis, ScD; Joel S. Weissman, PhD
[+] Author Affiliations

Author Affiliations: Center for Patient Safety, Dana-Farber Cancer Institute (Drs Weingart, Padolsky, Isaac, and Seger and Mr Simchowitz), Tufts University School of Medicine (Dr Padolsky), Division of General Medicine and Primary Care, Beth Israel Deaconess Medical Center (Drs Weingart, Isaac, and Davis), Division of General Medicine, Brigham and Women's Hospital (Dr Seger), Massachusetts College of Pharmacy and Health Sciences (Dr Seger), Institute for Health Policy, Massachusetts General Hospital (Dr Weissman), and Executive Office of Health and Human Services, Commonwealth of Massachusetts (Dr Weissman), Boston, Massachusetts; PatientsLikeMe, Cambridge, Massachusetts (Dr Massagli); and Department of Community and Family Medicine, University of Massachusetts Medical School, Worcester (Dr Weissman).


Arch Intern Med. 2009;169(16):1465-1473. doi:10.1001/archinternmed.2009.252.
Text Size: A A A
Published online

Background  Because ambulatory care clinicians override as many as 91% of drug interaction alerts, the potential benefit of electronic prescribing (e-prescribing) with decision support is uncertain.

Methods  We studied 279 476 alerted prescriptions written by 2321 Massachusetts ambulatory care clinicians using a single commercial e-prescribing system from January 1 through June 30, 2006. An expert panel reviewed a sample of common drug interaction alerts, estimating the likelihood and severity of adverse drug events (ADEs) associated with each alert, the likely injury to the patient, and the health care utilization required to address each ADE. We estimated the cost savings due to e-prescribing by using third-party–payer and publicly available information.

Results  Based on the expert panel's estimates, electronic drug alerts likely prevented 402 (interquartile range [IQR], 133-846) ADEs in 2006, including 49 (14-130) potentially serious, 125 (34-307) significant, and 228 (85-409) minor ADEs. Accepted alerts may have prevented a death in 3 (IQR, 2-13) cases, permanent disability in 14 (3-18), and temporary disability in 31 (10-97). Alerts potentially resulted in 39 (IQR, 14-100) fewer hospitalizations, 34 (6-74) fewer emergency department visits, and 267 (105-541) fewer office visits, for a cost savings of $402 619 (IQR, $141 012-$1 012 386). Based on the panel's estimates, 331 alerts were required to prevent 1 ADE, and a few alerts (10%) likely accounted for 60% of ADEs and 78% of cost savings.

Conclusions  Electronic prescribing alerts in ambulatory care may prevent a substantial number of injuries and reduce health care costs in Massachusetts. Because a few alerts account for most of the benefit, e-prescribing systems should suppress low-value alerts.

Figures in this Article

Although the value of electronic prescribing (e-prescribing) is well established in the acute care hospital, its safety benefits in ambulatory care are less well understood. Early studies1,2 showed that rudimentary electronic order entry, without advanced decision support such as drug interaction and allergy alerts, resulted in legible prescriptions but no difference in the rate of adverse drug events (ADEs) compared with paper-based prescribing. In subsequent studies,35 however, investigators discovered that clinicians with access to these features overrode as many as 91% of drug interaction and allergy alerts. Although overriding alerts may jeopardize the potential impact of these systems, it is possible that even the small number of accepted alerts may reduce patient harm, decrease unnecessary utilization of health care services, and save money.

To understand the potential benefits of medication safety alerts in ambulatory care, we conducted a multifaceted study of a commercial e-prescribing system serving 2321 Massachusetts ambulatory care providers in 2006. We hypothesized that the alerts that clinicians accepted would, in aggregate, benefit patients, lower health care costs, and help to validate the continued use of these systems.

CONCEPTUAL MODEL

To examine the impact of medication safety alerts, we modified a conceptual framework that characterizes prescribers' responses to drug safety alerts and the resulting consequences (Figure 1).6,7 When an alert is triggered, a clinician has the following 3 options: (1) override the alert and leave the prescription intact, (2) cancel the prescription, or (3) change the prescription to another medication. We combined canceled prescription orders and those changed to an alternate medication and termed these accepted alerts. Some accepted alerts prevent ADEs from occurring. The number of prevented ADEs is a function of the number of alerts generated, the acceptance rate, and the probability that the alerted interaction would have resulted in an ADE if allowed to reach the patient. The number of ADEs prevented should be discounted by the number of prescriptions that pharmacists do not fill or patients do not collect. The severity of the ADE may affect the type of injury, health care utilization, and cost of care.

Place holder to copy figure label and caption
Figure 1.

Conceptual model for estimating the patient safety impact, health care utilization, and cost savings attributable to accepted medication safety alerts. Alerts are generated in an electronic prescribing system. ADE indicates adverse drug event; DDI, drug-drug interaction; and ED, emergency department.

Graphic Jump Location
e-PRESCRIBING

We examined medication alerts generated by the users of PocketScript, an e-prescribing application developed by ZixCorp (Dallas, Texas) that allows clinicians to transmit prescriptions electronically to a pharmacy via a desktop computer or a handheld device. The system creates a profile of a patient's active medications based on previously written e-prescriptions. When a prescriber attempts to order a drug, the system checks whether the prescribed medication interacts with any medications on the patient's profile, drawing on a list of medication interactions maintained by Cerner Multum, Inc (Denver, Colorado). If an interaction is detected, a warning banner is displayed showing the severity of the interaction (high, medium, or low), and a description of the interaction is available through a drug reference guide.

SELECTION OF MEDICATION SAFETY ALERTS

We studied all e-prescriptions written and all medication safety alerts generated by 2321 eligible Massachusetts clinicians who used the PocketScript e-prescribing system for at least 1 e-prescription from January 1 through June 30, 2006. ZixCorp provided information on all drug-drug interaction (DDI) alerts generated during the study period, along with the prescribers' action on receiving the alert. Prescribers included physicians (79.2%) and nonphysicians (20.8%), including specialists in family medicine (14.5%), internal medicine (13.1%), pediatrics (13.7%), psychiatry (2.5%), and other specialties (26.5%). Specialty information was unavailable for 29.7% of prescribers.

These clinicians wrote 1 833 254 prescriptions for 60 352 patients and generated 279 476 drug interaction alerts during the study period. Multiple alerts may have appeared for the same prescription attempt if a prescribed drug interacted with more than 1 medication on the patient's profile. To avoid double-counting alerts and multiple prescription attempts for the same drug, we studied the last drug alert that the prescriber triggered sorted by physician, patient, prescribed generic drug name, date, and time.3 This approach eliminated 146 425 alerts, leaving 133 051. Each DDI belongs to a broader class-class interaction (CCI): bupropion hydrochloride with citalopram hydrobromide, for example, is a DDI within the smoking cessation agents and selective serotonin reuptake inhibitor antidepressant CCI. We selected the 100 most frequently accepted CCIs; these alerts represented 56.5% of all accepted alerts. Finally, we selected the most commonly accepted DDI within each CCI.

EXPERT PANEL ESTIMATES

Because there are no published data estimating the probability and severity of harm caused by most DDIs, we used a modified Delphi technique (a type of consensus method) to characterize the clinical utility, patient safety benefit, and cost savings associated with accepted alerts.810 We recruited a 7-member expert panel of 4 Massachusetts physicians and 3 pharmacists (including investigators S.N.W., T.I., and A.C.S.), inviting panelists based on their experience in primary care medicine and patient safety research.

Investigators oriented panelists to the project by means of a conference call and practiced sample cases. Panelists then reviewed the same 100 medication safety alerts and, informed by their experience, made a series of judgments about each alert. We provided information about each DDI taken from pharmaceutical reference texts, including Micromedex,11 Clinical Pharmacology,12 Lexi-Comp,13 Epocrates,14 and a research study by Malone et al.15

For each DDI, panelists estimated the probability that the interaction would result in an ADE—defined as an injury due to medication use—and, if so, whether the most likely ADE would be serious, significant, or minor.2,16,17 Serious ADEs could cause organ system dysfunction, such as a seizure or major gastrointestinal tract bleeding. Significant ADEs could cause symptoms such as a rash or fever and/or laboratory abnormalities such as thrombocytopenia or hyperkalemia. Minor ADEs could cause minimal injury, such as flushing or dyspepsia. Because alerts were generated for initial prescriptions rather than renewals, panelists assumed that the medications were being prescribed together for the first time. Panelists judged the severity of potential ADEs on the basis of the most likely reaction (rather than the worst-case scenario) of the typical patient who would receive the drug combination (in terms of age, comorbidities, and dosage). We asked panelists to classify each incident by likely frequency, injury, and health care utilization using the categories in Table 1. Finally, panelists estimated the probability that a given drug interaction would have been intercepted in a community pharmacy and the prescription not dispensed.

Table Graphic Jump LocationTable 1. Expert Panel Instructions for Rating Drug Interaction Alerts

Panelists scored each interaction independently. We selected 15 interactions with the greatest scoring discrepancies and presented them in a second conference call for discussion. Participants rescored these DDIs, and these final scores were used in the analyses.

COST ESTIMATES

We used published sources and payer data to estimate the costs to third-party payers associated with specified categories of health care utilization. The average cost of a medical hospitalization was estimated at $9000 (Adrienne Cyrulik, MPH, Blue Cross/Blue Shield of Massachusetts, written communication, June 17, 2008). Based on a study of the cost of emergency department visits,18 the average cost of an emergency department visit was calculated as $427, adjusted to 2006 dollars using the Consumer Price Index for medical services.19 The cost of a physician visit due to an ADE ($111) was derived from the average national charge in 2006 dollars for a 25-minute office visit.20 For physician visits that generated additional prescriptions, we estimated the average cost of a filled prescription at $58, calculated using the sales of prescription drugs in 2006 ($192 024 661 635) divided by the number of prescriptions dispensed (3 308 896 262).21 The cost of telephone calls with clinicians was assumed to be $0 because clinicians are not commonly reimbursed for this service.

STATISTICAL ANALYSES

We examined the interrater reliability of panelists' first-round judgments regarding the probability that each DDI would produce a serious, significant, or minor ADE; the type of injury to the patient; and the resulting health care utilization. Overall, agreement was satisfactory, with a range of 86% to 94% and κ scores of 0.49 to 0.69 (P < .001).

We used the panelists' median scores to estimate the type of ADE, injury, and health care utilization for each DDI. We then used the cost estimates, the number of CCI alerts, and the mathematical models shown in the eAppendixhttp://www.archinternmed.com to calculate the number of prevented ADEs, number of injuries, extent of health care utilization, and costs. We assumed that nonintercepted prescriptions were filled at a rate of 90% based on a previous study.6

Because our data set included 6 months of prescription information, we annualized the results to provide a more intuitive presentation of the data. We extrapolated the analysis of the 100 most commonly accepted DDIs, each selected from 1 of the top 100 CCIs, to the CCIs as a whole. In doing so, we assumed that each DDI presented to the panelists was representative of other DDIs within the same CCI subset. To estimate the total cost savings and the impact of medication safety alerts on patient safety in Massachusetts attributable to PocketScript prescribers, we then extrapolated from the 100 most common CCIs to the entire universe of alerts, assuming that weighted average rates would apply (Figure 2). A test of our assumptions supported this method: the average acceptance rate for the top 100 DDIs in the top 100 classes was 12.2%, whereas the acceptance rate for all other DDIs in the top 100 classes was 12.9%. The average acceptance rate for the 100 most commonly accepted CCIs was similar to that of the remaining CCIs (10.6% vs 10.0%).

Place holder to copy figure label and caption
Figure 2.

Flow diagram depicting selection of drug-drug interaction (DDI) alerts for analysis and extrapolation of the results of the analysis to all DDI alerts generated by a single electronic prescribing system. Data are from Massachusetts in 2006. ADE indicates adverse drug event; CCI, class-class interaction.

Graphic Jump Location

We performed sensitivity analyses to test the stability of our results over a range of assumptions. Given the limited number of reviewers for each DDI and the possibility of a skewed distribution of scores, we calculated upper- and lower-bound estimates of the likelihood of the type of ADE, injury, and health care utilization using the interquartile range (IQR) (25th-75th percentile) for each DDI. We varied our assumptions about the pharmacy interception rate (using the panel's estimate in the base case and 10% in an alternate model) and the rate with which patients filled their prescriptions (90% in the base case and 70% in an alternate model). We also examined the effect on savings of varying assumptions about the cost and likelihood of hospitalization. We used the average cost of a US hospitalization ($26 555) as a high-range estimate, calculated by dividing total hospital revenue from US inpatient hospital admissions in 2006 ($939 459 919 425) by the total number of admissions (35 377 659).22

This study was approved by the institutional review board of the Dana-Farber Harvard Cancer Center. Statistical analyses were performed using commercially available software (SAS [SAS Institute, Inc, Cary, North Carolina] and Stata 9.1 [StataCorp, College Station, Texas]).

ADEs AND RELATED INJURIES

During the 6-month study period, Massachusetts clinicians encountered DDI alerts in 7.3% of e-prescribing attempts and overrode 91.1% of the encountered DDI alerts. The crude rate of alert encounters—calculated without removing potential duplicate alerts—was 15.2%.

Table 2 shows the estimated number of ADEs that were prevented in 2006 when 2321 Massachusetts PocketScript users accepted medication safety alerts, based on the expert panel's assessment of risk and harm. In this base-case analysis, electronic alerts prevented an estimated 402 ADEs, of which 49 were serious or life threatening, 125 were significant, and 228 were minor (approximately 0.17 ADEs per year for each prescriber). The IQR (133-846 ADEs per year) shows the variability associated with these judgments.

Table Graphic Jump LocationTable 2. Number of Prevented ADEs and Injuries in Massachusetts in 2006 Owing to Accepted Medication Safety Alerts (Base Case) From a Single Electronic Prescribing System

The panel's estimates showed that many alerts were required to prevent a single ADE because relatively few DDIs posed a risk of harm. Overall, clinicians encountered 331 alerts to prevent a single ADE (of any severity) and 2715 alerts to prevent a single serious ADE. As shown in Figure 3, a small percentage of alerts prevented most ADEs. Ten percent of alerts were estimated to prevent 60% of ADEs, and only 6 of the 100 most commonly accepted DDI alerts were judged to have prevented at least 1 serious ADE per year (Table 3). Seventeen percent of alerts were judged unlikely to prevent ADEs of any severity.

Place holder to copy figure label and caption
Figure 3.

Cumulative number of serious, significant, and minor adverse drug events (ADEs) prevented by safety alerts. Data were obtained from a cohort of electronic prescribers in Massachusetts in 2006. A small percentage of alerts accounted for most of the estimated benefits.

Graphic Jump Location
Table Graphic Jump LocationTable 3. Selected DDI Alerts and Estimates of Serious ADEs Prevented

Of the 402 ADEs potentially prevented by medication alerts, the expert panel judged that alerts most commonly prevented symptoms that persisted less than 30 days (n = 272 [67.7%]) (Table 2). Of the 49 serious ADEs, accepted alerts may have prevented a patient death in 3 cases, permanent disability in 14, and temporary disability in 31. Clinicians encountered thousands of alerts (4292-44 350) to prevent a single potential death, disability, or case of prolonged symptoms.

COST SAVINGS

Preventing ADEs with e-prescribing may have resulted in 39 (IQR, 14-100) fewer hospitalizations, 34 (6-74) fewer emergency department visits, 267 (105-541) fewer office visits, and 60 (8-109) fewer telephone calls to clinicians. As shown in Table 4, e-prescribing alerts were judged to yield annual savings of $402 619 (IQR, $141 012-$1 012 386). Divided by the total number of eligible Massachusetts e-prescribers using the system (n = 2321), the average savings per clinician in 2006 was $173 (IQR, $61-$436). The bulk of the savings (86.8% of the total) was attributed to the 39 prevented hospitalizations. Although office visits were the most commonly averted service (n = 267 [66.4% of services]), the cost per hospitalization dominated the calculation.

Table Graphic Jump LocationTable 4. Prevented Health Care Costs in Massachusetts in 2006 Owing to Accepted Medication Safety Alerts (Base Case) From a Single Electronic Prescribing System

In a finding similar to the analysis of ADEs, a small number of alerts likely accounted for most of the savings. Ten percent of alerts were estimated to account for 78% of cost savings, derived largely from 32 prevented hospitalizations. Thirty-one percent of alerts resulted in savings of less than $100 annually, and 19% were judged to have no health care utilization impact from the perspective of third-party payers.

SENSITIVITY ANALYSES

Given the inherent uncertainty in estimating the likelihood of ADEs, associated injuries, and health care utilization, we analyzed the impact of alternate assumptions on our results. Compared with the base-case scenario, we found moderate differences (<30%, or <121 ADEs) in the number of ADEs when we varied the rate at which patients filled their prescriptions (70% in model 2 vs 90% in the base case), the likelihood that the pharmacist would intercept a potentially dangerous prescription (10% in model 3 vs the panelists' median estimate in the base case [median, 0.10%; IQR, 0%-1%]), and the combination of these factors (70% fill rate and 10% interception rate in model 4 [Figure 4]). These alternate assumptions resulted in estimates of 313, 367, and 285 prevented ADEs, respectively, relative to the base case (402 ADEs).

Place holder to copy figure label and caption
Figure 4.

Sensitivity analysis of the cumulative number of prevented adverse drug events (ADEs) owing to medication safety alerts. Varying assumptions about the rate with which patients filled prescribed medications (70% in model 2 vs 90% in the base case), the rate of pharmacy interceptions (10% in model 3 vs panelists' median estimates in the base case), and both factors (70% pharmacy fill rate and 10% pharmacy interception rate in model 4) did not result in drastically different estimates of prevented ADEs. The 75th and 25th percentile curves demonstrate the range of panelists' estimates regarding the frequency with which alerts prevented ADEs, using base-case assumptions.

Graphic Jump Location

Modifying these factors also had a moderate effect (<30%, or <$121 000) on cost savings ($313 000, $368 000, and $286 000 for models 2 through 4, respectively, compared with the base-case savings of $403 000). Savings were highly dependent on assumptions about the cost of hospitalization in Massachusetts. When used as an upper-bound estimate (rather than the base-case assumption of $9000 per hospitalization), the average cost of a US hospitalization ($26 555) yielded savings of $1.08 million (IQR, $383 000-$2.77 million). In addition to varying the cost of hospitalization, we also conducted sensitivity analyses using panelists' 25th and 75th percentile estimates of the likelihood of hospitalization for each DDI. Lower and upper hospitalization estimates resulted in projected cost savings of $146 000 and $487 000, respectively (Figure 5).

Place holder to copy figure label and caption
Figure 5.

Sensitivity analysis of the cumulative cost savings owing to medication safety alerts. Varying assumptions about the cost of hospitalization ($9000 for a Massachusetts [MA] admission and $26 555 for the average cost of a hospitalization in the United States), the likelihood of an adverse drug event (ADE) (panelists' median estimates in the base case vs the 25th and 75th percentile estimates in alternate scenarios), and the likelihood of hospitalization given an ADE (panelists' median estimates in the base case vs 25th and 75th percentile estimates in alternate scenarios) affected the estimated cost savings owing to electronic prescribing.

Graphic Jump Location

Using a modified Delphi technique and data on 1.8 million prescriptions and 135 051 unique alerts generated by ambulatory care clinicians, we estimated that e-prescribing alerts possibly averted 133 to 846 ADEs in Massachusetts in 2006, including 14 to 130 potentially serious ADEs that could have caused 2 to 13 deaths and 13 to 115 disabilities. Alerts may have prevented 125 to 715 hospitalizations, emergency department visits, and office visits, for a total savings to the health care system of $141 012 to $1 012 386. Extrapolating these results to all 38 847 prescribers (both inpatient and outpatient) in Massachusetts suggests that expanding e-prescribing statewide might prevent more than 6700 ADEs per year, including 50 deaths, and result in cost savings of approximately $6.7 million.

Our study provides perspective on earlier investigations of medication safety alerts. In previous studies, computerized prescribing in the hospital reduced nonintercepted serious medication errors by 55%, preventable ADEs by 17%, and hospital costs by $887 to $8958.16,2328 Decision analytic models similar to the approach used herein have been used to estimate the cost of drug-related morbidity in the United States at more than $177 billion in 2000.7 However, we are aware of no studies that have estimated the safety benefits or cost savings from the use of e-prescribing alerts in the ambulatory environment.

Given the recent passage of the Medicare Improvements for Patients and Providers Act of 2008,29 which provides federal incentives for the adoption of e-prescribing and imposes penalties on providers who fail to switch to the electronic systems, our findings offer timely information to government and private industry. The benefits of medication alerts in ambulatory care are likely derived from a small fraction of alerts encountered by frontline clinicians. Indeed, only 10% of drug interaction alerts were estimated to account for 60% of ADEs and 78% of cost savings in our study. Seventeen percent of alerts had no discernible patient safety benefit, and 31% made a small (<$100/y) contribution to cost savings. Accordingly, clinicians probably reviewed thousands of alerts to prevent a single serious ADE. In fact, our study may underestimate the alert burden by examining only a subset of alerts; we excluded more than 100 000 potentially redundant alerts and did not include formulary adherence or allergy alerts in our analyses.

The phenomenon of alert fatigue has been well described,3,15,3032 and the disproportionate relationship between the number of alerts and the patient safety and financial benefits of the system makes one wonder whether the juice is, in fact, worth the squeeze. Our findings suggest that the savings attributable to prevented ADEs may be insufficient to cover the costs to third-party payers of investing in e-prescribing systems. However, our estimates do not take into consideration savings that might accrue owing to improved formulary adherence, increased use of generic drugs, legible prescriptions, and the prevention of allergic reactions to prescribed drugs, and they do not account for the effect on the patients and families of lost wages and illness-related morbidity. Most important, we believe that the technology's ability to prevent ADEs makes it worthwhile, and our findings suggest that significant efficiencies could be gained by reducing overalerting. Doing so would mitigate alert fatigue, thereby increasing the percentage of clinically significant alerts accepted and the number of ADEs averted. Previous studies have demonstrated that tiering alerts and interrupting prescribers for only the most serious warnings are effective strategies for increasing alert acceptance rates.4,33

This study has several limitations. First, its generalizability may be restricted by the use of a single e-prescribing system and drug interaction alert database. However, in 2008, the PocketScript system was used by 8% of Massachusetts prescribers and approximately 4000 eligible prescribers in 18 states (Christopher Yu, MPH, ZixCorp, written communication, August 19, 2008). Its features, including required fields, pick lists with available dose forms, and DDI alerts, are common to many commercial and home-grown e-prescribing systems. In addition, Cerner Multum, Inc, which maintains the medication safety alerts for the PocketScript system, is 1 of several major commercial drug interaction databases used throughout North America.

Second, the decisions of Massachusetts clinicians to accept or override electronic alerts may not reflect the behavior of clinicians in other regions. However, in a related study of e-prescribing, we found that the rates and types of drug interaction alerts accepted by Massachusetts clinicians were similar to those of clinicians in New Jersey and Pennsylvania.34 Nevertheless, these studies examined clinicians who have generally opted into the system and therefore may reflect the experience of a relatively committed population of e-prescribers rather than what is expected from novice users.

Third, the structured implicit judgments of the expert panel were dependent on the makeup of the panel and the experience of its members. Although the use of a modified Delphi technique enhanced the reliability of this process, another panel with different members might have offered different estimates.

Finally, our analysis took a conservative approach to calculating the impact of e-prescribing in ambulatory care. The models did not consider the effect of alerts on clinician behavior other than aborting a prescription, such as counseling a patient or monitoring for drug interactions. We did not address benefits from e-prescribing that might derive from standardization of dosing, reduction in duplicate therapy, prevention of allergic reactions, and elimination of illegible prescriptions,35 nor did we attempt to assess the impact of alerts on patient satisfaction and drug adherence. We also focused on the estimated financial benefit to third-party payers, rather than the costs of medical injuries borne by patients and their families. Given the lack of data on these issues, our study used methods that are in line with those of other studies that have attempted to estimate the cost of illness and the benefits of e-prescribing.6,7,10,15 Future research should attempt to quantify the return on investment to medication safety alerts, including reduction in medical liability premiums and losses, to determine when the cost of these systems and the burden of excessive alerts detract from patient care. Studies could also examine differences between drug interaction databases and their effect on patient safety.

Our study suggests that drug alerts have the potential to prevent harm and reduce health care costs. To do so, however, clinicians need relief from alerts with little clinical value.

Correspondence: Saul N. Weingart, MD, PhD, Center for Patient Safety, Dana-Farber Cancer Institute, 44 Binney St, Boston, MA 02115 (saul_weingart@dfci.harvard.edu).

Accepted for Publication: May 15, 2009.

Author Contributions: Dr Weingart had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Weingart, Padolsky, and Weissman. Acquisition of data: Weingart and Seger. Analysis and interpretation of data: Weingart, Simchowitz, Padolsky, Isaac, Massagli, Davis, and Weissman. Drafting of the manuscript: Weingart, Simchowitz, and Padolsky. Critical revision of the manuscript for important intellectual content: Weingart, Padolsky, Isaac, Seger, Massagli, Davis, and Weissman. Statistical analysis: Weingart, Padolsky, Isaac, and Davis. Obtained funding: Weingart and Weissman. Administrative, technical, and material support: Weingart, Simchowitz, Padolsky, and Seger. Study supervision: Weingart.

Financial Disclosure: None reported.

Funding/Support: This study was supported by a grant from Blue Cross Blue Shield of Massachusetts.

Role of the Sponsor: Blue Cross Blue Shield of Massachusetts was not involved in the design and conduct of the study; the collection, management, analysis, and interpretation of data; or the preparation, review, or approval of the final manuscript.

Additional Contributions: Christopher Yu, MPH, Angus MacDonald, and Geoff Bibby, BC, of ZixCorp, provided the electronic prescription and alert data; Adrienne Cyrulik, MPH, and Jessica Fefferman, MPH, of Blue Cross Blue Shield of Massachusetts, assisted in conceptualizing the project and providing third-party-payer information.

Gandhi  TKWeingart  SNBorus  J  et al.  Adverse drug events in ambulatory care. N Engl J Med 2003;348 (16) 1556- 1564
PubMed
Gandhi  TKWeingart  SNSeger  AC  et al.  Outpatient prescribing errors and the impact of computerized prescribing. J Gen Intern Med 2005;20 (9) 837- 841
PubMed
Weingart  SNToth  MSands  DZAronson  MDDavis  RBPhillips  RS Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003;163 (21) 2625- 2631
PubMed
Shah  NRSeger  ACSeger  DL  et al.  Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006;13 (1) 5- 11
PubMed
Payne  THNichol  WPHoey  PSavarino  J Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp 2002;602- 606
PubMed
Johnson  JABootman  JL Drug-related morbidity and mortality: a cost-of-illness model. Arch Intern Med 1995;155 (18) 1949- 1956
PubMed
Ernst  FRGrizzle  AJ Drug-related morbidity and mortality: updating the cost-of-illness model. J Am Pharm Assoc (Wash) 2001;41 (2) 192- 199
PubMed
Stewart  JO’Halloran  CHarrigan  PSpencer  JABarton  RJSingleton  SJ Identifying appropriate tasks for the preregistration year: modified Delphi technique. BMJ 1999;319 (7204) 224- 229
PubMed
Fink  AKosecoff  JChassin  MBrook  RH Consensus methods: characteristics and guidelines for use. Am J Public Health 1984;74 (9) 979- 983
PubMed
Park  REFink  ABrook  RH  et al.  Physician ratings of appropriate indications for six medical and surgical procedures. Am J Public Health 1986;76 (7) 766- 772
PubMed
 Micromedex Healthcare Series (electronic version). Thomson Healthcare. http://www.thomsonhc.com. Accessed February 29, 2008
 Clinical Pharmacology Web site. Gold Standard Inc. http://www.clinicalpharmacology.com/. Accessed February 29, 2008
 Lexi-Comp Inc Web site. http://www.lexi-comp.com/institutions/products/online/. Accessed February 29, 2008
 Epocrates Web site. http://www.epocrates.com. Accessed February 29, 2008
Malone  DCAbarca  JHansten  PD  et al.  Identification of serious drug-drug interactions: results of the partnership to prevent drug-drug interactions. J Am Pharm Assoc (2003) 2004;44 (2) 142- 151
PubMed
Bates  DWLeape  LLCullen  DJ  et al.  Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998;280 (15) 1311- 1316
PubMed
Morimoto  TGandhi  TKSeger  ACHsieh  TCBates  DW Adverse drug events and medication errors: detection and classification methods. Qual Saf Health Care 2004;13 (4) 306- 314
PubMed
Dennehy  CEKishi  DTLouie  C Drug-related illness in emergency department patients. Am J Health Syst Pharm 1996;53 (12) 1422- 1426
PubMed
US Department of Labor, Bureau of Labor Statistics, Consumer Price Index for all urban consumers (CPI-U) for the US city average for medical care services, 1982-1984. http://www.bls.gov/cpi/home.htm#tables. Accessed March 24, 2008
Wasserman  Y Physicians' Fee Reference 2005. 22nd ed. Milwaukee, WI Medical Publishers Ltd2005;
 Top 200 brand-name drugs by retail sales in 2006 Drug Topics February19 2007.;http://drugtopics.modernmedicine.com/drugtopics/data/articlestandard/drugtopics/072007/405100/article.pdf. Accessed March 24, 2008
American Hospital Association, AHA Hospital Statistics, 2008 Edition.  Chicago, IL Jossey-Bass/AHA Press2008;
Bates  DWSpell  NCullen  DJ  et al. Adverse Drug Events Prevention Study Group, The costs of adverse drug events in hospitalized patients. JAMA 1997;277 (4) 307- 311
PubMed
Tierney  WMMiller  MEOverhage  JM McDonald  CJ Physician inpatient order writing on microcomputer workstations: effects on resource utilization. JAMA 1993;269 (3) 379- 383
PubMed
Mekhjian  HSKumar  RRKuehn  L  et al.  Immediate benefits realized following implementation of physician order entry at an academic medical center. J Am Med Inform Assoc 2002;9 (5) 529- 539
PubMed
Evans  RSPestotnik  SLClassen  DC  et al.  A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998;338 (4) 232- 238
PubMed
Schneider  PJGift  MGLee  YPRothermich  EASill  BE Cost of medication-related problems at a university hospital. Am J Health Syst Pharm 1995;52 (21) 2415- 2418
PubMed
Bates  DWTeich  JMLee  J  et al.  The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999;6 (4) 313- 321
PubMed
 Medicare Improvements for Patients and Providers Act of 2008, Pub L No. 110-275 (July 15, 2008) 
Glassman  PASimon  BBelperio  PLanto  A Improving recognition of drug interactions: benefits and barriers to using automated drug alerts. Med Care 2002;40 (12) 1161- 1171
PubMed
Ko  YAbarca  JMalone  DC  et al.  Practitioners' views on computerized drug-drug interaction alerts in the VA system. J Am Med Inform Assoc 2007;14 (1) 56- 64
PubMed
Magnus  DRodgers  SAvery  AJ GPs' views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther 2002;27 (5) 377- 382
PubMed
Paterno  MDMaviglia  SMGorman  PN  et al.  Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009;16 (1) 40- 46
PubMed
Isaac  TWeissman  JSDavis  RB  et al.  Overrides of medication alerts in ambulatory care. Arch Intern Med 2009;169 (3) 305- 311
PubMed
Fischer  MAVogeli  CStedman  MFerris  TBrookhart  MAWeissman  JS Effect of electronic prescribing with formulary decision support on medication use and cost. Arch Intern Med 2008;168 (22) 2433- 2439
PubMed

Figures

Place holder to copy figure label and caption
Figure 1.

Conceptual model for estimating the patient safety impact, health care utilization, and cost savings attributable to accepted medication safety alerts. Alerts are generated in an electronic prescribing system. ADE indicates adverse drug event; DDI, drug-drug interaction; and ED, emergency department.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.

Flow diagram depicting selection of drug-drug interaction (DDI) alerts for analysis and extrapolation of the results of the analysis to all DDI alerts generated by a single electronic prescribing system. Data are from Massachusetts in 2006. ADE indicates adverse drug event; CCI, class-class interaction.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 3.

Cumulative number of serious, significant, and minor adverse drug events (ADEs) prevented by safety alerts. Data were obtained from a cohort of electronic prescribers in Massachusetts in 2006. A small percentage of alerts accounted for most of the estimated benefits.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 4.

Sensitivity analysis of the cumulative number of prevented adverse drug events (ADEs) owing to medication safety alerts. Varying assumptions about the rate with which patients filled prescribed medications (70% in model 2 vs 90% in the base case), the rate of pharmacy interceptions (10% in model 3 vs panelists' median estimates in the base case), and both factors (70% pharmacy fill rate and 10% pharmacy interception rate in model 4) did not result in drastically different estimates of prevented ADEs. The 75th and 25th percentile curves demonstrate the range of panelists' estimates regarding the frequency with which alerts prevented ADEs, using base-case assumptions.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 5.

Sensitivity analysis of the cumulative cost savings owing to medication safety alerts. Varying assumptions about the cost of hospitalization ($9000 for a Massachusetts [MA] admission and $26 555 for the average cost of a hospitalization in the United States), the likelihood of an adverse drug event (ADE) (panelists' median estimates in the base case vs the 25th and 75th percentile estimates in alternate scenarios), and the likelihood of hospitalization given an ADE (panelists' median estimates in the base case vs 25th and 75th percentile estimates in alternate scenarios) affected the estimated cost savings owing to electronic prescribing.

Graphic Jump Location

Tables

Table Graphic Jump LocationTable 1. Expert Panel Instructions for Rating Drug Interaction Alerts
Table Graphic Jump LocationTable 2. Number of Prevented ADEs and Injuries in Massachusetts in 2006 Owing to Accepted Medication Safety Alerts (Base Case) From a Single Electronic Prescribing System
Table Graphic Jump LocationTable 3. Selected DDI Alerts and Estimates of Serious ADEs Prevented
Table Graphic Jump LocationTable 4. Prevented Health Care Costs in Massachusetts in 2006 Owing to Accepted Medication Safety Alerts (Base Case) From a Single Electronic Prescribing System

References

Gandhi  TKWeingart  SNBorus  J  et al.  Adverse drug events in ambulatory care. N Engl J Med 2003;348 (16) 1556- 1564
PubMed
Gandhi  TKWeingart  SNSeger  AC  et al.  Outpatient prescribing errors and the impact of computerized prescribing. J Gen Intern Med 2005;20 (9) 837- 841
PubMed
Weingart  SNToth  MSands  DZAronson  MDDavis  RBPhillips  RS Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003;163 (21) 2625- 2631
PubMed
Shah  NRSeger  ACSeger  DL  et al.  Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006;13 (1) 5- 11
PubMed
Payne  THNichol  WPHoey  PSavarino  J Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp 2002;602- 606
PubMed
Johnson  JABootman  JL Drug-related morbidity and mortality: a cost-of-illness model. Arch Intern Med 1995;155 (18) 1949- 1956
PubMed
Ernst  FRGrizzle  AJ Drug-related morbidity and mortality: updating the cost-of-illness model. J Am Pharm Assoc (Wash) 2001;41 (2) 192- 199
PubMed
Stewart  JO’Halloran  CHarrigan  PSpencer  JABarton  RJSingleton  SJ Identifying appropriate tasks for the preregistration year: modified Delphi technique. BMJ 1999;319 (7204) 224- 229
PubMed
Fink  AKosecoff  JChassin  MBrook  RH Consensus methods: characteristics and guidelines for use. Am J Public Health 1984;74 (9) 979- 983
PubMed
Park  REFink  ABrook  RH  et al.  Physician ratings of appropriate indications for six medical and surgical procedures. Am J Public Health 1986;76 (7) 766- 772
PubMed
 Micromedex Healthcare Series (electronic version). Thomson Healthcare. http://www.thomsonhc.com. Accessed February 29, 2008
 Clinical Pharmacology Web site. Gold Standard Inc. http://www.clinicalpharmacology.com/. Accessed February 29, 2008
 Lexi-Comp Inc Web site. http://www.lexi-comp.com/institutions/products/online/. Accessed February 29, 2008
 Epocrates Web site. http://www.epocrates.com. Accessed February 29, 2008
Malone  DCAbarca  JHansten  PD  et al.  Identification of serious drug-drug interactions: results of the partnership to prevent drug-drug interactions. J Am Pharm Assoc (2003) 2004;44 (2) 142- 151
PubMed
Bates  DWLeape  LLCullen  DJ  et al.  Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998;280 (15) 1311- 1316
PubMed
Morimoto  TGandhi  TKSeger  ACHsieh  TCBates  DW Adverse drug events and medication errors: detection and classification methods. Qual Saf Health Care 2004;13 (4) 306- 314
PubMed
Dennehy  CEKishi  DTLouie  C Drug-related illness in emergency department patients. Am J Health Syst Pharm 1996;53 (12) 1422- 1426
PubMed
US Department of Labor, Bureau of Labor Statistics, Consumer Price Index for all urban consumers (CPI-U) for the US city average for medical care services, 1982-1984. http://www.bls.gov/cpi/home.htm#tables. Accessed March 24, 2008
Wasserman  Y Physicians' Fee Reference 2005. 22nd ed. Milwaukee, WI Medical Publishers Ltd2005;
 Top 200 brand-name drugs by retail sales in 2006 Drug Topics February19 2007.;http://drugtopics.modernmedicine.com/drugtopics/data/articlestandard/drugtopics/072007/405100/article.pdf. Accessed March 24, 2008
American Hospital Association, AHA Hospital Statistics, 2008 Edition.  Chicago, IL Jossey-Bass/AHA Press2008;
Bates  DWSpell  NCullen  DJ  et al. Adverse Drug Events Prevention Study Group, The costs of adverse drug events in hospitalized patients. JAMA 1997;277 (4) 307- 311
PubMed
Tierney  WMMiller  MEOverhage  JM McDonald  CJ Physician inpatient order writing on microcomputer workstations: effects on resource utilization. JAMA 1993;269 (3) 379- 383
PubMed
Mekhjian  HSKumar  RRKuehn  L  et al.  Immediate benefits realized following implementation of physician order entry at an academic medical center. J Am Med Inform Assoc 2002;9 (5) 529- 539
PubMed
Evans  RSPestotnik  SLClassen  DC  et al.  A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998;338 (4) 232- 238
PubMed
Schneider  PJGift  MGLee  YPRothermich  EASill  BE Cost of medication-related problems at a university hospital. Am J Health Syst Pharm 1995;52 (21) 2415- 2418
PubMed
Bates  DWTeich  JMLee  J  et al.  The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999;6 (4) 313- 321
PubMed
 Medicare Improvements for Patients and Providers Act of 2008, Pub L No. 110-275 (July 15, 2008) 
Glassman  PASimon  BBelperio  PLanto  A Improving recognition of drug interactions: benefits and barriers to using automated drug alerts. Med Care 2002;40 (12) 1161- 1171
PubMed
Ko  YAbarca  JMalone  DC  et al.  Practitioners' views on computerized drug-drug interaction alerts in the VA system. J Am Med Inform Assoc 2007;14 (1) 56- 64
PubMed
Magnus  DRodgers  SAvery  AJ GPs' views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther 2002;27 (5) 377- 382
PubMed
Paterno  MDMaviglia  SMGorman  PN  et al.  Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009;16 (1) 40- 46
PubMed
Isaac  TWeissman  JSDavis  RB  et al.  Overrides of medication alerts in ambulatory care. Arch Intern Med 2009;169 (3) 305- 311
PubMed
Fischer  MAVogeli  CStedman  MFerris  TBrookhart  MAWeissman  JS Effect of electronic prescribing with formulary decision support on medication use and cost. Arch Intern Med 2008;168 (22) 2433- 2439
PubMed

Correspondence

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).
Submit a Comment

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 26

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Topics
PubMed Articles