Unintended Effects of a Computerized Physician Order Entry Nearly Hard-Stop Alert to Prevent a Drug Interaction:  A Randomized Controlled Trial FREE

Anticoagulants, especially warfarin, are the cornerstone of therapy for several diseases, including the prophylaxis and treatment of pulmonary embolism, venous thrombosis, and atrial fibrillation with embolization. Although anticoagulants confer significant benefits, they are associated with a high risk of adverse events, specifically bleeding, which is more likely in the setting of supratherapeutic anticoagulation.

Many medications can increase the anticoagulation effects of warfarin. In observational studies^1- 2 assessing risk factors for overanticoagulation, antibiotics have been a common culprit, with trimethoprim-sulfamethoxazole among the most common. In a retrospective cohort study³ of patients using warfarin who initiated any of several antibiotic therapies, 69% of patients using trimethoprim-sulfamethoxazole exhibited clinically significant elevations in the international normalized ratio to greater than 4. Adverse bleeding events developed in 13% of the patients exposed to trimethoprim-sulfamethoxazole and in none of the other antibiotic groups studied.³ A case-control study⁴ showed that recent initiation of trimethoprim-sulfamethoxazole therapy in patients receiving warfarin was associated with hospitalization for gastrointestinal bleeding (adjusted odds ratio, 1.46 [95% confidence interval (CI), 1.16-1.85] for a prescription filled 0-5 days before the hospitalization and 2.54 [2.08-3.10] for a prescription filled 6-10 days before the hospitalization). For several years, the Hospital of the University of Pennsylvania has, therefore, had in place a program of pharmacy interventions in which pharmacists telephoned prescribers to notify them of this interaction whenever a simultaneous order was written, that is, a pharmacist intervention program. Yet, some physicians still continued to prescribe these drugs concurrently.

In 1997, the University of Pennsylvania Health System implemented a computerized physician order entry (CPOE) system for inpatient care.⁵ Yet, numerous previous studies have shown that using preprogrammed alerts contained in the decision support as part of a CPOE system is often ineffective in changing prescribing^6- 11 and is frequently related to poor design characteristics of the alerts and insufficient consideration of human factors in the implementation.^12- 13 The results also may differ when using vendor-developed solutions vs homegrown solutions. Automatic order alerts are built into the CPOEs to protect patients against harmful administration of medications. Automatic order alerts may be either soft or hard. When soft order alerts appear on the computer screen, clinicians are alerted about potential problems associated with the particular prescription order and are presented with alternative treatments available for consideration. In contrast, when hard stop-order alerts appear on the screen, the clinician's order is blocked from further execution to avert potentially serious reactions. The goal of this study was to measure the effectiveness of a customized nearly hard stop alert in reducing concomitant orders for warfarin and trimethoprim-sulfamethoxazole compared with the standard practice of a pharmacist intervention program.

This study was a randomized controlled trial initially planned for 7 months, that was started on August 9, 2006, but was terminated early on February 13, 2007.

This study was conducted at the Hospital of the University of Pennsylvania and at Penn Presbyterian Medical Center, where all inpatient orders are entered using the Sunrise Clinical Manager (Eclipsys Corp, Atlanta, Georgia) CPOE system. The study participants were 1981 eligible clinicians involved in inpatient care, of whom 1971 were included in the final analysis (1872 resident physicians [RPs] and 99 nurse practitioners [NPs]).

The randomization procedure involved listing in a spreadsheet the names of all the RPs and NPs and assigning each a random number. We then rank-ordered the list based on the random numbers and selected the first half to be in the intervention group and the second half to be in the control group.

The intervention included clinicians subject to an automatic electronic hard-stop alert of a trimethoprim-sulfamethoxazole or warfarin order entered into the CPOE system whenever an RP or NP placed an order for trimethoprim-sulfamethoxazole with an already-active warfarin order, if warfarin was ordered for a patient already taking trimethoprim-sulfamethoxazole, or when ordering both simultaneously. The hard-stop alert appeared as a pop-up window that notified the clinician that the order could not be processed because of a significant potential drug-drug interaction. The specific text of the stop alert read as follows:

The prescription of warfarin and TMP/Sulfa together is completely prohibited except in cases of urgent need for the TMP/Sulfa.

If you are attempting to prescribe warfarin and the patient is already on TMP/Sulfa, discontinue the TMP/Sulfa and your order for warfarin will be processed.

If you are attempting to prescribe TMP/Sulfa and feel that your patient has an urgent need, then contact the inpatient pharmacy and you will be directed as to how to process the order for TMP/Sulfa.

Alerts could be overridden in 2 ways. One way, which did not involve pharmacist intervention, was to enter in the order the indication of Pneumocystis carinii pneumonia (PCP) prophylaxis. This triggered an acknowledge alert that read as follows: “Dr. xxx, you have certified that this patient has PCP. Click on ‘Acknowledge’ button to certify that this diagnosis is still active to proceed with this prescription.” The other way to override the alert was to bypass the CPOE altogether by calling the pharmacist directly.

The control group continued with the prevailing practice of the pharmacist telephoning prescribers to notify them of this interaction and recommend cessation for concurrent warfarin and trimethoprim-sulfamethoxazole orders, that is, a pharmacist intervention program.

The primary outcome was a new concurrent prescription order for trimethoprim-sulfamethoxazole and warfarin accepted through the electronic ordering system. In this context, not reordering the alert-triggering drug within 10 minutes after the alert fired (or would have fired in the control group) represented a desired response by the providers. This time frame was chosen because it was a reasonable period within which a clinician, having just received an alert, would have reacted to it. However, because 10 minutes was an arbitrary time frame, we also repeated the analysis using an outcome of 24 hours; the results were comparable and are not presented.

Two potential adverse outcomes of the computerized hard-stop alert were also of interest. The first was a delay in obtaining trimethoprim-sulfamethoxazole when the practitioner believed that an infection was best treated with trimethoprim-sulfamethoxazole and when the potential warfarin interaction was judged less important than the need for the antibiotic. The second was unintentional warfarin cessation in a patient previously undergoing long-term warfarin therapy. Therefore, the study also assessed the incidence of warfarin cessation on the day when an order of trimethoprim-sulfamethoxazole was attempted in a patient already receiving warfarin.

All the data were obtained from the CPOE systems of the 2 study hospitals. The unit of analysis for this study was the hard stop alerts that fired, or would have fired, whenever a prescription order for concurrent trimethoprim-sulfamethoxazole and warfarin was encountered during the inpatient stay. Alerts were attributed to the study group of the clinician who ordered the add-on prescription that triggered the alert.

Some alerts appeared to be firing multiple times, immediately one after another, presumably because providers attempted to reenter the order for the triggering medication to attempt to overrule the alert. Accordingly, we applied a 5-minute rule in defining units of analysis; we counted alerts firing within 5 minutes of each other as a single episode and alerts firing more than 5 minutes apart as separate episodes.

This study was reviewed by the University of Pennsylvania's institutional review board (IRB), receiving approval with exemption from obtaining the clinicians' consent and a waiver of Health Insurance Portability and Accountability Act authorization. All faculty and clinical staff at the University of Pennsylvania Health System receive annual notification that they may become part of research evaluating interventions designed to improve clinical practice, specifically, interventions involving electronic systems.

The IRB initially had concerns that depriving the control group of the intervention would be unethical and, in fact, was initially reluctant to approve the study. Another IRB concern was that the new alert could lead to delays in patient care: either that clinicians would discontinue warfarin because of the alert or that the alternative antibiotic for trimethoprim-sulfamethoxazole may not work as well in a given patient. Also, there may be reasons to give trimethoprim-sulfamethoxazole even with warfarin. To ensure that the risk remained minimal, the IRB requested a monitoring plan to look continuously for signals of delaying treatment (ie, inappropriate delay or stoppage of warfarin and inappropriate delay or stoppage of trimethoprim-sulfamethoxazole) resulting from the new alert. Accordingly, a monitoring committee was convened and charged with reviewing monthly reports of all events in which the electronic intervention was activated to determine the occurrences of inappropriate discontinuation or failure to begin therapy in situations in which warfarin and trimethoprim-sulfamethoxazole were appropriate.

The proportions of “desired responses” (ie, not reordering the alert-triggering drug within 10 minutes after alert firing) in the intervention and control groups were compared using logistic regression,¹⁴ accounting for clustering¹⁵ of RPs and NPs (ie, the nonindependence of outcomes in patients within the same clinic), using STATA version 10 (StataCorp LP, College Station, Texas).

Figure 1 shows the profile of the 1981 clinicians randomized to the 2 study groups (995 intervention clinicians receiving the active hard stop alerts and 986 control clinicians who would have received the alerts and did not), of whom 1971 were included in the final analysis after 10 were excluded because they erroneously appeared in both lists. During the study period, these providers ordered 8826 prescriptions through the CPOE system (3167 warfarin and 1036 trimethoprim-sulfamethoxazole prescriptions ordered by intervention clinicians, and 3630 warfarin and 993 trimethoprim-sulfamethoxazole prescriptions ordered by control clinicians). Seven patients in the intervention group were exempt because of PCP prophylaxis.

Overall, 437 alerts had fired. Of these, 342 alerts (194 in the intervention group and 148 in the control group) were analyzed as unique events after applying the 5-minute rule. Of the 194 hard stop alerts issued to the intervention group, the proportion of desired responses by the clinicians (ie, not reordering the alert-triggering drug within 10 minutes of firing) was 57.2% (n = 111). Of the 83 undesired responses in the intervention group, the alert-triggering drug ordered was warfarin in 78 and trimethoprim-sulfamethoxazole in 5. In contrast, the comparable proportion in the control group was 13.5% (n = 20). Of the 128 undesired responses in the control group, the alert-triggering drug ordered was warfarin in 121, trimethoprim-sulfamethoxazole in 2, and both drugs simultaneously in 5. Clinicians in the intervention group were less likely than control clinicians to reorder the alert-triggering drug after adjusting for provider type (RP or NP) as a confounder and accounting for clustering by provider (adjusted odds ratio, 0.12; 95% CI, 0.045-0.33). The unadjusted odds ratio was 0.12 (95% CI, 0.07-0.20). Adjustment for hospital was not possible because 1 hospital had only 1 observation in which the alerting medication was reordered within 10 minutes after the stop alert had fired. Note the high mean number of alerts per provider (3.53 in the intervention group and 3.29 in the control group), even after counting alerts triggered by repeated orders within 5 minutes as a single alert event. The greatest proportion of desired responses was observed in the first 3 months of the intervention, after which it steadily declined, suggesting that the effectiveness of the alert may have started to wear off (Figure 2). The groups remained different, however, at the end of the study.

Finally, the study was stopped early by the IRB because of 4 instances of unintended consequences identified in the intervention group. For a synopsis of the monthly monitoring of unintended consequences associated with the intervention—either a delay of treatment with trimethoprim-sulfamethoxazole when determined to be necessary for treatment or inadvertent warfarin discontinuation—see the Table. Review of the electronic records of all 4 individuals involved in these events showed that in no case could we identify specific infectious or thrombotic complications that could have been related to the delays in therapy in the adverse event reporting.

This study revealed that an electronic order intervention that is nearly a hard-stop alert can, indeed, be effective in reducing the undesired prescribing, at least initially. This intervention also resulted in delay in ordering trimethoprim-sulfamethoxazole when the practitioners believed that trimethoprim-sulfamethoxazole was necessary for their patients. This necessitated the early termination of this study for ethical reasons because of potential for harm in the intervention arm, a dramatic finding given the IRB's initial concern for the potential for harm by having a control group that was denied the intervention.

Alerts are typically communicated through pop-up warning messages on the computer screen (ie, simple information triggers). Such CPOE system interventions have reduced medication error rates,^16- 17 but even important electronic alerts are sometimes overridden,^6- 11 suggesting that there is still much to learn about the best way to present such alerts to providers.^10,18- 19 In inpatient settings, Hsieh et al²⁰ found that only 20% of physicians changed prescriptions in response to drug allergy alerts, and Payne et al¹¹ found that only 12% of critical drug interaction alerts and 31% of drug allergy interaction alerts resulted in a changed prescription order. The principal reasons for low response by clinicians to the automated alerts are many low-consequence alert firings and poor design of alerts.^12- 13 Clinicians tend to override the alerts because they are perceived to be nonspecific and lack the clinicians' additional knowledge of the clinical situation for the specific patient context.^12,21- 25 Given the perception among clinicians that most alerts are inappropriate and are a nuisance because of low clinical relevance, CPOE-based drug alerts may perhaps be more effective if they were customized to include only a limited number of critically important alerts.^12,24- 25 Another approach has been to design alerts that require from the clinician a specific response to the alert (eg, entering a reason for overriding).^20,26- 29 Alerts that require a response seem to be somewhat more effective,^12,26- 29 presumably because they engage the clinician's attention, although not always,^20,29 and they increase the clinician's time burden.^18,20

Still another approach is to design hard stop alerts that prevent the clinician from completing the prescription order entry, although some find it objectionable on the grounds that decision support should not replace the clinician's responsibility for their patients.¹⁰ Published reports of interventions that used hard stop alerts are scant.¹³

An important component of CPOE systems is to check prescription orders by clinicians to reduce medication errors and improve patient safety. As part of this support, CPOE systems are designed to trigger safety alerts to prevent adverse drug reactions. However, excessive alerting can disrupt clinician workflows,^18,30 and overdependence on CPOE systems may cause chaos when the system is down, with clinicians having trouble remembering medication contraindications, standard dosages, and hospital formulary recommendations.^10,30- 31 Such unintended consequences^5,32 and unintended therapy delays, such as occurred in a few cases in this study that were determined to be definitely or probably related to the alert intervention, provide yet another example of the need for formal evaluations of CPOE interventions rather than assuming that they are always necessarily of benefit.

Because many prescribing “errors” detected by automated systems may, in fact, represent reasonable prescribing decisions, it is proper to inquire about the clinical appropriateness of the simultaneous warfarin and trimethoprim-sulfamethoxazole orders that were entered despite the intervention. The review of these cases deemed them to represent delays in appropriate therapy being administered. One was a failure to prescribe appropriate trimethoprim-sulfamethoxazole prophylaxis for an otherwise critically ill patient. Another was a delay in initiation of antibiotic therapy recommended by the Infectious Diseases Consultation Service. The others were delays in initiation of warfarin therapy.

As noted in a recent review by Eslami et al,¹⁷ many trials of CPOE and medication safety were not randomized. The strength of this study was its randomized design and statistical power. It is generally accepted that randomization of at least 100 subjects will produce balance between the study groups,³³ and, of course, the present sample size is much larger than this.

The decision to randomize clinicians rather than patients was motivated by several considerations. First, because the order is written by clinicians, it is appropriate to consider each order by them as an opportunity for error. Second, each medical practitioner has a unique access code to use the electronic ordering system, and the order system menu can be varied by individual user. In addition, we wanted to keep each practitioner in the same study group for the duration of the study to minimize contamination between the 2 groups. However, there is the possibility that the difference in effect that we observed between the intervention and control groups for concurrent orders of trimethoprim-sulfamethoxazole and warfarin may have diminished across time during the study because RPs and NPs usually work in teams and may work alongside other physicians who may be assigned to a different group in the study. It is common for RPs and NPs to discuss a patient's care plan, which includes issues of ordering medications. Across time, the RPs and NPs from both groups may have learned not to order trimethoprim-sulfamethoxazole with an active order of warfarin because of awareness of the hard-stop alert. We attempted to reduce contamination by trying to complete this study as rapidly as possible. It was initially planned to last 7 months but had to be terminated early.

Finally, because we did not review the medical records of all the patients who, in fact, went on to receive the concurrent warfarin and trimethoprim-sulfamethoxazole medications and because we did not follow up on possible adverse events associated with this drug interaction in these patients, we do not have information to conclude that the benefits of reducing the critical warfarin and trimethoprim-sulfamethoxazole drug interaction outweigh the harms that were observed in this trial. Yet, the harm here was believed to be sufficient that the IRB terminated the study early.

In conclusion, we showed that a computerized decision support intervention—a nearly hard-stop alert—was markedly effective in reducing the prescribing of an undesired drug-drug combination. However, this intervention had unintended adverse consequences that were deemed sufficiently serious to warrant discontinuation of the intervention and early termination of the study. This emphasizes the need for formal evaluation and monitoring of programmatic interventions rather than simply assuming that they will be effective.

Correspondence: Brian L. Strom, MD, MPH, Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania School of Medicine, 824 Blockley Hall, 423 Guardian Dr, Philadelphia, PA 19104-6021 (bstrom@exchange.upenn.edu).

Accepted for Publication: March 20, 2010.

Author Contributions:Study concept and design: Strom, Schinnar, Aberra, Hennessy, and Pifer. Acquisition of data: Strom and Pifer. Analysis and interpretation of data: Strom, Schinnar, Aberra, Bilker, Hennessy, Leonard, and Pifer. Drafting of the manuscript: Strom and Schinnar. Critical revision of the manuscript for important intellectual content: Aberra, Bilker, Hennessy, Leonard, and Pifer. Statistical analysis: Bilker. Obtained funding: Strom. Administrative, technical, and material support: Schinnar and Leonard. Study supervision: Strom.

Financial Disclosure: None reported.

Funding/Support: This study was funded in part by the University of Pennsylvania Health System and in part by cooperative agreement U18HS016946 from the Agency for Healthcare Research and Quality.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

Previous Presentation: This study was presented at the 25th International Conference on Pharmacoepidemiology & Therapeutic Risk Management; August 18, 2009; Providence, Rhode Island.

Additional Contributions: Todd Barton, MD; Jeffrey Staab, MD, MS; and Robert Gross, MD, MSCE, University of Pennsylvania IRB Oversight Committee provided valuable suggestions and guidance; Jesse Chittams, MS; Asaf Hanish, MPH; Qing Liu, BS; and Catherine T. Williams-Smith, BS, Biostatistics Analysis Center of the Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania School of Medicine, created the analytic data set; Maggie Massary, MS, University of Pennsylvania Health System, helped develop the electronic alert and guided us through the elements of the electronic intervention and Sunrise data collection; and Anne Saint John, AAS, assisted with manuscript preparation.