0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Investigation | Health Care Reform

Efficacy of an Evidence-Based Clinical Decision Support in Primary Care Practices:  A Randomized Clinical Trial FREE

Thomas G. McGinn, MD, MPH1; Lauren McCullagh, MPH1; Joseph Kannry, MD2; Megan Knaus, MPH1; Anastasia Sofianou, MS2; Juan P. Wisnivesky, MD, DrPH2; Devin M. Mann, MD, MS3
[+] Author Affiliations
1Division of Internal Medicine, Department of Medicine, Hofstra North Shore–LIJ School of Medicine, Manhasset, New York
2Division of Internal Medicine, Department of Medicine, Mount Sinai School of Medicine, New York, New York
3Section of Preventive Medicine and Epidemiology, Department of Medicine, Boston University School of Medicine, Boston, Massachusetts
JAMA Intern Med. 2013;173(17):1584-1591. doi:10.1001/jamainternmed.2013.8980.
Text Size: A A A
Published online

Importance  There is consensus that incorporating clinical decision support into electronic health records will improve quality of care, contain costs, and reduce overtreatment, but this potential has yet to be demonstrated in clinical trials.

Objective  To assess the influence of a customized evidence-based clinical decision support tool on the management of respiratory tract infections and on the effectiveness of integrating evidence at the point of care.

Design, Setting, and Participants  In a randomized clinical trial, we implemented 2 well-validated integrated clinical prediction rules, namely, the Walsh rule for streptococcal pharyngitis and the Heckerling rule for pneumonia.

Interventions and Main Outcomes and Measures  The intervention group had access to the integrated clinical prediction rule tool and chose whether to complete risk score calculators, order medications, and generate progress notes to assist with complex decision making at the point of care.

Results  The intervention group completed the integrated clinical prediction rule tool in 57.5% of visits. Providers in the intervention group were significantly less likely to order antibiotics than the control group (age-adjusted relative risk, 0.74; 95% CI, 0.60-0.92). The absolute risk of the intervention was 9.2%, and the number needed to treat was 10.8. The intervention group was significantly less likely to order rapid streptococcal tests compared with the control group (relative risk, 0.75; 95% CI, 0.58-0.97; P = .03).

Conclusions and Relevance  The integrated clinical prediction rule process for integrating complex evidence-based clinical decision report tools is of relevant importance for national initiatives, such as Meaningful Use.

Trial Registration  clinicaltrials.gov Identifier: NCT01386047

Figures in this Article

Up to 30% of all medical care in the United States can be considered unnecessary.13 There is widespread consensus among health care experts, policymakers, and consumers that health information technology has great potential to help reduce this overtreatment, particularly electronic health records (EHRs) with clinical decision support (CDS). To this end, the federal government has distributed more than $6 billion in incentive payments to health care professionals and hospitals to implement CDS in conjunction with EHRs to improve performance on high-priority health conditions.4 Nevertheless, this potential has not been realized, highlighting the need to improve provider (including attending physicians, residents, fellows, and nurse practitioners) adoption and use of CDS.58

Most forms of CDS involve input of 1 or 2 simple data points. For example, influenza vaccination reminders embedded in EHRs use the time of the year to trigger providers to act, and colon cancer screening reminders use patient age as the trigger. More complex forms of decision support that are customized and pull data from various data fields hold great potential to reduce waste. Successful examples of CDS have reduced prescribing of brand-name antibiotics,9 improved lipid management in patients receiving renal transplants,10 enhanced compliance with guidelines for treating human immunodeficiency virus,1113 and decreased ordering of tests when costs were displayed,14 in addition to age-specific alerts that have minimized inappropriate prescribing in the elderly.1520 Yet, large-scale adoption rates continue to be a challenge.

Integrated clinical prediction rules (CPRs) represent a form of CDS used at the point of care. They are well-validated forms of evidence-based tools that help providers accurately estimate probabilities and have demonstrated in research trials the ability to reduce waste, while simultaneously enhancing quality of care.21 However, to integrate CPRs into an EHR requires data from various components of the history, physical examination, and basic laboratory test results instantaneously, making them difficult to include in the EHR. Prior attempts to integrate such complex forms of evidence at the point of care to influence frontline decision making have failed because of a lack of adoption by providers.22,23

We chose to examine the effect of new methods for evidence integration and to assess the influence this integration had on the management of upper respiratory tract infections. For this study, we examined 2 underused CPRs, the rule by Heckerling et al24 for pneumonia and the rule by Walsh et al25 for streptococcal pharyngitis (data not shown). Although there are other scores developed for these diseases, the Walsh and Heckerling rules have been validated in various settings and are considered the standard of care, yet they are not uniformly applied in clinical practice.2527

Our study sought to improve provider adoption of this complex CDS at the point of care in real time by seamlessly blending CPRs into provider work flow. Phase 1 of the study evaluated novel usability techniques that draw on low-cost technology and real ambulatory clinical settings and created a tool that includes CPRs in a widely used commercial EHR platform. This first phase of the development and testing is described in previous publications.28,29

For phase 2, we conducted a randomized clinical trial to study (1) the effect of the CPR tool on provider diagnostic and treatment patterns and (2) the adoption of the tool for each of 2 target conditions. Primary care providers were randomized to the new tool vs usual care.

The CPR study consisted of the following 2 phases: (1) the development and usability testing of the CPR tool and (2) a randomized clinical trial in which we evaluated the effectiveness of the CPR for reducing antibiotic ordering rates among enrolled providers, as well as the uptake of the tool. Phase 1 and the usability testing that were central to the development are described in greater detail elsewhere.28

Setting

The institutional review board at Mount Sinai School of Medicine approved the study protocols. The CPR randomized clinical trial was conducted within 2 large urban ambulatory primary care practices at Mount Sinai Medical Center in New York City with a mix of providers, including attending physicians, residents, fellows, and nurse practitioners. The patient population of the clinical practice was racially/ethnically diverse, with almost 56% of patients self-identifying as Hispanic, 35% as African American, 7% as white, and 2% as other races/ethnicities (data not shown).

Exclusion and Inclusion Criteria

Attendings, residents, fellows, and nurse practitioners working in the outpatient primary care clinic at the medical center were eligible to participate in the study. Providers were not enrolled if they were part of the study team.

Study Design

Primary care providers were randomized using a random number generator to a control group (usual care) or an intervention group (Figure 1). The intervention group participated in an in-person 1-hour training consisting of an overview of CPRs and the evidence supporting the Walsh and Heckerling CPRs in particular, as well as the study protocols, demonstration of the tool in the EHR, and a video of a simulated patient encounter in which the tool is used. Providers randomized to the control group received background information on the Walsh and Heckerling CPRs (Figure 2).

Place holder to copy figure label and caption
Figure 1.
Study Activities for Health Care Providers in the Control and Intervention Groups

EHR indicates electronic health record.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.
Integrated Clinical Prediction Rule (iCPR)

iCPR tool workflow in the electronic medical health record.

Graphic Jump Location

From November 1, 2010, through October 31, 2011, we activated the CPR tool for intervention providers by modifying their EHR user access settings. The CPR tool appeared on the provider’s screen during a clinical encounter when the provider entered 1 or more keywords in the fields for chief complaint, diagnosis, or orders (data not shown). Criteria for triggering included chief complaints and diagnoses associated with pharyngitis or pneumonia or a diagnosis and test order combination.28 Chief complaints relevant to pharyngitis included sore throat, throat discomfort, and streptococcal pharyngitis; for pneumonia, they included possible pneumonia and “chest hurts when breathing.”28 Providers were invited to complete an CPR risk score calculator and were given management recommendations based on the score. The recommendation for patients with a total score of –1 (unlikely to have streptococcal pharyngitis) was to provide supportive care. The recommendation for patients with a total score of 0 or 1 (intermediate likelihood of streptococcal pharyngitis) was to obtain a throat swab or symptom resolution before deciding about antibiotic therapy. The recommendation for patients with a total score of 2 or higher (high likelihood of streptococcal infection) was to start empirical antibiotic therapy, with no culture necessary. Similar recommendations, but specific to pneumonia, were given after completing the pneumonia risk score calculator. Patient encounters with the study providers were monitored, and data were collected from the EHR for the intervention period.

Outcomes
Primary Outcomes

We assessed changes in provider patterns of ordering antibiotics as the primary outcome. Antibiotic orders were evaluated by examining EHR data related to patient encounters in which the tool was triggered. The study team derived a list of antibiotics for pharyngitis and pneumonia used in the analysis, with consultation from other primary care physicians. We measured the frequency, rates, and type of antibiotics prescribed after using the CPR risk score calculator and bundled order sets. Additional outcomes measured were how often chest radiographs, rapid streptococcal tests, and pharyngitis throat cultures were ordered during eligible patient visits. We recorded the occurrence of 1 or more emergency department and outpatient clinic visits and antibiotics ordered 2 weeks after an CPR visit as a proxy for the appropriateness of care as a result of using the tool.

Process Outcomes

Step-by-step analysis of provider use of the CPR tool has been critical to understanding its uptake and to shaping its ongoing development. Therefore, we also assessed the uptake and use of the tool by measuring the rate of eligible visits in which (1) the provider opened the tool after it was triggered (based on the criteria in the Study Design subsection of the Methods section), (2) the provider completed the risk score calculator embedded in the tool, (3) the provider viewed the bundled order set associated with the score generated, and (4) the provider signed the associated bundled order set.

Provider demographics such as sex, age, and years of practice were collected in a self-reported survey and analyzed. Patient characteristics were selected for analysis on the basis of their potential to influence provider decisions. These included age, sex, smoking status, any allergies to antibiotics (penicillins, cephalosporins, quinolones, and macrolides), and relevant comorbidities (asthma, chronic obstructive pulmonary disease, diabetes mellitus, and congestive heart failure).

Statistical Analysis

Differences in the characteristics of patients seen by providers in the intervention and control groups were measured using t test and χ2 test as appropriate. Provider characteristics were compared using similar methods (data not shown). Process, primary, and secondary outcomes were compared for all intervention and control visits and were stratified by pneumonia or pharyngitis CPR tool. Absolute risk (risk difference) and relative risk (RR) ratios of antibiotic prescribing in the intervention group vs the control group were compared using a generalized estimating equation model (logit link) with robust standard errors to account for clustering of patient visits (encounters) within providers and subsequent correlated outcomes. Because the median age of patients differed in the intervention and control groups, we performed a secondary analysis adding age as a covariate in our final age-adjusted models. The same approach was applied for our secondary outcomes of how often chest radiographs, rapid streptococcal tests, and pharyngitis throat cultures were ordered. Similar generalized estimating equation analysis compared rates of antibiotic prescribing and secondary outcomes in the intervention group vs the control group 2 weeks after a visit. Analyses were conducted with statistical software (SAS, version 9.2; SAS Institute, Inc) using 2-sided P values.

From November 1, 2010, through October 31, 2011, a total of 168 primary care providers provided informed consent and were enrolled in the study. There were no statistically significant differences in demographics between providers in the intervention and control groups (data not shown).

In total, more than 40 003 patient visits occurred during the study period. The median ages of the patients seen at the study visits were statistically different for the intervention and control groups (P = .001). No statistically significant differences in sex or race/ethnicity were observed between the 2 groups. Similarly, no statistically significant differences were found in the patient health data on smoking status, relevant comorbidities, or relevant allergies to the antibiotics on the CPR list or other medication allergies (Table 1).

Table Graphic Jump LocationTable 1.  Patient Characteristics by Provider Randomization Status
Primary Outcomes

We found statistically significant differences in antibiotic prescriptions, rapid streptococcal test orders, and pharyngitis throat cultures between the control and intervention groups (Table 2). Providers in the intervention group were significantly less likely to order antibiotics (age-adjusted RR, 0.74; 95% CI, 0.60-0.92; P = .008). The absolute risk of the intervention was 9.2%, and the number needed to treat was 10.8.

Table Graphic Jump LocationTable 2.  Antibiotic and Test Orders by Provider Randomization Status

When stratified by pharyngitis and pneumonia, the RR of ordering remained significant for pneumonia (age-adjusted RR, 0.79; 95% CI, 0.64-0.98; P = .03). The intervention group was significantly less likely to order rapid streptococcal tests in pharyngitis encounters (age-adjusted RR, 0.75; 95% CI, 0.58-0.97; P = .03). No statistically significant differences were observed in chest radiograph ordering among patients with possible pneumonia (age-adjusted RR, 0.98; 95% CI, 0.60-1.62; P = .95) or in pharyngitis throat culture orders (age-adjusted RR, 0.54; 95% CI, 0.18-1.64; P = .28).

The RR ratio for ordering quinolones (broad-spectrum antibacterial drugs) in the intervention group compared with the control group was 0.50 (95% CI, 0.29-0.88; P = .02) (Table 3). No significant differences were observed between the other antibiotics, including penicillins, cephalosporins, and macrolides.

Table Graphic Jump LocationTable 3.  Antibiotic Orders by Provider Randomization Status

No significant differences were found between the intervention and control groups in the proportions of visits resulting in a patient returning to the emergency department (P > .99) or outpatient clinic (P = .10) for follow-up treatment (Table 4). Likewise, no differences were observed in rates of antibiotic orders between the intervention and control groups 2 weeks after CPR visits (P = .45).

Table Graphic Jump LocationTable 4.  Orders and Clinical Visits 2 Weeks After the Integrated Clinical Prediction Rule Visit
Process Outcomes

The CPR tool was triggered 984 times. Of these, 398 patients were seen by control providers and 586 by intervention providers (Table 4). Overall, the tool had high adoption rates in the intervention group, with 62.8% having opened the tool and 57.5% of providers accepting the tool. Among the encounters seen by intervention providers, the pharyngitis tool was more commonly used than the pneumonia tool (Table 5). In total, 74.3% of 374 pharyngitis encounters resulted in an initial acceptance of the triggered CPR tool compared with 42.5% of 212 pneumonia encounters (RR, 0.65; 95% CI, 0.54-0.78; P < .001). In addition, providers were significantly more likely to complete later steps for the pharyngitis tool than for the pneumonia tool. Examples include completion of the risk score calculator (RR, 0.69; 95% CI, 0.57-0.83) and completion of bundled order sets (RR, 0.60; 0.46-0.77) (P < .001 for both). For most patient visits, these scores categorized patients as being at low risk for pharyngitis (55.4%) and pneumonia (96.6%).

Table Graphic Jump LocationTable 5.  Integrated Clinical Prediction Rule Tool Use in the Intervention Group

Providers were more likely to open bundled order sets for pharyngitis visits than for pneumonia visits. For pharyngitis visits, 63.1% of bundled sets were opened, and 50.5% were completed. For pneumonia visits, 39.2% of bundled sets were opened, and 26.9% were completed.

The CPR is a form of complex CDS that holds great promise for reducing overtreatment and testing. Although numerous well-validated CPRs exist, few studies3033 have reported significant adoption of CPRs, and they have found little effect on provider behavior.

The CPR study brought CDS to the point of care in real time by embedding 2 well-validated CPRs in an ambulatory EHR. The intervention group had a high overall adoption rate (62.8%); orders for antibiotic therapy were significantly reduced (age-adjusted RR, 0.74; 95% CI, 0.60-0.92; P = .008), as were orders for rapid streptococcal tests (age-adjusted RR, 0.75; 95% CI, 0.58-0.97; P = .03). The antibiotics that were ordered were more appropriate (ie, narrow spectrum). When comparing pneumonia vs pharyngitis antibiotic ordering, a significant difference was observed only for pneumonia-related visits. The reason is unclear but is attributed to the CPR tool design or to the integration into provider work flow, and further investigation is required.

Prior investigations of CDS achieved adoption rates between 10% and 20%.34 In addition to our high overall adoption rate of 62.8%, we found that 57.5% of all relevant encounters resulted in the provider completing the risk score calculator and generating a risk score assessment. This exceeds the adoption rates for another CDS intervention that attempted to reduce antibiotic prescriptions for upper respiratory tract infections but resulted in a reduction of only 16%.35 We believe our results indicate that providers may have perceived the tool as being helpful with the clinical diagnosis of pharyngitis and pneumonia, enhancing clinical work flow and improving patient care, which are hallmarks of CDS interventions that are well received by physicians.23,36,37 We speculate that the high adoption rates were a result of the comprehensive user-centered development process we created for this study, including pilot testing with providers, usability testing, focused user training on CDS (rarely provided in other studies), and collaboration with a multidisciplinary team that included experts in usability and informatics and CDS specialists.29,38

The CPR tool does not guarantee a reduction in antibiotic use after implementation. If overuse of antibiotics was an issue at baseline, the tool should reduce use. Although some clinicians may have been triggered to order antibiotics, the overall rate of ordering antibiotics was reduced. This study was unusual in that it specifically addressed many of the challenges of integrated decision support identified in earlier studies (cited in the introduction). These included ensuring that the CDS tool used only off-the-shelf functionality from the vendor, minimizing the frequency of alerts and carefully balancing alerts in relation to work flow.28 In addition, the study team conducted usability testing and focus groups with providers, incorporated a supportive training program, and provided personalized on-call troubleshooting during the first weeks of implementation. Finally, this study was performed in the setting of a widely available commercial EHR rather than a proprietary system available to only a single institution. Resources to maintain the tool and implementation resources consisted of a part-time EHR programmer to assist with the design and troubleshoot the EHR program, significantly enhancing the potential for replication.39

Our study had several limitations. It was performed at 2 primary care practice sites within a large academic health center with significant numbers of trainees. Although off-the-shelf technology was used to facilitate exportation of the method to other sites, further dissemination studies are needed to identify the effect of clinical setting on CPR effectiveness. In addition, our study took place in an inner-city academic setting; therefore, its use in more diverse settings may offer different insight.

In conclusion, we demonstrated high adoption rates (>50%) of a complex decision support tool that integrates evidence-based diagnostic tools at the point of care. Successful integration and adoption were related to its user-centered design and implementation process, which resulted in a reduction in antibiotic ordering and in point-of-care testing and throat cultures. High adoption rates of CDS interventions that lead to changes in outcomes need to be further researched. Our study showed one such approach to successful CDS interventions in a commercial EHR.

Corresponding Author: Thomas G. McGinn, MD, MPH, Division of Internal Medicine, Department of Medicine, Hofstra North Shore–Long Island Jewish School of Medicine, 300 Community Dr, Manhasset, NY 11030 (tmcginn@nshs.edu)

Published Online: July 29, 2013. doi:10.1001/jamainternmed.2013.8980.

Author Contributions:Study concept and design: McGinn, Kannry, Wisnivesky, Mann.

Acquisition of data: Kannry, Mann.

Analysis and interpretation of data: McGinn, McCullagh, Knaus, Sofianou, Wisnivesky, Mann.

Drafting of the manuscript: McGinn, McCullagh, Kannry, Knaus, Sofianou, Mann.

Critical revision of the manuscript for important intellectual content: McGinn, Kannry, Wisnivesky, Mann.

Statistical analysis: Sofianou, Wisnivesky, Mann.

Obtained funding: McGinn.

Administrative, technical, and material support: McCullagh, Kannry, Knaus.

Study supervision: McGinn, Kannry.

Conflict of Interest Disclosures: Dr Wisnivesky is a member of the research board of EHE International; has received consulting fees from Merck, IMS Health, and UBS; and is the recipient of a research grant from GlaxoSmithKline.

Funding/Support: This study was supported by grant 5 R18 HS 018491 from the Agency for Health and Quality Research (Dr McGinn).

Role of the Sponsor: The Agency for Health and Quality Research participated in the design and conduct of the study and in the collection, management, analysis, and interpretation of the data.

Additional Contributions: Jacqui Arciniega, MPH, Diego Chiluisa, BA, Dan Edonyabo, MS, Alice Li, MD, and Lucas Romero, BA, assisted with the study.

Fisher  ES, Wennberg  DE, Stukel  TA, Gottlieb  DJ, Lucas  FL, Pinder  EL.  The implications of regional variations in Medicare spending, part 2: health outcomes and satisfaction with care. Ann Intern Med. 2003;138(4):288-298.
PubMed   |  Link to Article
Martin  AB, Lassman  D, Washington  B, Catlin  A; National Health Expenditure Accounts Team.  Growth in US health spending remained slow in 2010; health share of gross domestic product was unchanged from 2009. Health Aff (Millwood). 2012;31(1):208-219.
PubMed   |  Link to Article
Berwick  DM, Hackbarth  AD.  Eliminating waste in US health care. JAMA. 2012;307(14):1513-1516.
PubMed   |  Link to Article
Centers for Medicare & Medicaid Services. EHR Incentive Program: data and reports. 2012. http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/DataAndReports.html. Accessed July 27, 2012.
Blumenthal  D.  Launching HITECH. N Engl J Med. 2010;362(5):382-385.
PubMed   |  Link to Article
Linder  JA, Chan  JC, Bates  DW.  Evaluation and treatment of pharyngitis in primary care practice: the difference between guidelines is largely academic. Arch Intern Med. 2006;166(13):1374-1379.
PubMed   |  Link to Article
 Medicare and Medicaid programs: Electronic Health Record Incentive Program: stage 2: proposed rule. Fed Regist. 2012;77(45):13698-13827.
Federal Register. Electronic Health Record (EHR) Incentive Program (CMS-0033-F). http://www.federalregister.gov/regulations/0938-AP78/electronic-health-record-ehr-incentive-program-cms-0033-f-. Accessed September 5, 2010.
Bernstein  SL, Whitaker  D, Winograd  J, Brennan  JA.  An electronic chart prompt to decrease proprietary antibiotic prescription to self-pay patients. Acad Emerg Med. 2005;12(3):225-231.
PubMed   |  Link to Article
Garthwaite  EA, Will  EJ, Bartlett  C, Richardson  D, Newstead  CG.  Patient-specific prompts in the cholesterol management of renal transplant outpatients: results and analysis of underperformance. Transplantation. 2004;78(7):1042-1047.
PubMed   |  Link to Article
Safran  C, Rind  DM, Davis  RB,  et al.  Guidelines for management of HIV infection with computer-based patient’s record. Lancet. 1995;346(8971):341-346.
PubMed   |  Link to Article
Safran  C, Rind  DM, Davis  RM,  et al.  An electronic medical record that helps care for patients with HIV infection. Proc Annu Symp Comput Appl Med Care. 1993:224-228.
PubMed
Safran  C, Rind  DM, Sands  DZ, Davis  RB, Wald  J, Slack  WV.  Development of a knowledge-based electronic patient record. MD Comput. 1996;13(1):46-54, 63.
PubMed
Tierney  WM, Miller  ME, McDonald  CJ.  The effect on test ordering of informing physicians of the charges for outpatient diagnostic tests. N Engl J Med. 1990;322(21):1499-1504.
PubMed   |  Link to Article
Simon  SR, Smith  DH, Feldstein  AC,  et al.  Computerized prescribing alerts and group academic detailing to reduce the use of potentially inappropriate medications in older people. J Am Geriatr Soc. 2006;54(6):963-968.
PubMed   |  Link to Article
Shah  NR, Seger  AC, Seger  DL,  et al.  Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc. 2006;13(1):5-11.
PubMed   |  Link to Article
Tamblyn  R, Huang  A, Perreault  R,  et al.  The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ. 2003;169(6):549-556.
PubMed
Gaikwad  R, Sketris  I, Shepherd  M, Duffy  J.  Evaluation of accuracy of drug interaction alerts triggered by two electronic medical record systems in primary healthcare. Health Informatics J. 2007;13(3):163-177.
PubMed   |  Link to Article
Smith  DH, Perrin  N, Feldstein  A,  et al.  The impact of prescribing safety alerts for elderly persons in an electronic medical record: an interrupted time series evaluation. Arch Intern Med. 2006;166(10):1098-1104.
PubMed   |  Link to Article
Seidling  HM, Schmitt  SP, Bruckner  T,  et al.  Patient-specific electronic decision support reduces prescription of excessive doses. Qual Saf Health Care. 2010;19(5):e15. http://qualitysafety.bmj.com/content/19/5/e15.long. Accessed June 4, 2013.
Link to Article
McGinn  TG, Guyatt  GH, Wyer  PC, Naylor  CD, Stiell  IG, Richardson  WS; Evidence-Based Medicine Working Group.  Users’ guides to the medical literature, XXII: how to use articles about clinical decision rules. JAMA. 2000;284(1):79-84.
PubMed   |  Link to Article
Garg  AX, Adhikari  NK, McDonald  H,  et al.  Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223-1238.
PubMed   |  Link to Article
Krall  MA, Sittig  DF.  Clinician’s assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proc AMIA Symp. 2002:400-404.
PubMed
Heckerling  PS, Tape  TG, Wigton  RS,  et al.  Clinical prediction rule for pulmonary infiltrates. Ann Intern Med. 1990;113(9):664-670.
PubMed   |  Link to Article
Walsh  BT, Bookheim  WW, Johnson  RC, Tompkins  RK.  Recognition of streptococcal pharyngitis in adults. Arch Intern Med. 1975;135(11):1493-1497.
PubMed   |  Link to Article
Centor  RM, Witherspoon  JM, Dalton  HP, Brody  CE, Link  K.  The diagnosis of strep throat in adults in the emergency room. Med Decis Making. 1981;1(3):239-246.
PubMed   |  Link to Article
McGinn  TG, Deluca  J, Ahlawat  SK, Mobo  BH  Jr, Wisnivesky  JP.  Validation and modification of streptococcal pharyngitis clinical prediction rules. Mayo Clin Proc. 2003;78(3):289-293.
PubMed   |  Link to Article
Mann  DM, Kannry  JL, Edonyabo  D,  et al.  Rationale, design, and implementation protocol of an electronic health record integrated clinical prediction rule (iCPR) randomized trial in primary care. Implement Sci. 2011;6(1):e109. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3184082/. Accessed June 3, 2013.
PubMed   |  Link to Article
Li  AC, Kannry  JL, Kushniruk  A,  et al.  Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support. Int J Med Inform. 2012;81(11):761-772.
PubMed   |  Link to Article
Penaloza  A, Mélot  C, Dochy  E,  et al.  Assessment of pretest probability of pulmonary embolism in the emergency department by physicians in training using the Wells model. Thromb Res. 2007;120(2):173-179.
PubMed   |  Link to Article
Drescher  FS, Chandrika  S, Weir  ID,  et al.  Effectiveness and acceptability of a computerized decision support system using modified Wells criteria for evaluation of suspected pulmonary embolism. Ann Emerg Med. 2011;57(6):613-621.
PubMed   |  Link to Article
Kucher  N, Koo  S, Quiroz  R,  et al.  Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med. 2005;352(10):969-977.
PubMed   |  Link to Article
Penaloza  A, Laureys  M, Wautrecht  JC, Lheureux  P, Motte  S.  Accuracy and safety of pretest probability assessment of deep vein thrombosis by physicians in training using the explicit Wells clinical model. J Thromb Haemost. 2006;4(1):278-281.
PubMed   |  Link to Article
Bright  TJ, Wong  A, Dhurjati  R,  et al.  Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157(1):29-43.
PubMed   |  Link to Article
Mainous  AG  III, Lambourne  CA, Nietert  PJ.  Impact of a clinical decision support system on antibiotic prescribing for acute respiratory infections in primary care: quasi-experimental trial. J Am Med Inform Assoc. 2013;20(2):317-324.
PubMed   |  Link to Article
Magrabi  F, Westbrook  JI, Coiera  EW.  What factors are associated with the integration of evidence retrieval technology into routine general practice settings? Int J Med Inform. 2007;76(10):701-709.
PubMed   |  Link to Article
Weiner  M, Gress  T, Thiemann  DR,  et al.  Contrasting views of physicians and nurses about an inpatient computer-based provider order-entry system. J Am Med Inform Assoc. 1999;6(3):234-244.
PubMed   |  Link to Article
Patterson  ES, Doebbeling  BN, Fung  CH, Militello  L, Anders  S, Asch  SM.  Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods. J Biomed Inform. 2005;38(3):189-199.
PubMed   |  Link to Article
Wright  A, Sittig  DF, Ash  JS, Sharma  S, Pang  JE, Middleton  B.  Clinical decision support capabilities of commercially-available clinical information systems. J Am Med Inform Assoc.2009;16(5):637-644.
Link to Article

Figures

Place holder to copy figure label and caption
Figure 1.
Study Activities for Health Care Providers in the Control and Intervention Groups

EHR indicates electronic health record.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.
Integrated Clinical Prediction Rule (iCPR)

iCPR tool workflow in the electronic medical health record.

Graphic Jump Location

Tables

Table Graphic Jump LocationTable 1.  Patient Characteristics by Provider Randomization Status
Table Graphic Jump LocationTable 2.  Antibiotic and Test Orders by Provider Randomization Status
Table Graphic Jump LocationTable 3.  Antibiotic Orders by Provider Randomization Status
Table Graphic Jump LocationTable 4.  Orders and Clinical Visits 2 Weeks After the Integrated Clinical Prediction Rule Visit
Table Graphic Jump LocationTable 5.  Integrated Clinical Prediction Rule Tool Use in the Intervention Group

References

Fisher  ES, Wennberg  DE, Stukel  TA, Gottlieb  DJ, Lucas  FL, Pinder  EL.  The implications of regional variations in Medicare spending, part 2: health outcomes and satisfaction with care. Ann Intern Med. 2003;138(4):288-298.
PubMed   |  Link to Article
Martin  AB, Lassman  D, Washington  B, Catlin  A; National Health Expenditure Accounts Team.  Growth in US health spending remained slow in 2010; health share of gross domestic product was unchanged from 2009. Health Aff (Millwood). 2012;31(1):208-219.
PubMed   |  Link to Article
Berwick  DM, Hackbarth  AD.  Eliminating waste in US health care. JAMA. 2012;307(14):1513-1516.
PubMed   |  Link to Article
Centers for Medicare & Medicaid Services. EHR Incentive Program: data and reports. 2012. http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/DataAndReports.html. Accessed July 27, 2012.
Blumenthal  D.  Launching HITECH. N Engl J Med. 2010;362(5):382-385.
PubMed   |  Link to Article
Linder  JA, Chan  JC, Bates  DW.  Evaluation and treatment of pharyngitis in primary care practice: the difference between guidelines is largely academic. Arch Intern Med. 2006;166(13):1374-1379.
PubMed   |  Link to Article
 Medicare and Medicaid programs: Electronic Health Record Incentive Program: stage 2: proposed rule. Fed Regist. 2012;77(45):13698-13827.
Federal Register. Electronic Health Record (EHR) Incentive Program (CMS-0033-F). http://www.federalregister.gov/regulations/0938-AP78/electronic-health-record-ehr-incentive-program-cms-0033-f-. Accessed September 5, 2010.
Bernstein  SL, Whitaker  D, Winograd  J, Brennan  JA.  An electronic chart prompt to decrease proprietary antibiotic prescription to self-pay patients. Acad Emerg Med. 2005;12(3):225-231.
PubMed   |  Link to Article
Garthwaite  EA, Will  EJ, Bartlett  C, Richardson  D, Newstead  CG.  Patient-specific prompts in the cholesterol management of renal transplant outpatients: results and analysis of underperformance. Transplantation. 2004;78(7):1042-1047.
PubMed   |  Link to Article
Safran  C, Rind  DM, Davis  RB,  et al.  Guidelines for management of HIV infection with computer-based patient’s record. Lancet. 1995;346(8971):341-346.
PubMed   |  Link to Article
Safran  C, Rind  DM, Davis  RM,  et al.  An electronic medical record that helps care for patients with HIV infection. Proc Annu Symp Comput Appl Med Care. 1993:224-228.
PubMed
Safran  C, Rind  DM, Sands  DZ, Davis  RB, Wald  J, Slack  WV.  Development of a knowledge-based electronic patient record. MD Comput. 1996;13(1):46-54, 63.
PubMed
Tierney  WM, Miller  ME, McDonald  CJ.  The effect on test ordering of informing physicians of the charges for outpatient diagnostic tests. N Engl J Med. 1990;322(21):1499-1504.
PubMed   |  Link to Article
Simon  SR, Smith  DH, Feldstein  AC,  et al.  Computerized prescribing alerts and group academic detailing to reduce the use of potentially inappropriate medications in older people. J Am Geriatr Soc. 2006;54(6):963-968.
PubMed   |  Link to Article
Shah  NR, Seger  AC, Seger  DL,  et al.  Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc. 2006;13(1):5-11.
PubMed   |  Link to Article
Tamblyn  R, Huang  A, Perreault  R,  et al.  The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ. 2003;169(6):549-556.
PubMed
Gaikwad  R, Sketris  I, Shepherd  M, Duffy  J.  Evaluation of accuracy of drug interaction alerts triggered by two electronic medical record systems in primary healthcare. Health Informatics J. 2007;13(3):163-177.
PubMed   |  Link to Article
Smith  DH, Perrin  N, Feldstein  A,  et al.  The impact of prescribing safety alerts for elderly persons in an electronic medical record: an interrupted time series evaluation. Arch Intern Med. 2006;166(10):1098-1104.
PubMed   |  Link to Article
Seidling  HM, Schmitt  SP, Bruckner  T,  et al.  Patient-specific electronic decision support reduces prescription of excessive doses. Qual Saf Health Care. 2010;19(5):e15. http://qualitysafety.bmj.com/content/19/5/e15.long. Accessed June 4, 2013.
Link to Article
McGinn  TG, Guyatt  GH, Wyer  PC, Naylor  CD, Stiell  IG, Richardson  WS; Evidence-Based Medicine Working Group.  Users’ guides to the medical literature, XXII: how to use articles about clinical decision rules. JAMA. 2000;284(1):79-84.
PubMed   |  Link to Article
Garg  AX, Adhikari  NK, McDonald  H,  et al.  Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223-1238.
PubMed   |  Link to Article
Krall  MA, Sittig  DF.  Clinician’s assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proc AMIA Symp. 2002:400-404.
PubMed
Heckerling  PS, Tape  TG, Wigton  RS,  et al.  Clinical prediction rule for pulmonary infiltrates. Ann Intern Med. 1990;113(9):664-670.
PubMed   |  Link to Article
Walsh  BT, Bookheim  WW, Johnson  RC, Tompkins  RK.  Recognition of streptococcal pharyngitis in adults. Arch Intern Med. 1975;135(11):1493-1497.
PubMed   |  Link to Article
Centor  RM, Witherspoon  JM, Dalton  HP, Brody  CE, Link  K.  The diagnosis of strep throat in adults in the emergency room. Med Decis Making. 1981;1(3):239-246.
PubMed   |  Link to Article
McGinn  TG, Deluca  J, Ahlawat  SK, Mobo  BH  Jr, Wisnivesky  JP.  Validation and modification of streptococcal pharyngitis clinical prediction rules. Mayo Clin Proc. 2003;78(3):289-293.
PubMed   |  Link to Article
Mann  DM, Kannry  JL, Edonyabo  D,  et al.  Rationale, design, and implementation protocol of an electronic health record integrated clinical prediction rule (iCPR) randomized trial in primary care. Implement Sci. 2011;6(1):e109. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3184082/. Accessed June 3, 2013.
PubMed   |  Link to Article
Li  AC, Kannry  JL, Kushniruk  A,  et al.  Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support. Int J Med Inform. 2012;81(11):761-772.
PubMed   |  Link to Article
Penaloza  A, Mélot  C, Dochy  E,  et al.  Assessment of pretest probability of pulmonary embolism in the emergency department by physicians in training using the Wells model. Thromb Res. 2007;120(2):173-179.
PubMed   |  Link to Article
Drescher  FS, Chandrika  S, Weir  ID,  et al.  Effectiveness and acceptability of a computerized decision support system using modified Wells criteria for evaluation of suspected pulmonary embolism. Ann Emerg Med. 2011;57(6):613-621.
PubMed   |  Link to Article
Kucher  N, Koo  S, Quiroz  R,  et al.  Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med. 2005;352(10):969-977.
PubMed   |  Link to Article
Penaloza  A, Laureys  M, Wautrecht  JC, Lheureux  P, Motte  S.  Accuracy and safety of pretest probability assessment of deep vein thrombosis by physicians in training using the explicit Wells clinical model. J Thromb Haemost. 2006;4(1):278-281.
PubMed   |  Link to Article
Bright  TJ, Wong  A, Dhurjati  R,  et al.  Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157(1):29-43.
PubMed   |  Link to Article
Mainous  AG  III, Lambourne  CA, Nietert  PJ.  Impact of a clinical decision support system on antibiotic prescribing for acute respiratory infections in primary care: quasi-experimental trial. J Am Med Inform Assoc. 2013;20(2):317-324.
PubMed   |  Link to Article
Magrabi  F, Westbrook  JI, Coiera  EW.  What factors are associated with the integration of evidence retrieval technology into routine general practice settings? Int J Med Inform. 2007;76(10):701-709.
PubMed   |  Link to Article
Weiner  M, Gress  T, Thiemann  DR,  et al.  Contrasting views of physicians and nurses about an inpatient computer-based provider order-entry system. J Am Med Inform Assoc. 1999;6(3):234-244.
PubMed   |  Link to Article
Patterson  ES, Doebbeling  BN, Fung  CH, Militello  L, Anders  S, Asch  SM.  Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods. J Biomed Inform. 2005;38(3):189-199.
PubMed   |  Link to Article
Wright  A, Sittig  DF, Ash  JS, Sharma  S, Pang  JE, Middleton  B.  Clinical decision support capabilities of commercially-available clinical information systems. J Am Med Inform Assoc.2009;16(5):637-644.
Link to Article

Correspondence

CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
Submit a Comment

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 2

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections
JAMAevidence.com

Users' Guides to the Medical Literature
Clinical Resolution

Users' Guides to the Medical Literature
Clinical Scenario