0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.196.24.103. Please contact the publisher to request reinstatement.
Original Investigation | Health Care Reform

Timely Follow-up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting:  Are Electronic Medical Records Achieving Their Potential? FREE

Hardeep Singh, MD, MPH; Eric J. Thomas, MD, MPH; Shrinidi Mani, BA; Dean Sittig, PhD; Harvinder Arora, MD, MPH; Donna Espadas, BS; Myrna M. Khan, PhD, MBA; Laura A. Petersen, MD, MPH
[+] Author Affiliations

Author Affiliations: Department of Veterans Affairs Health Services Research & Development Service, Center of Excellence, and The Center of Inquiry to Improve Outpatient Safety Through Effective Electronic Communication, Michael E. DeBakey Veterans Affairs Medical Center, and the Section of Health Services Research, Department of Medicine, Baylor College of Medicine, Houston, Texas (Drs Singh, Khan, and Petersen, and Mss Mani and Espadas); University of Texas at Houston–Memorial Hermann Center for Healthcare Quality and Safety, Division of General Medicine, Department of Medicine, University of Texas Medical School at Houston (Dr Thomas); University of Texas School of Health Information Sciences and the University of Texas–Memorial Hermann Center for Healthcare Quality & Safety, Houston (Dr Sittig); and Department of Medicine, Baylor College of Medicine, Houston (Dr Arora).


Arch Intern Med. 2009;169(17):1578-1586. doi:10.1001/archinternmed.2009.263.
Text Size: A A A
Published online

Background  Given the fragmentation of outpatient care, timely follow-up of abnormal diagnostic imaging results remains a challenge. We hypothesized that an electronic medical record (EMR) that facilitates the transmission and availability of critical imaging results through either automated notification (alerting) or direct access to the primary report would eliminate this problem.

Methods  We studied critical imaging alert notifications in the outpatient setting of a tertiary care Department of Veterans Affairs facility from November 2007 to June 2008. Tracking software determined whether the alert was acknowledged (ie, health care practitioner/provider [HCP] opened the message for viewing) within 2 weeks of transmission; acknowledged alerts were considered read. We reviewed medical records and contacted HCPs to determine timely follow-up actions (eg, ordering a follow-up test or consultation) within 4 weeks of transmission. Multivariable logistic regression models accounting for clustering effect by HCPs analyzed predictors for 2 outcomes: lack of acknowledgment and lack of timely follow-up.

Results  Of 123 638 studies (including radiographs, computed tomographic scans, ultrasonograms, magnetic resonance images, and mammograms), 1196 images (0.97%) generated alerts; 217 (18.1%) of these were unacknowledged. Alerts had a higher risk of being unacknowledged when the ordering HCPs were trainees (odds ratio [OR], 5.58; 95% confidence interval [CI], 2.86-10.89) and when dual-alert (>1 HCP alerted) as opposed to single-alert communication was used (OR, 2.02; 95% CI, 1.22-3.36). Timely follow-up was lacking in 92 (7.7% of all alerts) and was similar for acknowledged and unacknowledged alerts (7.3% vs 9.7%; P = .22). Risk for lack of timely follow-up was higher with dual-alert communication (OR, 1.99; 95% CI, 1.06-3.48) but lower when additional verbal communication was used by the radiologist (OR, 0.12; 95% CI, 0.04-0.38). Nearly all abnormal results lacking timely follow-up at 4 weeks were eventually found to have measurable clinical impact in terms of further diagnostic testing or treatment.

Conclusions  Critical imaging results may not receive timely follow-up actions even when HCPs receive and read results in an advanced, integrated electronic medical record system. A multidisciplinary approach is needed to improve patient safety in this area.

Figures in this Article

Communication breakdown is consistently identified as a preventable factor in studies of adverse events16 and a significant contributor to outpatient diagnostic errors from a lack of follow-up of abnormal test results.712 The volume of outpatient care and nature of high-risk transitions between health care practitioners/providers (HCPs), settings, and systems of care makes timely communication particularly challenging.13 For example, a patient referred for diagnostic workup for respiratory symptoms by a primary care provider (PCP) may undergo several laboratory and imaging tests and a pulmonary consultation. Any abnormal findings such as a possible lung mass would need to be communicated rapidly and effectively to the treating HCPs to ensure adequate follow-up.

Electronic communication using alerts (computerized notifications of critical information such as abnormal diagnostic test results) can facilitate transmission and potentially a response and follow-up action by the ordering HCP—an advantage over paper-based reporting.14 For instance, the electronic medical record (EMR) used by the Department of Veterans Affairs (VA) mostly relies on a notification system (the “View Alert” window) to alert clinicians about critical test results, whereas only in selected life-threatening cases does the radiologist verbally communicate the abnormality to the HCP.14 However, effective communication involves more than just information transfer—it requires a response from the recipient, such as taking follow-up action and acknowledging receipt of the information to the sender.15 Previous studies raise concern that HCPs may not read all their alerts.14,1619 Data on outpatient drug-related alerts have shown that HCPs ignore alerts because of information overload,2023 often resulting in failure to act on important as well as clinically unimportant alerts.20 While no data exist to substantiate a similar problem for diagnostic test result alerts, HCPs often receive several hundred normal diagnostic test results and many abnormal alerts every week, suggesting that a similar problem could exist.24

Given that research in this area has thus far focused on critical results that either fail to reach the ordering HCPs or were somehow “missed” by HCPs after the delivery of information, we hypothesized that an EMR that facilitates the transmission and availability of critical imaging results through either automated notification or direct access of primary report would eliminate this problem. Because there is limited knowledge about why test results get lost to follow-up in EMR systems, we also determined predictors of communication breakdowns (ie, not reading or taking follow-up action) to help us gain insight about potential interventions to improve safety related to the follow-up of abnormal test results.13,14

We identified all alerts for critically abnormal radiographs, computerized tomographic (CT) scans, magnetic resonance images (MRIs), mammograms, and ultrasonograms transmitted electronically in the multispecialty ambulatory clinic of the Michael E. DeBakey VA Medical Center (MEDVAMC) and its 5 satellite clinics from November 2007 to June 2008. The local institutional review boards approved the study.

To confirm that HCPs actually received the alert, we queried the Alert Tracking File, the same repository used by the EMR to populate the HCP's View Alert window. We excluded all alerts related to imaging studies performed on inpatients. Being their only “inbox” for all types of notifications, HCPs in the VA are highly dependent on the View Alert system for their abnormal test results. Except for mandatory notifications, receiving various types of notifications is a configurable parameter (for instance, HCPs have an option to turn normal test result notifications off). We studied mandatory critical imaging alerts transmitted to all types of HCPs (attending physicians, allied HCPs, and trainees) working in a variety of specialties.

DATA COLLECTION

The Computerized Patient Record System (CPRS) is the EMR used at all VA facilities. It is maintained within the Veterans Health Information Systems and Technology Architecture (VISTA), an automated information system used to support ambulatory and inpatient care.25 The CPRS includes a notification system (the View Alert system) for alerting clinicians to clinically significant events such as abnormal diagnostic test results. Because of computerized provider order entry, the ordering clinician is always known and notified. When the PCP (usually a permanent staff HCP) is out of the office, they can assign covering surrogate HCPs to receive their alerts. When a trainee, a subspecialist, a surrogate, or a substitute HCP orders a test, a second alert is automatically transmitted to the PCP (ie, dual-alert communication). For the purposes of our study, any HCP who received an alert was considered responsible for timely follow-up (eg, if 2 HCPs are alerted, we considered them both responsible for timely follow-up).

Using predefined standardized codes, staff radiologists electronically coded abnormal imaging that required action as alerts,14 which were then transmitted to the View Alert window. The window is displayed when HCPs log on or switch between patient records and contains alerts on all of their patients, regardless of which record is viewed. Per institutional protocol, alerts stay active in the window for 2 weeks. Health care providers are expected to process alerts in a timely manner but have an option of ignoring the View Alert window to bypass it. Occasionally, while evaluating the medical record in response to other reasons, such as a patient visit or telephone call, HCPs may become aware of an abnormal imaging result and review the report without clicking on its corresponding alert.

Two weeks after alert transmission, we queried VISTA through an alert-management-tracking program to determine whether the alert had been “acknowledged,” (ie, any of the notified HCPs clicked on and opened the message). If alerts were acknowledged, they disappeared from the View Alert window and we considered them read. Each week, we downloaded both acknowledged and unacknowledged alerts corresponding to our time period (see timeline, Figure 1). The downloaded list included patient identifiers and ordering location, names of HCPs receiving the alert, and the date and type of imaging study. Outpatient alerts in which the patient was hospitalized after alert generation but prior to our medical chart review were categorized separately because the hospitalizations were not always related to the abnormal test result (hence, follow-up may not occur). Cases in which the radiologist verbally communicated the results (via telephone or face to face) to the HCP in addition to sending the alert were also categorized separately. Institutional protocol requires that documentation of verbal communication is always included in the imaging report.

Place holder to copy figure label and caption
Figure 1.

Timeline to assess study outcomes.

Graphic Jump Location

A reviewer (H.A.) blinded to acknowledgment status evaluated the medical records of patients to determine if there was any documented response to the alert. Response was defined as any documented evidence of follow-up action, such as contacting the patient or ordering a follow-up test or consultation or hospitalizing the patient for the test result and its presence was considered timely follow-up. If no response was documented, a second study investigator (S.M.) confirmed the findings in the medical record and then contacted the ordering HCP by telephone. We used this procedure to determine HCPs' awareness of test results and any follow-up action they took but failed to document. If the ordering HCP was unavailable, we called the PCP of the patient (who also receives the alert per protocol). These processes helped confirm whether the test result lacked timely follow-up at the end of 4 weeks after alert transmittal. Lack of timely follow-up was said to occur when there was no response (documented follow-up action at initial medical chart review) and when the HCP had not performed any undocumented follow-up action before our call. We allowed 4 weeks as the maximum time to follow up on non–life-threatening findings based on our previous work.14 When we called HCPs and lack of timely follow-up was confirmed, we encouraged them to initiate appropriate follow-up action and rereviewed their records a week later to check for action. Outcome variables were defined as communication breakdown at the electronic acknowledgment step or the timely follow-up step.

DATA ANALYSIS

We identified 2 groups of alerts corresponding to the 2 outcome variables representing communication breakdown in our study: (1) alerts lacking electronic acknowledgment vs acknowledged alerts at 2 weeks after alert transmission and (2) alerts lacking timely follow-up vs those receiving timely follow-up at 4 weeks after alert transmission. We compared the distribution (as proportions) of several independent variables such as ordering HCP specialty (primary care, medicine subspecialties, and surgery, among others), ordering HCP type (physician, trainee, and allied health professionals), type of imaging test (radiographs, CT scans, MRIs, mammograms, and ultrasonograms), use of additional verbal communication by the radiologist, and use of dual-alert communication between the groups. Dual-alert communication was counted as one instance of alerting in the denominator. If one of the alerted HCPs followed up the test result, the alert was considered followed up even if the second HCP did not perform any follow-up. The χ2 test was used for categorical variables, and the Fisher exact test was used when the assumptions for χ2 were not met. Multivariable logistic regression models accounting for clustering effect by HCPs were used to identify factors associated with the 2 outcome variables (ie, lack of acknowledgment, lack of timely follow-up). Predictor variables included all covariates with P values of <.20 in univariate testing. The models were fit using maximum likelihood estimation; odds ratios (ORs) and 95% confidence intervals (CIs) were calculated. Specialties from univariate analysis were combined into the following groups for multivariable testing: primary care, emergency medicine, surgery, and other subspecialties. To ensure that the model had stable and precise estimates of ORs, variables with small numbers of patients with a given characteristic (such as mammograms) were removed, and a final model was run on the remaining predictor variables.

Between November 2007 and June 2008, 123 638 outpatient studies (including radiographs, CT scans, ultrasonograms, MRIs, and mammograms) were performed, from which 1196 images (0.97%) generated alerts. Figure 2 illustrates the outcomes of these 1196 abnormal imaging reports. For 206 alerts (17.2%), the radiologist also communicated verbally with the HCP; for 172 (14.4%), the patient was subsequently admitted after alert transmission; and for 73 (6.1%), there was evidence of both. A total of 979 alerts (81.9%) were tracked as acknowledged and 217 (18.1%) were tracked as unacknowledged. No evidence of documented follow-up action was found in 131 of the alerts (11%). In 20 of these cases (1.7%), we determined that calling the HCP would be unnecessary or make no impact on outcome because the diagnosis was not new, the patient was already receiving appropriate care for the condition, or the patient had died. This decision was made after an additional investigator (H.S.) agreed with the initial reviewer, and these alerts were not considered to lack timely follow-up. In the remaining 111 cases, we called the HCPs to determine if undocumented follow-up care had occurred and if any intentional decision had been made to not pursue follow-up.

Place holder to copy figure label and caption
Figure 2.

Study flowchart of outcomes. *In 20 cases, calling health care providers was determined unnecessary.

Graphic Jump Location

Overall, 92 alerts (7.7%) lacked timely follow-up at 4 weeks; thus, the estimated rate of lack of timely follow-up was 0.07% of all diagnostic imaging tests performed. In these cases, we rereviewed the record a week after our initial call to determine if the HCPs took any follow-up action. We found that in 46 cases (50%), HCPs had executed the appropriate action.

LACK OF ACKNOWLEDGMENT

In Table 1 we compare types of communication, characteristics of ordering HCPs, and types of imaging between the acknowledged and unacknowledged alerts using univariate testing. Additional verbal communication by the radiologist and hospitalization of the patient within 2 weeks of alert transmission was not significantly associated with acknowledgment. Types and specialties of ordering HCPs and types of imaging tests were significantly different across the 2 groups. For instance, trainees and surgical specialties were less likely to acknowledge alerts compared with other types and specialties of ordering HCPs (P < .001 for both). Radiographs were more likely to be acknowledged (P < .001) than other imaging tests.

Table Graphic Jump LocationTable 1. Comparison of Types of Communication, Health Care Providers (HCPs), and Imaging for Acknowledged and Unacknowledged Diagnostic Imaging Alerts

In a logistic regression model for lack of acknowledgment, the following variables were significant (data not shown in the Tables): physician assistants as ordering HCPs compared with attending physicians (OR, 0.46; 95% CI, 0.22-0.98); trainees as ordering HCPs (OR, 5.58; 95% CI, 2.86-10.89); and when dual-alert as opposed to single-alert communication was used (OR, 2.02; 95% CI, 1.22-3.36).

Of 74 alerts unacknowledged by trainees that were followed by a documented response in the medical record at the time of review, the ordering trainee directly responded to the test result (without opening the alert) in 34 cases, 9 of which were prompted by a direct notification by the radiologist. In the remaining 40 cases, another HCP documented a response to the report, 4 of which were prompted by a direct notification by the radiologist.

LACK OF TIMELY FOLLOW-UP

Table 2 lists a univariate analysis comparing the following characteristics between the 92 alerts determined to lack timely follow-up at 4 weeks vs those that received timely follow-up: presence or absence of acknowledgment, types of communication, characteristics of ordering HCPs, and types of imaging. There was no significant difference in rates of lack of timely follow-up between the acknowledged and unacknowledged alerts (7.3% vs 9.7%; P = .22). Additional verbal communication and instances when the patient was hospitalized within 2 weeks of alert transmission were significantly more likely to be associated with timely follow-up (P < .001). No statistically significant differences in HCP type or specialty were seen, but certain tests (CT scans and MRIs) were more likely associated with timely follow-up (P = .02).

Table Graphic Jump LocationTable 2. Comparison of Types of Communication, Health Care Providers (HCPs), and Imaging for Diagnostic Imaging Alerts With and Without Timely Follow-up at 4 Weeks

In a logistic regression model for lack of timely follow-up, the following variables were significant predictors (Table 3): additional verbal communication (OR, 0.12; 95% CI, 0.04-0.38); a subsequent hospital admission after alert transmission (OR, 0.22; 95% CI, 0.08-0.38); alerts of CT scans (OR, 0.49; 95% CI, 0.29-0.85); and when dual-alert as opposed to single-alert communication was used (OR, 1.99; 95% CI, 1.06-3.48).

Table Graphic Jump LocationTable 3. Logistic Regression Analysis Adjusted for Clustering for Lack of Timely Follow-up of Abnormal Diagnostic Imaging Result Alerts (All Characteristics With P<.20 in Univariate Analysis)

Table 4 gives the types of near-miss abnormalities (defined as events that could have harmed the patient but did not cause harm as a result of chance, prevention, or mitigation26) in cases that lacked timely follow-up vs those that did not. Chest imaging showing a nonspecific density was more likely to be associated with a lack of timely follow-up (P = .04). All near-misses had potential to lead to missed or delayed diagnosis had we not intervened; the majority of these (62 of 92 [67.4%]) were related to some form of a suspected new malignant neoplasm. Half (50.0%) of the test results lacking timely follow-up were abnormal chest radiographs. Although obtaining qualitative data from practitioners was not a part of this study, 7 HCPs we called volunteered that someone else other than them (eg, the PCP) was the HCP responsible for follow-up.

Table Graphic Jump LocationTable 4. Types of Near-Miss Abnormalitiesa for Diagnostic Imaging Alerts With and Without Timely Follow-up at 4 Weeks

Subsequent evaluation of the importance and outcomes of the 92 imaging test results that did not receive timely follow-up was conducted, and data were collected on outcomes that were the direct result of the imaging result in question. Table 5 lists the outcomes determined at 9 months after the date of the test. Although we could not determine outcomes in 5 cases because the patient did not return to the institution, all other tests had a measurable clinical impact in terms of either diagnostic testing or treatment. In these 92 cases, more than one-fourth (n = 26 [28%]) led to a new diagnosis being established. Cancer was determined to be a new diagnosis in 11 of these cases.

Table Graphic Jump LocationTable 5. Diagnostic and Therapeutic Outcomes of 92 Critical Imaging Test Results That Lacked Timely Follow-up at 4 Weeksa

In a study of 1196 critical imaging notifications in an integrated EMR, we found 92 results (7.7%) without timely follow-up at 4 weeks after result transmission. Our findings suggest that an EMR that facilitates transmission and availability of critical imaging results to the HCP through either automated notification or direct access of primary report does not eliminate the problem of missed test results even when 1 or more HCPs read the results. Communicating alerts to 2 recipients, which occurred when tests were ordered by the HCP other than the regular PCP, significantly increased the odds that the alert would not be read and would not receive timely follow-up action. This could have resulted because the lines of responsibility for follow-up were less clear with dual-alert communication; the PCP may assume the ordering HCP was providing follow-up and vice-versa. Verbal communication by the radiologist in addition to the electronic communication strongly predicted timely response and follow-up, probably because radiologists called only for emergent and life-threatening findings. Nearly all abnormal results lacking timely follow-up were eventually found to have measurable clinical impact in terms of further diagnostic testing or treatment.

We unexpectedly found that imaging test results flagged as “critical” may not receive timely follow-up actions even when confirmed to have been read by HCPs. Therefore, even in the best of information systems that contain advanced notification features, patients with abnormal imaging test results are vulnerable to “falling through the cracks.” This underscores the need for a multidisciplinary approach involving human-computer interaction and informatics27,28 to complement the benefits achieved by automated notification and the need for continuous monitoring procedures to ensure follow-up even when HCPs “acknowledge,” ie, read abnormal results. The impact of several factors such as time pressures, organizational characteristics, factors related to the task of processing alerts, technological factors, and HCP factors should be considered in the design of interventions to improve timely follow-up.29,30 Future studies should address process-of-care issues leading to such communication breakdowns and guide the design and implementation of the next generation of computerized notification systems in ambulatory care.

Our study provides a new and more accurate estimate of prevalence of critical imaging test results without follow-up within a defined time interval. In our previous work, we calculated this to be approximately 4% of critical imaging results, but this figure was based only on unacknowledged alerts.14 Unexpectedly, our present study shows a substantial lack of timely follow-up even in alerts read by HCPs; hence, the increase from our previous estimate. However, we believe that our estimates of lack of timely follow-up are likely better than in systems that do not use computerized notification, although accurate and comparable data from such systems is lacking.9,31

Trainees had a higher likelihood of not reading their alerts but were not associated with a lack of timely follow-up. In our system, as in many others, residents spend only one-half to 1 day in a 2-week period in an outpatient setting usually in their own continuity clinics. In addition, duty hour rules also impact their clinic schedule. Therefore, an alert related to an abnormal imaging test ordered by a resident may be not be subsequently opened and read by the same resident in a timely manner. Rather, an alternate or supervising physician may either receive a call from the radiologist or access the primary report in response to an alert, in their absence. In more than half of alerts unacknowledged by residents where a documented response was found in the medical record, it was another HCP who documented that response (ie, alternate or supervising physician).

Several predictors provided important information for future work in improving patient safety in this area. First, dual-alert communication (defined as when an alert was transmitted to 2 HCPs), intended to be a safeguard to protect against loss of follow-up was unexpectedly associated with a lack of timely follow-up. Preliminary evidence from our study suggests that HCPs may believe responsibility for follow-up belonged to someone else. While institutional practices may determine whether the PCP or another ordering HCP (for instance the subspecialist) follows up an abnormality, this problem may be improved by better tracking systems,14,3234 using reminders built into the EMR, and care-coordination programs.27,3538 Second, direct (verbal) communication is important and will continue to play a valuable role even in sophisticated EMR systems. Currently at our institution, direct communication is required only when test results are life-threatening or require an emergent intervention. However, our findings suggest that HCPs may perceive a lack of urgency for test results that have less immediate consequential implications (eg, suspicious for a new cancer). Policies and procedures regarding the use of verbal communication for such results may need to be reevaluated.39,40

While the next generation of EMR-based notification systems are being designed, we propose several potential interventions based on our findings that can be used immediately to improve timely follow-up of abnormal imaging results. First, every institution must develop and publicize a policy regarding who is responsible (PCP vs the ordering HCP, who may be a consultant) for taking action on abnormal results. Second, unacknowledged alerts must stay active on the EMR screen for longer periods, perhaps even indefinitely, and should require the HCP's signature and statement of action before they are allowed to drop off from the screen. This would also ensure alerts do not “disappear” before the HCP has a chance to order the follow-up action. Third, interventions to reduce alert overload and improve signal to noise ratio of alerts should be explored. For instance, unnecessary alerts should be minimized and alerts should be retracted when the patients dies or if the radiologist calls, or the patient is admitted before the alert is acknowledged.41 Fourth, the EMR should be automated to track not just acknowledgment but the specific response documenting that appropriate action has been taken for each abnormal alert. This would save time and expense of performing medical chart reviews and calling HCPs to confirm their actions. Lastly, an audit and performance feedback system should be established to give HCPs information on timely follow-up of patients' test results on a regular basis.42

Our study has several limitations. Because sophisticated tracking such as the one we used is not easily possible in the non-EMR environment and similar data from the pre-EMR era in the VA does not exist, lack of a comparison group was a limitation. Because of the study population and site, our findings may not be generalizable outside the VA setting, especially to free-standing clinics that do not use integrated EMR systems. However, with recent emphasis of EMR adoption and implementation nationally, we believe that our findings are of great significance. In addition, we did not account for any problems with alert transmission failures such as situations when the radiologist inadvertently did not code the report as an abnormality. Conversely, many factors including a large sample size, involvement of several clinics, large number of patients (more than 120 000 veterans in southeast Texas) and HCPs (more than 500 from different specialties), rigorous methods to determine timely follow-up and outcomes at 9 months of alerts without timely follow-up, various types of diagnostic imaging tests, and an advanced integrated EMR used in VA facilities nationwide all add unique strength to our study.

In conclusion, an EMR that facilitates transmission and availability of critical imaging results to the HCP through either automated notification or direct access of primary report does not eliminate the problem of missed test results even when 1 or more HCPs read the results. Dual-alert communication, intended to be a safeguard to protect against loss of follow-up, was unexpectedly associated with lack of timely follow-up. Future multidisciplinary studies should address process-of-care issues leading to such communication breakdowns and guide the design and implementation of the next generation of computerized notification systems in ambulatory care.

Correspondence: Hardeep Singh, MD, MPH, VA Medical Center (152), 2002 Holcombe Blvd, Houston, TX 77030 (hardeeps@bcm.tmc.edu).

Accepted for Publication: May 22, 2009.

Author Contributions: Drs Singh and Khan had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Singh, Mani, and Petersen. Acquisition of data: Singh, Mani, Arora, and Espadas. Analysis and interpretation of data: Singh, Thomas, Mani, Sittig, Espadas, Khan, and Petersen. Drafting of the manuscript: Singh, Mani, Sittig, Arora, Espadas, and Khan. Critical revision of the manuscript for important intellectual content: Singh, Thomas, Mani, Sittig, Khan, and Petersen. Statistical analysis: Khan. Obtained funding: Singh and Petersen. Administrative, technical, and material support: Singh, Mani, Sittig, Arora, Espadas, and Petersen. Study supervision: Singh, Thomas, and Petersen.

Financial Disclosure: None reported.

Funding/Support: The study was supported by an NIH K23 career development award (K23CA125585) to Dr Singh, the VA National Center of Patient Safety, Agency for Health Care Research and Quality, and in part by the Houston VA HSR&D Center of Excellence (HFP90-020).

Role of the Sponsor: These sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.

Disclaimer: The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or the National Institutes of Health.

Additional Contributions: Meena Vij, MD, provided data support for our project.

Joint Commission Resources Staff, Joint Commission on Accreditation of Healthcare Organizations Inc, The Physician's Promise: Protecting Patients from Harm.  Chicago, IL Joint Commission Resources2002;
Arora  VJohnson  JLovinger  DHumphrey  HJMeltzer  DO Communication failures in patient sign-out and suggestions for improvement: a critical incident analysis. Qual Saf Health Care 2005;14 (6) 401- 407
PubMed
Graber  MLFranklin  NGordon  R Diagnostic error in internal medicine. Arch Intern Med 2005;165 (13) 1493- 1499
PubMed
Greenberg  CCRegenbogen  SEStuddert  DM  et al.  Patterns of communication breakdowns resulting in injury to surgical patients. J Am Coll Surg 2007;204 (4) 533- 540
PubMed
Kripalani  SLeFevre  FPhillips  COWilliams  MVBasaviah  PBaker  DW Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. JAMA 2007;297 (8) 831- 841
PubMed
Singh  HThomas  EJPetersen  LAStuddert  DM Medical errors involving trainees: a study of closed malpractice claims from 5 insurers. Arch Intern Med 2007;167 (19) 2030- 2036
PubMed
Gandhi  TK Fumbled handoffs: one dropped ball after another. Ann Intern Med 2005;142 (5) 352- 358
PubMed
Poon  EGGandhi  TKSequist  TDMurff  HJKarson  ASBates  DW “I wish I had seen this test result earlier!”: dissatisfaction with test result management systems in primary care. Arch Intern Med 2004;164 (20) 2223- 2228
PubMed
Poon  EGHaas  JSLouise Puopolo  A  et al.  Communication factors in the follow-up of abnormal mammograms. J Gen Intern Med 2004;19 (4) 316- 323
PubMed
Schiff  GD Introduction: communicating critical test results. Jt Comm J Qual Patient Saf 2005;31 (2) 63- 65, 61
PubMed
Hickner  JGraham  DGElder  NC  et al.  Testing process errors and their harms and consequences reported from family medicine practices: a study of the American Academy of Family Physicians National Research Network. Qual Saf Health Care 2008;17 (3) 194- 200
PubMed
Singh  HPetersen  LAThomas  EJ Understanding diagnostic errors in medicine: a lesson from aviation. Qual Saf Health Care 2006;15 (3) 159- 164
PubMed
Singh  HNaik  ARao  RPetersen  L Reducing diagnostic errors through effective communication: harnessing the power of information technology. J Gen Intern Med 2008;23 (4) 489- 494
PubMed
Singh  HArora  HSVij  MSRao  RKhan  MPetersen  LA Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007;14 (4) 459- 466
PubMed
Ash  JSBerg  MCoiera  E Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004;11 (2) 104- 112
PubMed
Choksi  VRMarn  CSBell  YCarlos  R Efficiency of a semiautomated coding and review process for notification of critical findings in diagnostic imaging. AJR Am J Roentgenol 2006;186 (4) 933- 936
PubMed
Wahls  THaugen  TCram  P The continuing problem of missed test results in an integrated health system with an advanced electronic medical record. Jt Comm J Qual Patient Saf 2007;33 (8) 485- 492
PubMed
Wahls  TLCram  PM The frequency of missed test results and associated treatment delays in a highly computerized health system. BMC Fam Pract 2007;832
PubMed
Kuperman  GJTeich  JMBates  DW  et al.  Detecting alerts, notifying the physician, and offering action items: a comprehensive alerting system Proc AMIA Annu Fall Symp 1996;704- 708
PubMed
van der Sijs  HAarts  JVulto  ABerg  M Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13 (2) 138- 147
PubMed
Taylor  LKTamblyn  R Reasons for physician non-adherence to electronic drug alerts. Stud Health Technol Inform 2004;107 (pt 2) 1101- 1105
PubMed
Magnus  DRodgers  SAvery  AJ GPs' views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther 2002;27 (5) 377- 382
PubMed
Feldstein  ASimon  SRSchneider  J  et al.  How to design computerized alerts to safe prescribing practices. Jt Comm J Qual Saf 2004;30 (11) 602- 613
PubMed
Poon  EGWang  SJGandhi  TKBates  DWKuperman  GJ Design and implementation of a comprehensive outpatient Results Manager. J Biomed Inform 2003;36 (1-2) 80- 91
PubMed
VA Information Resource Center, Veterans Health Information Systems and Technology Architecture (VistA)—Description. 2007;http://www.virec.research.va.gov/DataSourcesName/VISTA/VISTA.htm. Accessed January 25, 2007
Institute of Medicine of the National Academies, Patient Safety: Achieving a New Standard for Care.  Washington, DC National Academy Press2004;
Hickner  JMFernald  DHHarris  DMPoon  EGElder  NCMold  JW Issues and initiatives in the testing process in primary care physician offices. Jt Comm J Qual Patient Saf 2005;31 (2) 81- 89
PubMed
Horsky  JZhang  JPatel  VL To err is not entirely human: complex technology and user cognition. J Biomed Inform 2005;38 (4) 264- 266
PubMed
Carayon  PSchoofs Hundt  AKarsh  BT  et al.  Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006;15 ((suppl 1)) i50- i58
PubMed
Kushniruk  AWPatel  VL Cognitive evaluation of decision making processes and assessment of information technology in medicine. Int J Med Inform 1998;51 (2-3) 83- 90
PubMed
Cram  PRosenthal  GEOhsfeldt  RWallace  RBSchlechte  JSchiff  GD Failure to recognize and act on abnormal test results: the case of screening bone densitometry. Jt Comm J Qual Patient Saf 2005;31 (2) 90- 97
PubMed
Gandhi  TKSequist  TDPoon  EG  et al.  Primary care clinician attitudes towards electronic clinical reminders and clinical practice guidelines. AMIA Annu Symp Proc 2003;848
PubMed
Murff  HJGandhi  TKKarson  AK  et al.  Primary care physician attitudes concerning follow-up of abnormal test results and ambulatory decision support systems. Int J Med Inform 2003;71 (2-3) 137- 149
PubMed
Wei  EKRyan  CTDietrich  AJColditz  GA Improving colorectal cancer screening by targeting office systems in primary care practices: disseminating research results into clinical practice. Arch Intern Med 2005;165 (6) 661- 666
PubMed
Freeman  HPMuth  BJKerner  JF Expanding access to cancer screening and clinical follow-up among the medically underserved. Cancer Pract 1995;3 (1) 19- 30
PubMed
Freeman  HP Patient navigation: a community centered approach to reducing cancer mortality. J Cancer Educ 2006;21 (1) ((suppl)) S11- S14
PubMed
Ell  KVourlekis  BLee  PJXie  B Patient navigation and case management following an abnormal mammogram: a randomized clinical trial. Prev Med 2007;44 (1) 26- 33
PubMed
Battaglia  TARoloff  KPosner  MAFreund  KM Improving follow-up to abnormal breast cancer screening in an urban population: a patient navigation intervention. Cancer 2007;109 (2) ((suppl)) 359- 367
PubMed
Singh  HSethi  SRaber  MPetersen  LA Errors in cancer diagnosis: current understanding and future directions. J Clin Oncol 2007;25 (31) 5009- 5018
PubMed
Bates  DWLeape  LL Doing better with critical test results. Jt Comm J Qual Patient Saf 2005;31 (2) 61, 66- 67
PubMed
Kuperman  GJHiltz  FLTeich  JM Advanced alerting features: displaying new relevant data and retracting alerts. Proc AMIA Annu Fall Symp 1997;243- 247
PubMed
Kawamoto  KHoulihan  CABalas  EALobach  DF Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330 (7494) 765
PubMed

Figures

Place holder to copy figure label and caption
Figure 1.

Timeline to assess study outcomes.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.

Study flowchart of outcomes. *In 20 cases, calling health care providers was determined unnecessary.

Graphic Jump Location

Tables

Table Graphic Jump LocationTable 1. Comparison of Types of Communication, Health Care Providers (HCPs), and Imaging for Acknowledged and Unacknowledged Diagnostic Imaging Alerts
Table Graphic Jump LocationTable 2. Comparison of Types of Communication, Health Care Providers (HCPs), and Imaging for Diagnostic Imaging Alerts With and Without Timely Follow-up at 4 Weeks
Table Graphic Jump LocationTable 3. Logistic Regression Analysis Adjusted for Clustering for Lack of Timely Follow-up of Abnormal Diagnostic Imaging Result Alerts (All Characteristics With P<.20 in Univariate Analysis)
Table Graphic Jump LocationTable 4. Types of Near-Miss Abnormalitiesa for Diagnostic Imaging Alerts With and Without Timely Follow-up at 4 Weeks
Table Graphic Jump LocationTable 5. Diagnostic and Therapeutic Outcomes of 92 Critical Imaging Test Results That Lacked Timely Follow-up at 4 Weeksa

References

Joint Commission Resources Staff, Joint Commission on Accreditation of Healthcare Organizations Inc, The Physician's Promise: Protecting Patients from Harm.  Chicago, IL Joint Commission Resources2002;
Arora  VJohnson  JLovinger  DHumphrey  HJMeltzer  DO Communication failures in patient sign-out and suggestions for improvement: a critical incident analysis. Qual Saf Health Care 2005;14 (6) 401- 407
PubMed
Graber  MLFranklin  NGordon  R Diagnostic error in internal medicine. Arch Intern Med 2005;165 (13) 1493- 1499
PubMed
Greenberg  CCRegenbogen  SEStuddert  DM  et al.  Patterns of communication breakdowns resulting in injury to surgical patients. J Am Coll Surg 2007;204 (4) 533- 540
PubMed
Kripalani  SLeFevre  FPhillips  COWilliams  MVBasaviah  PBaker  DW Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. JAMA 2007;297 (8) 831- 841
PubMed
Singh  HThomas  EJPetersen  LAStuddert  DM Medical errors involving trainees: a study of closed malpractice claims from 5 insurers. Arch Intern Med 2007;167 (19) 2030- 2036
PubMed
Gandhi  TK Fumbled handoffs: one dropped ball after another. Ann Intern Med 2005;142 (5) 352- 358
PubMed
Poon  EGGandhi  TKSequist  TDMurff  HJKarson  ASBates  DW “I wish I had seen this test result earlier!”: dissatisfaction with test result management systems in primary care. Arch Intern Med 2004;164 (20) 2223- 2228
PubMed
Poon  EGHaas  JSLouise Puopolo  A  et al.  Communication factors in the follow-up of abnormal mammograms. J Gen Intern Med 2004;19 (4) 316- 323
PubMed
Schiff  GD Introduction: communicating critical test results. Jt Comm J Qual Patient Saf 2005;31 (2) 63- 65, 61
PubMed
Hickner  JGraham  DGElder  NC  et al.  Testing process errors and their harms and consequences reported from family medicine practices: a study of the American Academy of Family Physicians National Research Network. Qual Saf Health Care 2008;17 (3) 194- 200
PubMed
Singh  HPetersen  LAThomas  EJ Understanding diagnostic errors in medicine: a lesson from aviation. Qual Saf Health Care 2006;15 (3) 159- 164
PubMed
Singh  HNaik  ARao  RPetersen  L Reducing diagnostic errors through effective communication: harnessing the power of information technology. J Gen Intern Med 2008;23 (4) 489- 494
PubMed
Singh  HArora  HSVij  MSRao  RKhan  MPetersen  LA Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007;14 (4) 459- 466
PubMed
Ash  JSBerg  MCoiera  E Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004;11 (2) 104- 112
PubMed
Choksi  VRMarn  CSBell  YCarlos  R Efficiency of a semiautomated coding and review process for notification of critical findings in diagnostic imaging. AJR Am J Roentgenol 2006;186 (4) 933- 936
PubMed
Wahls  THaugen  TCram  P The continuing problem of missed test results in an integrated health system with an advanced electronic medical record. Jt Comm J Qual Patient Saf 2007;33 (8) 485- 492
PubMed
Wahls  TLCram  PM The frequency of missed test results and associated treatment delays in a highly computerized health system. BMC Fam Pract 2007;832
PubMed
Kuperman  GJTeich  JMBates  DW  et al.  Detecting alerts, notifying the physician, and offering action items: a comprehensive alerting system Proc AMIA Annu Fall Symp 1996;704- 708
PubMed
van der Sijs  HAarts  JVulto  ABerg  M Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13 (2) 138- 147
PubMed
Taylor  LKTamblyn  R Reasons for physician non-adherence to electronic drug alerts. Stud Health Technol Inform 2004;107 (pt 2) 1101- 1105
PubMed
Magnus  DRodgers  SAvery  AJ GPs' views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther 2002;27 (5) 377- 382
PubMed
Feldstein  ASimon  SRSchneider  J  et al.  How to design computerized alerts to safe prescribing practices. Jt Comm J Qual Saf 2004;30 (11) 602- 613
PubMed
Poon  EGWang  SJGandhi  TKBates  DWKuperman  GJ Design and implementation of a comprehensive outpatient Results Manager. J Biomed Inform 2003;36 (1-2) 80- 91
PubMed
VA Information Resource Center, Veterans Health Information Systems and Technology Architecture (VistA)—Description. 2007;http://www.virec.research.va.gov/DataSourcesName/VISTA/VISTA.htm. Accessed January 25, 2007
Institute of Medicine of the National Academies, Patient Safety: Achieving a New Standard for Care.  Washington, DC National Academy Press2004;
Hickner  JMFernald  DHHarris  DMPoon  EGElder  NCMold  JW Issues and initiatives in the testing process in primary care physician offices. Jt Comm J Qual Patient Saf 2005;31 (2) 81- 89
PubMed
Horsky  JZhang  JPatel  VL To err is not entirely human: complex technology and user cognition. J Biomed Inform 2005;38 (4) 264- 266
PubMed
Carayon  PSchoofs Hundt  AKarsh  BT  et al.  Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006;15 ((suppl 1)) i50- i58
PubMed
Kushniruk  AWPatel  VL Cognitive evaluation of decision making processes and assessment of information technology in medicine. Int J Med Inform 1998;51 (2-3) 83- 90
PubMed
Cram  PRosenthal  GEOhsfeldt  RWallace  RBSchlechte  JSchiff  GD Failure to recognize and act on abnormal test results: the case of screening bone densitometry. Jt Comm J Qual Patient Saf 2005;31 (2) 90- 97
PubMed
Gandhi  TKSequist  TDPoon  EG  et al.  Primary care clinician attitudes towards electronic clinical reminders and clinical practice guidelines. AMIA Annu Symp Proc 2003;848
PubMed
Murff  HJGandhi  TKKarson  AK  et al.  Primary care physician attitudes concerning follow-up of abnormal test results and ambulatory decision support systems. Int J Med Inform 2003;71 (2-3) 137- 149
PubMed
Wei  EKRyan  CTDietrich  AJColditz  GA Improving colorectal cancer screening by targeting office systems in primary care practices: disseminating research results into clinical practice. Arch Intern Med 2005;165 (6) 661- 666
PubMed
Freeman  HPMuth  BJKerner  JF Expanding access to cancer screening and clinical follow-up among the medically underserved. Cancer Pract 1995;3 (1) 19- 30
PubMed
Freeman  HP Patient navigation: a community centered approach to reducing cancer mortality. J Cancer Educ 2006;21 (1) ((suppl)) S11- S14
PubMed
Ell  KVourlekis  BLee  PJXie  B Patient navigation and case management following an abnormal mammogram: a randomized clinical trial. Prev Med 2007;44 (1) 26- 33
PubMed
Battaglia  TARoloff  KPosner  MAFreund  KM Improving follow-up to abnormal breast cancer screening in an urban population: a patient navigation intervention. Cancer 2007;109 (2) ((suppl)) 359- 367
PubMed
Singh  HSethi  SRaber  MPetersen  LA Errors in cancer diagnosis: current understanding and future directions. J Clin Oncol 2007;25 (31) 5009- 5018
PubMed
Bates  DWLeape  LL Doing better with critical test results. Jt Comm J Qual Patient Saf 2005;31 (2) 61, 66- 67
PubMed
Kuperman  GJHiltz  FLTeich  JM Advanced alerting features: displaying new relevant data and retracting alerts. Proc AMIA Annu Fall Symp 1997;243- 247
PubMed
Kawamoto  KHoulihan  CABalas  EALobach  DF Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330 (7494) 765
PubMed

Correspondence

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).
Submit a Comment

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 50

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Topics