Author Affiliations: Divisions of Critical Care Medicine (Dr McCambridge) and General Internal Medicine (Drs Sussman and Etchason), Department of Community Health, Health Studies, and Education (Drs Jones and Etchason and Mss Paxton and Baker), Lehigh Valley Health Network, Allentown, Pennsylvania; Department of Medicine, Penn State College of Medicine, Pennsylvania State University, Hershey (Dr McCambridge); and Department of Medicine, College of Medicine, University of South Florida, Tampa (Dr Sussman).
Little evidence exists to support implementing various health information technologies, such as telemedicine, in intensive care units.
A coordinated health information technology bundle (HITB) was implemented along with remote intensivist coverage (RIC) at a 727-bed academic community hospital. Critical care specialists provided bedside coverage during the day and RIC at night to achieve intensivist coverage 24 hours per day, 7 days per week. We evaluated the effect of HITB-RIC on mortality, ventilator and vasopressor use, and the intervention length of stay. We compared our results with those achieved at baseline.
A total of 954 control patients who received care for 16 months before the implementation of HITB-RIC and 959 study patients who received care for 10 months after the implementation were included in the analysis. Mortality for the control and intervention groups were 21.4% and 14.7%, respectively. In addition, the observed mortality for the intervention group was 75.8% (P < .001) of that predicted by the Acute Physiology and Chronic Health Evaluation IV hospital mortality equations, which was 29.5% lower relative to the control group. Regression results confirm that the hospital mortality of the intensive care unit patients was significantly lower after implementation of the intervention, controlling for predicted risk of mortality and do-not-resuscitate status. Overall, intervention patients also had significantly less (P = .001) use of mechanical ventilation, controlling for body-system diagnosis category and severity of illness.
The use of HITB-RIC was associated with significantly lower mortality and less ventilator use in critically ill patients.
Because of the complexities of caring for critically ill patients, as well as an ongoing shortage of intensive care unit (ICU) specialists, a number of promising technologies are being advanced in the hope that they can support the provision of higher-quality, more efficient care to an increasing number of patients. The number of studies1- 9 of several types of health information technology (HIT), such as telemedicine and computer-assisted physician order entry, among others, is limited and results have been mixed. Very little evidence exists related to the effect of some of these technologies, specifically in ICU settings,8,9 and, furthermore, how they might function together as an integrated package or bundle to improve the quality of care. Given the considerable expense of implementing such technologies, this lack of evidence represents a significant barrier to their adoption.
To date, the most convincing evidence to guide quality improvements in ICU settings pertains to physician specialty training and medical staffing. A substantial body of evidence demonstrates improvements in outcomes for ICU patients who receive care from specially trained and certified critical care physicians (intensivists).10- 16 Furthermore, closed ICU staffing models, in which management of all ICU patients is transferred to intensivists, produce better outcomes than open models, in which intensivists comanage some or none of the patients.16- 21
Although the evidence for closed ICUs is compelling, whether to have intensivists present in the ICU on a continuous basis is still an unsettled issue for many experts.22- 24 Despite equivocal evidence, there is pressure from organizations such as the Leapfrog Group25 to provide continuous coverage by intensivists in closed ICUs. However, the proponents of this goal admit that its attainment is challenged by an ongoing shortage of specially trained and credentialed intensivists. To make matters worse, with only 10% to 20% of ICUs staffed by dedicated intensivists,26,27 the current shortage will likely become exacerbated.28- 30
This workforce dilemma has led to speculation that HIT might help expand patients' access to high-quality critical care medicine;31 in fact, remote ICU telemonitoring by intensivists is being used in a limited number of institutions to fill gaps in staffing coverage. However, the evidence for the effectiveness of doing so, much like for other technologies such as computer-assisted physician order entry, is still preliminary and limited.2 Only 3 studies32- 34 of remote ICU telemonitoring have emerged in the peer-reviewed literature. Two of those studies32,33 demonstrated improvements in mortality and length of stay (LOS), while the third34 reported no overall effect on either outcome. However, all 3 of these previous studies share the limitation of not having used a closed ICU staffing model as the baseline comparator. Hence, the existing evidence does not address the question of whether advanced HIT, including telemedicine applications, can effectively improve outcomes in ICUs with use of a closed staffing model, ie, the model whose use is considered to be evidence-based best practice.
This article reports the results of installing a health information technology bundle (HITB) in conjunction with remote intensivist coverage (RIC). We conducted a study to evaluate the effectiveness of HITB-RIC with respect to mortality, ventilator and vasopressor use, and LOS. We compared the results of this intervention with those achieved at baseline by a closed ICU staffing model without HIT enhancements and 24 hours per day, 7 days per week on-site coverage.
We conducted an interrupted time series observational study. This study was deemed exempted research by the institutional review board of Lehigh Valley Health Network.
Patients were included in the study if they were aged 18 years or older and had been admitted to an ICU for a medical diagnosis. To maintain independent observations, we considered only the first admission to the ICU during hospitalization. The comparison group consisted of patients admitted consecutively to the ICU for 16 months preceding the implementation of HITB-RIC, which began January 1, 2004, and was completed September 30, 2004. The study group consisted of patients admitted consecutively to the ICU for the 10 months after the implementation had been completed.
The study was conducted in 3 ICUs that contained a total of 36 monitored beds, all part of a 727-bed academic community hospital. Two of the ICUs shared staff and management, whereas the third ICU had a separate staff and nurse manager but operated in the same manner as the other 2 ICUs with respect to staffing, work hours, protocols, and standards of care. During the study, no changes occurred in the physical layout of the units, nursing staff ratios, or unit structure. Before the implementation of HITB-RIC, the ICU was a closed model staffed by board-certified intensivists who were physically present from 7 AM to 11 PM, with call coverage during off hours.
With the goal of increasing intensivist coverage to 24 hours a day, a team of caregivers, administrators, and information systems experts worked for approximately 12 months to implement HITB-RIC.35 By involving bedside caregivers in the planning process, project leaders ensured that the eventual bundle would serve to improve their ability to treat patients and not disrupt their workflow.35 The implementation team determined that all components of HITB-RIC were necessary for the successful reengineering of ICU care. Therefore, all components were selected as part of an integrated program of care enhancement. The HITB-RIC consisted of an ICU electronic medical record (EMR) with an electronic algorithmic event system (MetaVision and MVcentral; iMDsoft, Needham, Massachusetts); computer-assisted physician order entry, an electronic medication administration record, and barcoded medication administration (LastWord; GE Healthcare, Fairfield, Connecticut); a radiographic picture archiving and communication system (Centricity RIS-IC; GE Healthcare); and a 2-way audio and 1-way video remote monitoring system (Vistacom Inc, Allentown, Pennsylvania).
The telemedicine team (an intensivist and a critical care nurse), located off-site, used the audiovisual equipment in each ICU room to interact with patients and caregivers and had real-time access to all the components of the HITB, including the EMR, current and prior medical transcriptions, and ancillary data (LastWord; GE Healthcare). From 7 PM to 7 AM, the telemedicine team admitted new patients, responded to telephone calls from ICU nurses about their patients, and responded to computer-generated events, as identified by the EMR's algorithmic event system. These events included, but were not limited to, critical changes in heart rate, blood pressure, laboratory values, mechanical ventilator parameters, and central venous and pulmonary artery catheter values. The telemedicine team also responded to radiographic abnormalities and managed other patient care issues while performing rounds for all monitored patients every 2 hours, proactively looking for changes in clinical status that would warrant intervention.
Data were collected with regard to 1000 control and 1000 study patients. For the controls, demographic information was obtained from administrative records and clinical data were manually abstracted from medical records. Data regarding HITB-RIC patients were retrieved from the MetaVision database and supplemented by manual medical record review. Clinical measures for all patients' first 24 hours of ICU stay were obtained from the medical record and used to calculate each patient's Acute Physiology and Chronic Health Evaluation (APACHE; Cerner Corporation, Kansas City, Missouri) Acute Physiology Score (APS).36 The APS is part of the APACHE IV hospital mortality equations (hereafter, APACHE IV),37 which comprise the Cerner Corporation's most up-to-date model for predicting group hospital mortality among critically ill patients. Patients were assigned body-system diagnostic categories according to the APACHE IV classification system contained in the calculation template.
APACHE APS and APACHE IV have not been validated for patients with burn diagnoses, patients staying in the ICU less than 4 hours, or patients transferred from other ICUs.36- 38 Two study patients had burn diagnoses; 18 control and 3 study patients stayed less than 4 hours in the ICU and were, therefore, excluded from analysis. In addition, 28 controls and 38 study patients who had been transferred from other ICUs were also excluded. Altogether, 46 controls and 41 HITB-RIC patients were excluded. Because 2 patients who had been transferred from other ICUs also stayed fewer than 4 hours, the total number of patients excluded in the study period was 41. Results for 954 control and 959 HITB-RIC patients were analyzed.
We analyzed the effect of HITB-RIC on several outcome variables. The primary outcome of interest was hospital mortality. Secondary outcome measures were mechanical ventilator and vasopressor use (both measured dichotomously as used or not used) and ICU and hospital LOS. Control variables included the severity of illness, diagnosis, and do-not-resuscitate (DNR) status.
APACHE IV was used to calculate the standardized mortality ratio (SMR). The SMR is calculated as the observed hospital mortality of the study sample divided by the average hospital mortality predicted by APACHE IV (ie, the observed/predicted mortality ratio).36- 39 To test the statistical significance of differences in observed vs predicted mortality, we used a simple comparison of the observed proportion of mortalities to the predicted benchmark proportion because SMRs in general are not comparable to one another.
We used binary logistic regression to explore the difference in mortality before and after HITB-RIC, controlling for risk of mortality (with APACHE IV) and DNR status. Differences in use of ventilators and vasopressors before and after HITB-RIC were explored with binary logistic regression, controlling for severity of illness (APACHE APS), body-system diagnostic category, and DNR status (and substituted do-not-intubate status for DNR status in ventilator use analysis). Differences in hospital and ICU LOS were explored by regression analysis, controlling for severity of illness (APACHE APS) and body-system diagnostic category.
We also used an interrupted time series design to analyze outcomes over time to control for trends in study outcomes that may have started in the period before HITB-RIC. Because mortality and ventilator use are relatively rare, rates of each were meaningful only for 1 month or longer. We had too few data points before and after HITB-RIC to analyze an interrupted time series with monthly observations. However, we analyzed time within our binary logistic regressions, in which patients are the unit of analysis, by controlling for number of months after data collection began. Also, we analyzed the time trends in monthly and bimonthly mortality and ventilator use rates over time using linear regression. In both cases, we were exploring whether the observed changes in mortality and ventilator use rates were indeed a discontinuity at the implementation period.
No statistically significant differences (α = .05) were found between the 2 groups with respect to demographic characteristics or clinical measures except diagnostic category (Table 1). Overall, we observed a decrease in crude mortality from 21.4% in the control period to 14.7% in the study period. During the control period, mortality was not significantly different from the predicted mortality (SMR, 1.075). However, during the study period, observed mortality was significantly lower (P < .001) than predicted by APACHE IV (SMR, 0.758) (Table 2). The HITB-RIC group SMR was 29.5% lower than the SMR of the control group. Holding APACHE IV (which also controls for diagnosis) and DNR status constant, regression analysis demonstrates that HITB-RIC group mortality was not only significantly less than predicted (as shown by the SMRs) but also significantly less than the control group mortality (Table 3).
Mortality, predicted mortality, and SMR were increasing in the period before HITB-RIC, using monthly and bimonthly rates; when controlling for predicted mortality, observed mortality demonstrated a positive trend but was statistically insignificant for monthly data (P = .19) and negative but statistically insignificant for bimonthly data (P = .76). In separate binary logistic regressions of the patients before and after HITB-RIC, the monthly time trend variable was statistically insignificant (P=.40 and .73, respectively), whereas APACHE IV and DNR status remained significant predictors (P < .001) of mortality. Mortality was found to decrease significantly over time when the full sample was analyzed; however, it was insignificant when an after-implementation indicator variable (ie, dummy variable) was added, suggesting that the effect was a discontinuous drop in mortality levels (intercept) at the HITB implementation period (an interaction term suggests no significant change in slope).
Subanalyses of mortality were also conducted for body-system diagnostic categories of sufficient size for analysis (ie, cardiovascular, neurologic, gastrointestinal, and respiratory). In each diagnostic category, the SMR of the study group was reduced by at least 19.0%. The HITB-RIC group's observed mortality was significantly lower than predicted for patients with neurologic (P =.002) and gastrointestinal (P = .04) diagnoses. Regression analyses of the neurology subgroup showed that mortality was significantly (P = .02) lower in the HITB-RIC group, controlling for severity of illness with APACHE IV. Within the patients with neurologic diagnoses, observed mortality for patients without stroke was significantly less than predicted at P < .001 (SMR, 0.50). However, the observed mortality for HITB-RIC patients with stroke was not significantly less than predicted (SMR, 0.89).
Regarding ventilator use, 36.1% of patients before HITB-RIC required ventilators compared with 31.5% of patients after HITB-RIC. Ventilator use was significantly less (P = .001) in the HITB-RIC group, controlling for severity of illness and body-system diagnosis (Table 4). Neither DNR nor do-not-intubate status was a significant predictor. In general, patients with respiratory and neurologic diagnoses were more likely to require use of a ventilator than those with cardiologic diagnoses (the reference diagnosis category). In subanalyses by diagnostic category (eg, neurology, cardiology, etc), ventilator use was significantly less in the HITB-RIC group, controlling for severity of illness and body-system diagnosis, for patients with neurologic (P < .001) and respiratory (P = .04) diagnoses.
Before HITB-RIC implementation, the trend in ventilator use rates was positive and statistically significant for monthly (P < .001) and bimonthly (P = .01) rates, indicating increasing ventilator use. The time trend in ventilator use rates when controlling for severity with APS was also positive and significant for monthly (P = .001) and bimonthly (P = .02) rates. In a separate binary logistic regression explaining ventilator use and including only patients before HITB-RIC implementation, the monthly time trend variable was positive and significant (P < .001), whereas the APS and neurologic or respiratory diagnosis remained significant predictors (P < .001). Using only patients after HITB-RIC, the monthly time variable was not significant (P = .32). However, using all patients, the monthly time variable was only significant when the after-implementation indicator variable was also included, suggesting that there was a discontinuous drop in ventilator use levels at the HITB implementation period. We found no significant difference in vasopressor use between control and study patients when controlling for diagnostic category and severity of illness.
The mean ICU LOS was 4.06 days before and 3.77 days after HITB-RIC; the mean hospital LOS was 9.15 days before and 9.21 days after HITB-RIC. Differences in distributions for ICU and hospital LOS were not statistically significant using Mann-Whitney tests. Controlling for severity of illness and diagnostic category, no significant differences were detected between the groups in either ICU LOS or overall hospital LOS.
This observational study took place in a 727–licensed bed community teaching hospital. The implementation of HITB-RIC, consisting of an ICU EMR, an electronic algorithmic event system, computer-assisted physician order entry, an electronic medication administration record, bar-coded medication administration, a radiographic picture archiving and communication system, and an ICU telemedicine program, was associated with a 29.5% reduction in ICU mortality. We also observed a significant decrease in patients' use of mechanical ventilation after implementation of HITB-RIC. This result was driven primarily by the decrease in ventilator use observed in the subgroups of patients with respiratory and neurologic diagnoses. It is likely that the frequency of ventilator use by patients with other diagnoses in our sample was insufficient to produce statistically meaningful results within those subgroups. We believe our study is the first to contribute evidence suggesting that the use of such a comprehensive and integrated HIT application, combined with RIC to provide off-hours coverage, improves outcomes beyond those achieved by a closed ICU staffing model without coverage 24 hours a day, 7 days a week.
Three previous studies32- 34 examining the associations among various teleintensivist arrangements and patient outcomes, such as mortality and LOS, have produced mixed results. However, our study differs from the previous ones in 2 important ways. First, we examined the effects of a teleintensivist program implemented as part of a comprehensive, integrated HIT application (HITB-RIC). The previous studies had only limited or no other advanced HIT to support the teleintensivists. Second, our study settings used closed ICU staffing models exclusively and all patients were under the care of teleintensivists during off hours. The previous studies incorporated a variety of staffing models with varying degrees of involvement of teleintensivists. These differences may help to explain the lack of overall effect reported in the most recently published study, in which almost two-thirds of the patients had minimal exposure to the tele-ICU intervention and the teleintensivists did not have access to the computerized physician order entry system or EMR in the ICUs.34
Although the evidence is mixed, 2 reviews suggest that intensivist-led ICU staffing models40 or high-intensity ICU physician staffing41 result in reduced ICU LOS. In addition, 2 of the 3 previously reported studies32,33 on the effect of teleintensivist interventions on LOS showed significant reductions in ICU LOS. Our study cohort in the period after HITB-RIC implementation experienced a decrease in mean ICU LOS of 0.29 day, but this difference was not statistically significant and is in contrast to data from the previously cited studies. One potential explanation for this difference is that our intervention was compared with a baseline LOS achieved under a closed ICU staffing model, and those of the previous studies were not. If HITB-RIC is capable of contributing to further reductions in LOS beyond those achieved by a closed ICU staffing model, our study did not have sufficient power to detect it.
The generalizability of our results is limited because they were produced at a single institution using the technologies unique to specific vendors. Given that different vendors promote nonstandardized technologies, further comparative effectiveness research should be conducted to reach more definitive conclusions about ICU technology.2 Our results could have also been affected by the differences in data collection methods between case patients (electronic based) and control patients (paper based), although this variance was inevitable given that an improved data collection and storage system is one of the components of the intervention that was studied. During the control and study periods, no additional interventions, such as other quality improvement initiatives or staffing changes, were put in place that might have influenced the outcomes.
Because we used a multifaceted intervention, it is not possible to comment on the effectiveness of individual components of HITB-RIC. The team that designed and implemented the study intervention thought that the reengineering of ICU care could not be maximally effective without the implementation of all aspects of HITB-RIC. Therefore, the intention of our study was to evaluate the effectiveness of a comprehensive program for providing continuous intensivist coverage of ICU patients, not the program's individual components.
Our evaluation is based on an observational study of actual practice in a 727–licensed bed community teaching hospital rather than on a randomized controlled experiment. Unlike some nonrandomized studies, ours benefited from the use of a case-mix adjustment model to compare outcomes. Therefore, this article contributes to establishing the effectiveness of the intervention in a real-world environment rather than its efficacy under highly controlled experimental conditions.
We believe the evidence of effectiveness presented herein is compelling despite the potential limitations of our study design and that evidence of the efficacy of HITB-RIC will be difficult, if not impossible, to obtain. The significant resources needed to implement the comprehensive bundle of components comprising HITB-RIC and the major investment of human resources, particularly nursing resources,35 would make a multicenter randomized controlled trial extremely expensive. In addition, to perform a multicenter randomized controlled trial, standardized technologies would be necessary but difficult to implement given the different vendors and programs for each component of the HITB.
We believe the HITB-RIC model represents a significant advancement in the quality of critical care medicine. Although there are still unanswered questions, especially concerning the most efficient use of these technologies, our study suggests that implementing HITB-RIC may be a viable strategy to improve access to high-quality ICU care in the setting of an inadequate supply of qualified intensivists.
Correspondence: Hannah D. Paxton, RN, MPH, Department of Community Health, Health Studies, and Education, Lehigh Valley Health Network, PO Box 7017, Allentown, PA 18105-7017 (Hannah_D.Paxton@lvhn.org).
Accepted for Publication: November 10, 2009.
Author Contributions: Dr McCambridge had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: McCambridge, Jones, Paxton, Baker, and Etchason. Acquisition of data: McCambridge and Baker. Analysis and interpretation of data: McCambridge, Jones, Paxton, Sussman, and Etchason. Drafting of manuscript: McCambridge, Jones, Paxton, Sussman, and Etchason. Critical revision of the manuscript for important intellectual content: Jones, Paxton, Baker, Sussman, and Etchason. Statistical analysis: Jones and Paxton. Obtained funding: Etchason. Administrative, technical, and material support: McCambridge, Paxton, and Baker. Study supervision: McCambridge, Sussman, and Etchason.
Financial Disclosure: None reported.
Funding/Support: Dr Jones was supported by a grant from the Dorothy Rider Pool Health Care Trust, Allentown, Pennsylvania.
Role of the Sponsor: The funding organization had no role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; or the preparation or approval of the manuscript.
Previous Presentation: Results of this study were presented at the AcademyHealth Annual Research Meeting; June 29, 2009; Chicago, Illinois.
Additional Contributions: Steve Matchett, MD (formerly of Lehigh Valley Health Network), and the caregivers and staff of the Lehigh Valley Health Network's Advanced Intensive Care Unit facilitated and assisted with this research. The following employees of Lehigh Valley Health Network also provided help: Carol Foltz, PhD, assisted with the study design and protocol; Margaret Kornuszko-Story, MHA, and Joseph Tracy, MS, reviewed the manuscript; and Jackie Laudenslager assisted with revisions and proofreading. These individuals did not receive extra compensation for their assistance.
Thank you for submitting a comment on this article. It will be reviewed by JAMA Internal Medicine editors. You will be notified when your comment has been published. Comments should not exceed 500 words of text and 10 references.
Do not submit personal medical questions or information that could identify a specific patient, questions about a particular case, or general inquiries to an author. Only content that has not been published, posted, or submitted elsewhere should be submitted. By submitting this Comment, you and any coauthors transfer copyright to the journal if your Comment is posted.
* = Required Field
Disclosure of Any Conflicts of Interest*
Indicate all relevant conflicts of interest of each author below, including all relevant financial interests, activities, and relationships within the past 3 years including, but not limited to, employment, affiliation, grants or funding, consultancies, honoraria or payment, speakers’ bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued. If all authors have none, check "No potential conflicts or relevant financial interests" in the box below. Please also indicate any funding received in support of this work. The information will be posted with your response.
Some tools below are only available to our subscribers or users with an online account.
Download citation file:
Web of Science® Times Cited: 30
Customize your page view by dragging & repositioning the boxes below.
Users' Guides to the Medical Literature
Users' Guides to the Medical Literature
Regression Modeling With Dichotomous Target Variables
All results at
Enter your username and email address. We'll send you a link to reset your password.
Enter your username and email address. We'll send instructions on how to reset your password to the email address we have on record.
Athens and Shibboleth are access management services that provide single sign-on to protected resources. They replace the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session. It operates independently of a user's location or IP address. If your institution uses Athens or Shibboleth authentication, please contact your site administrator to receive your user name and password.