Author Affiliations: Program on Prevention Outcomes and Practices, Stanford Prevention Research Center, Stanford University School of Medicine (Mr Romano and Dr Stafford), and Program in Human Biology, Stanford University School of Arts and Sciences (Mr Romano), Stanford, California.
American physicians often fail to provide outpatient care that is recommended by clinical guidelines,1,2 and many stakeholders identify health information technology (HIT) as a potential solution to low-quality care.3 Since 1991, the Institute of Medicine has repeatedly called for increasing electronic health record (EHR) use to improve health care quality.3,4 Clinical practices implementing outpatient EHRs self-report improved clinical decisions and resulted in easier communication with other providers and patients, faster and more accurate access to medical records, and avoidance of medication errors. While US physicians have been slow to adopt outpatient EHRs,5,6 their use is likely to accelerate because of the Health Information Technology for Economic and Clinical Health (HITECH) provisions of the American Reinvestment and Recovery Act of 2009.7,8 Nonetheless, evidence linking increased national use of outpatient EHRs to improved care quality is lacking. While past studies within specific institutions have demonstrated better quality as a result of EHR implementation,9,10 using 2004 national data Linder et al11 found no quality difference between ambulatory care provided with and without EHRs. Several recent studies have also failed to observe an association between EHR use and improved quality of care.12,13
This lack of association between EHRs and national outpatient quality of care may reflect early patterns of EHR use in past studies and the heterogeneous functionality of EHRs studied. One EHR function of key relevance to quality is clinical decision support (CDS), a feature that that alerts, reminds, or directs health care providers according to clinical guidelines. Past evaluations present conflicting results regarding CDS's effects on quality, although most previous studies have had small sample sizes and have focused on specific diseases in a limited number of institutions.12,14- 20 Broader evaluations of CDS effects on quality across several institutions and diseases have had variable results.13
Using nationally representative, federally collected 2005-2007 data, we reexamined the impact of EHRs on outpatient care in the United States. We hypothesized that CDS functionality is associated with higher-quality outpatient care compared with EHR use without CDS.
Focusing on ambulatory care provided in physicians' offices, hospital outpatient departments, and emergency departments, we examined patient and physician characteristics associated with the use of EHRs and CDS. We also determined whether EHR and CDS use predicted better outpatient quality of care using an existing set of performance indicators, while accounting for potential confounders.
We used the most recent data available from the National Ambulatory Medical Care Survey (NAMCS, 2005-2007) and the National Hospital Ambulatory Medical Care Survey (NHAMCS, 2005-2007), both conducted by the National Center for Health Statistics (NCHS, Hyattsville, Maryland). These surveys gather information on ambulatory medical care provided by nonfederal, office-based, direct-care physicians (NAMCS)21 and provided in emergency and outpatient departments affiliated with nonfederal general and short-stay hospitals (NHAMCS).22 These federally conducted, national surveys are designed to meet the need for objective, reliable information about US ambulatory medical care services.23 These data sources have been widely used by government and academic research to report on patterns and trends in outpatient care.
The unit of analysis derived from NAMCS/NHAMCS is the patient visit. Patient visit data are collected using a 3-stage (NAMCS) or a 4-stage (NHAMCS) sampling procedure, selecting geographic primary sampling units, hospitals or physicians within each primary sampling unit, clinics within each hospital (NHAMCS only), and patient visits for each provider or clinic. Response rates were 59% to 62% among physicians invited to participate in NAMCS (2005-2007) and 80% to 88% among hospital emergency departments and outpatient departments invited to participate in NHAMCS (2005-2007).21,22 In 2005, there were 24 627 visit records available from NAMCS, 26 806 from the NHAMCS outpatient department component, and 29 747 from the NHAMCS emergency department component.
To each patient visit, NCHS assigns a statistical weight derived from the sampling procedure and independent national annual estimates of outpatient visits. These weights allow extrapolation to a national level. Data from NHAMCS and NAMCS were combined, consistent with NCHS guidelines. The combination of these data sets widened the scope of practices covered, increased the study's generalizability, and more than tripled the sample sizes available compared with NAMCS alone, thereby increasing statistical power. We included visits to emergency departments because they are a key source of care and a setting in which EHRs have been more widely adopted. We excluded visits resulting in hospitalization.
The NCHS Research Ethics Review Board annually approves NAMCS and NHAMCS and has waived informed consent requirements and authorization for medical record release.21,22 Because no identifiers are publically available, this study was conducted under an exemption from the institutional review board of Stanford University, Stanford, California.
Information is available regarding the clinical setting as well as individual visits. The NCHS conducts a 35- to 60-minute induction interview with each participating physician or hospital to gather information on practice characteristics (eg, specialty, size of practice, physician specialty/clinic type, geographic region, and ownership). The in-person induction interviews precede visit record data collection and have been refined since the surveys began in the 1970s. We defined primary care visits as those involving the specialties of family medicine, internal medicine, geriatrics, pediatrics, and general practice (NAMCS) or as those taking place in the “general medical clinic,” “internal medicine,” and “pediatrics” (NHAMCS).
The induction surveys since 2005 ask, “Does your practice use electronic medical records (not including billing records)?” We coded responses of “yes, all electronic,” and “yes, part paper and part electronic,” as EHRs. Information collected about EHR characteristics includes whether the EHR provides CDS: “reminders for guideline-based interventions and/or screening tests.”24 Practices that had CDS but had turned off these functions were considered not to have CDS. We excluded patient visits at which answers to the EHR or CDS questions were missing or “don't know.” We use the term EHR rather than electronic medical record because EHR is rapidly becoming standard terminology.8
Each visit record requires 5 to 10 minutes to complete and includes information on the reason for the patient's visit, physician diagnoses, new and continued medications, and demographic data. Reasons for visits are coded using NCHS's own classification system. Diagnostic information is coded according to the International Classification of Diseases, Ninth Revision, Clinical Modification.25 In 2005, medications were coded using NCHS's own system, but since 2006 Lexicon Plus (Cerner Multum, Denver, Colorado) has been used. For each visit, health care providers or their staff members manually enter text fields on paper surveys, which are subsequently coded by trained medical coders.
Nonresponse rates for most questions on both surveys were generally below 5%, with the exception of race and ethnicity. For records lacking age, race, ethnicity, and sex data, NCHS imputed values based on multiple imputation using physician specialty, geographic region, and 3-digit International Classification of Diseases, Ninth Revision, Clinical Modification codes for primary diagnosis. National Center for Health Statistics quality control for medical and drug coding involved a 2-way independent verification procedure for 10% of records in each survey year. For records with coding discrepancies, records were reviewed and adjudicated by both reviewers. Coding error rates for various items ranged from 0.3% to 4.5% in 2005, from 0.2% to 1.4% in 2006, and from 0.0% to 1.7% in 2007.
Our analysis of quality of care used a selected set of 20 quality indicators that had previously been used to assess quality using NAMCS/NHAMCS26 but that had been updated to reflect changes in clinical guidelines. Each indicator represents a care guideline whose adherence can be measured using the visit-based information available from NAMCS/NHAMCS visit records. The indicators were developed using broad criteria established by the Institute of Medicine (clinical importance, scientific soundness, and feasibility for indicator selection) and specific criteria based on the NAMCS/NHAMCS data sources. The indicators fall into 5 categories: (1) pharmacological management of common chronic diseases, including atrial fibrillation, coronary artery disease, heart failure, hyperlipidemia, asthma, and hypertension (9 indicators); (2) appropriate antibiotic use in urinary tract infection and viral upper respiratory infections (2 indicators); (3) preventive counseling regarding diet, exercise, and smoking cessation (5 indicators); (4) appropriate use of screening tests for blood pressure measurement, urinalysis, and electrocardiography (3 indicators); and (5) inappropriate prescribing in elderly patients (1 indicator).
Performance on each quality indicator was defined as the proportion of eligible patients receiving guideline-congruent care so that a higher proportion represents greater concordance with care guidelines. Attention was paid to excluding those patients with comorbidities that would complicate guideline adherence (eg, asthma in assessing the use of β-blockers in coronary artery disease). Also, in some instances, care was adherent to the quality indicator if a similar therapy was provided (eg, warfarin rather than aspirin in coronary artery disease).
We performed a retrospective, cross-sectional analysis of ambulatory care visits from the NAMCS (2005-2007) and NHAMCS (2005-2007) data sets using weighted visits. Based on each clinical practice's induction information, visits were categorized into 3 mutually exclusive groups for the comparison: (1) EHR and CDS use in the clinical practice, (2) use of an EHR but without CDS, and (3) no use of an EHR. We developed 2 hierarchical statistical models to examine the likelihood of EHR use among all visits and of CDS use among visits using EHRs. We developed a series of hierarchical statistical models of identical form applied to each quality indicator that evaluated the likelihood of quality performance for the 2 EHR groups together compared with visits not associated with an EHR. We also developed a series of hierarchical statistical models comparing the likelihood of favorable quality in CDS visits with that in visits without CDS, measured among EHR visits, for each indicator. We conducted an additional, planned analysis of quality and EHR/CDS separately for NAMCS (private offices) and NHAMCS (hospital outpatient and emergency departments).
Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, North Carolina). We used the SURVEYLOGISTIC procedure to statistically control for potentially confounding variables via logistic regression modeling.27 Control variables for all models included patient sex, patient age group, patient race/ethnicity, specialty/clinic type, clinic ownership/size, geographical region, and data set of origin. Clinical characteristics were not included in these models under the argument that quality indicators should be applicable to all patients, except where specific complicating diagnoses are defined. All hypothesis testing was 2-tailed, and P < .05 was considered statistically significant. We did not adjust statistical significance for multiple comparisons, but interpretation of the findings reflects recognition of this issue. Our statistical power to detect differences in quality associated with EHR and CDS varied widely across indicators and was largely dependent on visit sample size. In comparing EHR with no EHR, the statistical power to detect a 5% absolute change in indicator performance (eg, 66% vs 71%) at a statistical significance level of P = .05 (2-tailed) was 90% or greater for 11 indicators, 75% to 89.9% for 3 indicators, 50% to 74.9% for 5 indicators, and less than 50% for 1 indicator. Similarly, for CDS vs no CDS among EHR visits, the statistical power was 90% or greater for 7 indicators, 75% to 89.9% for 2 indicators, 50% to 74.9% for 3 indicators, and less than 50% for 8 indicators.
Between 2005 and 2007, the NAMCS and the NHAMCS contained data on 243 478 unweighted patient visits not resulting in a hospital admission with EHR status available. These data represented a projected 3.18 billion patient visits nationwide, or 1.06 billion annually. The EHRs were used in 29.6% of all patient visits, and the CDS was used in 56.3% of these EHR visits (16.7% of all patient visits). The use of both EHR and CDS increased over time.
The use of EHRs and CDS in ambulatory visits varied significantly across patient and provider characteristics (Table 1). The EHRs were used in 41% of patient visits in the West, a rate much higher than that of other regions (25%-30%). The use of EHRs also was higher in group practices/community health centers (28%) and hospital-affiliated practices (48%) compared with solo practice offices (16%). Compared with primary care (29%), EHR use was higher for emergency medicine (56%) and lower among other specialties (20%). These same characteristics were statistically significant as predictors of ambulatory visit EHR use in multivariable logistic regression models of EHR use (Table 1).
Among ambulatory visits at which EHRs were used, CDS was more likely to be present in the West (66% of EHR visits) compared with other regions (47%-56%). Surgeons (46%) and other specialists (35%) had the lowest CDS use compared with other physicians (51%-67%). Electronic health record visits by male patients had lower rates of CDS use than those by female patients (54% vs 58%). These same characteristics were statistically significant predictors of CDS use among EHR visits in multivariable logistic regression (Table 1).
Ambulatory quality of care was suboptimal for many indicators (Table 2). The use of EHRs was not consistently associated with higher-quality care. Among all outpatient visits, those at which an EHR was present showed significantly better performance in only 1 of 20 quality indicators based on unadjusted data (diet counseling in high-risk adults, 28% vs 20%; adjusted odds ratio [OR] 1.65; 95% confidence interval [CI], 1.21-2.26; P = .002). In the other 19 indicators, there was no statistically significant difference in quality based on the presence or absence of an EHR. These patterns were present both in bivariate analysis and in multivariate logistic regression models. Nonstatistically significant differences of greater than 5% favored EHR use in 3 instances and lack of EHR use in 2 instances. In the context of multiple comparisons across quality indicators, these findings suggest no association between EHR use and quality.
Clinical decision support also was not associated with higher-quality care. Among EHR visits, CDS was associated with better performance on only 1 of 20 ambulatory care quality indicators: avoidance of unnecessary electrocardiography during routine examinations (98% vs 93%; adjusted OR, 2.88; 95% CI, 1.69-4.90; P = .001). In the other 19 indicators, there were no significant differences in quality. Nonstatistically significant differences of greater than 5% favored CDS in 1 instance and EHRs without CDS in 3 instances. Given the multiple comparisons, our findings indicate no association between CDS and quality.
Separate planned analysis of office-based (NAMCS) and hospital-based (NHAMCS) patient visits showed similar results. Among EHR office-based visits, CDS use was associated with higher quality for 1 indicator: fewer inappropriate electrocardiograms (OR, 3.78; 95% CI, 1.72-8.31). The remaining 19 quality indicators showed no variation in quality by CDS status. Among hospital-based visits, CDS use was favorably associated with smoking cessation counseling (OR, 3.09; 95% CI, 1.57-6.09) and unfavorably associated with lower use of inhaled corticosteroids in adults with asthma (OR, 0.62; 95% CI, 0.43-0.87) and lower routine blood pressure monitoring (OR, 0.57; 95% CI, 0.35-0.90). Given the finding of greater EHR and CDS use in the West, we examined post hoc the independent association of Western region with care quality and found no consistent relationship. Western visits had higher quality than other regions for 2 indicators (antibiotic use in respiratory infection and inappropriate electrocardiography ordering), worse quality for 3 indicators (urinary tract infection antibiotic selection, diet counseling, and exercise counseling), and similar quality for the remaining 15 indicators.
In a nationally representative survey of physician visits, neither EHRs nor CDS was associated with ambulatory care quality, which was suboptimal for many indicators. We noted no association between EHR use and care quality for 19 indicators and a positive relationship for only 1 indicator. We also found CDS use associated with better quality for only 1 of 20 quality indicators, refuting our hypothesis that CDS would be associated with improved care quality. Some studies suggest that certain guidelines and clinical areas lend themselves well to CDS intervention, while others do not. However, the lack of association between EHR and CDS and care quality in our study fails to support such conclusions.28- 31
The breadth of data from ambulatory practices across the United States sets this study apart from past CDS literature. The NAMCS/NHAMCS data provide a superior source of information for resolving discrepancies arising between institution-specific studies. Clinical decision support is an often-cited EHR mechanism of quality improvement, yet this study failed to detect significant improvements in health care quality among all US practices using CDS systems. While our findings do not rule out the possibility that the use of CDS may improve quality in some settings, they cast doubt on the argument that the use of EHRs is a “magic bullet” for health care quality improvement, as some advocates imply.
Political, economic, and clinical interest in health information technology has come much later to the United States compared with other nations. A large majority (often more than 90%) of primary care physicians in 10 other Western industrialized countries use EHRs in their practices. We report the use of EHRs in only 29% of primary care visits, although recent federal actions and incentives will no doubt increase EHR use.5,7,32
The American Reinvestment and Recovery Act stimulus bill set aside $19.2 billion to promote HIT use in the United States, with the underlying assumption that more HIT is better. The majority of HITECH funding goes to raising reimbursement rates for Medicare and Medicaid services delivered with the “meaningful EHR use.” Qualifying EHRs must have a minimum number of CDS guidelines.33,34
While our study cannot predict the broad impacts of a government HIT incentive program, 2 interpretations are available. Given that we observed no difference in quality between visits with and without CDS, our findings raise doubts about the ability of broad CDS adoption promoted by HITECH to unilaterally improve outpatient care quality. Several anecdotal articles describe how CDS can disrupt care and decrease care quality; however, further empirical research is needed.35,36 In the absence of broad evidence supporting existing CDS systems, planned investment should be monitored carefully and its impact and cost evaluated rigorously.
At the same time, our findings may suggest a need for greater attention to quality control and coordinated implementation to realize the potential of EHRs and CDS to improve health care. In the absence of governmental impetus and standards, current adoption patterns may have fostered incomplete implementation and use of less effective technologies. Systematic federal intervention through HITECH may be needed to realize the potential of these technological advances.
Our analysis extends Linder and colleagues’9 finding of no significant association between EHR use and national ambulatory care quality. Electronic health records can vary enormously in their functionality, so Linder and coauthors hypothesized that examination of overall EHR use may have been too broad. To remedy this overly inclusive categorization, we compared a subset of advanced-function EHRs with more basic, limited-function EHRs but still noted no difference in quality. Furthermore, the present analysis assesses the predictors of EHR and CDS use and considers statistical power in assessing outpatient quality.
We also observed variations in the use of EHRs and CDS according to patient and physician characteristics. Being located in the West was a prominent predictor of EHR and CDS uptake, as was receiving services in larger care settings. These findings suggest the substantial economies of scale that are faced in costly HIT implementation. Lack of disparities by patient race may be attributable to adoption of EHRs and CDS in emergency departments and hospital outpatient departments, settings in which minority patients are more likely to receive care than white patients.37,38
This study has several limitations. The use of NAMCS/NHAMCS data examines only a sample of patients and excludes those receiving care in federal facilities and nonparticipating practices. The most recently available data from 2005 through 2007 may not reflect current practice patterns. Use of cross-sectional, visit-level data does not allow assessment of longitudinal care, which may be more important than process- and visit-based quality indicators. We used multivariate analyses to control for variables that are independently associated with CDS adoption and care quality, but other factors that are not available in NAMCS/NHAMCS may be confounders. The quality targets of CDS were not reported, and the relationship of these targets to the selected quality indicators is not known. The report of an EHR or a CDS being used in a clinical practice does not translate into use at every visit, although we did account for CDS functionality being turned off. There are many steps between the availability of CDS and its effective use to improve quality, and our assessment is not able to pinpoint specific barriers within this complex process. Although cultural and technical factors are likely involved, distinct barriers may apply to different institutional settings.
For several quality measures, we lacked sufficient statistical power to detect moderate-sized differences in quality. Among nonstatistically significant comparisons for which quality indicator differences were greater than 5%, however, there was no tendency to favor EHRs or CDS. Despite these limitations, these data remain the best source of information available to examine the connection between EHR/CDS use and national outpatient quality of care.
Future research should investigate why the CDS benefits in randomized controlled trials have not translated into national quality improvement. Research also is needed to elucidate the factors influencing HIT adoption. This information will be vital to federal decisions about HIT implementation in the coming years. As in this assessment, further research should continue to evaluate the role of EHRs and CDS outside of academic medical centers in the smaller-scale settings in which most Americans receive outpatient health services.
In conclusion, despite the promise of better quality, the clinical benefits of EHRs and CDS are not evident in our quality indicators. Given the growing institution-specific evidence that CDS use increases the quality of health care services, our results raise doubts about past implementation of costly EHR technologies nationally. While EHRs offer substantial administrative efficiency over paper records, current patterns of EHR and CDS use do not appear to translate into better outpatient quality of care.
Correspondence: Max J. Romano, BA, 2634 Guilford Ave, Baltimore, MD 76132 (email@example.com).
Accepted for Publication: November 13, 2010.
Published Online: January 24, 2011. doi:10.1001/archinternmed.2010.527
Author Contributions: Both authors had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Romano and Stafford. Acquisition of data: Romano and Stafford. Analysis and interpretation of data: Romano and Stafford. Drafting of the manuscript: Romano and Stafford. Critical revision of the manuscript for important intellectual content: Romano and Stafford. Statistical analysis: Stafford. Obtained funding: Romano and Stafford. Administrative, technical, and material support: Romano and Stafford. Study supervision: Stafford.
Financial Disclosure: Dr Stafford has a consulting relationship with Mylan Pharmaceuticals and within the past 5 years has consulted for Bayer Corporation and has had grants through Stanford University with Procter and Gamble, GlaxoSmithKline, Toyo Shinyaku, and Wako.
Funding/Support: Dr Stafford's involvement in this project was funded by a National Heart, Lung, and Blood Institute mid-career mentoring award (K24HL086703). Mr Romano's funding was provided by a Stanford University Office of Undergraduate Advising and Research major grant award.
Role of the Sponsor: The funders of this research study had no role in the design and conduct of the study; in the collection, analysis, and interpretation of the data; or in the preparation, review, or approval of the manuscript.
Additional Contributions: Special thanks to Jeffrey A. Linder, MD, MPH (Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts), Jun Ma, MD, PhD (Palo Alto Medical Foundation Research Institute, Palo Alto, California), and G. Caleb Alexander, MS, MD (University of Chicago, Chicago, Illinois), for their consultation and helpful suggestions on an earlier draft of the manuscript.
Thank you for submitting a comment on this article. It will be reviewed by JAMA Internal Medicine editors. You will be notified when your comment has been published. Comments should not exceed 500 words of text and 10 references.
Do not submit personal medical questions or information that could identify a specific patient, questions about a particular case, or general inquiries to an author. Only content that has not been published, posted, or submitted elsewhere should be submitted. By submitting this Comment, you and any coauthors transfer copyright to the journal if your Comment is posted.
* = Required Field
Disclosure of Any Conflicts of Interest*
Indicate all relevant conflicts of interest of each author below, including all relevant financial interests, activities, and relationships within the past 3 years including, but not limited to, employment, affiliation, grants or funding, consultancies, honoraria or payment, speakers’ bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued. If all authors have none, check "No potential conflicts or relevant financial interests" in the box below. Please also indicate any funding received in support of this work. The information will be posted with your response.
Some tools below are only available to our subscribers or users with an online account.
Download citation file:
Web of Science® Times Cited: 94
Customize your page view by dragging & repositioning the boxes below.
Enter your username and email address. We'll send you a link to reset your password.
Enter your username and email address. We'll send instructions on how to reset your password to the email address we have on record.
Athens and Shibboleth are access management services that provide single sign-on to protected resources. They replace the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session. It operates independently of a user's location or IP address. If your institution uses Athens or Shibboleth authentication, please contact your site administrator to receive your user name and password.