Author Affiliations: Departments of Health Policy and Management (Ms Chatterjee and Drs Joynt and Jha) and Biostatistics (Dr Orav), Harvard School of Public Health, Boston, Massachusetts; Divisions of Cardiovascular (Dr Joynt) and General Internal Medicine (Drs Orav and Jha), Brigham and Women's Hospital, Boston; VA Boston Healthcare System, Boston (Drs Joynt and Jha); and Division of Cardiovascular Medicine (Dr Joynt), Department of Biostatistics (Dr Orav), and Division of General Internal Medicine (Dr Jha), Harvard Medical School, Boston. Ms Chatterjee is a medical student at Harvard Medical School.
Background Whether safety-net hospitals (SNHs) provide patient-centered care has important implications both for patient outcomes and for how these hospitals will fare under value-based purchasing (VBP). We sought to determine performance and improvement on measures of patient-reported hospital experience among SNHs compared with non-SNHs.
Methods Our sample consisted of 3096 US hospitals. We defined safety-net hospitals as those hospitals in the highest quartile of the Disproportionate Share Hospital (DSH) index, and we used national data on patient experience from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey in 2007 and 2010 to examine overall hospital performance and improvement over time.
Results Safety-net hospitals had lower performance than non-SNHs on nearly all measures of patient experience. The greatest differences were in overall hospital rating, where patients in SNHs were less likely to rate the hospital a 9 or 10 on a 10-point scale compared with patients in non-SNHs (63.9% vs 69.5%; P < .001). Gaps were also sizeable for the proportion of patients who reported receiving discharge information (2.6 percentage point difference; P < .001) and who thought they always communicated well with physicians (2.2 percentage point difference; P < .001). Although both groups of hospitals improved from 2007 through 2010, the gap between SNHs and non-SNHs increased (3.8% in 2007 vs 5.6% in 2010; P = .08). Finally, SNHs had a 60% lower odds of meeting VBP performance benchmarks for hospital payments (odds ratio, 0.4; 95% CI, 0.3-0.5; P < .001) compared with non-SNHs.
Conclusions Safety-net hospitals have lower performance than non-SNHs on metrics of patient-reported experience, improved somewhat more slowly under public reporting, and are likely to fare poorly under VBP.
Patient-reported experience with health care is an essential measure of how well a health care system functions. As such, improving patient experience in hospitals has become a major priority for both clinical leaders and policy makers. Prior studies in multiple health care settings have shown that poor self-reported experiences with the health care system are associated with slower recovery from illness and a lower likelihood of adherence to prescribed treatment regimens.1- 3 Consequently, suboptimal patient experience has important implications not only for the health of patients but also for health care costs, which increase when patients use more health care services because of poor recovery and nonadherence.4
Optimizing patient experience may be particularly important in safety-net hospitals (SNHs), which care for vulnerable and typically poor populations. These patients, on average, are sicker5 and tend to have lower levels of trust in the health care system.6,7 Therefore, they may be more susceptible to the negative effects of substandard experiences with health care providers. Beyond the importance of providing patient-centered care for this population, the changing fiscal landscape for hospital payments under the Patient Protection and Affordable Care Act8 has made the need to improve patient experience even more pressing. Value-based purchasing (VBP), a program run by the Centers for Medicare and Medicaid Services (CMS), now ties part of each hospital's payments to its performance on a set of quality measures.9 Under VBP, approximately 1% to 3% of total Medicare payments will be held back, and hospitals will receive some portion of that money based on how well they perform on the VBP metrics. A meaningful part of each hospital's performance score will be determined using metrics of patient-reported experience from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey.10 For SNHs, ensuring high performance under VBP will be particularly critical to their economic viability.11 Yet, without robust data on how SNHs perform on measures of patient experience and how these hospitals have responded to 4 years of public reporting on these metrics, it is difficult to project how they are likely to fare under the new payment scheme.
Therefore, in this study, we used HCAHPS data to answer 4 questions: First, how well do SNHs perform on patient experience compared with non-SNHs? Second, are there certain subgroups of hospitals for which being an SNH is associated with particularly poor levels of performance? Third, over the 4 years during which patient experience has been publicly reported, do SNHs show any evidence of narrowing the gap with other hospitals? And finally, how do SNHs currently fare in relation to achievement thresholds in the VBP program? Understanding the degree to which being an SNH is associated with worse performance, independent of hospital characteristics that are largely out of the hospital's control (such as its size or location), is critically important to devising policies that ensure greater accountability while maintaining equity.
Our hospital sample was based on hospitals reporting HCAHPS and Disproportionate Share Hospital (DSH) index12 data to CMS in 2010. Of the 4742 acute-care hospitals in the United States, we excluded 1547 hospitals that did not report DSH index to CMS. Of these excluded hospitals, 1267 were small, rural critical access hospitals, which are paid using different formulas by CMS. Of the remaining 3195 hospitals, 99 did not report HCAHPS scores in 2010. Our sample included the remaining 3096 hospitals, which provide 91% of all the acute care for elderly Medicare fee-for-service beneficiaries in the US comparisons between hospitals that reported both HCAHPS scores and DSH index vs hospitals that reported neither HCAHPS scores nor DSH index (eTable 1).
The HCAHPS survey includes 27 questions regarding patients' experiences during hospitalization for a medical or surgical condition. Responses, scored as “never,” “sometimes,” “usually,” and “always,” are grouped into a series of composite quality measures: (1) communication with physicians, (2) communication with nurses, (3) communication about medications, (4) quality of nursing services, (5) presence of discharge planning, (6) pain management, (7) cleanliness, and (8) quietness of the hospital environment. In addition, there are 2 global measures of patient experience: an overall rating on a scale from 0 to 10, and an assessment of whether the patient would recommend the hospital to family and friends (definitely no, probably no, probably yes, and definitely yes). The data were adjusted for patient-level demographics and the survey administration mode.13 The VBP program will use scores on the 8 HCAHPS measures in their calculation of incentive payments—including the 6 composite domains (composite quality measures 1-6) and a combined measure of cleanliness and quietness of the hospital environment (composite quality measures 7 and 8)—and an overall rating of the hospital.9
We used the American Hospital Association survey14 to obtain hospital characteristics including size, ownership, location (region and urban vs rural), teaching status, nurse-staffing levels, percentage of patients with Medicaid insurance, percentage of patients with Medicare insurance, and presence or absence of a medical intensive care unit. We calculated the proportion of each hospital's discharged elderly patients who self-identified as being black from Medicare inpatient files. We used a standard measure of nurse-staffing levels by dividing the number of full-time–equivalent nurses on staff by 1000 patient-days.15- 17 We obtained data on Medicare cost margins (net hospital income divided by total hospital revenue) to characterize hospitals' financial status.
Because there is no consensus on how to define an SNH, we used multiple approaches. Our primary approach, which has been widely used,18 defines hospitals in the highest quartile of DSH index as SNHs. The DSH index is reported by each hospital to CMS and is a combination of the proportion of elderly patients who are receiving Supplemental Security Income and the proportion of nonelderly patients who are receiving Medicaid, as represented in the following formula:
DSH Index = (Medicare Supplemental Security Income days/total Medicare days) + (Medicaid, non-Medicare days/ total patient days).
One advantage to the DSH index is that it identifies poor patients irrespective of age. Although the proportion of Medicaid patients can also be used to assess a hospital's safety-net status, this method fails to identify elderly patients who are poor because their hospital charges are covered by Medicare. Still, we performed sensitivity analyses using proportion Medicaid to determine safety-net status; the results were qualitatively similar and here we present only those results under the DSH index classification.
Our primary outcome was the patient-reported overall hospital rating. Because the 2 measures of overall experience (overall hospital rating, and whether the patient would recommend the hospital) are highly correlated15 and the VBP rules only include the overall rating as a performance metric, we used the overall rating as our primary outcome of interest. However, we also examined the proportion of patients who would definitely recommend the hospital. The results were qualitatively very similar and are not presented. Secondary outcomes included each of the individual composite measures (eg, communication with nurses, communication with physicians).
We first compared hospital and patient characteristics across DSH index quartiles using χ2 tests, analyses of variance, and Kruskal-Wallis tests, as appropriate.
To examine the relationship between HCAHPS performance and DSH index as a continuous variable, we used a nonparametric locally weighted bivariate Loess curve. We then divided the hospitals in our sample into DSH index quartiles for ease of presentation, treating DSH index quartile as a categorical variable. We assessed bivariate relationships between DSH index quartile and HCAHPS scores and then created multivariate generalized linear regression models including those variables that we believed were largely outside the hospitals' control, including the number of hospital beds, teaching status, profit status, location, census region, presence or absence of a medical intensive care unit, nurse-to-patient ratio, and the percentage of patients with Medicare as the source of payment. We repeated the analyses excluding nurse-to-patient ratio or proportion of Medicare patients and the results were qualitatively similar.
We hypothesized, based on prior literature,15,19 that the impact of safety-net status on patient experience might be less pronounced among larger, public, teaching hospitals, or institutions located in the Northeast. We serially tested for interactions and show subgroups in instances where the interactions were significant.
Next, given our interest in examining how SNHs have fared under 4 years of public reporting, we used regression models to examine the HCAHPS overall hospital rating in 2007 and 2010 and assessed whether the changes over time varied by DSH index quartile. On the basis of prior evidence that poor baseline performers improve the most under public reporting,20 we further adjusted for baseline (2007) performance. Finally, we examined the characteristics of SNHs that were associated with the greatest improvements over the 4-year period. All analyses were performed with DSH index quartile as an ordinal variable and repeated using DSH index as a continuous variable; the continuous results are available in eTable 2.
Finally, because VBP includes a “consistency” score on which hospitals score maximum points if they perform at or above the national median score for all 8 HCAHPS measures, we assessed the odds of meeting this benchmark for SNHs vs non-SNHs. We used bivariate instead of multivariate models because under VBP performance payment policies, there will be no adjustments made for hospital characteristics. We also calculated the odds of failing to score at or above the median on all of the 8 measures, which, under VBP, would correspond with a withholding of the entire incentive payment allotted to a hospital.
Statistical analyses were performed using Stata software (version 12; StataCorp). The project was considered exempt by the human subjects committee of the Harvard School of Public Health.
Of the 3096 hospitals included in our analyses, 769 were in the highest DSH index quartile and thus composed the SNH group (Table 1). Safety-net hospitals were more likely than non-SNHs to be large hospitals that were for profit or publicly owned. Safety-net hospitals were more likely to be major teaching hospitals (18% in SNHs vs 4% among hospitals in the lowest DSH index quartile; P < .001) and were more often located in the South (52% vs 31%; P < .001) and West (27% vs 16%; P < .001). Safety-net hospitals had substantially lower nurses per 1000 patient-days than hospitals in the lower DSH index quartiles (5.8 vs 6.8; P < .001). Safety-net hospitals had fewer Medicare patients and more Medicaid and black patients than other hospitals. When we examined Medicare cost margins, we found that hospitals in the highest DSH index quartile had about 40% lower margins (P < .001 for difference across quartiles) than hospitals in the lowest DSH index quartile (Table 1).
When we examined the relationship between DSH index as a continuous variable and HCAHPS scores, we found that higher DSH index was associated with worse performance on overall hospital rating (Figure 1).
Figure 1. Loess curve of overall hospital rating scores in 2010 on the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey10 by Disproportionate Share Hospital (DSH) index.12 Vertical lines represent separation into DSH quartiles at indices of 0.17, 0.24, and 0.34 units, consecutively.
When the DSH index was examined in quartiles, SNHs had the lowest HCAHPS scores on all measures of patient experience but one (quietness of hospital environment; Table 2). The greatest differences in scores between SNHs and hospitals in the lowest DSH index quartile were on measures of the overall hospital rating (63.9% vs 69.5%, 5.6 percentage point difference; P < .001) and receipt of discharge information (2.6 percentage point difference; P < .001). Analyses treating DSH index as a continuous variable were similar (eTable 3).
When we tested for effect modification, we found that only the interaction terms for hospital size and region were statistically significant. We found modest differences in performance on the overall rating between SNHs and non-SNHs based on size (Table 3) but larger differences based on region. However, contrary to our hypothesis, gaps between SNHs and non-SNHs were smallest in the South (difference of 0.6% in overall rating) and largest in the Northeast (difference of 6.0 percentage points; P < .001 for interaction).
Safety-net hospitals had worse performance on overall hospital rating than non-SNHs in 2007 (the beginning of the public reporting period). Over the subsequent 3 years, all groups of hospitals improved, but the improvement was somewhat smaller among SNHs compared with non-SNHs (3.0% vs 4.8% among the lowest DSH index quartile, although the P value for difference was nonsignificant [P = .08]; Figure 2), leading to a wider performance gap in 2010 (5.6%) than in 2007 (3.8%). Analyses accounting for baseline performance and analyses treating DSH index as a continuous variable were similar (eTable 3a and eTable 3b).
Figure 2. Changes in performance on overall hospital rating among Disproportionate Share Hospital (DSH)12 index quartiles from 2007 through 2010. P = .08 for difference between DSH index quartiles in scores from 2007 through 2010. Adjusted for size, teaching status, profit status, ownership, region, location, presence or absence of a medical intensive care unit, nurse-to-patient ratio, and the percentage of patients with Medicare as a source of payment.
When we examined which SNHs improved more (vs less) than the median, we found that characteristics associated with greater improvement included being larger, being for profit (and not being a public hospital), being a teaching hospital, and serving a smaller proportion of black patients (eTable 4).
We found that in 2010, while 26% of hospitals in the lowest DSH index quartile scored at or above the median on all 8 HCAHPS measures of patient experience (a key metric under VBP), only 11% of SNHs did so (Table 4). Safety-net hospitals had a 60% lower odds of achieving this benchmark (odds ratio, 0.4; 95% CI, 0.3-0.5; P < .001) than hospitals in the lowest DSH index quartile. Similarly, SNHs were far more likely to fail to score at or above the median on any of the 8 HCAHPS measures (Table 4). The results from multivariate models were similar (eTable 5).
In this national sample of US hospitals, we found that SNHs received lower ratings on measures of patient experience than non-SNHs. The gaps in performance between hospitals in the highest and lowest quartiles of DSH index were sizeable and are increasing over time. These differences, which varied depending on the type of SNH, translated into a striking disparity in the odds of scoring at or above the median on all 8 HCAHPS measures used in pay-for-performance policies, suggesting that these hospitals may fare poorly under the VBP payment scheme.
Our findings have important implications for the health and health outcomes of patients seeking care in these hospitals. Prior studies have shown that hospitals with the highest HCAHPS scores perform approximately 2 to 4 percentage points higher on measures for completion of clinical processes than hospitals with the lowest HCAHPS scores,15 and the differences we observed based on safety-net status were much larger than differences based on any other hospital characteristics we could identify. Patients who receive care in SNHs tend to be of lower socioeconomic status and are more often underrepresented minorities.21 These groups are at higher risk of nonadherence to treatment regimens,22- 24 experience delays in care and longer hospital stays,25 and have been reported to have lower levels of trust in the health care system.6,7 Given their risk profile, these vulnerable patients would likely benefit the most from high-quality patient-centered experience, which has been shown to be associated with several measures of improved health status in many care settings.26,27 Despite the greater need to deliver patient-centered care for vulnerable populations at SNHs, our findings suggest that these hospitals are failing to do so.
The lower performance of SNHs on HCAHPS measures may have meaningful consequences under VBP, particularly since many SNHs operate under very small financial margins.28 While tying payments to quality metrics should promote better care, whether these financially constrained hospitals will be able to respond effectively is unclear.29 There is some evidence that under financial incentives, SNHs are able to improve their performance on processes of care, such as prescribing aspirin to patients with acute myocardial infarction.30 However, improving patients' experience may be substantially more challenging. Although the new VBP rules pay for improvements as well as achievement, our findings suggest that SNHs face challenges on both fronts. If SNHs are unable to substantially improve patients' experiences over the next several years, hospital-based incentive programs are likely to disproportionately penalize these institutions.
While prior studies have demonstrated that SNHs perform poorly on quality metrics based on processes of care,31- 33 to our knowledge, there are no prior studies examining patient experience in SNHs among a nationally representative group of institutions. Although an earlier study showed that hospitals with a higher proportion of Medicaid patients tend to have lower performance on HCAHPS,34 this was at a time when nearly half of hospitals had chosen not to publicly disclose their performance. Our study examines data over a 4-year period and uses a larger, more representative sample of US hospitals.
Our study has limitations. First, while we chose a priori to define “safety-net hospitals” according to DSH index quartiles, this is an imperfect measure.35,36 Nonetheless, we found a consistent relationship between safety-net status and HCAHPS performance even when we used other approaches to identifying SNHs, which was reassuring. Second, we were only able to examine data over a 4-year period and it is possible that hospital efforts to improve their performance may take longer. Tracking how this group of hospitals fares over time will be critically important.
Another limitation is that HCAHPS measures are, by their nature, subjective. Therefore, we cannot differentiate between actual differences in care delivery vs differences in expectations of the underlying patient population. However, we are unaware of any data that suggest that poor patients have higher expectations; we suspect they may actually have lower expectations of service quality. If so, we may be underestimating the true gap in performance between SNHs and other hospitals. The HCAHPS measures are adjusted for patient self-reported health status and survey mode,13 which helps mitigate differences in performance based on variations in the patient population. However, other factors such as case-mix and race may still influence hospital performance on these measures.
In conclusion, we found that US SNHs performed more poorly than other hospitals on nearly every measure of patient experience and that gaps in performance were sizeable and persistent over time. These findings have important consequences for patients who receive care at these institutions and should renew our focus on helping these hospitals improve. Given that hospital payments are now tied to performance on these measures, we need renewed efforts to track performance of SNHs under VBP and may need specific quality-improvement programs targeting these organizations. Safety-net hospitals play a critical role in providing medical care to vulnerable populations, and ensuring that efforts to improve the quality of care at US hospitals do not worsen existing disparities will be a key challenge to policy makers.
Correspondence: Ashish K. Jha, MD, MPH, Department of Health Policy and Management, Harvard Scholl of Public Health, 677 Huntington Ave, Boston, MA 02115 (firstname.lastname@example.org).
Accepted for Publication: May 12, 2012.
Published Online: July 16, 2012. doi:10.1001/archinternmed.2012.3158
Author Contributions: Ms Chatterjee had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Chatterjee, Joynt, and Jha. Acquisition of data: Chatterjee and Jha. Analysis and interpretation of data: Chatterjee, Joynt, Orav, and Jha. Drafting of the manuscript: Chatterjee and Joynt. Critical revision of the manuscript for important intellectual content: Chatterjee, Joynt, Orav, and Jha. Statistical analysis: Chatterjee, Orav, and Jha. Study supervision: Jha.
Financial Disclosure: None reported.
Thank you for submitting a comment on this article. It will be reviewed by JAMA Internal Medicine editors. You will be notified when your comment has been published. Comments should not exceed 500 words of text and 10 references.
Do not submit personal medical questions or information that could identify a specific patient, questions about a particular case, or general inquiries to an author. Only content that has not been published, posted, or submitted elsewhere should be submitted. By submitting this Comment, you and any coauthors transfer copyright to the journal if your Comment is posted.
* = Required Field
Disclosure of Any Conflicts of Interest*
Indicate all relevant conflicts of interest of each author below, including all relevant financial interests, activities, and relationships within the past 3 years including, but not limited to, employment, affiliation, grants or funding, consultancies, honoraria or payment, speakers’ bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued. If all authors have none, check "No potential conflicts or relevant financial interests" in the box below. Please also indicate any funding received in support of this work. The information will be posted with your response.
Chatterjee P, Joynt KE, Orav EJ, Jha AK. Patient experience in safety-net hospitals: implications for improving care and value-based purchasing [published online July 16, 2012. Arch Intern Med.. Published online July 23, 2012. doi:10.1001/archinternmed.2012.3158.
eTable 1. Comparison of Hospital Characteristics Between Reporting and Nonreporting Hospitals
eTable 2. Bivariate Estimates of HCAHPS Scores in 2010
eTable 3. Multivariate regression estimates of HCAHPS scores in 2010
eTable 3a: Predicted Improvements in Performance Over Time, 2007-2010, With DSH Percent as Continuous Predictor, With and Without Controlling for Baseline Performance
eTable 3b. Supplementary methods.
eTable 4. Improvement of SNHs Over Time (n=492 SNHs Reporting in Both 2007 and 2010)
eTable 5. Hospitals Reaching VBP Achievement Thresholds by DSH Quartile (Multivariate Models)
Some tools below are only available to our subscribers or users with an online account.
Download citation file:
Web of Science® Times Cited: 11
Customize your page view by dragging & repositioning the boxes below.
Enter your username and email address. We'll send you a link to reset your password.
Enter your username and email address. We'll send instructions on how to reset your password to the email address we have on record.
Athens and Shibboleth are access management services that provide single sign-on to protected resources. They replace the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session. It operates independently of a user's location or IP address. If your institution uses Athens or Shibboleth authentication, please contact your site administrator to receive your user name and password.