11 Oct Hospital Ranking Systems Need Better Validation and Transparency
MedicalResearch.com Interview with:
Bala N. Hota, MD
Department of Internal Medicine
Rush Medical College
Chicago, IL
MedicalResearch.com: What is the background for this study?
Response: There has been a proliferation of online ranking systems that seek to rate the quality of health care systems. Rush University Medical Center (RUMC) has consistently scored highly on patient safety measurement systems. For example, RUMC has received nine consecutive “A” grades for safety from the Leapfrog Group, and was recently ranked “4 stars” in the CMS 2016 star ranking system.
In the 2015-2016 US News and World Report Ranking System, however, Rush received a low score for quality, a 1 out of 5 possible points, which was a surprise. To understand these results, the RUMC quality team began a process to validate the data and methods of the US News hospital ranking system.
What was found was a surprise – the data backing the ranking system produced by US News had flaws, leading to the low score. Specifically, data were missing from the data set used for the US News analysis, including whether a condition was present on admission in 10% of cases; dates of service were also missing from all cases. The quality team at RUMC then conducted an analysis using national data, and simulated the impact of these data flaws on national rankings.
MedicalResearch.com: What are the main findings?
Response: In this sensitivity analysis, RUMC found that the gaps in the US News and World Report data set would lead to bias in the published rankings, with a negative skew of results in hospitals with higher transfer or admission rates.
As a result of these findings and others, US News and World Report changed their methodology for the 2016-2017 ranking year. RUMC noted an increase from a 1 to a 4 out of 5 in its safety score with the changes.
MedicalResearch.com: What should readers take away from your report?
Response: It is critical that health systems and quality teams develop expertise in the validation of publicly reported data. Transparency in the methods and data used for benchmarking would also add more rigor to the measurement of quality. Small deficits in the data used for these measures can lead to big challenges in the resulting benchmarked rates.
MedicalResearch.com: What recommendations do you have for future research as a result of this study?
Response: Further study of the impact of bias on the analytic results of benchmarking would help to understand the situations where publicly reported rates have inherent flaws. In addition, an evaluation of the understanding among health care consumers of the limitations of these methods would also be of interest.
MedicalResearch.com: Is there anything else you would like to add?
Response: Better validation of and transparency in the methods used to benchmark institutions, and the data used for these approaches, would be instrumental in getting to the ultimate goal: measuring and improving quality.
MedicalResearch.com: Thank you for your contribution to the MedicalResearch.com community.
Citation:
Consumer Rankings and Health Care: Toward Validation and Transparency
Authors: Hota, Bala; Webb, Thomas A.; Stein, Brian D.; Gupta, Richa; Ansell, David; Lateef, Omar
Source: The Joint Commission Journal on Quality and Patient Safety, Volume 42, Number 10, October 2016, pp. 439-446(8)
Note: Content is Not intended as medical advice. Please consult your health care provider regarding your specific medical condition and questions.
More Medical Research Interviews on MedicalResearch.com
[wysija_form id=”5″]
Last Updated on October 11, 2016 by Marie Benz MD FAAD