Consumer Rating Systems for Hospitals May Use Faulty Data Sets and Metric

More transparency and validation is needed for consumer-based benchmarking methods, according to a new study in Joint Commission Journal on Quality & Patient Safety. The study found discrepancies in data and high false-positive event rates, problems that especially affect high-transfer and high-volume hospitals.

After Rush University Medical Center in Chicago received the lowest ranking (1 out of 5) for patient safety in the 2015-2016 US News & World Report “Best Hospitals” rankings, Bala Hota, MD, MPH, and coauthors at Rush University Medical Center compared the Patient Safety Indicators (PSIs) used by US News with findings derived from Rush University Medical Center’s own internal billing data as well as sensitivity analyses using a simulated data set derived from the Healthcare Cost and Utilization Project state inpatient data sets.

The researchers reported that discrepancies were found for PSIs for pressure ulcer rates, perioperative hemorrhage or hematoma rate, and postoperative respiratory failure rate in the US News & World Report’s data compared with Rush University Medical Center’s data: an excess of 0.72, 0.63, and 0.26 cases of these three PSIs, respectively. The sensitivity analysis, which included missing present on admission (POA) flags and dates, resulted in an increase of rates by 1.83 (pressure ulcer rate), 2.72 (perioperative hemorrhage or hematomas), and 3.89 (postoperative respiratory failure) PSIs.

“People have argued and theorized that if you take care of a lot of very sick people or have a lot of transfers, then the data could be biased,” Hota said in a statement. “This study proves that. It shows that data is not perfect, and misunderstandings in the data can negatively portray a hospital.”

The researchers noted that their study’s findings as well as concerns raised by others resulted in US News & World Report in 2016 making changes to its methodology and data sources for the future issues of the magazine’s Best Hospitals rankings. The 2016-17 US News & World Report’s hospital ranking issue, released in August 2016, stated that the rankings help patients make more informed healthcare decisions, comparing nearly 5000 medical centers nationwide in 25 specialties, procedures, and conditions.

According to Becker’s Hospital Review, the new US News rankings methodology places less emphasis on the Agency for Healthcare Research and Quality’s (AHRQ) PSIs when assessing hospitals for quality. Other changes include exclusion of the PSI measure for pressure ulcers when assessing a hospital for safety because of billing inaccuracies related to this PSI, and a change in which the patient safety score’s weighted influence on overall quality will be reduced from 10% to 5%. The 5% weight reduction from PSIs will be offset with added weight to risk-adjusted survival, increasing this measure’s significance from 32.5% to 37.5%.

Hota believes that the rules used by hospital ratings systems are too complex for most consumers to navigate, which is why he and his colleagues urge ratings systems, such as US News to be more transparent about hospital quality ratings. In the meantime, Rush plans to continue measuring, tracking, and sharing patient safety data.

In an editorial that accompanied the study, authors David M. Shahian, MD, Elizabeth Mort, MD, MPH, and Peter J. Pronovost, MD, PhD, concluded, “Just as healthcare providers have ethical and moral responsibilities to the public they serve, rating organizations and journalists that grade providers have similar obligations—in their case, to ensure measure validity and methodological transparency.”