As a certified data junkie, I have spent a considerable amount of time studying the composition of the CMS Overall Hospital Quality Star Rating program. While I understand and appreciate what CMS is trying to provide patients and the public at large, I am not a fan of the current methodology, which I feel doesn’t result in an accurate reflection of the quality of care hospitals are delivering across our country.
I am not the only one who sees the flaws in the methodology. CMS acknowledged the issues in its calculations and had pledged to improve. With that promise in mind, I was excited to see the news that CMS had made updates to the methodology. My excitement was short-lived.
The recent publicly released CMS Hospital Compare in October does not correct any of the criticisms leveled with the initial release and continues to illustrate and validate concerns previously noted.
- Academic medical centers (AMCs) and complex teaching hospitals are disproportionately represented as “Poorly Performing Hospitals” (one or two stars out of five)
- Unique AMC patient population characteristics were not considered. Characteristics such as multiple co-morbidities, acute patient transfers to a higher level of care and low socioeconomic status were not adequately accounted for or risk adjusted in key measures such as readmissions and mortality.
- Improvement efforts hospitals made in April 2016 in key measure groups (roughly 64 percent of all groups) would not be reflected to the public for two more years due to data latency
The data for only about one half of the metrics has been updated by one quarter but continues to lag by a minimum of one year and up to two years. The annually updated metrics are now an additional three months delayed, giving even less of a contemporary picture.
The latent variable modeling caused significant change in weighting of 10 out of 64 metrics (more than 15 percent of the metrics), so the same performance yields a different score. Imagine if you take a test and the exact same questions are weighted differently. One’s grade could be impacted drastically. Metrics with significantly increased weights included 30-day hospital readmissions, HCAHPS Patient Satisfaction “likely to recommend,” 30-day stroke readmission rate and a venous thromboembolism core measure. Some hospital-acquired infections and the hip/ knee replacement complication measure weights were significantly decreased.
The overlap of the same condition being counted in more than one metric persists and in some cases becomes worse with the weighting changes in areas such as readmissions.
As expected, hospital star ratings changed more due to flawed methodology and less to changes in performance. The changes due to actual performance reflect changes that occurred more than one year ago due to the data lag.
I remain hopeful that CMS will continue to refine the methodology and make the ratings a more useful tool for consumers to inform decisions and for hospitals to improve quality. While we wait, my recommendation is for hospitals to continue to focus on areas where they consistently under-perform rather than change strategies to address issues that surfaced based on this single update.
