Ensuring the Integrity and Transparency of Public Reports: How a Possible Oversight Model Could Benefit Healthcare

O ver the last decade, the US healthcare system has seen tremendous growth in the public reporting of provider performance, including privately and publicly issued rankings and awards.¹ Further growth is expected with the expansion of value-based purchasing programs in the Medicare program, the introduction of state health exchanges, and the development of accountable care organizations, each with their own public reporting requirements.² The growth in the number of organizations measuring provider performance and the increase in the number of public reports have highlighted the wide range of methodologies used for publicly reporting provider performance.³

While efforts to advance performance measurement should be applauded, performance measurement experts and policy analysts have raised concerns that many measures in public reports, and the public reports themselves, may not meet a standard for quality.⁴ Concerns focus on the validity and reliability of measures used in public reports, which are often unknown or poor, potentially misinforming consumers and discouraging clinicians from using performance measures to improve care.⁵

The process of developing a public report is complex and involves collecting data, calculating performance measures from the data, and constructing the report from the measures. Report sponsors often approach the creation of their reports very differently in terms of definitions, provider attribution, and data sources.⁶ This "Wild West" approach to publicly reporting provider performance has created a public reporting landscape that is highly variable and highlights the lack of standards and transparency.

So why should healthcare stakeholders care about a lack of standards and transparency in public reports? For consumers, the lack of standards is a barrier to their effectively using public reports for decision making: they have no assurances about the integrity of the reports and report findings often conflict with each other.⁷ For healthcare purchasers, the variability in reports is an obstacle to their creating meaningful incentives to recognize quality providers and to encouraging performance improvement. And for providers, the lack of standards challenges them in deciding which performance measures and reports to focus on, and which measures should guide internal improvement efforts. For all, the lack of standards and the transparent reporting of performance against such standards, including a measure’s validity and reliability, require stakeholders to trust that the report sponsors have made appropriate decisions. This trust extends to both traditional reports issued by public and private entities (eg, Hospital Compare, U.S. News & World Report) and to marketing materials issued by healthcare providers themselves (eg, an ad touting "Our doctors are the best," with no data supporting this claim).

The variation in the methodological rigor and the lack of standards in existing public reports suggest a potential need for an oversight process to ensure the scientific integrity of public reports. We have previously discussed the need for an oversight agency that would serve a transparency function, similar to the function of the Securities and Exchange Commission (SEC) for publicly held securities, and for an independent body to set standards, similar to the Federal Accounting Standards Board, to whom the SEC has delegated authority for setting standards for financial reporting.⁸

We believe the public reporting of healthcare data needs to follow the model of reporting financial data with the triad of 1) an independent body to set standards, 2) trained and certified professionals to review and audit compliance with those standards, and 3) an external entity to enforce the standards. This combination would significantly advance the public reporting of provider performance with greater standardization and transparency in the collection of the underlying data, the calculation of performance measures, and the construction of public reports. Such a model would offer a consistent definition of performance on which to report and would hopefully change the conversation from debating the accuracy of performance measures to discussing how well measures perform, gaps in performance, and the marginal costs and benefits of producing better measures.

In conceptualizing a structure that could serve the oversight role, key questions will need to be explored, such as: Should compliance with standards be voluntary or mandatory? Should public reports only include specific measures, or can any measure be included as long as the specifications and scientific properties of the measure are made transparent? Who should have a role in creating the standards? And what will be the financing mechanism for an oversight agency?

Options for an oversight agency range from a strictly voluntary review model on one end to a more formal regulatory model on the other. Each model has its advantages and disadvantages, in terms of its potential efficacy to introduce standards and transparency to public reports, the political will to implement the program, and the financing that would be required. For a more formal regulatory model to work, an entity such as the SEC will likely be needed. In the absence of an SEC-like entity, a standard-setting body alone would likely make little progress.

While other models may exist, government is a likely candidate for the role of standards enforcer. The standard-setting body could be a private entity, if a structured funding source could be identified. The work the National Quality Forum has done to date with endorsing performance measures serves as a strong model for the setting of standards. Additional structures may be needed to support this model, including professionals in the mold of certified public accountants, who would be certified to offer public attestation on healthcare quality data. Currently, the skills and training of individuals who collect and report provider performance data vary widely, with few having any formal training in measurement science.

One tension with developing standards, especially at a local level, is maintaining the ability to measure performance while not stifling innovation. A potential concern of reporting being overseen at a national level is that reporting will become a “one-size-fits-all” approach, which is problematic given the diverse needs of widely varying communities. To minimize the concern, numerous regional collaboratives have formed in the United States to produce public reports for a single state or a region within a state. These collaboratives have proved successful in providing communities the opportunity to tailor reports to meet local needs and priorities. And they should continue. However, they do not provide a uniform national data collection and measurement process.⁹ The end goal of this effort is to have standards for public reports, not standard public reports.

The ideas presented here are intended to serve as a starting point for further conversation. We recognize that much more needs to be considered and will require broader conversations across all stakeholders on the strengths and weaknesses of our current reporting system and the concerns that a new structure would present. While no one structure can address all the concerns of all groups, the principles of transparency and inclusiveness will be extremely important to promote buy-in of a final model.

We recommend that policy makers establish a multistakeholder commission that would evaluate different structures. This commission would report the results of their evaluation back to policy makers who, in turn, could make an informed decision on the best path for moving forward. Such an initiative will surely be fraught with challenges and controversy. Nevertheless, policy makers have been successful in garnering broad support for the recognition of performance measurement as an important component for improving performance and enhancing accountability in healthcare. Now they need to ensure the performance reports are up to the task.Source of Funding: None.

Author Disclosures: Dr Austin discloses grant or contract support from The Leapfrog Group (research on hospital performance measurement); the Agency for Healthcare Research and Quality (research on consumer-centered public reporting and performance measurement); and The Commonwealth Fund (research on intensive care unit performance measures). Dr Young discloses serving as a member on a National Quality Forum expert panel. Dr Pronovost discloses grant or contract support from the Agency for Healthcare Research and Quality; the Gordon and Betty Moore Foundation (research related to patient safety and quality of care); the National Institutes of Health (acute lung injury research); and the American Medical Association (on improving blood pressure control). Dr Pronovost also discloses consulting fees from the Association of Professionals in Infection Control and Epidemiology, Inc; honoraria from various hospitals, health systems, and the Leigh Bureau to speak on quality and patient safety; book royalties from the Penguin Group; and board membership to the Cantel Medical Group.

Address correspondence to: J. Matthew Austin, PhD, Armstrong Institute for Patient Safety and Quality, Johns Hopkins Medicine, 750 E Pratt St, 15th Fl, Baltimore, MD 21202. E-mail: jmaustin@jhmi.edu.REFERENCES

1. Jha AK. Hospital rankings get serious. An Ounce of Evidence / Health Policy blog. http://blogs.sph.harvard.edu/ashish-jha/hospital-rankings-get-serious/. Published August 14, 2012. Accessed February 8, 2014.

2. ACO Public Reporting Guidance. CMS website. http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/sharedsavingsprogram/Downloads/ACO-Public-Reporting-Guidance.pdf. Updated September 16, 2014. Accessed December 2, 2014.

3. Berenson RA, Pronovost PJ, Krumholz HM. Achieving the potential of health care performance measures. http://www.rwjf.org/content/dam/farm/reports/reports/2013/rwjf406195. Published May 2013. Accessed February 8, 2014.

4. Damberg CL, Hyman D, France J. Do public reports of provider performance make their data and methods available and accessible? Med Care Res Rev. 2014;71(5 suppl):81S-96S.

5. Scheurer D. Consumer Reports’ hospital quality ratings dubious. The Hospitalist website. http://www.the-hospitalist.org/details/article/5026531/Consumer_Reports_Hospital_Quality_Ratings_Dubious.html. Published August 1, 2013. Accessed February 8, 2014.

6. Luft HS. Advancing public reporting through a new ‘aggregator’ to standardize data collection on providers’ cost and quality. Health Aff (Millwood). 2012;31(3):619-626.

7. Shahian DM, Wolf RE, Iezzoni LI, Kirle L, Normand SL. Variability in the measurement of hospitalwide mortality rates. N Engl J Med. 2010;363(26):2530-2539.

8. Mathews SC, Pronovost PJ, Herzlinger RE. Focus on quality: an opportunity to execute health care reform. Am J Med Qual. 2011;26(3):239-240.

9. Young GJ. Multistakeholder regional collaboratives have been key drivers of public reporting, but now faces challenges. Health Aff (Millwood). 2012;31(3):578-584.

Download PDF: Ensuring the Integrity and Transparency of Public Reports: How a Possible Oversight Model Could Benefit Healthcare