Use of Patient-Reported Outcomes and Satisfaction for Quality Assessments

ABSTRACT

Objectives: Recent focus on patient-reported outcomes (PROs) has created a new challenge as we learn how to integrate these outcomes into practice along with other quality metrics. We investigated the relationship between PROs and satisfaction among spine surgery patients. We hypothesized that there would be significant disparities between patient satisfaction and PROs at the 1-year postoperative time point.

Study Design: Retrospective cohort study of adults undergoing elective lumbar spine surgery at 12 hospitals participating in the Spine Surgical Care and Outcomes Assessment Program.

Methods: Satisfaction, pain, and function scores were collected at 1 year post operation, along with clinical information, to determine the relationship between PROs and satisfaction at the patient level.

Results: Among 520 patients (mean age = 63 ± 13 years; 47% male), the majority of patients (82%) reported being satisfied with surgery. Satisfaction was associated with both improvement in pain (odds ratio [OR], 1.33; 95% CI, 1.17-1.51) and function (OR, 1.06; 95% CI, 1.04-1.08). However, even among patients who did not improve in pain or function, more than half (59%) reported being satisfied.

Conclusions: Overall, patients undergoing elective lumbar spine surgery reported being satisfied with outcomes, but the reported responses in PROs were much more variable. As the expectations increase to include PRO measures as valid quality indicators, it is necessary to dedicate time and consideration to understanding the relationships among these measures to support meaningful translations into healthcare policy.

Am J Manag Care. 2017;23(10):618-622Takeaway Points

The findings from this study indicate that the quality profile of a hospital is highly dependent on the domain of measurement.

• Recent focus on patient-reported outcomes (PROs) has created a new challenge as we learn how to integrate these outcomes into practice along with other quality metrics.
• Understanding the relationships among PROs, satisfaction, and quality is the first step in drawing meaningful conclusions that can then be translated into policy.

Traditionally, clinical outcomes, such as the rates of adverse events or mortality, served as the metrics by which we measured treatment success or failure. Although these objective measures of quality allow for comparisons across hospitals and physicians, the sole use of clinical outcomes may provide an incomplete appraisal of healthcare quality. Within the last few years, payers, policy makers, and healthcare quality organizations focused on patient-reported outcomes (PROs) as important markers of quality distinct from assessments of patient satisfaction and clinical outcomes.^1-4 PROs are defined as measures of health that are directly reported by the patient, rather than interpreted by a clinician or reported in the medical record.⁵ This newer metric is becoming financially relevant as a result of the Merit-Based Incentive Payment System (MIPS), passed as part of the Medicare Access and CHIP Reauthorization Act of 2015.⁶ PROs have been proposed by policy makers and payers as a required component of quality assessments and reimbursement schedules.^7,8 As clinicians and hospitals are charged with adding PRO measures to established clinical and satisfaction metrics, a challenge exists to integrate PROs in a manner that both improves patient care and provides a meaningful assessment of quality.

Many PRO scores assess a patient’s perception of improvement after an intervention, but others aim to quantify a patient’s level of satisfaction with their care. However, it is not clear that these patient satisfaction measures correlate with the delivery of safe and effective care as defined by clinicians and caregivers. Satisfaction surveys are often criticized for focusing on the “wrong” outcomes, such as the hospital noise level or the variety of food choices offered to patients, rather than on the clinical result of a treatment or intervention.⁹ Patient satisfaction also may not reflect the total patient experience, as one patient might report high levels of satisfaction with their overall care despite a poor clinical outcome¹⁰ and another patient might report dissatisfaction with their care while simultaneously benefiting from a positive clinical outcome.

Spine surgery represents 1 relevant scenario in which integration of both PRO and clinical outcome measures improves the holistic view of an individual’s outcome following surgery. Functional impairment and pain are often the primary drivers for patients considering spine surgery; therefore, understanding which patients are most likely to see improvement in these symptoms can inform the decision to be made between a surgical and a nonsurgical option. Using PROs to inform direct patient care versus measures for assessing changes in a patient population is increasingly called for in today’s healthcare environment.^11,12 In Washington state, healthcare providers are experiencing demands to implement and interpret PROs in spine surgery due to recent policy recommendations from the Bree Collaborative, proposed in 2015. The Bree Collaborative calls for the use of the Oswestry Disability Index (ODI) and the Patient-Reported Outcomes Measurement Information System (PROMIS) 10, measures of functional and general quality of life, respectively, for all patients undergoing single-level lumbar spine fusion procedures. This policy is intended to allow appropriate selection of surgical candidates from the vast population of adults with low back pain and to limit expensive surgical interventions to those patients most likely to benefit from the procedure.¹³

Given the differences between PROs, which typically focus on functional ability and pain, and patient satisfaction measures, it is not clear that one or the other necessarily provides a true measure of outcome or that these measures are robust enough to allow for policy and reimbursement decisions. To address this, we investigated the relationship between PRO measures and patient satisfaction at 1 year following spine surgery. We hypothesized that there would be significant disparities between patient satisfaction and PRO measures, suggesting that the use of satisfaction scores as a primary measure of quality is flawed.

METHODS

The Comparative Effectiveness Research Translation Network (CERTAIN) is a research and analytic platform aligned with the Spine Surgical Care and Outcomes Assessment Program (Spine SCOAP).^11,14,15 Spine SCOAP is a collaborative of hospitals within Washington state whose mission is to aggregate data from hospitals, healthcare providers, and patients to generate evidence-based best-practice measures that can be dispersed among the collaborative to improve the care of patients in the state and beyond. We conducted a retrospective cohort study of adults undergoing elective lumbar spine surgery from 2012 to 2014 at 12 Spine SCOAP hospitals who also participated in CERTAIN PRO data collection.

Eligible participants completed a survey that included measures of pain, function, and satisfaction both before surgery and at 1 year after. Pain was measured using a 10-point numeric rating scale with “0” representing “no pain” and “10” representing “worst pain possible.” Function was measured using the ODI, a composite measure that grades function on a scale of 0 to 100, with higher scores indicating worse disability.¹⁶ Improvement in pain or function was established by a minimally clinically important difference between baseline and follow-up (for pain, a change of at least 2 points from baseline; for function, a change of at least 15 points from baseline).¹⁷ We measured satisfaction using a single question: “How satisfied are you with the overall result of your spine operation so far?” The satisfaction question was scored using a 4-point modified Likert scale (Table 1).

The University of Washington Human Subjects Division approved the retrospective review of these data. All analyses were carried out using Stata version 11 (STATA Corp; College Station, Texas).

We analyzed patient-reported satisfaction, pain, and function at 1 year post operation to determine the association between patient characteristics and a positive outcome in any or all domains. Patient characteristics were obtained through Spine SCOAP and included information about sociodemographics, underlying diagnosis (eg, spinal stenosis, herniated disc), and operative characteristics (eg, fusion). An invasiveness index based on the type of intervention at each vertebral level as well as the number of operated levels was included, as this has been shown to be correlated with clinical outcomes in spine surgery patients.^18,19 The American Society of Anesthesiologists Physical Classification System score was included as an additional measure of physical status at the time of the operation. This is a commonly used score applied to every patient undergoing surgery and is intended to provide an additional description of patient risk.²⁰ We selected the 1-year postoperative time point because PROs and satisfaction in the early postoperative period are not necessarily reflective of findings at 1 year,^21,22 which are felt to be a more reliable indicator of long-term satisfaction.

To determine the relationship among the 3 domains, we used a Poisson regression model stratified by pain and function improvement that was predictive of satisfaction (somewhat or very satisfied) at 12 months following spine surgery. A subset of patients (n = 69; 13%) did not have clinical information available through Spine SCOAP and was excluded from the regression analysis. To account for the fact that other patient characteristics are known to correlate with overall satisfaction,²³ we included sociodemographic factors, diagnostic factors, and operative characteristics as measured by SCOAP in the regression model. The resulting model was clustered by hospital to account for correlated data. In cases such as this, where multiple observations are drawn from the same hospital, there is a concern that these results from within each individual hospital are correlated due to extrinsic factors that are not measured and that this effect varies between hospitals. Failure to account for correlation can result in a biased estimate and an incorrect inference.

RESULTS

At the 1-year follow-up point, 520 patients (mean age = 63 ± 13 years; 47% male) had complete baseline and follow-up survey data and 451 (87%) had clinical information available from Spine SCOAP. More than half of patients reported that they were very satisfied (n = 301; 58%) while one-fourth of patients reported that they were somewhat satisfied (n = 124; 24%) with their spine surgery result. Table 2 demonstrates the interaction among patient-reported pain and function improvement and patient satisfaction. Patients with a higher magnitude of pain improvement more often reported that they were somewhat or very satisfied: 62% patients with no pain improvement or worse pain were satisfied, whereas 96% of those with 5 or more points of pain improvement were satisfied. We found a similar trend in satisfaction and function: 61% of those with no functional improvement or worse function reported being satisfied and 80% of those whose function improved more than 30 points reported being satisfied. Interestingly, even among patients who had no change in their symptoms or reported that they were worse, 59% of patients were still somewhat or very satisfied.

In our stratified analysis, adjusting for patient characteristics, patients who had no improvement in back pain were 1.7 times (95% CI, 1.27-2.16) and 2.3 times (95% CI, 1.71-3.11) more likely to be satisfied if they experienced a minor improvement in function (ODI score reduced 15-29 points) or major improvement in function (ODI score reduced ≥30 points), respectively, compared with patients who had no improvement or worse function. Of the patients with improved back pain at 1 year, a mild improvement in function was associated with 1.3 times (95% CI, 1.00-1.71) the likelihood of being satisfied, and a major improvement in function was associated with 1.4 times (95% CI, 1.08-1.81) the likelihood of being satisfied compared with patients who had no improvement or worse function outcomes (Table 3).

DISCUSSION

In this analysis, we sought to determine how PRO measures compare with other quality measures, such as patient-reported satisfaction. We analyzed data from a statewide clinical registry encompassing a collaborative of hospitals in Washington state that allowed for an assessment of both hospital-level and patient-level outcomes. The majority of patients reported being satisfied with their spine surgery result, and we found a strong association between patient-reported improvements in pain and function and satisfaction. Despite this understandable finding, more than half of the patients with no improvement or worse outcomes in pain or function were satisfied with their surgery. This incongruence suggests that satisfaction ratings may be based on nonclinical aspects of care not captured by this survey. Reasons for this incongruence may be related to the patient’s perception that their care was especially respectful or well coordinated. Conversely, there was a group of patients who did improve but were not satisfied, which may indicate that they had other expectations beyond pain or functional improvement. We can conclude that the “quality” profile of a hospital is highly dependent on the domain of measurement and that understanding the items measured by a particular instrument is critical for interpretation of results.

Previous studies have explored the relationship between patient satisfaction and other outcome measures (eg, clinical outcomes, patient-reported functional outcomes), but conclusions have varied among studies. In one prospective study, Godil et al reported that patient satisfaction measures at 90 days post operation were not correlated with clinical outcomes, such as complications and readmissions, and as such, they should not be used as a proxy for overall quality.²⁴ These findings were confirmed in a subsequent review article.⁸ The time frame of measurement is important, as well: PROs 3 months post operation are not necessarily predictive of PROs at 1 year.^21,22 This latter finding may be clinically relevant as satisfaction metrics are frequently measured at time points close to the date of care delivery, rather than at later time points, which might better reflect long-term improvements or outcomes. These studies highlight the unresolved issues related to the use of PROs and satisfaction data in quality assessments.

More generally, there is evidence that satisfied patients are more likely to utilize healthcare resources and thereby incur costs, but it is not clear that the increased healthcare utilization is related to better outcomes.²⁵ Among surgical patients, one study found no association between satisfaction and measures of quality as defined by the Surgical Care Improvement Program, such as adequate antibiotic prophylaxis before surgery,²⁶ again pointing to the concern that patients base satisfaction ratings on nonclinical factors that are not reflected in clinical or administrative data. Our analysis found a correlation between patient-reported improvement in pain and function and satisfaction in the patient-level analysis, but there was a relatively large proportion of individuals who reported being somewhat or very satisfied who also had a negative outcome in either pain or function. Potential reasons for this include appreciation of the nursing care that the patient received, the organization of their care, or attentiveness of their care team as a whole. Future work should focus on understanding the domains of care that are most correlated with satisfaction.

Limitations

The survey utilized in this study was created under the auspices of quality improvement to help surgeons and hospitals understand more about their patients. The evidence is still emerging as to the complete set of questions that are relevant to support patient- and hospital-level decision making, as well as to provide quality assessments. One limitation of this analysis is the potential inability to capture all domains that are important to patients, and thus it may be biased to those areas deemed clinically relevant to the survey creators. A second limitation is the restriction of the analysis to lumbar surgery patients alone, meaning the results may not be reflective of all clinical conditions. We encourage future research endeavors that focus on how measures of the patient experience correlate in other health conditions. Because this analysis included only those with complete data, these results may not be representative of outcomes from nonresponders. We do not have information regarding nonresponders, and it may be that patients who respond to surveys are systematically different from those who do not. It may also be that improvement in pain or function is representative of better patient selection. Although this is outside the scope of this research, future work should focus on understanding which patients will benefit most from spine surgery. Finally, the survey supported through CERTAIN is currently only available in English, which may further bias our results.

CONCLUSIONS

In Washington state, current policy and payment recommendations require health systems to capture PRO data for patients undergoing lumbar spine surgery. It is imperative that we understand how to interpret this PRO information to ensure that patients get the right care at the right time. With the ongoing concerns about PRO implementation, collection, interpretation, and application,^27,28 it is necessary to focus on how these important data can inform decisions about patient care. Successful implementation depends not only on the available infrastructure for collection and analysis, but also on the commitment of healthcare providers to the concept that PROs add value to the ongoing quality discussion. Without this focus, we risk more measurement and more data without the ability to transform patient care. There is considerable work to be done before we fully understand the appropriate role of PROs in healthcare.

The implications of this work are far-reaching, especially with the recent passage of MIPS, which emphasizes quality and value over volume.⁶ Results such as this may serve to inform policy makers who are deciding on the most appropriate measures to use in quality assessments. Understanding the relationships among PROs, satisfaction, and general quality is the first step in drawing meaningful conclusions that can then be translated to policy. From a patient’s perspective, our study is important because it may help patients better interpret the significance of a particular physician or hospital having high satisfaction ratings. Furthermore, this work requires the inclusion of patients, providers, payers, and policy makers to ensure that we are selecting the appropriate domains on which to judge quality.Author Affiliations: Department of Surgery (APE, DRF, DCL), and Surgical Outcomes Research Center (SK), and Department of Orthopaedics and Sports Medicine (AMC), University of Washington, Seattle, WA; Neuroscience Institute, Virginia Mason Medical Center (J-CAL), Seattle, WA; Rainer Orthopedic Institute, Proliance Surgeons Inc (NSS), Puyallup, WA; Swedish Neuroscience Institute, Swedish Medical Center (RJO), Seattle, WA.

Source of Funding: Dr Ehlers was supported by a training grant from the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health under Award Number T32DK070555. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Data used for this project were collected with grant number R01HS020025 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. The Surgical Care and Outcomes Assessment Program (SCOAP) is a Coordinated Quality Improvement Program of the Foundation for Health Care Quality. CERTAIN is a program of the University of Washington, the academic research and development partner of SCOAP.

Author Disclosures: Dr Cizik is currently supported by a grant from the NIH (grant number R21AR068009) investigating nonresponse in spine surgery. The data included in this manuscript submission are not reflective of any data collected as part of the disclosed study. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article. Portions of this work were presented at Concordium (September 2015; Washington, DC) and the International Society for Quality of Life Annual Conference (October 2015; Vancouver, BC, Canada).

Authorship Information: Concept and design (APE, SK, NSS, DRF, DCL); acquisition of data (SK, NSS, RJO, DRF, DCL); analysis and interpretation of data (APE, SK, AMC, J-CAL, NSS, RJO, DRF, DCL); drafting of the manuscript (APE, J-CAL, DRF, DCL); critical revision of the manuscript for important intellectual content (APE, SK, AMC, J-CAL, DRF, DCL); statistical analysis (SK, DCL); provision of patients or study materials (NSS, RJO); obtaining funding (DRF, DCL); administrative, technical, or logistic support (AMC, NSS, RJO, DCL); and supervision (AMC, NSS, DCL).

Address Correspondence to: Anne P. Ehlers, MD, MPH, University of Washington, 1107 NE 45th St, Ste 502, Box 354808, Seattle, WA 98195. E-mail: apugel@uw.edu. REFERENCES

1. 2012 Annual Progress Report to Congress: National Strategy for Quality Improvement in Health Care. Agency for Healthcare Research and Quality website. https://www.ahrq.gov/workingforquality/reports/2012-annual-report.html. Published April 2012. Accessed May 2016.

2. Patient Protection and Affordable Care Act, 2010. Government Publishing Office website. http://www.gpo.gov/fdsys/pkg/PLAW-111publ148/pdf/PLAW-111publ148.pdf. Accessed May 2016.

3. Institute of Medicine; Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press; 2001. doi: doi.org/10.17226/10027.

4. NQF-endorsed measures for person- and family-centered care: phase I technical report. National Quality Forum website. http://www.qualityforum.org/Publications/2015/03/Person-_and_Family-Centered_Care_Final_Report_-_Phase_1.aspx. Published March 4, 2015. Accessed May 2016.

5. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims. FDA website. https://www.fda.gov/downloads/drugs/guidances/ucm193282.pdf. Published December 2009. Accessed May 2016.

6. H.R.2.—Medicare Access and CHIP Reauthorization Act of 2015. Congress.gov website. https://www.congress.gov/bill/114th-congress/house-bill/2. Accessed May 2016.

7. HCAHPS fact sheet. Hospital Consumer Assessment of Healthcare Providers and Systems website. http://www.hcahpsonline.org/files/August%202013%20HCAHPS%20Fact%20Sheet2.pdf. Published August 2013. Accessed February 2016.

8. Tevis SE, Schmocker RK, Kennedy GD. Can patients reliably identify safe, high quality care? J Hosp Adm. 2014;3(5):150-160.

9. Casarett D. Lessons in end-of-life care from the V.A. The New York Times website. https://opinionator.blogs.nytimes.com/2015/11/11/lessons-in-end-of-life-care-from-the-v-a/. Published November 11, 2015. Accessed May 2016.

10. Robbins A. The problem with satisfied patients. The Atlantic website. http://www.theatlantic.com/health/archive/2015/04/the-problem-with-satisfied-patients/390684/. Published April 17, 2015. Accessed May 2016.

11. Lee MJ, Shonnard N, Farrokhi F, et al; Spine SCOAP-CERTAIN Collaborative. The Spine Surgical Care and Outcomes Assessment Program (Spine SCOAP): a surgeon-led approach to quality and safety. Spine (Phila Pa 1976). 2015;40(5):332-341. doi: 10.1097/BRS.0000000000000750.

12. Weinstein JN, Lurie JD, Tosteson TD, et al. Surgical vs nonoperative treatment for lumbar disk herniation: the Spine Patient Outcomes Research Trial (SPORT) observational cohort. JAMA. 2006;296(20):2451-2459.

13. Spine/low back pain topic: report & recommendations. The Bree Collaborative website. http://www.breecollaborative.org/wp-content/uploads/spine_lbp.pdf. Published November 2013. Accessed May 2016.

14. Devine EB, Alfonso-Cristancho R, Devlin A, et al; CERTAIN Collaborative. A model for incorporating patient and stakeholder voices in a learning health care network: Washington State’s Comparative Effectiveness Research Translation Network. J Clin Epidemiol. 2013;66(suppl 8):S122-S129. doi: 10.1016/j.jclinepi.2013.04.007.

15. Flum DR, Fisher N, Thompson J, Marcus-Smith M, Florence M, Pellegrini CA. Washington state’s approach to variability in surgical processes/outcomes: Surgical Clinical Outcomes Assessment Program (SCOAP). Surgery. 2005;138(5):821-828. doi: 10.1016/j.surg.2005.07.026.

16. Fairbank JC, Pynsent PB. The Oswestry Disability Index. Spine (Phila Pa 1976). 2000;25(22):2940-2952; discussion 2952.

17. Parker SL, Adogwa O, Paul AR, et al. Utility of minimum clinically important difference in assessing pain, disability, and health state after transforaminal lumbar interbody fusion for degenerative lumbar spondylolisthesis. J Neurosurg Spine. 2011;14(5):598-604. doi: 10.3171/2010.12.SPINE10472.

18. Cizik AM, Lee MJ, Martin BI, et al. Using the spine surgical invasiveness index to identify risk of surgical site infection: a multivariate analysis. J Bone Joint Surg Am. 2012;94(4):335-342. doi: 10.2106/JBJS.J.01084.

19. Mirza SK, Deyo RA, Heagerty PJ, Turner JA, Lee LA, Goodkin R. Towards standardized measurement of adverse events in spine surgery: conceptual model and pilot evaluation. BMC Musculoskelet Disord. 2006;7:53. doi: 10.1186/1471-2474-7-53.

20. ASA Physical Classification System. American Society of Anesthesiologists website. https://www.asahq.org/resources/clinical-information/asa-physical-status-classification-system. Published October 15, 2014. Accessed May 2016.

21. Parker SL, Asher AL, Godil SS, Devin CJ, McGirt MJ. Patient-reported outcomes 3 months after spine surgery: is it an accurate predictor of 12-month outcome in real-world registry platforms? Neurosurg Focus. 2015;39(6):E17. doi: 10.3171/2015.9.FOCUS15356.

22. McGregor AH, Doré CJ, Morris TP. An exploration of patients’ expectation of and satisfaction with surgical outcome. Eur Spine J. 2013;22(12):2836-2844. doi: 10.1007/s00586-013-2971-6.

23. Bible JE, Kay HF, Shau DN, O’Neill KR, Segebarth PB, Devin CJ. What patient characteristics could potentially affect patient satisfaction scores during spine clinic? Spine (Phila Pa 1976). 2015;40(13):1039-1044. doi: 10.1097/BRS.0000000000000912.

24. Godil SS, Parker S, Zuckerman SL, et al. Determining the quality and effectiveness of surgical spine care: patient satisfaction is not a valid proxy. Spine J. 2013;13(9):1006-1012. doi: 10.1016/j.spinee.2013.04.008.

25. Fenton JJ, Jerant AF, Bertakis KD, Franks P. The cost of satisfaction: a national study of patient satisfaction, health care utilization, expenditures, and mortality. Arch Intern Med. 2012;172(5):405-411. doi: 10.1001/archinternmed.2011.1662.

26. Lyu H, Wick EC, Housman M, Freischlag JA, Makary MA. Patient satisfaction as a possible indicator of quality surgical care. JAMA Surg. 2013;148(4):362-367. doi: 10.1001/2013.jamasurg.270.

27. Bilimoria KY, Cella D, Butt Z. Current challenges in using patient-reported outcomes for surgical care and performance measurement: everybody wants to hear from the patient, but are we ready to listen? JAMA Surg. 2014;149(6):505-506. doi: 10.1001/jamasurg.2013.5285.

28. Lavallee DC, Chenok KE, Love RM, et al. Incorporating patient-reported outcomes into health care to engage patients and enhance care. Health Aff (Millwood). 2016;35(4):575-582. doi: 10.1377/hlthaff.2015.1362.

Download PDF: Use of Patient-Reported Outcomes and Satisfaction for Quality Assessments