The Impact of 1-Star Physician Ratings on New Patient Volume

ABSTRACT

Objectives: To quantify the impact of 1-star reviews across multiple physician rating websites (PRWs) on new patient volume.

Study Design: Retrospective analysis of 1.12 million new patient appointments and 12,882 physician reviews from a proprietary data set from a large single-specialty practice in the New York and New Jersey area.

Methods: We compiled new patient appointments scheduled and kept between January 1, 2015, and April 25, 2018, and the reviews of the practice’s affiliated physicians from 10 PRWs. Assuming that reviews are read prior to appointment creation, an ordinary least squares regression model was run with a time series analysis to compare patient volume in the period immediately prior to the posting of a 1-star review with patient volume in the period immediately after a 1-star review was posted. An additional sensitivity analysis was performed at 4, 6, 8, 10, 12, 14, and 16 weeks to validate a robust effect.

Results: The majority of reviews on PRWs were overwhelmingly positive, with only 6.7% of reviews (n = 733) rating a physician with 1 star. A mean of 6.2 new patient appointments were made per half-day session. The mean new patient volume decreased 2.3% to 2.6% following a 1-star review, with effects of the 1-star review affecting patient volume for at least 16 weeks.

Conclusions: Given the limited yet longitudinal negative impact of 1-star reviews and the growing influence of PRWs, physicians should consider the magnitude of the effect as they consider responding to bad reviews.

Am J Manag Care. 2023;29(10):528-531. https://doi.org/10.37765/ajmc.2023.89441

Takeaway Points

A retrospective analysis of appointments scheduled at a single-specialty practice and the corresponding reviews of its physicians finds the following:

• In the first study to evaluate the aggregated effect of online reviews across multiple review platforms, we find that 1-star reviews have a limited but longitudinal influence on new patient volume.
• Given the limited fiscal consequence of poor care and negative reviews, managed care organizations and policy makers must seek alternate avenues to incentivize physicians to further invest in their doctor-patient relationships and ensure that quality health care services are delivered.

Web-based physician ratings have a significant influence on patient attitudes toward practitioners.^1,2 Physician review websites (PRWs) have rapidly grown in recent years, both in terms of the number of physicians with ratings and the number of ratings per physician.^3,4 Although nearly 90% of these ratings are positive, physicians remain wary of these sites.^5-9

Two flaws of web-based physician rating platforms are that some do not require an individual to have had a physician interaction to post a review⁸ and that demonstrably false information cannot be corrected by physicians.¹⁰ Although PRWs can capture valuable information about a physician’s bedside manner, these evaluations are often influenced by external factors, such as a physician’s gender or race, the office’s wait times, and ancillary staff’s service.^11,12 Further, it has been established that individuals are more driven to write reviews if they have extreme positive or negative emotions about their experience.¹³ Accordingly, the literature has found that reviews on PRWs are often not consistent with reviews by physician peers and may not correlate with formal institutional evaluation scales.^14,15

Recent studies have found that receiving a 5-star rating, as opposed to a 1-star rating, could boost a physician’s patient volume by 4% to 8%.¹⁶ Even a half-star increase in a physician’s rating could increase the probability of an appointment being filled by 10%.¹⁷ However, the literature has limited its focus to the average rating across all patients’ submissions on a single PRW, for all patients. This does not account for the fact that returning patients base their provider selection on their prior experiences.¹⁷ The influence of negative reviews on patient volume, in the broader context of multiple web-based physician review platforms, remains unclear. Findings of studies in the restaurant and tourism industries have suggested that consumers are loss averse, finding 1-star reviews the most useful because they allow them to avoid bad experiences.^18,19 Given the importance of negative reviews in other sectors, this study thus aims to quantify the impact of 1-star reviews on new patient volume using a proprietary data set from a large specialty practice containing 1.12 million new patient appointments and 10,999 physician reviews.

METHODS

A single-specialty practice in the New York and New Jersey area provided deidentified data on the number of appointments scheduled and kept over the span of 3 years: January 1, 2015, to April 25, 2018. We also compiled reviews from 10 platforms (Zocdoc, Vitals, Healthgrades, RateMDs, Google, UCompareHealthCare, WebMD, Wellness, Yelp, and Facebook) of all affiliated physicians who practiced during this period. Physicians were identified by their National Provider Identifier (NPI). All physicians who saw patients during the study period and who had a 1-star review were included. Providers who had an inconsistent work schedule—defined as having fewer than 2 new patient appointments in 10 or more weeks of a year between 2015 and 2017—were excluded. Because only 18 weeks of 2018 were included in our study, this standard was reduced such that in 2018 any physicians who had fewer than 2 new patient appointments in 6 or more weeks were excluded from the year’s analysis. Lastly, if a new patient appointment did not have any recorded appointment data, a temporal relationship could not be established between the publishing of the 1-star review and the creation of the appointment, so it was also excluded from further analysis.

The number of appointments scheduled and kept per day and the data from reviews (star ratings, character counts of text reviews) were aggregated to a weekly level. The point of observation for the study is the number of new patients seen by a provider in a given week (NPI-week). For example, the first observation would be the number of reviews and appointments for NPI 1234567890 between January 1 and January 7, 2015. This process is repeated for each physician for the remaining 172 weeks. Because the last week of the calendar year does not necessarily have 7 days, the 53rd week of each year was excluded from this analysis. This constituted dropping 1 day from our analysis in 2015 and 2017 and 2 days in 2016, which was a leap year.

Assuming that reviews are read prior to appointment creation, an ordinary least squares regression model was run on Stata 14 (StataCorp LLC), with a time series analysis to compare new patient volume in the period immediately prior to a 1-star review with patient volume in the period immediately after a 1-star review (eAppendix Equation [eAppendix available at ajmc.com]). New patient volume is estimated by the total number of new patient appointments made per 4-hour half-day session, regardless of whether the appointment was kept. Year and quarter fixed effects were included to capture patient trend. Given the correlations among poor performance, low volume, and online evaluation ratings, the ordinary least squares regression model included a fixed effect for physicians to control for unobserved physician variation, examining differences before and after a 1-star review by provider and averaging them.^17,20 To ensure that reviews were not capturing previously existing negative trends, we reviewed mean new appointment trends in the 52 weeks prior to the 1-star review and found a trend line slightly positively sloped at 0.0064. Office-specific effects were considered, but because offices generally consist of 2 to 4 providers, office-specific effects and physician fixed effects overspecified the model. The method by which new patients made their appointments (either online or by phone), the specific platform of the review, the quarter, number of reviews, mean star rating, and number of characters in written reviews were also controlled for.¹² An additional sensitivity analysis was performed at 4, 6, 8, 10, 12, 14, and 16 weeks to determine the period in which a 1-star review had the greatest influence.

RESULTS

The study included a total of 209 physicians and 1.12 million scheduled new patient appointments across this 3-year span. After dropping observations without appointment data and from the 53rd week of each year, there were 30,728 NPI-week observations. The limited availability of 5 providers between 2015 and 2018 reduced our analysis to 204 physicians and 30,271 provider-weeks. An additional 63 providers (30%) and 9044 NPI-weeks were excluded because the corresponding physicians had zero 1-star reviews during the study interval. Ultimately, 21,227 NPI-weeks of 141 physicians were included in the analysis.

There were 10,999 applicable reviews gathered from 10 PRWs; 6 platforms—Zocdoc, Vitals, Healthgrades, RateMDs, Google, and Yelp—had 99% of reviews. The remaining 1% of reviews were divided among 4 platforms, Wellness, Facebook, UCompareHealthCare, and WebMD. Assuming the lack of reviews was representative of lack of consumer use, these reviews were excluded, leaving 10,894 reviews.

The majority of physician reviews on PRWs were overwhelmingly positive, with only 6.7% (n = 733) giving 1 star, 2.3% (n = 254) giving 2 stars, 3.3% (n = 359) giving 3 stars, 7.2% (n = 793) giving 4 stars, and 81.3% (n = 8860) giving 5 stars. A mean of 6.2 new patient appointments were made per half-day session (Figure 1). A sensitivity analysis revealed that a 1-star review caused a decrease in a physician’s new patient appointments within 4 weeks, but only by a mean of 0.08 appointments (Figure 2). Per the ordinary least squares regression model, 12 weeks after a 1-star review, the CI captured exclusively nonzero values for the first time. At this point, the new patient volume decreased by a mean of 0.14 appointments, or 2.3%. Per the sensitivity analysis, 12 weeks after a 1-star review, the new patient volume decreased by an average of 0.16 appointments, or 2.6%. The CI did not capture exclusively nonzero values until 14 weeks after a 1-star review had been posted. Per both analyses, the mean new patient volume decreased 2.3% to 2.6% following a 1-star review, with effects of the 1-star review affecting patient volume for at least 16 weeks (Figure 2).

DISCUSSION

We believe this study is the first to quantify the impact of 1-star reviews on new patient volume and the persistence of their impact on new patient volume over time. We found that although only 6.7% of ratings on PRWs were 1-star reviews, these strongly negative ratings had a small but real impact on new patient volume that could persist for 16 weeks. A mean of 6.2 new patient appointments are made in a half-day session, but 12 weeks after a 1-star review, there is a 2.3% decline in new patient volume (eAppendix Table and eAppendix Figure). The sensitivity analysis suggests that the effect of 1-star reviews may be more immediate because the mean change in new patient appointments is negative within 4 weeks in the analyses with and without Zocdoc (Figure 2). Based on the mean difference in new patient appointments and CIs, both analyses also indicate that the effect of the 1-star review persists for at least 16 weeks.

Although studies have identified the limited impact of CMS’ Hospital Care Compare star rating and New York State’s public reporting system for coronary artery bypass surgery on hospitals’ market share, few studies have delved into the impact of popular consumer-facing websites.^21-23 This is the first study to our knowledge to quantify both the timescale and extent of a 1-star review’s impact on exclusively new patient volume.

Our findings suggest that although physicians should be cognizant of the influence of negative reviews on new patient volume, they should not preoccupy themselves with the repercussions of these reviews.^7,9,24,25 Although poor reviews have the potential to improve the quality and value of physician care, their impact on new patients is tangible but not financially impactful at less than 1 fewer new patient per week.²⁶

PRWs, hospital systems, and private physician practices can take steps to make negative reviews a better resource for both physicians and patients. At present, PRWs are not necessarily reflective of evaluations by physician peers or formal institutional evaluation scales.^13,14 Findings from recent studies suggest that if PRWs were to use star ratings with systemic data collection tools, such as surveys, in lieu of free-response forms, the quantity and quality of reviews would improve.²⁷ Basing ratings on more than a handful of reviews and adopting a more fact-oriented style via surveys may give physicians a clearer idea of how to modify their practice and increase the credibility of the reviews to patients.^27-29 Given that current projections indicate that patient usage of PRWs, and hence PRW credibility, will improve with time, it is critical that hospital systems and private physician practices pursue such reforms.³⁰

Limitations

The major limitation of our study is the restricted generalizability of its findings. Our data set was derived from a single-specialty practice with locations throughout New York and New Jersey, although its large scale allowed us to observe a great number of physicians and appointments across multiple office-based centers. Other limitations include the unaccounted influence of the office space and ancillary support staff on the star rating, as well as that of varying appointment capacities of physicians or the interaction effect of multiple 1-star ratings. Further studies may consider categorizing details of the written comments to address whether certain text elements of 1-star reviews differentially influence patient volume trends.

CONCLUSIONS

Although most physician reviews are overwhelmingly positive, our study found that 1-star ratings could cause a decrease in new patient volume by less than 3% within 12 weeks after a 1-star review, and these effects could persist for more than 16 weeks. This study tested only new patient appointments, which account for approximately one-third of the practice volume, so assuming no impact on existing patient appointments, there would be approximately a 1% decline in total volume. The limited but longitudinal influence of negative ratings of physicians could reflect patients’ desire to avoid negative experiences and consequent utilization of PRWs, despite the limited credibility they attribute to them. The finding suggests that the financial impact of negative reviews is real but limited and that physician focus on reviews should be proportional. 

Author Affiliations: Division of Health Policy and Management, Department of Public Health (AEB), and School of Medicine (MR), New York Medical College, Valhalla, NY; University of British Columbia (EC), Montreal, Quebec, Canada.

Source of Funding: None.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (AEB); acquisition of data (AEB); analysis and interpretation of data (AEB, MR, EC); drafting of the manuscript (MR); critical revision of the manuscript for important intellectual content (AEB, MR, EC); statistical analysis (AEB, EC); administrative, technical, or logistic support (MR); and supervision (AEB).

Address Correspondence to: Medha Reddy, BA, New York Medical College, 1413 Old Farm Rd, Valhalla, NY 10595. Email: mreddy3@student.nymc.edu.

REFERENCES

1. Burkle CM, Keegan MT. Popularity of internet physician rating sites and their apparent influence on patients’ choices of physicians. BMC Health Serv Res. 2015;15:416. doi:10.1186/s12913-015-1099-2

2. The patient perspective 2019: online reputation survey. PatientPop. April 4, 2019. Accessed October 2, 2022. https://www.patientpop.com/blog/patient-perspective-survey-2019/

3. López A, Detz A, Ratanawongsa N, Sarkar U. What patients say about their doctors online: a qualitative content analysis. J Gen Intern Med. 2012;27(6):685-692. doi:10.1007/s11606-011-1958-4

4. Mueller MA, Glassett TS, Burke SM, Nichter LS. Protecting a plastic surgeon’s reputation: damage control for dishonest online reviews. Plast Reconstr Surg. 2013;131(4):670e-671e. doi:10.1097/prs.0b013e318282792a

5. Emmert M, Sander U, Pisch F. Eight questions about physician-rating websites: a systematic review. J Med Intern Res. 2013;15(2):e24. doi:10.2196/jmir.2360

6. Chang JT, Hays RD, Shekelle PG, et al. Patients’ global ratings of their health care are not associated with the technical quality of their care. Ann Intern Med. 2006;144(9):665-672. doi:10.7326/0003-4819-144-9-200605020-00010

7. Lee VS. Why doctors shouldn’t be afraid of online reviews. Harvard Business Review. March 29, 2016. Accessed October 2, 2022. https://hbr.org/2016/03/why-doctors-shouldnt-be-afraid-of-online-reviews

8. Ellimoottil C, Leichtle SW, Wright CJ, et al. Online physician reviews: the good, the bad, and the ugly. Bulletin of the American College of Surgeons. September 1, 2013. Accessed October 2, 2022. https://bulletin.facs.org/2013/09/online-physician-reviews/

9. Remove online reviews of doctors! Change.org. June 27, 2017. Accessed October 2, 2022. https://www.change.org/p/yelp-remove-online-reviews-of-doctors

10. Ornstein C. Doctors fire back at bad Yelp reviews — and reveal patients’ information online. Washington Post. May 27, 2016. Accessed October 2, 2022. https://www.washingtonpost.com/news/to-your-health/wp/2016/05/27/docs-fire-back-at-bad-yelp-reviews-and-reveal-patients-information-online/

11. Marrero K, King E, Fingeret AL. Impact of surgeon gender on online physician reviews. J Surg Res. 2020;245:510-515. doi:10.1016/j.jss.2019.07.047

12. Daskivich TJ, Houman J, Fuller G, Black JT, Kim HL, Spiegel B. Online physician ratings fail to predict actual performance on measures of quality, value, and peer review. J Am Med Inform Assoc. 2018;25(4):401-407. doi:10.1093/jamia/ocx083

13. Clemons E. Why fake news campaigns are so effective. Knowledge at Wharton. October 3, 2018. Accessed December 20, 2022. https://knowledge.wharton.upenn.edu/article/build-fake-news-campaign/

14. McGrath RJ, Priestley JL, Zhou Y, Culligan PJ. The validity of online patient ratings of physicians: analysis of physician peer reviews and patient ratings. Interact J Med Res. 2018;7(1):e8. doi:10.2196/ijmr.9350

15. Widmer RJ, Maurer MJ, Nayar VR, et al. Online physician reviews do not reflect patient satisfaction survey responses. Mayo Clinic Proc. 2018;93(4):453-457. doi:10.1016/j.mayocp.2018.01.021

16. Chen Y. User-generated physician ratings—evidence from Yelp. Stanford University. November 8, 2018. Accessed October 3, 2022. https://stanford.edu/~yiwei312/papers/jmp_chen.pdf

17. Luca M, Vats S. Digitizing doctor demand: the impact of online reviews on doctor choice. American Economic Association. October 31, 2013. Accessed October 2, 2022. https://www.aeaweb.org/conference/2014/retrieve.php?pdfid=55

18. Park S, Nicolau JL. Asymmetric effects of online consumer reviews. Ann Tour Res. 2015;50:67-83. doi:10.1016/j.annals.2014.10.007

19. Beaton C. Why you can’t really trust negative online reviews. New York Times. June 13, 2018. Accessed October 2, 2022. https://www.nytimes.com/2018/06/13/smarter-living/trust-negative-product-reviews.html

20. Xu Y, Armony M, Ghose A. The interplay between online reviews and physician demand: an empirical investigation. Manage Sci. 2021;67(12):7344-7361. doi:10.1287/mnsc.2020.3879

21. Jha AK, Epstein AM. The predictive accuracy of the New York State coronary artery bypass surgery report-card system. Health Aff (Millwood). 2006;25(3):844-855. doi:10.1377/hlthaff.25.3.844

22. Kaye DR, Norton EC, Ellimoottil C, et al. Understanding the relationship between the Centers for Medicare and Medicaid Services’ Hospital Compare star rating, surgical case volume, and short-term outcomes after major cancer surgery. Cancer. 2017;123(21):4259-4267. doi:10.1002/cncr.30866

23. Segal J, Sacopulos M, Sheets V, Thurston I, Brooks K, Puccia R. Online doctor reviews: do they track surgeon volume, a proxy for quality of care? J Med Internet Res. 2012;14(2):e50. doi:10.2196/jmir.2005

24. Lagu T, Haskell J, Cooper E, Harris DA, Murray A, Gardner RL. Physician beliefs about online reporting of quality and experience data. J Gen Intern Med. 2019;34(11):2542-2548. doi:10.1007/s11606-019-05267-1

25. Menon AV. Do online reviews diminish physician authority? the case of cosmetic surgery in the U.S. Soc Sci Med. 2017;181:1-8. doi:10.1016/j.socscimed.2017.03.046

26. Lee V. Transparency and trust — online patient reviews of physicians. N Engl J Med. 2017;376(3):197-199. doi:10.1056/nejmp1610136

27. Lagu T, Metayer K, Moran M, et al. Website characteristics and physician reviews on commercial physician-rating websites. JAMA. 2017;317(7):766-768. doi:10.1001/jama.2016.18553

28. Grabner-Kräuter S, Waiguny MKJ. Insights into the impact of online physician reviews on patients’ decision making: randomized experiment. J Med Internet Res. 2015;17(4):e93. doi:10.2196/jmir.3991

29. Lu W, Wu H. How online reviews and services affect physician outpatient visits: content analysis of evidence from two online health care communities. JMIR Med Inform. 2019;7(4):e16185. doi:10.2196/16185

30. Gao GG, McCullough JS, Agarwal R, Jha AK. A changing landscape of physician quality reporting: analysis of patients’ online ratings of their physicians over a 5-year period. J Med Internet Res. 2012;14(1):e38. doi:10.2196/jmir.2003