Success of Automated Algorithmic Scheduling in an Outpatient Setting

Objective

To determine if algorithmically generated double-booking recommendations could increase patient volume per clinical session without increasing the burden on physicians.

Study Design

A randomized controlled trial was conducted with 519 clinical sessions for 13 dermatologists from December 1, 2011, through March 31, 2012.

Methods

Sessions were randomly assigned to “Smart-Booking,” an algorithm that generates double-booking recommendations using a missed appointment (no-shows + same-day cancellations) predictive model (c-statistic 0.71), or to a control arm where usual booking rules were applied. The primary outcomes were the average number and variance of arrived patients per session, after controlling by physician. In addition, physicians received a survey after each session to quantify how busy they felt during that session.

Results

257 sessions were randomized to Smart-Booking and 262 sessions were randomized to control booking. Using a generalized multivariate linear model, the average number of arrived patients per session was higher in the Smart-Booking intervention arm than the control (15.7 vs 15.2, difference between groups 4.2; 95% CI, 0.08-0.75; P = .014).The variance was also higher in the intervention than control (3.72 vs 3.33, P = .38).The survey response rate was 92% and the physicians reported being similarly busy in each study arm.

Conclusions

Algorithmically generated double-booking recommendations of dermatology clinical sessions using individual physician assumptions and predictive modeling can increase the number of arrived patients without overburdening physicians, and is likely scalable to other settings.

Am J Manag Care. 2014;20(7):570-576 Algorithmically generated booking recommendations based on customizable physician assumptions and predictive modeling modestly increased productivity without overburdening physicians in a randomized controlled trial.

• Predictive modeling can identify patients likely to be no-shows.

• Advance scheduling systems can be successfully implemented.

• The role of clinical scheduling decision support should be further studied.

Rising healthcare costs and federal budget deficits continue to put pressure on physicians to more ^1-3 efficiently deliver care. Patients who miss outpatient appointments without prior adequate notification, colloquially called “no-shows,” are a frequent source of complaint because they decrease efficiency, have a negative financial impact, and waste appointment slots that could be used by others. One study at a family practice residency clinic concluded that no-shows resulted in a 3% to 14% revenue loss.⁴ In an era in which access to primary care and several other specialties is constrained, optimal utilization of physician time is paramount.^5,6

No-shows have a variety of causes. Logistical issues such as an inability to miss work, find child care, or find transportation are reasons for many patients.^7-9 Simply forgetting is another obvious problem^8-11 and often the most frequent cause of missed appointments (48%9 and 39%8 in 2 studies, for instance). Interestingly, patients’ perceptions or concerns about their visit also affect adherence. A hesitation to hear bad news, endure an uncomfortable procedure, and encounter perceived disrespect by the medical establishment have been reported.^7,10 Self-resolving symptoms have also been also cited as a cause for no-shows.^7,8,11 And not surprisingly, the greater the number of “wait days”—the days between the scheduling of the appointment and the appointment date— the greater the risk of a no-show.^7,11,12

There are numerous no-show interventions, but most rely on appointment reminders to decrease no-shows or on double booking to compensate for them. Reminder interventions discussed in the literature include staff phone reminders, automated phone reminders, text messages, mailed reminders, other electronic reminders, and financial penalties, all of which have been shown to work to some degree in some settings.^13-15 A systematic review demonstrated a 39% reduction in the non-arrival rate after manual reminders and a 29% reduction for automated reminders; the reminders were shown to be cost-effective.¹³ Several randomized controlled trials (RCTs) have also concluded that reminders meaningfully reduce no-show rates.^8,9,16-21 For example, a 3-arm outpatient RCT resulted in no-show rates of 23.1% with no reminders, 17.3% for automated phone reminders, and 13.6% with staff phone reminders.¹⁶ However, in some settings automated reminders have been ineffective.^8,22,23

Overbooking is a common strategy, but most clinics do so with a nonscientific approach, such as, “If we double book early, we will catch up later.” Two groups have published the results of nontraditional scheduling system implementations that used overbooking. A pediatric ophthalmology clinic implemented a system that algorithmically opened appointment slots based on predicted patient demand, physician supply, and scheduling rules; however, their implementation was preliminary at the time of publication.²⁴ In another study, Israeli dermatologists reduced the baseline non attendance rate from 32.9% to 27.9% using managed overbooking and service centralization.²⁵

Open access, predictive modeling, and advanced scheduling models have been proposed to improve clinic efficiency. A systematic review of 24 open access studies concluded that no-show rates were lower in practices with a prior baseline >15%; however, other outcome measures were mixed.²⁶ No-show prediction models have typically been designed using association rules,²⁷ logistic regression,^28,29 and a combination approach.³⁰ Two studies reported C statistics of 0.82 and 0.84; however, neither was externally validated.^28,29 Others have designed and validated advanced scheduling systems that maximize clinic utility through computer simulation and calculation.^{24,27,28,31-39}

Given the relative lack of published outcomes of advanced scheduling system implementations, we developed and validated a model to predict no-shows and same-day cancellations and conducted a randomized controlled trial to determine the following: Can an automated algorithmic approach to double-booking dermatology appointments increase the number of arrived patients per session without overburdening the physicians?

METHODS

Study Setting

A randomized controlled trial was conducted from December 1, 2011, through March 31, 2012 at the Department of Medical Dermatology at Massachusetts General Hospital (MGH) under the approval of the Partners Institutional Review Board. The department hosts approximately 80 medical dermatology clinical sessions per week, for a total volume of approximately 50,100 patient visits per year. At the time of the study, 8 full-time physicians averaged 6 sessions weekly, and 22 physicians worked 1 to 5 sessions weekly; the practice also employs residents. Clinical sessions run from 8:00 am to 12:00 pm or 1:00 pm to 5:00 pm, and each appointment slot is 15 minutes long except for 30-minute procedures. Physician compensation is based on Relative Value Unit productivity.

Prior to the study, the missed appointment rate was 16.5%, and given this rate, some clinicians routinely double booked their sessions. Appointments were booked using the electronic booking system IDX by 10 schedulers employed by the practice at the front desk or in a small call center adjacent to the practice. Prior to the intervention, schedulers would doublebook by attempting to spread additional patients out evenly across the schedule, and there was an average of 55 days between the scheduling and arrival of each new patient.

Eligibility

Sixteen of 30 physicians were excluded from the study for the following reasons: 7 worked 1 session a week and lacked booking flexibility, 4 were in another scheduling study, 3 worked primarily at other practice sites, 1 was on maternity leave, and 1 new physician’s schedule was not yet standardized. The other 14 physicians were asked to join the study, and all consented to participate. Double booking in designated urgent access, procedural, evening, and weekend clinics was excluded from this pilot.

Develop Missed Appointment Predictive Model

A missed appointment (no-shows + same-day cancellations) predictive model was developed. Potentially predictive variables were identified through literature review, discussion with the practice leadership, and evaluation of existing administrative data sources. The variables identified were: appointment type, day of week, wait days, language, ethnicity, age, historical diagnoses, insurance, and appointment arrival history. One year of data about approximately 54,000 dermatology appointments were collected to develop and validate the model (see eAppendix A, available at www.ajmc.com for details of the methodology). In addition, missed appointment models were developed using the same methodology for 5 additional departments to quantify the predictive model’s generalizability.

Develop Smart-Booking Algorithm

The Smart-Booking approach was developed to algorithmically generate double booking recommendations using individual physician assumptions and missed appointment probabilities. The resulting system was a stochastic algorithm that outputted reports identifying the slots in which appointments should be doublebooked, appointment types to book in open slots, and where open slots should be blocked.

Intervention

Clinic sessions were randomized 1:1 to the intervention and control, stratified by physician, using a computer-generated list of random numbers. In the control, appointments were booked based on prior methodology; in the intervention, schedulers booked using the Smart-Booking recommendations. Physician scheduling assumptions were collected from all participating physicians and then adjusted based on input from practice leadership.

The Smart-Booking system was used to generate 2 reports. The Follow-up Double-Booking Report (eAppendix B) identified the control and intervention sessions, and in the intervention sessions, time slots were identified to double book follow-up appointments for the next 2 to 60 days. The Next-Day Report (eAppendix C) identified intervention time slots to double book new and follow-up appointments, to convert empty time slots into new or follow-up appointments, and to block empty slots because the schedule was overbooked.

Booking recommendations were electronically delivered at 5:30 am each weekday to the department. Every day the schedulers printed the Follow-up Double-Booking report and booked accordingly. In addition, the Next-Day report was used to modify the next day’s schedule. Schedulers did double-book in the control arm based on each physician’s historical booking maximums. Additionally, patients were double-booked in both arms beyond recommended limits on occasion by physician request or for urgent issues.

Study Outcomes

Data collected from the IDX scheduling system were used to calculate the number of normalized arrived patients per 4-hour session (referred to as arrived patients) (eAppendix D). The primary outcomes were the average and variance of the arrived patients in each study arm. As a secondary outcome, the average arrived patients were calculated for each individual physician. In addition, the following survey was sent to each physician to quantify how busy they felt after each session.

Your clinic session was:

-3 Too slow

-2

-1

0 Neither too slow nor too busy

+1

+2

+3Too busy

Sample Size

Based on historical data, the average number of patients arrived per session was 14.0 with a standard deviation of 3.0. This study was powered to detect the difference of 1 additional arrived patient per physician session with an alpha = 0.05 and power = 0.80. This resulted in an initial sample size of 141 sessions per study arm. After accruing 141 sessions in each arm, there were no physician or staff complaints. Therefore, we continued the study for an additional 10 weeks, which resulted in 654 randomized clinical sessions to gain further physician-specific data.

Data Analysis

We fit a generalized multivariate linear model to determine the impact of the intervention. Arrived patients was the dependent variable, the arm was the independent variable, and the physician was the random effect. An F-test was used to determine the significance of the difference of the variance in each arm. The physician survey results were plotted for all sessions and summarized descriptively.

One physician was removed from the primary analysis because Smart-Booking was incorrectly calculated and implemented. The threshold for total number of booked patients in the intervention arm was set lower than intended; therefore, when the physician wanted to add extra patients into a schedule, they were systematically added to the control sessions. Sensitivity analyses were performed that included this physician’s data.

RESULTS

The final Smart-Booking dermatology missed appointment predictive model used age, wait days, insurance, and patient arrival history to calculate a patient’s missed appointment probability. The model had a C statistic of 0.71, and a detailed description of the model is in eAppendix A. Using the same methodology, similarly discriminative models were designed for the following MGH departments: gastroenterology, physical therapy, primary care, psychiatry, and gynecology (Table 1).

The flow diagram of study sessions is given in Figure 1. After exclusion of 38 sessions due to scheduling changes and all sessions (n = 95) of the physician for whom the intervention arm was incorrectly implemented, there was a total of 519 evaluable sessions.

Figure 2

The primary outcome of the average arrived patients was 15.7 in the intervention and 15.2 in the control. Using a generalized multivariate linear model, the difference between the intervention and control arms was 0.42 (95% CI, 0.08-0.75; P = .014). The variance of the arrived patients was 3.72 in the intervention and 3.33 in the control (P = .38). The physicians reported being similarly busy in each study arm ().

Figure 3

Table 2 contains results for the average arrived patients separately for each physician. Physicians 1 and 2 had statistically significant increases in arrived patients per session and reported similar numbers of very busy sessions ().

The excluded physician had 95 total sessions (45 intervention, 50 control), and averaged 17.20 patients in the intervention and 17.71 patients in the control. The difference was —0.51 patients per session (95% CI, –1.50 to 0.47; P = .31). Including the sessions of the excluded physician, the average number of arrived patients was 15.90 in the intervention arm and 15.61 in the control. Using a generalized multivariate linear model, the difference between the intervention vs control arm was 0.27 (95% CI, —0.04 to 0.60; P = .093). The variance of the arrived patients was 4.01 in the intervention and 3.73 in the control (P = .54).

DISCUSSION

Rising healthcare costs and federal budget deficits are increasingly putting pressure on physicians to more efficiently deliver care.^1-3 Patient no-shows have a negative impact on practices and decrease efficiency.^4-6,22,40-42 This study shows that even in a setting where overbooking routinely occurs, physician utilization can be increased, schedules can be smoothed to improve flow, and the negative impact of no-shows can be reduced without overburdening physicians. A 3% increase in arrived patients could result in an extra 1300 appointments per year for this department. More importantly, the increase in arrived patients occurred in a department where many clinicians were already using double booked schedules, and for certain physicians, especially those who had not maximized their booking prior to the study, the intervention resulted in substantial increases in productivity.

This approach appears to be unique in medical scheduling. Many authors who have designed advanced scheduling systems have not published results of an implementation^{24,27,28,31,32,34,36-39} or finalized results.²⁴ A strength of this study was also that it implemented an advanced scheduling system with the scientific rigor of a randomized blinded controlled trial. Physicians were not made aware of which sessions were Smart-Booked. The booking recommendations were sent directly to the schedulers, and the physician’s schedule view did not identify the session arm. Moreover, there was double booking in both study arms, so noting that a double booking had occurred did not imply that it was part of the intervention arm.

Because the Smart-Booking system is calibrated for each physician, the quality of results is highly dependent on this calibration. For example, the increase in arrived patients varied by physician (1.5 to —0.4). This is likely because the Smart-Booking assumptions were calibrated either more or less aggressively than the physician’s prior thresholds. It is essential that physicians understand their assumptions so they can be adjusted to meet their productivity goals.

For physicians 1 and 2, this approach substantially improved their productivity and smoothed their schedules. Despite seeing more patients, their survey scores indicated they were busier but not overburdened because extra patients were booked at times that others were likely to miss their appointment. Physician 1 (Figure 3) had an increase of 1.4 patients per session and rarely double booked, so there was a straightforward comparison to the control. Upon viewing the study results, physician 1 was satisfied with the increase in arrived patient and survey results, and even requested that the Smart-Booking assumptions be modified to accommodate an additional patient per session. Physician 2 (Figure 3) had an increase of 1.4 patients per session. Upon viewing the study results, physician 2 was also satisfied with the number of arrived patients per session and wanted to continue Smart-Booking appointments. For these 2 physicians, it appears the model was appropriately calibrated to increase productivity without generating stressful sessions for the physician.

Lastly, there was an unanticipated benefit of generating appointment availability information for the staff. Instead of spending time analyzing schedules and guessing where to place additional patients, they had clear information at hand that allowed them to immediately know the best place to add someone to the schedule. Physicians were also no longer burdened with the need to constantly examine their schedules in response to queries about where in a session to add patients outside the scheduling template. While we did not quantify the time saved for staff in this study, anecdotally, they found this approach easier and faster, and this is a potential benefit worth further examination.

The Smart-Booking system is potentially generalizable to other departments. Using the same variables in the dermatology missed appointment predictive model, models designed for 5 other MGH departments were similarly discriminative (Table 1) and similar in content. Moreover, the variables in the predictive model, age, wait days, insurance, and patient arrival history are likely attainable from existing hospital databases. The remaining Smart-Booking inputs are customized to each physician’s work speed and are not specific to dermatology.

While this study illustrates the potential for implementing Smart-Booking in departments outside of dermatology at MGH, a number of issues may limit the generalizability of our study. It was implemented in a single department, had 14 physicians, and not all appointment types were included. There was also an outlier physician who was removed from the study because the protocol was improperly calculated and implemented. Also, it was also not always possible to adhere to the double-booking recommendations, because patients were sometimes booked beyond the recommended limits in both arms for urgent issues or by physician request.

CONCLUSIONS

Algorithmically generated double-booking recommendations of dermatology clinical sessions using individual physician assumptions and predictive modeling increased the number of arrived patients without overburdening physicians. For clinicians, who already substantially overbook their schedules, this approach can result in a smoother clinical session with additional patients booked where non attendance is likely to occur. Since the variables used to derive this approach are commonly available, it is likely relevant and scalable to other settings.Acknowledgments: The authors would like to thank Pat Sullivan, Kevin Donahue, MBA, and James B. Meigs, MD, for their contributions to this research.

Author Affiliations: Massachusetts General Hospital, Practice Improvement Division, Boston (PRC); Harvard Medical School, Massachusetts General Hospital, Department of Dermatology, Boston (ABK).

Source of Funding: Massachusetts General Physicians Organization.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (PRC, ABK); acquisition of data (PRC); analysis and interpretation of data (PRC, ABK); drafting of the manuscript (PRC, ABK); critical revision of the manuscript for important intellectual content (PRC, ABK); statistical analysis (PRC); provision of study materials or patients (PRC); obtaining funding (ABK); administrative, technical, or logistic support (PRC).

Address correspondence to: Patrick R. Cronin, MA, 50 Staniford St, 7th Fl, Boston, MA 02114. E-mail: prcronin@partners.org.1.The Commonwealth Fund Commission on a High Performance Health System. The Performance Improvement Imperative: Utilizing a Coordinated, Community-Based Approach to Enhance Care and Lower Costs for Chronically Ill Patients. NewYork:The Commonwealth Fund; 2012.

2. Guterman S, Davis K, Schoen C, Stremikis K. Reforming Provider Payment: Essential Building Block For Health Reform. NewYork:The Commonweath Fund; 2009.

3. Blumenthal D. Performance improvement in health care--seizing the moment. N Eng J Med. 2012;366(21):1953-1955.

4. Moore CG, Wilson-Witherspoon P, Probst JC. Time and money: effects of no-shows at a family practice residency clinic. Fam Med. 2001;33(7): 522-527.

5. Colwill JM, Cultice JM, Kruse RL. Will generalist physician supply meet demands of an increasing and aging population? Health Aff (Millwood). 2008;27(3):w232-w241.

6. SequistTD. Ensuring equal access to specialty care. N Eng J Med. 2011;364(23):2258-2259.

7. Lacy NL, Paulman A, Reuter MD, Lovejoy B. Why we don’t come: patient perceptions on no-shows. Ann Fam Med. 2004;2(6):541-545.

8. Maxwell S, Maljanian R, Horowitz S, Pianka MA, CabreraY, Greene J. Effectiveness of reminder systems on appointment adherence rates. J Health Care Poor Underserved. 2001;12(4):504-514.

9. Woods R. The effectiveness of reminder phone calls on reducing no- show rates in ambulatory care. Nurs Econ. 2011;29(5):278-282.

10. Husain-Gambles M, Neal RD, Dempsey O, Lawlor DA, Hodgson J. Missed appointments in primary care: questionnaire and focus group study of health professionals. Br J Gen Pract. 2004;54(499):108-113.

11. Cosgrove MP. Defaulters in general practice: reasons for default and patterns of attendance. Br J Gen Pract. 1990;40(331):50-52.

12. Whittle J, Schectman G, Lu N, Baar B, Mayo-Smith MF. Relationship of scheduling interval to missed and cancelled clinic appointments. J Ambul Care Manage. 2008;31(4):290-302.

13. Hasvold PE, Wootton R. Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review. J TelemedTelecare. 2011;17(7):358-364.

14. Macharia WM, Leon G, Rowe BH, Stephenson BJ, Haynes RB. An overview of interventions to improve compliance with appointment keeping for medical services. JAMA.1992;267(13):1813-1817.

15. Bech M.The economics of non-attendance and the expected effect of charging a fine on non-attendees. Health Policy. 2005;74(2):181-191.

16. Parikh A, Gupta K, Wilson AC, Fields K, Cosgrove NM, Kostis JB. The effectiveness of outpatient appointment reminder systems in reducing no-show rates. Am J Med. 2010;123(6):542-548.

17. O’Brien G, Lazebnik R.Telephone call reminders and attendance in an adolescent clinic. Pediatrics.1998;101(6):E6.

18. Stockwell MS, Kharbanda EO, Martinez RA, Vargas CY, Vawdrey DK, Camargo S. Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: a randomized controlled trial. JAMA. 2012;307(16):1702-1708.

19.Taylor NF, Bottrell J, Lawler K, Benjamin D. Mobile telephone short message service reminders can reduce nonattendance in physical therapy outpatient clinics: a randomized controlled trial. Arch Phys Med Rehabil. 2012;93(1):21-26.

20. Irigoyen MM, Findley S, Earle B, Stambaugh K, Vaughan R. Impact of appointment reminders on vaccination coverage at an urban clinic. Pediatrics. 2000;106(4 suppl):919-923.

21. Hashim MJ, Franks P, Fiscella K. Effectiveness of telephone reminders in improving rate of appointments kept at an outpatient clinic: a randomized controlled trial. J Am Board of Fam Pract.2001;14(3):193-196.

22. Satiani B, Miller S, Patel D. No-show rates in the vascular laboratory: analysis and possible solutions. J Vasc Interv Radiol. 2009;20(1):87-91.

23. SequistTD, Zaslavsky AM, Colditz GA, Ayanian JZ. Electronic patient messages to promote colorectal cancer screening: a randomized controlled trial. Arch Intern Med. 2011;171(7):636-641.

24. Guo M, Wagner M, West C. Outpatient clinic scheduling - a simulation approach: proceedings of the 2004 Winter Simulation Conference. 2004;2,1981-1987.

25. Bibi Y, Cohen AD, Goldfarb D, Rubinshtein E, Vardy DA. Intervention program to reduce waiting time of a dermatological visit: managed overbooking and service centralization as effective management tools. Int J Dermatol. 2007;46(8):830-834.

26. Rose KD, Ross JS, Horwitz LI. Advanced access scheduling outcomes: a systematic review. Arch Intern Med. 2011;171(13):1150-1159.

27. Glowacka K, Henry RM, May JH. A hybrid data mining/simulation approach for modelling outpatient no-shows in clinic scheduling. JORS. 2009;60:1056-1068.

28. Daggy J, Lawley M, Willis D, et al. Using no-show modeling to improve clinic performance. Health Informatics J. 2010;16(4):246-259.

29. Lee VJ, Earnest A, Chen MI, Krishnan B. Predictors of failed attendances in a multi-specialty outpatient centre using electronic databases. BMC Health Serv Res. 2005;6(5):51.

30. Alaeddini A, Yang K, Reddy C, Yu S. A probabilistic model for predicting the probability of no-show in hospital appointments. Health Care Manag Sci. 2011;14(2):146-157.

31. Chakraborty S, Muthuraman K, Lawley M. Sequential clinical scheduling with patient no-shows and general service time distribu- tions. IIETransactions. 2010;42(5):354-367.

32. RohlederTR, Klassen KJ. Rolling horizon appointment scheduling: a simulation study. Health Care Manag Sci. 2002;5(3):201-209.

33. Joustra PE, de Wit J, Struben VM, Overbeek BJ, Fockens P, Elkhuizen SG. Reducing access times for an endoscopy department by an iterative combination of computer simulation and linear programming. Health Care Manag Sci. 2010;13(1):17-26.

34. Lin J, Muthuraman K, Lawley M. Optimal and approximate algorithms for sequential clinical scheduling with no-shows. IIETransactions on Healthcare Systems Engineering. 2011;1(1):20-36.

35. Sobolev BG, Sanchez V, Kuramoto L, Levy AR, Schechter M, Fitzgerald JM. Evaluation of booking systems for elective surgery using simulation experiments. Healthc Policy. 2008;3(4):113-124.

36. Liu N, Ziya S, Kulkarni VG. Dynamic scheduling of outpatient appointments under patient no-shows and cancellations. Manufacturing & Service Operations Management. 2010;12(2):347-364.

37. Zeng B,Turkcan A, Lin J, Lawley M. Clinic scheduling models with overbooking for patients with heterogeneous no-show probabilities. Ann Oper Res. 2010;178:121-144.

38. LaGanga LR, Lawrence SR. Clinic overbooking to improve patient access and increase provider productivity. Decision Sciences. 2007; 38(2):251-276.

39. Muthuraman K, Lawley M. A stochastic overbooking model for outpatient clinical scheduling with no-shows. IIETransactions. 2008;40(9): 820-837.

40. Hixon AL, Chapman RW, Nuovo J. Failure to keep clinic appointments: implications for residency education and productivity. Fam Med.1999;31(9):627-630.

41. Nguyen DL, Dejesus RS. Increased frequency of no-shows in residents’ primary care clinic is associated with more visits to the emergency department. J Prim Care Community Health. 2010;1(1):8-11.

42. Capko J.The price you pay for missed appointments. J Med Pract Manage. 2007;22(6):368.