Optimizing Number and Timing of Appointment Reminders: A Randomized Trial

ABSTRACT

Objectives: To compare missed appointment rates for patients receiving a single reminder either 3 days prior to a primary care visit, 1 day prior to the visit, or both 3 days and 1 day prior to the visit.

Study Design: Three-armed randomized controlled trial.

Methods: Text messages or interactive voice response calls were sent to patients with appointments at 25 primary care clinics in an integrated delivery system. A multivariable prediction model was developed to identify patients at high risk of missing appointments, based on prior appointment-keeping history and other variables from electronic health records.

Results: Among 54,066 randomized patients, those who received reminders both 3 days and 1 day prior to the visit were less likely to miss their appointment than those who received only a 3-day or 1-day reminder (4.4% vs 5.8% vs 5.3%, respectively; P <.001). In patients at high risk, 20.5% of those who received 2 reminders missed their visit, compared with 25.0% and 24.2% of those with only 3-day or 1-day reminders, respectively (P <.001). Visit satisfaction was unaffected by providing an additional reminder.

Conclusions: Two automated reminders were more effective than 1 in reducing missed appointments and did not reduce visit satisfaction. A predictive model based on clinical characteristics and prior appointment history can accurately identify patients who are at highest risk of missing appointments. These individuals may benefit more from multiple reminders, but additional strategies are necessary to further reduce their rates of missed appointments.

Am J Manag Care. 2018;24(8):377-384Takeaway Points

Text message or phone reminders 3 days and 1 day before a primary care appointment were more effective in reducing missed appointments than a single reminder, particularly in patients whom a clinical prediction rule identified as at high risk of missing appointments.

• Decision makers posed the question to fine-tune existing reminder protocols through collaboration with researchers in a learning health system.
• Because prior randomized trials had not directly compared either the timing or number of reminder calls, the findings may apply to other organizations.

Healthcare delivery systems use many strategies to remind patients of upcoming clinic appointments. A 1992 meta-analysis found that mailed reminders or telephone calls from clinic staff consistently reduced missed appointments.¹ Newer tools, such as interactive voice response (IVR) telephone calls and text messages, have also been effective while improving the efficiency and reducing the cost of appointment reminders.^2-5

Although the effectiveness of visit reminders is well established, little is known about the optimal number or timing of reminders. A reminder several days in advance provides patients an opportunity to cancel their appointment with sufficient lead time for a clinic to schedule other patients into vacated slots. On the other hand, a reminder the day prior to an appointment may allow the patient to better assess whether the visit remains necessary or conflicts with other obligations.⁵ Although multiple reminders may be more effective than a single reminder, patients may perceive them as unnecessary or intrusive.

Learning health systems combine clinical data, quality improvement methods, and research skills to address operational questions rapidly and at low cost.^6,7 When operational interventions are randomized, the resulting information is both directly actionable and potentially generalizable to other delivery systems. We previously reported the results of a collaborative trial between operational leaders and researchers in an integrated healthcare system, which found that a single reminder delivered by text or IVR message 1 day prior to a primary care appointment reduced the rate of missed appointments compared with no reminder.⁸ To refine the approach to delivering appointment reminders, we conducted a second randomized controlled trial to evaluate 3 strategies for providing primary care visit reminders: (1) a single reminder 3 days prior to the visit, (2) a single reminder 1 day prior to the visit (the standard approach based on the initial trial), or (3) reminders both 3 days and 1 day prior to the visit. We also assessed the effect of these 3 approaches on visit satisfaction and the impact of the intervention in the subgroup of patients at highest risk of missing their appointments.

METHODS

Study Setting

This project took place at Kaiser Permanente Colorado (KPCO), an integrated healthcare delivery system with more than 650,000 members in the Rocky Mountain West. Patients received care at 27 primary care clinics that were staffed by family physicians, internists, pediatricians, nurse practitioners, and physician assistants. Patients could make appointments for primary care visits in person, by telephone, or through a web portal. Since 2005, KPCO's research department has operated a visit reminder system that initiates telephone calls to landlines or cellular phones and text messages to cellular phones. Text messages are the primary communication modality unless the recipient has requested voice messages. We designate this system as IVR-T (interactive voice response or text). This system has been used in numerous interventions, several of which have been deployed operationally after completion of the research.^8-11

At the time of this study, the delivery system sent a single IVR-T reminder to each primary care patient 1 day before their scheduled visit, based on the results of our prior randomized trial.⁸ To fine-tune the approach to visit reminders, operational leaders then asked whether a 3-day reminder might be preferable to a 1-day reminder, because it provided greater opportunity to fill slots vacated through cancellations. Although leaders were also interested in determining whether 2 reminders would further reduce missed appointments, they raised concerns that multiple calls might be burdensome to patients and encouraged us to use existing visit satisfaction data to explore this issue.

Intervention Design

In response to these questions, we designed a pragmatic intervention trial. Stratifying by clinic site, we randomly assigned all patients with primary care appointments between October 15, 2016, and December 15, 2016, to 1 of 3 groups: a single IVR-T reminder 3 calendar days prior to the appointment; a single IVR-T reminder 1 calendar day prior to the appointment; or 2 IVR-T reminders, with one made 3 days and the other 1 day prior to the appointment. Calls were made on weekends, as well as business days. All patients were eligible if they had a primary care appointment and had made their appointment 3 or more days in advance of the scheduled visit. Patients who previously requested not to be contacted by phone or text message were excluded. One clinic was excluded because its staff made personal calls rather than using IVR-T reminders, whereas a second site was excluded because it opted to continue using the 1-day reminder protocol. We excluded scheduled telephone visits, visits for office procedures, and visits to non—primary care departments, such as radiology and ophthalmology. Patients who made appointments online through the KPCO patient portal and received email reminders were included in the intervention. Although information about email reminders was unavailable, the randomized design is likely to have distributed individuals who received these reminders equally among the 3 groups.

All text messages or calls were delivered in English. Those receiving a text message were asked to respond with their medical record number if they wished to cancel the appointment. Patients who answered the IVR call confirmed their identity by providing their medical record number, then received a short message confirming the date and time of the appointment and were offered the opportunity to cancel their appointment through the appointment line. If an answering machine responded, a recorded message provided the same information.

We used a randomization algorithm in the program that managed the IVR database to assign each visit to 1 of the 3 groups. Because randomization took place at the visit level, patients with multiple visits during the study period could receive the IVR-T intervention for some visits but not others. We limited the analysis to the first appointment randomized.

Study Measures: Electronic Health Records and Visit Satisfaction Survey

We used appointment records to determine the primary outcome: whether the appointment was missed, cancelled, or completed. We selected patient characteristics as covariates and potential predictors of missed appointments based on the findings of prior studies.^8,12-16 These variables included age, sex, race/ethnicity, marital status, employment, day and time of the appointment, number of comorbid conditions in the 2 years prior to the visit,¹⁷ health insurance type, duration of enrollment in KPCO (a proxy for the level of familiarity with the system), lead time to appointment (defined as the number of days between scheduling the appointment and the date of that appointment), the sum of the number of emergency department (ED) visits and hospitalizations within KPCO or other institutions reimbursed by KPCO health insurance within 6 months of the index appointment, and the number of missed primary care appointments within 6 months prior to the index appointment.

KPCO developed and conducted ongoing surveys to assess member satisfaction after completed primary care visits. These surveys were distinct from standard surveys of patient experience, such as the Hospital Consumer Assessment of Healthcare Providers and Systems survey.¹⁸ Members who had missed or cancelled their appointments were not surveyed. The 58-item survey included a global item on visit satisfaction: “Thinking just about your visit on [date] with [clinician name], and everything that happened from the time you scheduled the appointment until the time you left the medical office, how would you rate it?” Response options were excellent, good, adequate, and fair to poor. We added 4 items to surveys administered between October 1, 2016, and December 31, 2016. These items (with response options in parentheses) were: (1) Did you receive a reminder message before your visit? (yes/no/don’t remember); (2) Approximately how many reminder messages did you receive (1/2/3/4 or more/don’t remember); (3) How satisfied were you with the timing of reminders we sent you before your visit? (extremely satisfied/very satisfied/somewhat satisfied/not very satisfied/not at all satisfied/don’t know/not applicable); and (4) How satisfied were you with the number of reminders we sent you before your visit? (same response options as for question 3).

The survey vendor attempted to complete 50 surveys from each primary care department (internal medicine, family medicine, or pediatrics) at each site. Surveys were initially administered by email. If the member did not respond, they received a reminder email. Patients without email addresses received a phone survey with up to 5 call attempts. Surveys were typically completed within a week of the visit, with a historical response rate of 35%.

Sample Size Estimation

We defined a 1.0% absolute reduction in the rate of missed appointments (eg, from 6.0% to 5.0%) as an operationally significant effect. We calculated that 100,000 primary care visits, equally apportioned among study arms, would detect this difference, and projected that 2 months would enable us to reach this sample size. To estimate the power to detect differences in visit satisfaction, we used pretrial data that 84% of patients rated their visit as excellent or good to determine that 726 surveys per study arm would detect a 5% change in satisfaction.

Statistical Methods

To compare patient demographics, clinical characteristics, and intervention outcomes among the 3 intervention groups, we used Wilcoxon rank sum tests for nonnormally distributed continuous variables and χ² tests for discrete variables. We conducted a prespecified subgroup analysis to compare the effectiveness of the reminder interventions in the quartile of patients at highest risk of missing appointments with that in the 3 quartiles at lower risk. We developed a predictive model for the risk of missing an appointment based on previously described variables from the electronic health record. We excluded cancelled visits from this model. The clinical department (designated as an integrated primary care department, family medicine, internal medicine, or pediatrics) and the 25 clinic sites were entered into the model as fixed effects to address clustering of patients within departments and departments within clinic site. Linear and quadratic terms for continuous predictor variables were assessed, and both terms were included in the final model if statistically significant. Missing values for each predictor were included in the model as a separate category. Backward selection with Wald χ² tests guided the selection of predictors. The final prediction model included an indicator variable for treatment group and all covariates with P values <.05. The discrimination of the model was assessed with the C statistic, defined as the probability that in any 2 individuals randomly drawn from the sample, the predicted risk of a missed appointment is higher among the individual who actually missed the appointment.¹⁹

This project was approved by the KPCO Institutional Review Board with a waiver of individual informed consent. Because the project was conducted to address an operational issue, it was not registered on clinicaltrials.gov.

RESULTS

A total of 65,038 appointments were scheduled for 55,829 patients at 25 primary care clinics over the 2-month study period. After exclusions shown in the Figure, we analyzed 54,066 visits for 54,066 patients. Text messages were sent to 41,339 individuals (76.4%), IVR calls were made to 8038 (14.9%), and 4689 (8.7%) could not be reached by either modality. Of these 54,066 patients, 18,135 were randomized to receive a 3-day reminder call, 18,029 to receive a 1-day reminder call, and 17,902 to receive both 3-day and 1-day reminders. Patient characteristics were balanced across intervention arms (Table 1 [part A and part B]).

Overall, 5.2% of appointments were missed, 15.2% were cancelled, and 79.7% were completed. As shown in Table 2, the group receiving both 3-day and 1-day reminders had the lowest rate of missed visits (P <.0001). The number needed to treat (NNT) was 76; in other words, 1 fewer appointment was missed for every 76 patients who received both reminders compared with those who received a single reminder. In separate comparisons between missed and completed appointments and between cancelled and completed appointments, the intervention reduced missed appointments (P <.0001) but had no effect on cancellations (P = .53). The 3 groups did not differ in satisfaction with the number or timing of reminders or in overall visit satisfaction (Table 2). Individuals who received 2 IVR-T reminders did report receiving more reminder messages (a mean of 1.8 reminders vs 1.5 reminders in the other 2 groups; P <.0001).

Text messages and telephone calls were equally effective in reducing the rate of missed appointments. Among those who received text messages, rates of missed appointments were 5.6%, 4.9%, and 4.2% for the 3-day, 1-day, and both-days intervention groups, respectively, whereas among those who received calls, rates of missed appointments were 4.8%, 4.4%, and 3.8% (P <.001).

The statistical model for missed appointments included 45,873 patients after excluding 8193 individuals who cancelled appointments. Predictors of missing an appointment are shown in Table 3 (part A and part B). The strongest individual predictor was the number of missed outpatient appointments within the prior 6 months. The multivariable model showed excellent discrimination (C statistic = 0.93). A model limited to linear and quadratic terms for the number of prior missed visits also showed excellent discrimination (C statistic = 0.92).

The missed appointment rate in the highest-risk quartile was 23.3%, compared with 0.4% in the 3 lower-risk quartiles. In the highest-risk quartile, 2 reminders reduced the rate of missed appointments by 4.5% compared with the 3-day reminder and by 3.7% compared with the 1-day reminder (Table 4). The 2-reminder intervention had an NNT of 25 compared with a single reminder. In the 3 lower-risk quartiles, 2 reminders remained significantly more effective than a single reminder, but the NNT was 1328. The interaction among subgroups was statistically significant (P = .047).²⁰

DISCUSSION

In this 3-armed randomized trial, we found that 2 reminders, delivered by text message or telephone 3 days and 1 day prior to a primary care visit, were more effective than a single reminder delivered either 3 days or 1 day prior to a visit. The intervention was effective despite a low baseline rate of missed appointments.^1,5 Member satisfaction with the reminder process and the primary care visit was similar in the 3 study arms. The effectiveness of the intervention was substantially greater in patients at highest risk of missing their appointments, as defined by a multivariable prediction model.

The effectiveness of a single visit reminder in medical settings is well established.^1-5 Less is known about the effect of varying the timing or number of reminders. One randomized trial in an endoscopy clinic found that IVR reminders 3 days versus 7 days prior to the appointment were equally effective.²¹ Systematic reviews have found that the effectiveness of a single reminder was comparable when delivered between 1 and 7 days prior to a visit, although they did not report any trials that directly compared the timing or number of reminders.^4,5 Thus, our finding that 2 automated reminders were more effective than a single reminder in reducing missed appointments suggests a simple approach to increase reminder effectiveness.

The patient-level predictors of missed appointments in Table 3 are similar to those found in prior studies.^8,12-16 The C statistic of 0.93 indicates that the model was able to accurately discriminate between individuals who completed their appointments and those who missed them.¹⁹ Other organizations have also reported high predictive accuracy for such models.^13,16 Although several sociodemographic and clinical variables contributed to the model, a model including just the number of prior missed appointments had a C statistic of 0.92, which suggests that attendance at prior appointments alone can accurately identify individuals at risk of subsequent missed visits.

Our analysis of high-risk patients was particularly informative. We developed a multivariable risk score rather than assessing intervention impact on multiple subgroups as defined by individual risk factors.^20,22,23 This analysis showed that among 75% of patients, the risk of a missed appointment was less than 1% (Table 4). Although 2 reminders were slightly more effective than a single reminder in this group, the NNT of 1328 demonstrated that reminders for primary care visits were rarely beneficial. In contrast, the model identified a high-risk quartile of patients whose rate of missed appointments was 23%, comparable with rates reported in other settings.^1,5 In this subgroup, 2 reminders were substantially more effective than 1 (NNT of 25), but they only reduced the rate of missed appointments to 20.5%. Automated text or phone messages will need to be augmented by more resource-intensive interventions, such as staff outreach or patient navigation,^16,24 to further reduce missed appointments in these individuals.

Strengths and Limitations

The study had several strengths. It demonstrated that randomization can be used to rapidly address questions of operational importance using data from administrative and clinical sources. The marginal costs of the intervention were low because operational data, the IVR-T system, and the survey protocol were already in place. The results are consistent with those of other recent studies and systematic reviews^2,5,14 and should be replicable in other healthcare systems that provide automated reminders by telephone or text message.

The study also had limitations, many of which illustrate the constraints of conducting randomized trials in an operational environment. Our intervention took place within a single integrated delivery system with a low baseline rate of missed appointments. Thus, our findings may not be generalizable to other settings. Fewer individuals were randomized into the trial than projected. Although the sample size could easily have been increased by extending the trial, we honored our original commitment to organizational leaders to complete the intervention within 2 months. Consequently, the number of completed surveys was also smaller than predicted, limiting statistical power to identify differences in satisfaction. Because only individuals who kept their appointments were included in the visit satisfaction survey, we could not assess whether those who missed or cancelled their visits had different attitudes about reminders. Additionally, although the response rate was not calculated in our study, about 35% of members typically respond to visit satisfaction surveys, raising concerns about response bias. The study was conducted over 2 months, and we did not assess the sustained effectiveness of the intervention. Some variables in our prediction model, such as the number of prior outpatient appointments, ED visits, and hospitalizations, may be more difficult to assess outside of integrated healthcare delivery systems. Although patients who made appointments online received email reminders in addition to IVR-T reminders, these individuals were likely distributed equally among the 3 study arms and did not affect the relative differences we observed. Finally, we did not determine whether appointments cancelled after the 3-day reminder were more often filled by other patients than those cancelled after the 1-day reminder.^5,15

CONCLUSIONS

If other studies confirm that multiple reminders are more effective than a single reminder in reducing missed appointments, system leaders will face the decision about whether to send multiple reminders to all patients with upcoming appointments or restrict this approach to patients at highest risk. An argument in favor of the first approach is that multiple reminders did not appear to reduce visit satisfaction. However, if local satisfaction surveys suggest that patients view repeated reminders as unnecessary or intrusive or if an external IVR-T vendor bases their charges on the number of reminders sent, restriction of multiple reminders to high-risk patients might be justified despite the additional analytic effort necessary to implement the risk algorithm in real time.

In conclusion, we found that multiple text message or telephone reminders were more effective than a single reminder in reducing missed primary care visits, particularly in patients at highest risk. This study also demonstrates the value of a learning health system collaboration between operational leaders and researchers to address pragmatic questions of immediate concern.⁷Author Affiliations: Institute for Health Research, Kaiser Permanente Colorado (JFS, MRS, SX), Denver, CO; Kaiser Foundation Health Plan of Colorado (JZD, AK), Denver, CO.

Source of Funding: This project was funded by the Kaiser Permanente Health Plan of Colorado.

Author Disclosures: Dr Steiner received internal grant support from Kaiser Permanente to conduct this study. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (JFS, MRS, JZD, AK, SX); acquisition of data (JFS, MRS, JZD, SX); analysis and interpretation of data (JFS, MRS, SX); drafting of the manuscript (JFS, AK, SX); critical revision of the manuscript for important intellectual content (JFS, MRS, JZD); statistical analysis (SX); provision of patients or study materials (JFS, MRS); obtaining funding (JFS, AK); administrative, technical, or logistic support (AK); and supervision (JFS, SX).

Address Correspondence to: John F. Steiner, MD, MPH, Institute for Health Research, 10065 E Harvard Ave, Ste 300, Denver, CO 80231. Email: john.f.steiner@kp.org.REFERENCES

1. Macharia WM, Leon G, Rowe BH, Stephenson BJ, Haynes RB. An overview of interventions to improve compliance with appointment keeping for medical services. JAMA. 1992;267(13):1813-1817. doi: 10.1001/jama.1992.03480130129038.

2. Free C, Phillips G, Watson L, et al. The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLoS Med. 2013;10(1):e1001363. doi: 10.1371/journal.pmed.1001363.

3. Peterson K, McCleery E, Anderson J, Waldrip K, Helfand M. Evidence brief: comparative effectiveness of appointment recall reminder procedures for follow-up appointments. In: VA Evidence-based Synthesis Program Evidence Briefs. Washington, DC: Department of Veterans Affairs (US); 2011.

4. Hasvold PE, Wootton R. Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review. J Telemed Telecare. 2011;17(7):358-364. doi: 10.1258/jtt.2011.110707.

5. McLean SM, Booth A, Gee M, et al. Appointment reminder systems are effective but not optimal: results of a systematic review and evidence synthesis employing realist principles. Patient Prefer Adherence. 2016;10:479-499. doi: 10.2147/PPA.S93046.

6. Olsen L, Aisner D, McGinnis JM, eds. The Learning Healthcare System: Workshop Summary. Washington, DC: The National Academies Press; 2007.

7. Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157(3):207-210. doi: 10.7326/0003-4819-157-3-201208070-00012.

8. Steiner JF, Shainline MR, Bishop MC, Xu S. Reducing missed primary care appointments in a learning health system: two randomized trials and validation of a predictive model. Med Care. 2016;54(7):689-696. doi: 10.1097/MLR.0000000000000543.

9. Shoup JA, Madrid C, Koehler C, et al. Effectiveness and cost of influenza vaccine reminders for adults with asthma or chronic obstructive pulmonary disease. Am J Manag Care. 2015;21(7):e405-e413.

10. Kempe KL, Shetterly SM, France EK, Levin TR. Automated phone and mail population outreach to promote colorectal cancer screening. Am J Manag Care. 2012;18(7):370-378.

11. Bender BG, Cvietusa PJ, Goodrich GK, et al. Pragmatic trial of health care technologies to improve adherence to pediatric asthma treatment: a randomized clinical trial. JAMA Pediatr. 2015;169(4):317-323. doi: 10.1001/jamapediatrics.2014.3280.

12. Goldman L, Freidin R, Cook EF, Eigner J, Grich P. A multivariate approach to the prediction of no-show behavior in a primary care center. Arch Intern Med. 1982;142(3):563-567. doi: 10.1001/archinte.1982.00340160143026.

13. Lee VJ, Earnest A, Chen MI, Krishnan B. Predictors of failed attendances in a multi-specialty outpatient centre using electronic databases. BMC Health Serv Res. 2005;5:51. doi: 10.1186/1472-6963-5-51.

14. Parikh A, Gupta K, Wilson AC, Fields K, Cosgrove NM, Kostis JB. The effectiveness of outpatient appointment reminder systems in reducing no-show rates. Am J Med. 2010;123(6):542-548. doi: 10.1016/j.amjmed.2009.11.022.

15. Perron NJ, Dao MD, Kossovsky MP, et al. Reduction of missed appointments at an urban primary care clinic: a randomised controlled study. BMC Fam Pract. 2010;11:79. doi: 10.1186/1471-2296-11-79.

16. Percac-Lima S, Cronin PR, Ryan DP, Chabner BA, Daly EA, Kimball AB. Patient navigation based on predictive modeling decreases no-show rates in cancer care. Cancer. 2015;121(10):1662-1670. doi: 10.1002/cncr.29236.

17. Quan H, Sundararajan V, Halfon P, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43(11):1130-1139. doi: 10.1097/01.mlr.0000182534.19832.83.

18. Jha AK, Orav EJ, Zheng J, Epstein AM. Patients’ perception of hospital care in the United States. N Engl J Med. 2008;359(18):1921-1931. doi: 10.1056/NEJMsa0804116.

19. Meurer WJ, Tolles J. Logistic regression diagnostics: understanding how well a model predicts outcomes. JAMA. 2017;317(10):1068-1069. doi: 10.1001/jama.2016.20441.

20. Wallach JD, Sullivan PG, Trepanowski JF, Sainani KL, Steyerberg EW, Ioannidis JP. Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials. JAMA Intern Med. 2017;177(4):554-560. doi: 10.1001/jamainternmed.2016.9125.

21. Griffin JM, Hulbert EM, Vernon SW, et al. Improving endoscopy completion: effectiveness of an interactive voice response system. Am J Manag Care. 2011;17(3):199-208.

22. Hayward RA, Kent DM, Vijan S, Hofer TP. Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis. BMC Med Res Methodol. 2006;6:18. doi: 10.1186/1471-2288-6-18.

23. Burke JF, Hayward RA, Nelson JP, Kent DM. Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials. Circ Cardiovasc Qual Outcomes. 2014;7(1):163-169. doi: 10.1161/CIRCOUTCOMES.113.000497.

24. Shah SJ, Cronin P, Hong CS, et al. Targeted reminder phone calls to patients at high risk of no-show for primary care appointment: a randomized trial. J Gen Intern Med. 2016;31(12):1460-1466. doi: 10.1007/s11606-016-3813-0.

Download PDF: Optimizing Number and Timing of Appointment Reminders: A Randomized Trial