Publication
Article
Robustness of a Computer-assisted Diabetes Self-management Intervention Across Patient Characteristics, Healthcare Settings, and Intervention Staff
Author(s):
Background: A major problem in the dissemination of most interventions found to be efficacious is that they are of limited or unknown generalizability.
Objective: To document the "robustness," or external validity, of a computer-assisted diabetes self-management program across different patient characteristics, healthcare settings (mixed payer vs health maintenance organization), intervention staff, and outcomes.
Study Design: A randomized controlled trial evaluating a computer-assisted behavior change program for adult patients with type 2 diabetes mellitus (n = 217) vs a computerized health risk assessment.
Methods: Outcomes were identified using the RE-AIM framework and included program adoption among physicians, reach across patient groups, implementation, and behavioral (fat intake and physical activity) and biological (glycosylated hemoglobin and lipid levels) effectiveness measures.
Results: The program achieved 41% patient participation, variable adoption across healthcare settings (76% of health maintenance organization physicians vs 18% of non-health maintenance organization physicians participated), good implementation, and improvement in behavioral outcomes. There were few significant interactions between treatment condition and patient characteristics, type of healthcare setting, or interventionist experience on effectiveness measures.
Conclusions: Patients and physicians were willing to participate in a computer-assisted dietary and physical activity goal-setting intervention, although participation varied by healthcare setting. Interventionists from different backgrounds successfully delivered the intervention, and the results appear robust across various patient and delivery characteristics.
(Am J Manag Care. 2006;12:137-145)
Given the well-documented gap between research and clinical practice for diabetes mellitus and many other health conditions,1 there has been a call for practical clinical trials2,3 that may be more generalizable. There is concern about the relevance of many intervention studies because of their unknown or limited applicability to the situations of different patients, clinicians, and decision makers.
This article evaluates the generalizability of a computer-assisted program to help primary are physicians and patients address the challenges of diabetes self-management. Given that physicians have limited time to address prevention issues4,5 and that little diabetes self-management support is delivered in most clinical settings,6-8 we hypothesized that a computer-assisted program might have several advantages. If appropriately constructed using patient-centered principles,9-11 computer-assisted technologies have great potential to inform, leverage, and support patient-provider communication and to enhance behavior change.12,13 Legitimate questions have been raised, however, about the applicability and effectiveness of computer technology across various settings and different patient subgroups.14
From a conceptual and analytic perspective, these questions concern "effect modification" or "interaction effects"15 and are related to the external validity of a study. We refer to the consistency of effects across these various facets of generalizability16 as the "robustness" of an intervention. To facilitate dissemination of research findings, it is necessary to extend the scope of our evidence base beyond a sole focus on efficacy.17,18
The RE-AIM framework considers 5 dimensions that are important for evaluating the potential public health effects and generalizability of an intervention.17-19 This study used the RE-AIM model to identify outcomes for robustness analyses. The 5 components of the RE-AIM model (and their application to this program) are as follows: (1) reach (the percentage and representativeness of primary care patients with diabetes mellitus who were willing to participate in a computer-assisted, self-management program), (2) effectiveness (intervention effects on targeted outcomes, including behavioral and physiological measures, and whether effects were robust across patient subgroups and interventionists), (3) adoption (the percentage and representativeness of primary care physicians willing to sponsor and refer their patients to this program), (4) implementation (how consistently the various intervention components were delivered by diverse intervention personnel and to different patient subgroups), and (5) maintenance (the extent to which patients continued their participation in the program). Maintenance also has indicants at the setting level (see http://www.re-aim.org), but such indexes were not available for this article.
The objective of this study was to evaluate the robustness of effects of a computer-assisted diabetes self-management program. The analyses focus on behavior change and biological outcomes. Less frequently studied outcomes are also included, such as willingness of patients and physicians to participate in the program, consistency of intervention delivery, and outcomes produced by different staff members.
METHODS
Design and Recruitment
The Diabetes Health Connection project is a randomized study of a diabetes self-management intervention relative to a computer-assisted control condition. The intervention uses an in-person, computer-assisted behavior change program to facilitate healthful dietary and physical activity practices. Primary outcomes are dietary patterns and physical activity levels. In addition, the study is collecting biological outcomes (eg, glycosylated hemoglobin [A1C]) and lipid levels).
Participating physicians were recruited from the Denver metropolitan area. Using slightly different approaches, 45 Kaiser Permanente Colorado (KPCO) health maintenance organization (HMO) physicians and 44 physicians in mixed-payer settings were contacted (Figure). The non-managed care physicians were drawn from a list provided by Copic Insurance Company, Denver, which provides malpractice insurance for most independent practices in Colorado and sponsors several popular quality improvement initiatives. These physicians were mailed letters (cosigned by the Copic Insurance Company director of quality improvement) briefly describing the study and were then telephoned to determine their interest.
Because several of us are employed by KPCO, internal channels were used to approach KPCO physicians, including contacting the family medicine and internal medicine department heads at 2 large, strategically located clinics and providing a study description to each physician on these 4 teams. Kaiser Permanente Colorado physicians were asked to return an assent form. Identification of KPCO patients was facilitated by a centralized diabetes registry.
Non-HMO patients were recruited via invitation letters from their physicians, and KPCO patients were sent invitations on KPCO letterhead, signed by the primary investigators (REG) conducting the study (Figure). The content of both letters was the same, and a postage-paid refusal postcard was provided in both cases. All patients who did not return a postcard were contacted by telephone to describe the study in detail, ascertain interest and eligibility, and enroll them in the study if possible. Participants were then mailed informed consent forms and baseline assessment materials, with instructions to complete all paperwork and bring it to their first study visit. All procedures were approved by the local institutional review board.
Participants
Forty-two primary care physicians agreed to participate, including 34 HMO and 8 mixed-payer physicians. Study patients were 217 adults with type 2 diabetes mellitus (mean + SD age, 61.0 + 10.7 years; 45% female; and 70% white or non-Hispanic), 28% of whom had a high school education or less and 59% of whom had an annual family income of less than $50 000 (Table 1). Fourteen percent of patients were Hispanic, and 12% were African American. Most participants had 2 or more chronic illnesses in addition to diabetes mellitus.
Interventions
The intervention was conducted outside of the primary care setting by research staff members ("Health Connection coaches") trained in motivational interviewing techniques.10 Seven coaches (interventionists) with varied credentials and experience were trained. Five coaches had master's degrees (in public health, health education, occupational therapy, genetic counseling, and dietetics/nutrition), and 2 coaches had bachelor's degrees. Four coaches had at least 2 years of health education or patient counseling experience. Four coaches had experience working among populations with chronic illness, but none had experience coaching people with diabetes mellitus.
Participants were randomly assigned to the intervention or to the "enhanced standard care" condition. Randomization was conducted using an assignment sequence developed by the project statistician and performed within physician to control for potential provider effects. Health Connection coaches conducted individual sessions that involved a health behavior assessment and an education program via computer for all participants, but the content of the visits differed by treatment condition.
Intervention participants received a 30- to 40-minute computer-assisted, tailored self-management (TSM) session with multimedia, personalized feedback and goal setting focused on patient-selected strategies to improve physical activity and diet.20 The program generated a 2-page printout for the participant, a 1-page summary for his or her physician that included A1C and lipid data, and a more detailed printout that the coach used to counsel the participant. The coach reviewed the personalized behavior change plan with the participant and discussed the goals and strategies chosen to ensure that the action plan was achievable and was personally relevant. Participants were also given a strength training plan, tailored to specific ability levels, that included instructions on the use of Therabands (Hygiene Corp, Akron, Ohio) to perform strengthening exercises, an illustrated instruction book, and a videotape. The TSM participants also received brief follow-up telephone calls from their coach 1 and 4 weeks after the visit to check on progress and to revise goals and strategies, as needed. Finally, a tailored newsletter was mailed to participants 3 weeks after the second telephone call.
Control participants randomized to enhanced standard care also had a 1-on-1 session with a coach but received a health risk appraisal (HRA) multimedia computer-assisted session that provided more general age- and sex-appropriate information on preventive health measures (eg, immunizations, wearing seat belts, and cancer screening). Health risk appraisal participants did not receive follow-up telephone calls or a newsletter. Physicians of control patients were sent the results of the A1C and lipid panels only, and laboratory data were sent electronically from the participating laboratory, so this did not require additional staff time.
Measures
Reach.
Reach was determined by calculating the percentage and representativeness of eligible patients who took part in the project. An eligibility percentage was calculated based on the number of potential participants who responded to questions regarding eligibility in recruitment telephone calls (eligibility rate = eligible participants/[eligible participants + ineligible participants]). This rate was applied to potential participants who declined by postcard or who refused at the telephone call before eligibility could be determined. The participation rate was then calculated by dividing the number of patients agreeing to be in the study by the estimated number of eligible patients. Patient representativeness variables for reach analyses were limited to age, sex, and healthcare setting, because other data were not available on nonparticipants.
Effectiveness.
Effectiveness was evaluated by improvement on measures of fat intake, physical activity, and A1C and lipid levels. Effectiveness and implementation were evaluated across the 7 interventionists, who were crossed with experimental condition. Dietary change was assessed by the dietary screen for high fat intake by Block et al,21 which estimates dietary fat intake based on 15 high-fat food items. Physical activity was measured using the CHAMPS22 activities questionnaire for older adults, a 45-item, self-report instrument that assesses the frequency per week of specific activities in multiple areas (eg, social, recreational, housework, and walking). The frequency per week of all moderate-intensity activities was estimated.
Biological effectiveness was evaluated by the A1C level and by the lipid ratio (ratio of total cholesterol to high-density lipoprotein cholesterol). Glycosylated hemoglobin tests were performed using a National Glycohemoglobin Standardization Program-certified BioRad Variant 2 analyzer (BioRad Laboratories, Hercules, CA), correlated to an index of glycemic control (reference range, 4.1%-6.5%) established during the Diabetes Control and Complications Trial. Fasting lipid profiles were determined using Roche (Hoffmann-La Roche Inc., Nutley, NJ) methods. Enzymatic methods were used to determine total cholesterol and triglyceride levels. Highdensity lipoprotein level was determined using a direct homogeneous enzymatic process.
Adoption.
Adoption was assessed by the percentage and the representativeness of physicians who participated in the program, compared with those who were invited but who declined participation. We calculated participation rates separately for HMO and non-HMO physicians (Figure).
Implementation.
Implementation was measured by the percentage of patients in the intervention condition who received each of the key treatment elements (ie, computer assessment and intervention, collaborative goal setting, and follow-up telephone calls).
Maintenance/Attrition.
Given the short time frame of the study, patient-level maintenance was assessed by completion rate (or, conversely, attrition) at the 2-month follow-up.
Potential moderator variables included sociodemographic characteristics (age, sex, Hispanic vs non-Hispanic ethnicity, annual family income, and marital status), medical characteristics (number of comorbid medical conditions, whether taking insulin, baseline body mass index, and whether medical care was from the HMO or not), and baseline scores on 2 psychosocial measures. These measures were (1) autonomy support, a measure of the patient's perception of support for diabetes self-management from his or her healthcare team,23 and (2) autonomy (or self-efficacy), a measure of the patient's self-confidence in the ability to self-manage his or her diabetes mellitus.23
Analyses
Preliminary Analyses.
All variables were examined for skewness and for kurtosis. Transformations were performed on 3 outcome variables having unacceptable distributional characteristics. The square root was used to transform the physical activity and the lipid ratio outcomes. The base-10 logarithm was used to transform A1C levels. Descriptive statistics were computed for all variables in the analyses, and variables were checked for colinearity. Continuous potential moderator variables were mean centered15 for regression analyses, and centered values were used in the construction of interaction terms to reduce multicolinearity and to aid in interpretation of results.
Outcome Analyses.
Potential treatment condition- ×-interventionist effects were examined using analyses of covariance. Hierarchical stepwise multiple regression analyses were performed to determine the effectiveness and the robustness of the intervention on fat intake, physical activity, A1C level, and lipid ratio. In each analysis, the baseline value of the dependent variable was entered in the first step, treatment condition was entered in the second step, potential moderator variables were entered stepwise in the third step if significant, and potential moderator variable-×-treatment condition variables were entered stepwise in the fourth step if significant.
RESULTS
Reach
P
The overall patient participation rate was 41% but varied by the type of setting from which patients received their care. A smaller percentage of HMO patients (38%) than non-HMO patients (54%) participated (< .001). The only other information available on nonparticipants was age and sex. There were no significant differences between participants and nonparticipants in either of these factors.
Effectiveness
Effectiveness analyses focused on the consistency of effects across different patient and intervention staff factors. As seen in Table 2 (and in Table 1), there was adequate variability in all of the potential moderator variables at baseline and in the dependent variables, which generally demonstrated improvement between baseline and the 2-month follow-up.
Table 3 summarizes the results of the regression analyses. For the dependent variable of estimated daily dietary fat intake, one of the potential moderator variables (sex) produced a significant main effect (greater fat intake was associated with male sex), and none of the 11 interactions with treatment condition was significant.
For physical activity (Table 3), there were no significant main or interaction effects for any of the potential moderator variables. For A1C level, there was a significant main effect of baseline body mass index, with those having greater body mass index showing less improvement in A1C level. There were also 2 significant interactions with treatment condition. Higher baseline self-efficacy was associated with A1C improvement in the HRA condition, while lower baseline self-efficacy was associated with A1C improvement in the TSM condition. Also, A1C level improved for middle-aged people but worsened for older people in the TSM group, while A1C level improved in younger people and worsened for middle-aged people in the HRA group.
Finally, for lipid ratio, there was a significant main effect of income (higher income was associated with an improved lipid ratio) (Table 3). There were no significant interaction effects.
Adoption
P
P
There was a large difference in participation rates between HMO and non-HMO physicians. More than three quarters (76%) of invited HMO physicians participated in the project, compared with only 18% of non- HMO physicians (< .001). There were no differences in participation rates between internal medicine and family practice physicians. Among the non-HMO physicians, physicians in single-physician practices were less likely to participate (1/24) than those in larger practices (7/20) (< .01). All HMO physicians practiced in multiphysician clinics.
Implementation and Interventionist Effects
P
The intervention was implemented consistently by all intervention staff, and there were no differences across the 7 interventionists in the application of any of the 3 implementation variables (computer assessment and intervention, collaborative goal setting, and followup telephone calls). All TSM patients participated in the computer-assisted program and reviewed their plan with their interventionist. Follow-up telephone calls were completed with 96% of the TSM group subjects. In terms of outcomes, there were no significant treatment group-×-interventionist effects on analyses of covariance for any of the 4 dependent variables, except for fat intake (= .048). For 5 interventionists, the HRA and TSM patients reduced their fat intake from baseline to the 2-month follow-up, with a greater reduction among the TSM patients. For 2 interventionists, the TSM patients reduced their fat intake, but the HRA patients increased their fat intake.
Maintenance/Attrition
P
The only measure available for the analysis related to maintenance was attrition. There was little attrition during the first 2 months of the study, and attrition did not differ by treatment condition (11% of TSA group subjects vs 7% of HRA group subjects, = .32). Because attrition was so low, analyses of characteristics of dropouts vs those completing follow-up were not conducted.
DISCUSSION
One reason for the gap between research and practice in chronic illness management is related to the types of questions asked by researchers and data synthesizers on one hand vs clinicians and decision makers on the other.2 Effects of potential moderating variables are seldom reported for clinical trials and in systematic reviews. However, clinicians and healthcare planners want to know whether programs will work in their settings, what kinds of patients will benefit from an intervention, and whether their staff can successfully deliver a program. To answer such questions, this study evaluated several potential patient, healthcare setting, and intervention staff factors that could potentially moderate outcomes.
The RE-AIM framework24 was used to study the effects of potential moderator variables because selfmanagement programs could appeal to different types of patients or healthcare providers, be implemented differentially for patients or by staff with different types of training, or have different effects for patient subgroups. Data on such issues help to establish the generalizability of an intervention at several levels of program effects, from recruitment to maintenance.
For the reach component of the RE-AIM model, the overall patient participation rate was reasonable (41%) but lower than that found in similar programs conducted in conjunction with usual medical care.25,26 It was surprising that non-HMO patients were more likely to participate than HMO patients (54% vs 38%). This may have been due to the greater number of alternative services available to HMO patients, such as initial diabetes education, case management, and weight loss programs. Because of Health Insurance Portability and Accountability Act of 1996 regulations, little information was available regarding characteristics of nonparticipants. Participants and nonparticipants did not differ on the 2 patient variables for which information was available, age and sex. Computer-assisted self-management programs have been found to produce high and representative participation rates,26,27 but our ability to investigate characteristics of participants vs nonparticipants was limited in this study.
By contrast, there was a large difference between HMO physicians and non-HMO physicians in the adoption rate component of the RE-AIM model (76% vs 18%). This may be a result of the more centralized management structure in the HMO, in which most primary care physicians follow recommendations by the HMO directors of disease management. In non-HMO settings, identification of eligible patients also typically required staff time to generate lists, but this information was readily available in the HMO setting. These adoption results should not be overgeneralized, as we had only one HMO setting with which to compare physicians in mixedpayer settings, and slightly different physician recruitment procedures were used in HMO and non-HMO settings. More research is needed, but these characteristics seem applicable to many staff-model HMOs and other centralized healthcare settings.
In terms of behavior change outcomes, the TSM intervention seems reasonably robust across the range of patient, healthcare setting, and interventionist factors investigated. Overall effects on behavior change measures did not translate into treatment effects on A1C level or lipid ratio. This may have been partially due to the good baseline levels of these variables (Table 2). Few factors moderated intervention effectiveness in the RE-AIM model, despite including a large number of potential moderator variables in the regression analyses on treatment effects. No patient demographic, medical condition, or healthcare setting variable was associated with differential effects on more than 1 of the 4 dependent variables. Given the large number of analyses, the few significant interactions detected (2/44) could be due to chance. One possible reason for the lack of significant interaction effects may be low statistical power, given that the sample size was based on power calculations for anticipated main effects. The sample size required to detect interactions is greater than that needed to detect a main effect of the same size.15 However, any sizable or consistent differences associated with potential moderator variables should have been detected.
Although the data are encouraging regarding the consistency of effects across different subgroups, the overall magnitude of effect was modest on health behaviors and was not significant on biological outcomes. Future research might focus on enhancing treatment effects via more frequent or more extensive intervention contacts.
A great deal of recent attention has been paid to health disparities,28 sex differences,29 and "digital divide" issues.14 This study provides encouraging findings in these areas, as neither sex nor ethnicity nor income nor other demographic variables (with the possible exception of age on physical activity levels) moderated the intervention effects. Two potential reasons for this may be the computer-assisted nature of much of the intervention and the structured protocol of activities to be completed, which led to almost universal RE-AIM model implementation, as intended.
Except for perhaps the initial lengthy meeting with a health counselor, the program was designed to be generalizable and broadly applicable, relying to a large extent on computer technology. The fact that the meeting occurred in a centralized setting outside of the medical office visit can be viewed as facilitating generalizability and adoption (our perspective) or as limiting generalizability (one reviewer's perspective). The present program is probably most appropriate for facilities such as group practices, managed care organizations, Veterans Affairs settings, or practices that use disease management services.
It is encouraging that intervention staff members with different backgrounds and experience were able to successfully implement the program. The use of computer-assisted interventions helps ensure that a program is consistently delivered.12,13 Similar results were found in a previous study26 of counseling following computer-assisted goal setting.
This investigation has several strengths, including its prospective design, use of the RE-AIM model to evaluate different outcomes related to potential dissemination, assimilation of data from different healthcare settings and interventionists, and assessment of several patient demographic, medical, and psychosocial variables that could potentially moderate effects. Limitations include the lack of long-term follow-up that is needed to deter-mine maintenance and attrition, moderate sample size that may preclude detection of small interaction effects, and absence of data regarding previous computer experience of patients.
Future research is encouraged to investigate the robustness of different self-management, diabetes, and chronic care programs. It is important to research not only patient characteristics but also healthcare setting and delivery factors that may determine the conditions under which a program will be effective. We especially encourage investigation of the characteristics of intervention staff associated with high levels of implementation and effectiveness, identification of programs that have high reach across patient groups and adoption among physicians, and evaluation of the effects of computer- assisted vs strictly staff-administered programs.
Acknowledgments
We thank Copic Insurance Company for their collaboration in introducing the project to private physician offices and thank the participating clinicians and primary care offices.
From the Clinical Research Unit, Kaiser Permanente Colorado, Penrose (REG, DKK, AKR, MJ); Oregon Research Institute, Eugene (LAS, DJT); and Department of Medicine and Center for Research Strategies, University of Colorado, Denver (PAN).
This study was funded by grant DK35524 from the National Institute of Diabetes & Digestive & Kidney Diseases, Bethesda, Md.
Address correspondence to: Russell E. Glasgow, PhD, Clinical Research Unit, Kaiser Permanente Colorado, 335 Road Runner Lane, Penrose, CO 81240. E-mail: russg@ris.net.
