Correspondence regarding, "Asthma Disease Management: Regression to the Mean or Better?"

TO THE EDITORS

Tinkelman and Wilson¹ have done a laudable job addressing some of the concerns around measurement of return on investment of disease management programs, including seasonality and regression to the mean. However, their evaluation of the success of their program to diminish asthma expense in the Colorado Medicaid population has introduced substantial potential biases that diminish the strength of their conclusions.

Tinkelman and Wilson have chosen to have a "convenience" control group, as opposed to using a randomized control group. This control group is on average 4 years older than the intervention group, raising the question of overrepresentation of the disabled in the control group compared to the intervention group. There is substantial evidence that disabled and Temporary Assistance for Needy Families (TANF) Medicaid patients are very different populations,² and they should each be analyzed separately when reviewing efficacy of medical management programs. The need to eliminate 2.8% of the control group for "high use of nonasthma claims"also suggests that there were systematic differences between the intervention and control groups.

The control group was chosen because the researchers were unable to reach the provided home phone number. It is very likely that there is a systematic difference between those who researchers can reach and those who they cannot. The authors surmise that "those who were not at home were healthy and working, while those who were at home were sicker and not working."It is just as likely that those for whom the researchers did not have workable phone numbers were actually more impoverished, and therefore nonequivalent to the intervention group.

The authors also removed noncompliant patients from the intervention group, while they could not remove such patients from the control group. Clearly, those who had "lost telephone service"might have been at higher risk of adverse outcomes of their asthma. It would have been more appropriate to evaluate the intervention group on an "intention to treat"basis, since there was no similar elimination of members of the control group.

The authors acknowledge that their evaluation fails to consider the incremental cost of higher use of disease modifying inhalation agents in the intervention group. In fact, this additional cost might offset a considerable portion of the stated savings from this program.

Tinkelman and Wilson have done us a service to identify substantial regression to the mean (30.7%) in the control group of their population. More rigorous studies will be required to ascertain whether there was really substantially more decrease in cost in a comparable intervention group.

Jeffrey Levin-Scherz, MD, MBA

Boston, Mass

REFERENCES

Am J Manag Care.

1. Tinkelman D, Wilson S. Asthma disease management: regression to the mean or better? 2004;10:948-954.

2. KCMU (Kaiser Commission on Medicaid and the Uninsured). Medicaid program at a glance. Fact sheet. Washington: KCMU. January, 2004. Available at: http://www.kff.org/medicaid/loader.cfm?url=/commonspot/security/getfile.cfm&PageID=30463. Accessed December 31, 2004.

TO THE EDITORS

The foundation of Tinkelman & Wilson's conclusion¹ on the effectiveness of an asthma disease management (DM) program is based on the assumption that the reference group was equivalent to the population that received disease management services. Unfortunately, the authors do not present sufficient credible evidence to support that essential assumption.

The authors do assert that answering the phone on the first call for both the participants and non-participants was a "random variable"and that the populations selected for DM and the comparison were "statistically and clinically well-matched as cases and controls."Yet, no supporting evidence from the literature is presented to back-up such a contention. Indeed, evidence available in the paper itself reveals that the 2 populations were quite obviously different.

t

P

First, the investigators state that "individuals with extreme non-asthma use"were present in the reference group and were removed from the analysis, but no such outliers were cited as being in the intervention group. This fact is an obvious contradiction to the idea that the "random variable"caused, by itself, a group of "well-matched"controls. Second, the evidence presented in Table 1 to show comparability on age, in fact shows just the opposite. A simple statistical test using the values presented on age (mean and standard deviation) calculates to a statistic of 3.511 and a statistically significant difference on this important baseline characteristic (<.001).

This study would have been far more persuasive if the authors had 1) presented credible evidence from the scientific literature suggesting that their selection method (answering the telephone on the first attempt) has been shown to produce equivalent comparison groups elsewhere; and 2) if they increased the number of baseline metrics available in claims data (eg, comorbid conditions, inpatient and other utilization statistics, etc) in both the reference and intervention groups and offered appropriate statistical tests of their differences.

The "equivalence"principle is perhaps the most essential component of any study.² Like many before us, we recommend that investigators studying defined population interventions include a self-critical discussion of the "strength of evidence"supporting their conclusions,³ especially the degree to which selection bias was avoided.⁴ The extent to which a study achieved "internal validity"(and the extent to which it did not) should be supported by evidence from the literature and/or data produced by the study itself.

Wilson Research, LLC

Ariel Linden, DrPH, MS

Hillsboro, Ore

REFERENCES

Am J Manag Care.

1. Tinkelman D, Wilson S. Asthma disease management: regression to the mean or better? 2004; 10(12):948-954.

Dis Management.

2. Wilson TW, MacDowell M. Framework for assessing causality in disease management programs: principles. 2003;6(3):143-158.

Proc R

Soc Med.

3. Bradford-Hill A. The environment and disease: association or causation? 1965; 58:295-300.

Am J Manag Care.

4. Linden A, Roberts N. A user's guide to the disease management literature: recommendations for reporting and assessing program outcomes. 2005;11(2):81-90.

IN REPLY

The correspondents have brought up some important points regarding the limitations of our study. So as to not "throw the baby out with the bath water,"however, we would like to address some of these concerns. While acknowledging that both the methods and the data may not be perfect, at the same time we submit that some useful information can be garnered from these results.

We would argue that both Levin-Scherz and Wilson and Linden have perhaps overstated the likelihood of systematic differences between the treatment and control groups. While it is correct that the control group was not formally randomized and thus, could reasonably be construed as one of "convenience,"at the same time, the manifest similarity of the 2 groups suggests that they likely were not systematically dissimilar in several important ways. This methodology is consistent with that described previously by Haley.¹ We certainly agree, in accordance with that principle, "since the members of a convenience sample were not selected by a true random sampling method, one will not have the same degree of assurance that the results can be generalized to the target universe as one would have with a random sample. However, many convenience samples are free enough of obvious biases to provide accurate estimates of their target universes."^1,p54

Another issue raised in both letters relates to the age of the populations. While the ages were statistically different in the 2 groups (with the given group sizes and standard deviations, a difference of 2.1 years would have been statistically significant), the mean and range were very similar. This was also true for the other measured variables (sex, asthma utilization, and non-asthma utilization). We acknowledge it is possible that there were more disabled individuals in the control group or that the control group was more impoverished. However, it is difficult to conceive how these differences might not affect costs in the baseline year, but then go on to affect costs in the treatment year. Thus, we would argue that this supposition is possible but not likely.

Dr. Levin-Scherz makes an excellent point about using an intention to treat analysis, at least on the individuals who were excluded from the treatment group for reasons other than loss of eligibility for insurance. We would absolutely remedy this oversight in future studies. We also agree with his conclusion that replication of these findings is important step to further substantiating the benefits of asthma disease management. In addition, we agree with all of the valuable suggestions from Wilson and Linden, which indeed would have strengthened this paper significantly. We were constrained by space limitations by the journal and had to limit our presentation of these points in the Discussion. We thank both Drs. Levin-Scherz and Wilson for sharing their valued insights.

Steven M. Wilson, MA

National Jewish Medical and Research Center

REFERENCES

Techniques of Patient-Oriented Research.

1. Haley RW. Designing Clinical Research. In: Pak CYC, Adams PM, eds. New York: Raven Press; 1994:47-80.