Validity of the Asthma Control Test™ Completed at Home

Objective: To provide additional validity data for the Asthma Control Test^TM (ACT) using a different criterion measure, setting, and population.

Study Design: Cross-sectional survey.

Methods: Questionnaires were completed at home by a random sample of 570 members of a large integrated healthcare organization who were 35 years or older with utilization suggestive of active asthma. The questionnaires included the ACT; another validated asthma control questionnaire (Asthma Therapy Assessment Questionnaire™ [ATAQ]), which was used as the criterion measure; a validated quality-of-life tool (Mini Asthma Quality of Life Questionnaire [Mini-AQLQ]); a validated symptom frequency scale (Asthma Outcomes Monitoring System); and information regarding demographics.

Results: The ACT score was statistically significantly correlated with findings on the ATAQ (ρ = -0.73), Mini-AQLQ (ρ = 0.77), and symptom frequency scale (ρ = -0.69). The optimal ACT cutoff for well-controlled asthma (ATAQ level, 0) was confirmed to be 20 or higher (sensitivity, 78.1%; specificity, 83.8%), and the optimal ACT cutoff for poorly controlled asthma (ATAQ level, 3-4) was confirmed to be 15 or lower (sensitivity, 90.4%; specificity, 80.9%).

Conclusion: These data further support the validity of the ACT in the home setting among a random sample of patients with asthma.

(Am J Manag Care. 2007;13:661-667)

The lack of a clinical gold standard for determining asthma control may contribute to poorly controlled asthma. Validated questionnaire tools have been developed to measure asthma control.

The Asthma Control Test (ACT) is a previously validated tool, although optimal validation requires data from a variety of settings and populations and the use of various criterion and construct measures.

The ACT score was statistically significantly correlated with findings on the Asthma Therapy Assessment Questionnaire, Mini Asthma Quality of Life Questionnaire, and Asthma Outcomes Monitoring System. These data support the validity of the ACT in the home setting.

Asthma is a common chronic condition that is associated with substantial morbidity.¹ Despite guidelines that enumerate the goals of asthma control and define management strategies, asthma remains poorly controlled in most patients.^2-4 The lack of a clinical gold standard for determining asthma control, as well as inadequate recognition of uncontrolled asthma by patients and physicians, may contribute to this situation.⁴ An approach to this problem has been the development of validated questionnaire tools to measure asthma control in the clinical setting.^5-7

One such tool is the Asthma Control Test™ (ACT), a 5-item survey that assesses interference with activity, shortness of breath, nocturnal symptoms, rescue medication use, and self-rating of asthma control. It was developed using asthma specialist (allergist or pulmonologist) assessment of asthma control as the criterion measure of asthma control in a population of patients followed up by these specialists⁷ and was validated in a separate sample of patients new to these specialists.⁸ In these studies, a score of 20 or higher was found to be the most discriminating cutoff to define totally controlled or well-controlled asthma, and a score of 15 or lower was identified as the most discriminating cutoff to define asthma that was poorly controlled or not controlled at all.^7,8 However, optimal validation is a process that requires data to be obtained in a variety of settings, among different populations, and using various criterion and construct measures. The objective of this study was to provide additional validity data for the ACT completed at home by a random sample of patients with asthma. The Asthma Therapy Assessment Questionnaire (ATAQ) was chosen as the criterion measure for this study because it has been previously validated and was shown to be related to asthma-specific quality of life and to subsequent healthcare utilization.^5,9

METHODS

Study Subjects

Study participants were members of the Kaiser Permanente Colorado Region, a large managed care organization. Because this study was part of a larger study of discrepancies in asthma mortality between men and women that occur in patients 35 years or older, the study population was limited to this age group. Health plan members were eligible for the study if they were at least 35 years of age and had in the 2-year period preceding the survey (1) at least 1 documented medical encounter with a diagnosis of asthma (International Classification of Diseases, Ninth Revision code of 493.xx) and (2) at least a 6-month supply of asthma medication dispensed. A search of electronic medical records identified 4609 individuals who met these criteria, and we mailed surveys to a randomly selected subset of 800 subjects in the fall of 2002. Of 688 persons (86.0%) who completed the questionnaire, 65 denied having physician-diagnosed asthma and were excluded from further analysis. An additional 53 did not have complete data on the ACT questions, leaving a final sample of 570 participants. Completion of the questionnaires was assumed to indicate consent to participate in the study. The study was approved by the Kaiser Permanente Colorado Region Institutional Review Board.

Study Design

This study was a cross-sectional survey of a random sample of patients with asthma. Survey information included validated patient-reported outcome tools and demographic information.

Questionnaires

The survey contained the following information:

1. The 5-question ACT⁷ assessing the prior 4 weeks, with scores ranging from 5 to 25 (higher scores indicate better asthma control).

2. The 4-question ATAQ,^5,9 another validated asthma control tool, with scores ranging from 0 to 4 (higher scores indicate more control problems). The 4 potential control problems assessed during the prior 4 weeks by the ATAQ are (1) missed work, school, or normal activity; (2) waking up at night because of asthma; (3) highest number of puffs of rescue inhaler used exceeds 12 for any day; and (4) self-rating of well-controlled asthma.

3. The 15-question Mini Asthma Quality of Life Questionnaire (Mini-AQLQ),¹⁰ an asthma-specific quality-of-life questionnaire reflecting the prior 2 weeks, with a total score and 4 domain scores (symptom, emotion, activity, and environment), with each score ranging from 1 to 7 (higher scores indicate better quality of life).

4. The 5-question Asthma Outcomes Monitoring System symptom frequency scale,¹¹ assessed during the prior 4 weeks, with scores ranging from 1 to 4, analogous to the 4-level National Asthma Education and Prevention Program¹² severity classification. Higher scores indicate increased symptom frequency, with 1 indicating intermittent symptoms and 4 indicating severe persistent symptoms.

5. Self-assessment of general health (excellent, very good, good, fair, or poor) and asthma severity (very mild, mild, moderate, severe, or very severe).

6. Demographics, including age, sex, race/ethnicity, highest grade in school completed (<high school, high school, technical school, college graduate, or postgraduate or professional), and annual family household income (<$20 000, $20 000-$34 999, $35 000-$49 999, $50 000-$74 999, or ≥$75 000).

Statistical Analysis

Internal consistency reliability of the ACT in this sample was tested using Cronbach a. Hypothesis testing regarding the relationships between the ACT and demographic factors was by means of Wilcoxon rank sum test (2 samples) or Kruskal- Wallis (>2 samples) testing. The ATAQ was used as the criterion measure for determining criterion validity, namely, “the correlation of a scale with some other measure of the trait or disorder under study, ideally a gold standard that has been used and accepted in the field."^13(p147) Relationships between the ACT and the ATAQ were assessed by correlating the ACT score with the ATAQ level (Spearman rank correlation coefficient, ρ) and by determining the relationship between the mean ACT score and the ATAQ level (Kruskal-Wallis test).

Well-controlled asthma in this study was defined as an ATAQ level of 0, and poorly controlled asthma was defined as an ATAQ level of 3 or 4 because patients with ATAQ levels of 3 or 4 were 3.5 times more likely to experience subsequent acute asthmatic episodes compared with those without control problems.⁹ Sensitivities, specificities, positive predictive values, negative predictive values, and areas under the receiver operating characteristic curve (the C statistics in logistic regression models) were calculated for various ACT score cutoffs in relation to well-controlled and poorly controlled asthma, as already defined by ATAQ levels. The optimal cutoffs were chosen based on the cutoffs that produced the highest area under the receiver operating characteristic curve for wellcontrolled and for poorly controlled asthma. Convergent validity tests the relationship between a scale and other variables to which it should be related.¹³

Convergent validity was tested by determining the correlations of the ACT with the following characteristics to which asthma control should be related: (1) patient general health rating, (2) patient self-rating of asthma severity, (3) total and domain scores of the Mini-AQLQ, and (4) asthma symptom frequency scale. Potential effects of demographic factors on the validity of the ACT were assessed by evaluating the following in demographic subgroups: (1) Spearman rank correlation coefficients between the ACT score and the total score of the Mini-AQLQ, (2) the mean ACT score in the 4 ATAQ levels (Kruskal-Wallis test), and (3) the mean ACT score in the 4 levels of the symptom frequency scale (Kruskal-Wallis test). These subgroups were defined as age younger than 65 years, age 65 years or older, men, women, white race/ethnicity, other race/ethnicity, education less than high school, high school or technical education, college or postgraduate education, annual income of less than $35 000, and annual income of $35 000 or higher.

Nominal statistical significance was set at P < .05 using 2-sided tests. All analyses were performed using statistical software (SAS version 8.2 for Windows; SAS Institute Inc, Cary, NC).

RESULTS

The demographics of the study population are given in Table 1. Participants tended to be middle-aged, of white race/ethnicity, well educated, and nonpoor. Patients with less educational attainment and lower incomes had statistically significantly lower ACT scores. The internal consistency reliability of the ACT in this sample was 0.85. Asthma and health characteristics of the sample are given in Table 2.

Criterion Validity

The analyses of the area under the receiver operating characteristic curve identified the optimal cutoff for wellcontrolled asthma as 20 or higher (Table 3) and the optimal cutoff for poorly controlled asthma as 15 or lower (Table 4). The ACT score was statistically significantly correlated with findings on the ATAQ (ρ = -0.73, P < .001), and the mean ACT score statistically significantly varied by ATAQ level (Table 5). In addition, the likelihood of well-controlled asthma (ACT score ≥20) statistically significantly decreased with an increasing number of control problems identified by the ATAQ; conversely, the likelihood of poorly controlled asthma (ACT score ≤15) statistically significantly increased with an increasing number of control problems identified by the ATAQ.

Convergent Validity

The ACT was strongly correlated with the patient self-rating of asthma severity; total score of the Mini-AQLQ; symptom frequency scale; and symptom, activity, and emotion domains of the Mini-AQLQ (Table 6). Only modest correlations were seen with the patient general health rating and the environment domain of the Mini-AQLQ. The ACT score remained statistically significantly related (P < .001, Kruskal-Wallis test) to ATAQ and asthma symptom frequency scale levels in all demographic subgroups, and correlations between the ACT score and the total score of the Mini-AQLQ were similar and statistically significant in all subgroups (data not shown).

DISCUSSION

The importance of defining and achieving asthma control is being increasingly recognized.^14,15 Simple validated tools that measure asthma control are recommended as a means of defining asthma control,^14,15 but the validity of such tools needs to be confirmed using a spectrum of validation criteria in different settings and among various populations. The present study adds to the body of data supporting the validity of the ACT. It shows for the first time (to our knowledge) that the ACT correlates strongly with the ATAQ (a criterion measure) and with several related asthma characteristics, including asthma-specific quality of life and symptom severity. Unlike the published validation data to date,^7,8 the ACT questionnaires in the present study were completed at home. This increases our confidence in the ability to use the ACT as a home assessment tool, whether completed before an office visit or independent of an office visit.

In addition to using a variety of criterion and construct measures and settings to optimally validate a patient-reported outcome tool, different populations must be assessed to confirm generalizability. In contrast to the prior validation studies,^7,8 the present study assessed a general population of patients with asthma identified by random sampling of an integrated healthcare organization database. The patients in our study had evidence of active asthma based on utilization, while no specific severity criteria are mentioned in prior validation reports.^7,8 Patients in the present study were older (mean age, 58 years) than patients previously studied in the development (mean age, 45 years)⁷ or longitudinal validation (mean age, 35 years)⁸ studies. An important related issue is the degree to which the validity of the tool depends on educational or other demographic characteristics of the population. The criterion and convergent validity results in this study did not seem to vary among demographic subgroups (data not shown). However, this sample of members from an integrated healthcare organization did not represent a demographically heterogeneous population. Therefore, further validation data for the ACT among more diverse populations, especially regarding educational attainment and income, would be important.

Although the present validation study differed in setting, criterion measures, and populations from the prior studies,^7,8 it is important to note that previously defined optimal cutoffs for well-controlled asthma (ACT score ≥20)^7,8 and poorly controlled asthma (ACT score ≤15)⁸ were confirmed. This increases our confidence in using these cutoffs to identify patients in need of intervention in various settings. For example, this study specifically supports the use of the ACT as a mail outreach tool in managed care organizations to identify patients with well-controlled, not well-controlled, and poorly controlled asthma as defined in the new National Asthma Education and Prevention Program¹² guidelines. Furthermore, the measurement properties of the ACT identified in this study would allow the use of different cutoffs for well-controlled or poorly controlled asthma if increased sensitivity, specificity, or positive predictive value was desired in any specific circumstance (Tables 3 and 4).

If the goal of asthma control assessment is the definition of well-controlled, not well-controlled, or poorly controlled asthma, findings from this study suggest that the ACT and the ATAQ could be similarly useful. However, because the ACT is a 25-point scale, compared with the 5-level scale of the ATAQ, the ACT score expressed as a continuous measure should provide more precision than the ATAQ level. Responsiveness of the ACT has been demonstrated in another sample,⁸ and one might expect the ACT expressed as a continuous measure to be more responsive than the ATAQ. This property would be especially useful in clinical trials, although it could also be useful for serial measurements as part of longitudinal follow-up in a clinical practice setting. Although (to our knowledge) the actual responsiveness of the ACT vs the ATAQ has not been compared, we have shown that the ACT as a continuous measure correlates more strongly than the ATAQ with the Mini-AQLQ.¹⁵

A potential limitation of this study is that spirometry measurements were not obtained. Forced expiratory volume in the first second of expiration (FEV₁) is considered a measure of asthma control¹⁴ that has been shown to reflect dimensions different from symptoms or quality of life.¹⁶ However, spirometry is not available in many clinical settings and is not usually available at home; therefore, a valid measure of asthma control that does not require spirometry would be useful. In addition, the importance of the lack of spirometry measurements in this study is mitigated by 2 additional observations. First, FEV₁ results have been shown to be statistically significantly correlated with ACT scores,⁷ and that study demonstrated that the ACT scores correlate better than pulmonary function test results with specialist determination of the level of asthma control. Second, Juniper et al¹⁷ showed that the Asthma Control Questionnaire, another validated asthma control tool, demonstrated similar measurement properties whether or not spirometry results were included. Although the diagnosis of asthma was not confirmed by clinical or spirometric evaluation, identifying physician-diagnosed asthma is a valid way to identify patients with asthma in epidemiologic studies.¹⁸

In summary, this study provides further reliability and validation information regarding the ACT, supporting its use as a practical tool in the home setting to assess disease control among the general population of patients with asthma. It is anticipated that the results of this study will lead to improved asthma control in patients with asthma by encouraging the use of validated control tools to identify inadequately controlled asthma that needs additional intervention.

Author Affiliations: From the Department of Allergy, Kaiser Permanente, San Diego, Calif (MS, RSZ); Center for Health Research, Kaiser Permanente, Portland, Ore (DMM, WMV, EO); QualityMetric Corporation, Lincoln, RI (MK); and Clinical Research Unit, Kaiser Permanente Colorado, Denver (DJM).

Funding Source: This study was supported by grant 200-95-0953 from the Centers for Disease Control and Prevention.

Author Disclosures: Dr Schatz reports serving as a paid consultant to GlaxoSmithKline; receiving grants from sanofiaventis, GlaxoSmithKline, and Merck; receiving lecture fees from GlaxoSmithKline and Genentech; and having a grant pending from Schering-Plough. The authors (DMM, MK, WMV, DJM, EO, RSZ) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (MS, RSZ); acquisition of data (WMV, DJM, EO); analysis and interpretation of data (MS, MK, WMV, EO, RSZ); drafting of the manuscript (MS, DMM, MK, WMV, EO, RSZ); critical revision of the manuscript for important intellectual content (MS, DMM, MK, WMV, DJM, EO, RSZ); statistical analysis (MS); provision of study materials or patients (EO); obtaining funding (WMV, DJM); and administrative, technical, or logistic support (RSZ).

Address correspondence to: Michael Schatz, MD, Chief, Department of Allergy, Kaiser-Permanente Medical Center, 7060 Clairemont Mesa Blvd, San Diego, CA 92111. E-mail: michael.x.schatz@kp.org.

1. Mannino DM, Homa DM, Akinbami LJ, Moorman JE, Gwynn C, Redd SC. Surveillance for asthma: United States, 1980-1999. MMWR Surveill Summ. 2002;51:1-13.

2. Adams RJ, Fuhlbrigge A, Guilbert T, Lozano P, Martinez F. Inadequate use of asthma medication in the United States: results of the asthma in America national population survey. J Allergy Clin Immunol. 2002;110:58-64.

3. Fuhlbrigge AL, Adams RJ, Guilbert TW, et al. The burden of asthma in the United States: level and distribution are dependent on interpretation of the National Asthma Education and Prevention Program guidelines. Am J Respir Crit Care Med. 2002;166:1044-1049.

4. Boulet LP, Phillips R, O’Byrne P, Becker A. Evaluation of asthma control by physicians and patients: comparison with current guidelines. Can Respir J. 2002;9:417-423.

5. Vollmer WM, Markson LE, O’Connor E, et al. Association of asthma control with health care utilization and quality of life. Am J Respir Crit Care Med. 1999;160:1647-1652.

6. Juniper EF, O’Byrne PM, Guyatt GH, Ferrie PJ, King DR. Development and validation of a questionnaire to measure asthma control. Eur Respir J. 1999;14:902-907. 7. Nathan RA, Sorkness CA, Kosinski M, et al. Development of the Asthma Control Test: a survey for assessing asthma control. J Allergy Clin Immunol. 2004;113:59-65. 8. Schatz M, Sorkness CA, Li J, et al. Asthma Control Test: reliability, validity, and responsiveness in patients not previously followed by asthma specialists. J Allergy Clin Immunol. 2006;117:549-556.

9. Vollmer WM, Markson LE, O’Connor E, Frazier EA, Berger M, Buist AS. Association of asthma control with health care utilization: a prospective evaluation. Am J Respir Crit Care Med. 2002;165:195-199.

10. Juniper EF, Guyatt GH, Cox FM, Ferrie DJ, King DR. Development and validation of the Mini Asthma Quality of Life Questionnaire. Eur Respir J. 1999;5:35-46. 11. Bayliss MS, Espindle DM, Ware JE. Asthma Outcomes Monitoring System (AOMS) Administration, Scoring, and Interpretation Manual. Lincoln, RI: QualityMetric Inc; 1999.

12. National Heart, Lung, and Blood Institute Web site. Guidelines for the diagnosis and management of asthma (National Asthma Education and Prevention Program Expert Panel report 3). http://www.nhlbi.nih.gov/guidelines/asthma/. Accessed October 16, 2007.

13. Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to their Development and Use. Oxford, England: Oxford University Press; 2000.

14. Joint Task Force on Practice Parameters, American Academy of Allergy, Asthma and Immunology; American College of Allergy, Asthma and Immunology and Joint Council of Allergy, Asthma and Immunology. Attaining optimal asthma control: a practice parameter [published correction appears in J Allergy Clin Immunol. 2006;117:262]. J Allergy Clin Immunol. 2005;116:S3-S11.

15. Schatz M, Mosen DM, Kosinski M, et al. The relationship between asthma-specific quality of life and asthma control. J Asthma. 2007;44:391-395.

16. Juniper EF, Wisniewski ME, Cox FM, Emmett AH, Nielsen KE, O’Byrne PM. Relationship between quality of life and clinical status in asthma: a factor analysis. Eur Respir J. 2004;23:287-291.

17. Juniper EF, Svennson K, Mork AC, Stahl E. Measurement properties and interpretation of three shortened versions of the Asthma Control Questionnaire. Respir Med. 2005;99:553-558.

18. Toren K, Brisman J, Jarvholm B. Asthma and asthma-like symptoms in adults assessed by questionnaires: a literature review. Chest. 1993;104:600-608.

Download PDF: Validity of the Asthma Control Test&trade; Completed at Home