Insights Into Problems Obtaining Care Believed Necessary as Measured by the Medicaid Consumer Assessment of Health Plans Survey (CAHPS)

Background: The Consumer Assessment of Health Plans Survey (CAHPS) is widely used to evaluate health plans; however there are few reports of Medicaid health plan efforts to improve performance as measured by CAHPS.

Objective: Data from CAHPS were analyzed to help plan administrators determine how they might address member reports of problems obtaining care they or their doctor believed necessary.

Study Design: Secondary analysis of cross-sectional survey data obtained from adults and children enrolled in 3 Medicaid health plans.

Methods: Cross-tabulations of CAHPS responses and follow-up questions asking enrollees to describe the problems they had obtaining care believed necessary.

Results: Problems obtaining care believed necessary were among the most frequently reported problems (13%-17% of adults, 9% of children). Problems obtaining a satisfactory personal doctor; receiving help when calling a physician's office; securing routine, urgent, and specialist care appointments as soon as desired; receiving referrals to specialists; obtaining behavioral health services and prescription medications; and problems with doctor communication were related to problems obtaining care believed necessary. Enrollees' descriptions of the problems corroborated observed relationships between CAHPS responses, and revealed dissatisfaction with doctors' care and lack of coverage for various services as additional contributors.

Conclusions: Analyses of relationships between enrollee-reported problems obtaining care believed necessary and their responses to other CAHPS items and follow-up questions identified a number of plausible causes of this problem. Given the multifaceted nature of problems with obtaining care believed necessary, a plan of action that might substantially improve this predicament was not apparent. Additional information about how health plan operations influence CAHPS results and the effect of health plan interventions on CAHPS measurements are needed to facilitate use of these data for quality improvement.

(Am J Manag Care 2003;9:797-803)

The Consumer Assessment of Health Plans Survey (CAHPS) was developed to help consumers make valid comparisons of health plans.¹ The survey assesses a variety of member experiences that are important to consumers including obtaining care believed necessary; obtaining care as soon as desired; how well providers listen to their patients and explain care; and courtesy and helpfulness of providers' office staff and health plan customer service representatives.^2,3 Data from CAHPS may be used to support health plan accreditation, and for purchasers and potential enrollees to evaluate health plans. Thus, health plans have a vested interest in achieving and maintaining good performance as measured by CAHPS.

Published studies of CAHPS data have focused on differences between health plans.^4-7 Questions about members' healthcare experiences are typically combined into a smaller number of composite scores to facilitate comparisons.⁸ Reports of relative performance identify general areas in which a plan has failed to achieve average or better performance. To formulate effective action plans and improve performance as measured by CAHPS, plans need to understand the problems their members report and determine how these problems relate to processes the health plan can affect. Representatives of health plans have expressed a need for additional analysis of CAHPS data to help them better understand their data and target improvement efforts.⁹ Therefore, a secondary analysis of CAHPS data was conducted to determine whether relationships between responses to CAHPS questions might help health plans understand a problem and formulate a quality improvement plan. In addition, the usefulness of follow-up questions asking enrollees to describe a problem they experienced was examined.

METHODS

Participating Health Plans

Six health plans affiliated with a multistate managed care organization commissioned surveys of their Medicaid enrollees in 2000, using the adult version (2.0H) of the CAHPS questionnaire. Five plans also used the version of CAHPS that focuses on children's care. Each plan received a report from a survey vendor that compared ratings and composite scores derived from their enrollees' with national benchmarks. Three of the 6 plans chose to participate in this secondary analysis that specifically examined reports of problems obtaining care believed necessary. Plan A (45,000 members), located in the Northeast, and plan B (19,000 members), in the Mid-West, were in states that mandated Medicaid beneficiaries enroll in a managed care plan. Beneficiaries in plan C (24,000 members), in a North Central state, could choose between primary care case management and managed care plans. Plans A and C were fee-for-service primary care physician gatekeeper models. Plan B was a capitated gatekeeper model. Enrollees in all plans needed to obtain a referral from a primary care physician to access specialist care. Physicians could make referrals without plan authorization. Enrollees were provided a list of primary care physicians during enrollment and encouraged to select or be assigned to one.

Survey Administration

Data were collected in accordance with Health Plan Employer Data and Information Set specifications developed by the National Committee for Quality Assurance (NCQA).¹⁰ The sampling frames were adults (18 years or older) or children (aged 12 years or younger) who had Medicaid as their primary source of health insurance and were continuously enrolled in the health plan during the 6-month period ending December 1999. Samples were drawn randomly with 1 member per household.

An NCQA-certified vendor administered the survey. A pre-notification letter was mailed in February 2000. Three days later the first survey packet containing the CAHPS questionnaire; a cover letter; a postcard written in Spanish to be returned if a Spanish version of the survey materials were needed; and a prepaid return envelope was mailed to the sample. Three days later a thank you/reminder postcard was sent. Second and third survey packets were sent out 21 and 38 days after the first mailing. Six telephone calls were attempted to collect data from survey recipients who did not respond by mail. Data collection was completed in May 2000. The survey referent period was the previous 6 months. Respondents verified that Medicaid was their primary source of healthcare coverage and they were current members of the health plan.

Additional information about the nature of problems obtaining care believed necessary was obtained from an open-ended question the investigators had incorporated into the plans' Medicaid CAHPS conducted the previous year (1999). Both 1999 and 2000 surveys asked members if they had a problem obtaining care they or a doctor believed necessary. The follow-up question added on the 1999 survey was "In the last 6 months, if you had a problem getting care you or your doctor believed necessary, please describe what care you wanted and the problem you had getting this care."

Data Analysis

P

P

P

The number of completed adult surveys available for analysis in plans A, B, and C was 451 (39% response rate), 598 (35%), and 356 (31%), respectively. There were 445 (38%) and 395 (33%) responses to the child surveys conducted by plans A and B. Compared with all members who were eligible to be surveyed, fewer adult respondents were 18 to 24 years old (plan A 28% vs. 18%, plan B 27% vs. 13%, and plan C 33% vs. 12%, < .001) or male in plans A and C (19% vs. 13% and 21% vs. 15%, < .01). Children aged 0 to 2 years were underrepresented (< .001) among respondents in plan A (39% vs. 21%) and plan B (36% vs. 20%). Sex was representative in the children's survey data.

Relationships between reports of problems obtaining care believed necessary and other CAHPS data were examined by cross-tabulation of response categories and Pearson chi-square or Fisher exact statistical tests. In other words, members who reported a problem obtaining care believed necessary were compared with those who reported no problem. Initially, cross-tabulations used all of the CAHPS response categories. Subsequently, "big" problem and "small" problem categories were collapsed into a single "problem" category to avoid extremely small cell counts. Similarly, the "never," "sometimes," "usually," or "always" response format were collapsed into dichotomized categories of "sometimes or never" versus "usually or always." This approach to analyzing CAHPS data did not appear to mask important differences, although many cells were too sparsely populated for a conclusive assessment. Respondents' ratings of their health plan, healthcare, personal doctor or nurse, and specialist on a 0 (worst) to 10 (best) scale were summarized as the percentage of members who gave a rating of 8 or better as recommended by the NCQA for comparing health plans.

The CAHPS questionnaire has several gate questions that instruct members to skip subsequent items about specific problems if they did not need or experience a service during the 6-month referent period. Respondents who inappropriately skipped a question or marked more than 1 answer were excluded from cross-tabulations on a pair-wise basis. Missing data ranged from 4% to 6% per item except for some demographic items in which data were missing for 8% to 20% of respondents. Percentages in this report included members who appropriately skipped a question in the denominator. Multivariable analysis was not done because of concerns about how the missing data and planned skip patterns in the CAHPS questionnaire would affect multivariable relationships. In addition, identification of variables that were "independently" associated with problems obtaining care believed necessary would not address the needs of plans to understand how much each correlate may have contributed to the problem, directly or indirectly.

Health Plan Feedback

To determine if the secondary analyses including responses to the open-ended follow-up question about problems obtaining care believed necessary provided useful insights into ways to improve performance, each plan was given a written report of the results. The investigators then discussed reports via teleconferences with personnel from the Medicaid, quality improvement, and customer service departments at each plan.

RESULTS

Incidence of Problems Obtaining Care Believed Necessary and Relationship to Plan Ratings

Problems obtaining care believed necessary was the fourth most common problem reported on both the adult (13%) and child (9%) surveys of the Medicaid population in plan A. In plans B and C, problems obtaining care necessary was the second most common problem reported by adult members (17% in both plans). It was the third most common problem in plan B's child survey and was reported by 9% of respondents. Percentages that reported a "big" problem obtaining care believed necessary ranged from 2% to 6% across plans and age groups.

P

P

P

P

P

Problems obtaining care believed necessary were associated with significantly lower health plan ratings. For the adult survey, the percentages rating their plan an 8 or better (higher) in the groups that did versus did not report a problem obtaining care believed necessary were 58% versus 77% (< .05) in plan A, 35% versus 68% (< .05) in plan B, and 44% versus 75% (< .05) in plan C. Similar differences were observed on the children's survey for plan A (62% vs. 85%, < .05) and plan B (44% vs. 70%, < .05).

Characteristics of Members Reporting Problems Obtaining Care Believed Necessary

As shown in Table 1, members of plan A who reported problems obtaining care believed necessary were more commonly Hispanic or Latino. On the children's survey, significantly fewer used English as their primary language. Use of languages other than English was also associated with problems obtaining care believed necessary in plan C, although only 12% of members who reported this problem did not use English as their primary language.

Relationships to Problems Obtaining Access to Doctors

P

Table 2 summarizes relationships in the adult survey between problems obtaining care believed necessary and need to access doctors as well as specific problems accessing doctors. Except where noted in the text, results of the children's survey were similar to the adult survey and are not tabulated. Fewer adult members in each plan who had problems obtaining care believed necessary had a personal physician, although this difference was not significant. Differences in percentages of groups who did versus did not report problems obtaining care believed necessary that had a personal physician were significant (< .05) among children in plans A (68% vs. 85%) and B (68% vs. 83%).

Problems obtaining a personal doctor the member was happy with were associated with problems obtaining care believed necessary in all 3 plans. Obtaining a new doctor after joining the plan was significantly related to problems obtaining necessary care in plan B only. Open-ended questions identified several reasons why members might not be happy with their physicians, including dissatisfaction with a diagnosis or treatment and their physician's apparent unwillingness to make referrals to specialists.

Among members who expressed a need to see a specialist, higher percentages reported problems obtaining care believed necessary than those reporting no problem obtaining needed care. Similarly, among respondents reporting a problem obtaining a specialist referral, higher percentages also reported a problem obtaining care believed necessary. Specialist problems were described in the open-ended follow-up question as hassles and delays created by needing to see a physician gatekeeper for a referral, and disagreements with the primary physician about the need for a specialist. Problems finding a particular type of specialist in the Medicaid network and obtaining timely appointments were also described.

Members who reported a problem obtaining care believed necessary more frequently called doctors offices for advice and made appointments for routine and urgent care, and greater proportions of those who sought care had problems obtaining advice and appointments. Many mentioned not obtaining appointments as soon as they desired when asked to describe the problem they had obtaining care believed necessary. More people who did versus did not report a problem obtaining care believed necessary in the 3 plans indicated they waited longer than 30 days for regular appointments (7%-14% vs. 2%-5%) and longer than 2 days when they felt they needed care right away (23%- 33% vs. 6%-7%).

Relationships to Problems in Doctors' Offices

Problems obtaining care believed necessary among adults are shown in relation to problems encountered during doctor visits in Table 3. Greater percentages of members who had problems obtaining care believed necessary gave their doctors low ratings and reported communication problems. Non-English language was not as common a problem as doctors not listening to members, not respecting what members had to say, not explaining things in ways members could understand, or not spending enough time with members. Few members reported they needed an interpreter.

Members who had problems obtaining care believed necessary more frequently reported that their doctor's staff was not helpful. In follow-up questions members described instances in which they called physicians' offices for advice, appointments, or referrals and phone calls were not returned or they received unsatisfactory help; sometimes the respondents reported that the office staff were rude or disrespectful.

Relationships to Problems With Health Plan Operations

As shown in Table 4, most health plan operations assessed by CAHPS did not differentiate between members of plan A who did or did not report a problem obtaining care believed necessary. One exception was delays obtaining care while waiting for plan approval, which was also associated with problems obtaining care believed necessary in plans B and C. Health plan delays were not described in the open-ended responses except for 1 case of waiting for unspecified jaw surgery.

Problems with plan paperwork were associated with problems obtaining care believed necessary in plans B and C. Respondents described only a few instances of problems obtaining a plan identification card due to confusion about Medicaid eligibility and problems with bills for uncovered services or providers. Members with problems obtaining care believed necessary in plans B and C also had more difficulty than others with written materials and customer service.

Relationships to Optional CAHPS Questions

All 3 plans used optional questions in the CAHPS battery to gather information about pregnancy care and transportation needs. Neither of these topics was significantly associated with problems obtaining care believed necessary. Plans A and B also asked optional questions about problems obtaining prescription medications and behavioral health services; these problems were significantly more common in the group that reported problems obtaining care believed necessary than the group that did not.

Responses to Open-ended Follow-up Questions

Only 29%, 42%, and 28% of plan A, B, and C respondents, respectively, provided informative responses when asked to describe problems obtaining care believed necessary in 1999. Problems that were described were consistent with relationships noted in 2000 CAHPS surveys. Open-ended queries revealed a fair amount of dissatisfaction with doctors' diagnoses and treatments that was not apparent from standard CAHPS questions, and identified a variety of difficulties obtaining care because some healthcare services were not a covered benefit.

DISCUSSION

Problems obtaining care believed necessary was among the most frequently reported problems on CAHPS surveys of Medicaid enrollees in the 3 health plans. However, only 4% to 6% of respondents reported "big" problems, and the true incidence might be lower if members who experienced problems were more likely to complete the survey. Nevertheless, enrollees who reported problems obtaining care believed necessary gave their health plans significantly lower ratings than enrollees who did not report this problem. Plan personnel did not know specifically what enrollees meant when they indicated they had a problem obtaining care believed necessary, and therefore without additional information could not formulate a plan of action that would have a reasonable likelihood of improving this CAHPS assessment.

Analyses that examined item relationships within a plan's CAHPS data and responses to additional open-ended follow-up questions suggested that "the problem" was a myriad of problems. Problems obtaining care believed necessary were significantly related to problems obtaining a satisfactory personal doctor, obtaining help when calling physician offices, obtaining routine and urgent care appointments as soon as desired, obtaining referrals to and appointments with specialists, and problems with doctor communication. Problems obtaining prescription medications and behavioral health services were also related to problems obtaining care believed necessary, as were problems with services not covered by the health plan. Asking about "problems obtaining care believed necessary" seemed to be a catchall question.

Although observed associations could not establish cause and effect, relationships between other CAHPS items and problems obtaining care believed necessary appeared to be guiding the plans in the right direction(s) to address the problems. Observed relationships had face validity, and descriptions of problems obtaining necessary care collected by open-ended follow-up questions corroborated many relationships. Given the small size of respondent subgroups who reported some problems and the multitude of comparisons, some relationships may be spurious. Plan personnel have to be careful not to spend limited quality improvement resources on trying to reduce random variation.

Unfortunately, analysis of CAHPS data could not indicate which of the many related problems contributed most to problems obtaining care believed necessary or reveal where an intervention should focus to yield substantial impact. Other researchers have noted that results based on CAHPS data alone may not be actionable because provider practices are not identified to efficiently target performance improvement initiatives.¹¹ Study of Medicare plans has suggested that better CAHPS performance was achieved by plans that collected consumer-oriented information from several additional sources representing different plan operations and individual practice sites.¹²

Follow-up questions should give plans more direct information about the causes of a problem than analysis of relationships within CAHPS data. Because CAHPS data are anonymous, plans have to select a problem a priori to add questions to the survey rather than elicit further information from those who report the problem. The open-ended questions used in this evaluation usually did not pinpoint the problem, and the low response rate did not provide quantitative estimates of how much each issue or practice site contributed to the problem.

In summary, analysis and interpretation of CAHPS data pose several difficult challenges for health plans, including potential non-response bias, conducting analysis with missing data and small numbers, and data anonymity. Despite these difficulties, relationships between problems obtaining care believed necessary and other standard CAHPS items and problem descriptions obtained by follow-up questions identified several potential contributing factors that health plans might address. Given the multifaceted nature of problems obtaining care believed necessary and the lack of information about the most influential factors, a plan of action to reduce these problems with an acceptable likelihood of success was not apparent to participating health plans. Additional information about how plan operations influence CAHPS results and the effect of health plan interventions on CAHPS measures are needed to facilitate use of these data for quality improvement.

Acknowledgments

The contributions of key staff at the 3 health plans who reviewed and discussed our analyses of their CAHPS data are gratefully appreciated. We also thank Karen Scott-Collins, MD, for valuable comments made regarding a previous version of this article.

Download PDF: Insights Into Problems Obtaining Care Believed Necessary as Measured by the Medicaid Consumer Assessment of Health Plans Survey (CAHPS)