Using CAHPS Patient Experience Data for Patient-Centered Medical Home Transformation

ABSTRACT

Objectives: To examine how primary care practices use the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey and its patient-centered medical home (PCMH) items during their PCMH transition.

Study Design: Qualitative study of practices’ use of patient experience data during PCMH transformation, based on a random sample of primary care practices engaged in PCMH transformation, stratified by region, practice size, PCMH recognition history, and use of the CAHPS PCMH survey.

Methods: We interviewed 105 practice leaders from 294 sampled practices (36% response rate) and used content analysis to identify themes about uses of patient experience data for practice improvement during PCMH transformation.

Results: Patient experience data were used primarily to assess and track improvements toward PCMH goals and as quality improvement (QI) metrics. CAHPS measures were used most often to discuss best practices, share data with patient advisory councils, and improve provider performance. The CAHPS PCMH survey helped practices improve patient-centeredness, particularly in coordinating care and supporting patient self-management and communication. The CAHPS PCMH items that assisted practices most were about patient self-management and provider referrals. Most practice leaders using the CAHPS PCMH survey felt that its items were actionable for standardizing PCMH changes or making structural changes. Practices administering CAHPS surveys focused on a more diverse set of QI areas.

Conclusions: CAHPS surveys were considered actionable for PCMH transformation and used in standardizing and coordinating care. The CAHPS PCMH items were considered integral to the continuous QI needed for moving beyond formal PCMH recognition and maximizing transformation. This supports the National Committee for Quality Assurance’s recommendation to administer the CAHPS or CAHPS PCMH survey for PCMH transformation.

Am J Manag Care. 2021;27(9):e330-e335. https://doi.org/10.37765/ajmc.2021.88745

Takeaway Points

Prior research has assessed how providers undergoing patient-centered medical home (PCMH) transformation use patient experience data to improve care experiences and make care more patient centered, but little research has examined the use of the Consumer Assessment of Healthcare Providers and Systems (CAHPS) PCMH survey.

• The CAHPS PCMH survey was used in standardizing and coordinating care.
• Practices used patient experience data most often to assess improvements in patient experience related to PCMH goals; CAHPS measures were used to discuss best practices, share data with patient advisory councils, and improve provider performance.
• Practices administering the CAHPS PCMH survey focused on a diverse set of patient experience improvement areas and found CAHPS data actionable.

The patient-centered medical home (PCMH) may help reduce the fragmentation and inefficiency of the US health care system.^1-4 The PCMH model focuses on team-based care to address patient needs and improve care experiences and outcomes while reducing costs.⁵

Although several organizations provide PCMH recognition,^6,7 more than 13,000 US primary care practices, or 15% to 18% of all primary care practices, are recognized as PCMHs under the medical home standards of the National Committee for Quality Assurance (NCQA).^8,9 PCMH implementation requires changes to multiple aspects of primary care practice.¹⁰ Full transformation may take years¹¹ and requires resources from leaders and staff.^12,13 Adopting the PCMH model entails a fundamental shift in orientation and culture.¹⁴

NCQA included performance and quality improvement (QI) measurements as part of its sixth standard to drive more comprehensive measurement and use of patient experience data. Patient experience has been assessed using the Consumer Assessment of Healthcare Providers and Systems Clinician and Group (CG-CAHPS) survey supplemented with PCMH items—that is, the CAHPS PCMH survey. This survey includes questions assessing specific aspects of PCMH delivery such as access to care and self-management support.

Research has assessed how health care providers and systems use patient experience survey data to improve patient care experiences^15-19 and make care more patient centered. Studies have documented that organizations and health care leaders encounter challenges in driving change. Case studies have shown how large systems use CAHPS data primarily for QI and have focused PCMH transformation on care delivery.^19-23

However, there has been little research examining how practice leaders use CAHPS and CAHPS PCMH survey data to improve patient experiences and support PCMH transformation. This paper examines the PCMH transformation experiences of a nationwide sample of 105 primary care practices. We investigate how practices administering different patient experience surveys used the resulting data for PCMH transformation and related QI efforts.

METHODS

Design and Sample

We created a stratified random sample of 294 practices that had applied for NCQA PCMH recognition, and 105 participated (36% response rate).¹² Participating sites differed significantly from other eligible NCQA sites on use of CAHPS surveys, PCMH Level 3 history, number of physicians, and state, but nonresponse weights increased noise without reducing differences.

We obtained consent and collected information on practices in hour-long phone interviews from June 2017 to June 2018 with practice leaders knowledgeable about the practice’s PCMH history and patient experience data. We discussed the practice’s PCMH history, PCMH transformation decisions, and change efforts. We asked how they selected and used patient experience data. We recorded and transcribed the interviews and provided a $75 honorarium to each participant.

Analytic Approach

We entered transcripts into Dedoose (SocioCultural Research Consultants), a web application for analyzing qualitative data. We established structural codes that mapped to the research questions.²⁴ We developed a code structure and codebook using systematic, inductive procedures²⁵ and content analysis to develop emerging themes.^26,27 We coded early transcripts independently, noting topics related to the research questions.^24,28

Our coding team used meetings to reach consensus on topics, identify discrepancies, refine concepts, and define and add codes.²⁹ Coders resolved discrepancies through interrater reliability exercises and obtained a pooled κ coefficient of 0.93, indicating “very good” coding agreement.^30,31 We employed ongoing training among the coding team on emerging subcodes using the Dedoose training module.

Study protocols were approved by RAND’s Human Subjects Protection Committee (IRB Assurance No. FWA00003425; IRB No. IRB00000051) and the Office of Management and Budget (No. 0935-0236).

RESULTS

Practice Characteristics

The 105 practices were distributed throughout the United States and had a range of PCMH history (Table 1). The pattern of practices’ PCMH history suggests that practices adopt the CAHPS PCMH survey during the critical transformation period that leads to the practice receiving PCMH Level 3 certification, and then practices drop the CAHPS PCMH items as their PCMH transformation status matures.

Patient Experience Survey Characteristics

Overall, 31% administered the CAHPS PCMH survey, 29% administered a “homegrown” survey, 22% administered another standardized survey, and 17% administered the core CG-CAHPS survey. Table 2 provides aggregated detail on practices’ patient experience surveys and the eAppendix Table (available at ajmc.com) shows further refined information about these surveys.

Of the 30 practices administering a homegrown survey, 13 (43%) included CAHPS items, either “exactly plagiarizing CAHPS” (5 practices) or by adapting them (8 practices). Eight of all 105 practices (8%) reported administering the CAHPS PCMH survey and other surveys: Three fielded a Medicaid survey, 3 fielded an additional homegrown survey (eg, focused on visit type or procedure), and 2 fielded an additional standardized survey (eg, BluePrint survey).

Central Themes

Four main themes emerged on how practices used patient experience data for PCMH transformation.

Theme 1: Practices used patient experience data most often to assess improvements related to PCMH goals; CAHPS measures were used to discuss best practices, share data with patient advisory councils, and improve provider performance.

The most common uses of patient experience data across all practices were monitoring and assessing changes in patient experience related to the PCMH change process (90%). This aligns with the PCMH standard for “performance measurement and quality improvement.”³² This was similar for practices currently, previously, or never administering the CAHPS PCMH survey. One leader described how they share their data:

We monitor trending reports on patient experience quarterly, comparing overall CAHPS scores and picking out particular items that we really want to focus on improving.…[T]hose CAHPS scores are disseminated across the entire practice and discussed at monthly staff meetings.…We also share them with our QI collaborative, which has representatives from all our primary care clinics, specialty clinics, and from throughout the hospital.…[That] is a nice opportunity for people [who] are not right in the thick of it to have a look at CAHPS data and make suggestions.…We also share these data at the clinic level.…[W]e have an expectation that patient experience is always on the agenda. – Current_1071

Another leader noted using CAHPS to start QI efforts:

We monitor the information on the CAHPS PCMH survey including the comments. The first place I send the data is to the CEO.…Then [they go] to the monthly [quality assurance]/QI committee where we compare [them with data from] previous surveys.…CAHPS data help us get things set up by asking, “What do you identify from the trends and what do we need to work on?” Then we always pick several areas we desire to increase and start new QI efforts. We do this for at least 3 or 4 findings from the CAHPS PCMH, as these data really jumpstart our QI process and QI changes touch every department. – Current_18415

Table 3 lists quotes on how practices use patient experience data to meet PCMH standards. Table 4 lists uses of patient experience data, including sharing performance trends with patient advisory councils (55%) and leadership (20%). These uses vary by CAHPS PCMH survey administration. Practices currently using the CAHPS PCMH survey were most likely to use them for monitoring and QI but also to discuss best practices with staff or to improve provider performance. Practices that previously used the CAHPS PCMH survey were more likely than others to track performance.

Theme 2: Practices used patient experience data for QI.

All practices reported using patient experience data for QI, PCMH changes, and tracking their progress. Most often, practices—including 90% of those previously using the CAHPS PCMH survey and 67% of those currently or never using it—used a general QI methodology to implement and track changes over time. Twenty percent also reported using a specific QI method such as Plan-Do-Study-Act (PDSA) cycles, root-cause analysis, Pareto analysis, or Lean methodologies for implementing PCMH. One leader reported:

Every practice gets a monthly CAHPS report including responses to each question. Every site has a multidisciplinary site-level QI team that works on specific quality measures for NCQA, and CAHPS metrics are the measures they work on improving. So every month, they’d see the CAHPS data and talk about [them], come up with an improvement plan, and roll the plan out in their practice and track future trend data…using a PDSA cycle approach. – Previous_4139

Another leader noted using patient narrative comments to assess whether practice changes improved care:

We’ve been monitoring our newly implemented PCMH processes for wait times…adjusting and tweaking processes as we need to as part of QI. We look at our patient experience data regularly, also looking at any comments to see if we’re still getting the same written comments from patients. – Never_11421

All practices reported using their patient experience data extensively for QI during PCMH transformation.

Theme 3: Practices had various reasons for choosing their patient experience survey.

Most practices (81%) reported that leadership, rather than those in charge of implementing PCMH changes, selected their patient experience survey. Twenty percent of practice leaders also reported that their practice chose their patient experience survey to meet the quality measurement PCMH standard. This was most common in practices that previously implemented the CAHPS PCMH survey and for practices that never had administered the CAHPS PCMH survey but administered their own homegrown surveys.

Practices that never implemented or previously implemented the CAHPS PCMH survey were more likely to report using the core CAHPS survey before PCMH implementation. Practices currently administering the CAHPS PCMH survey chose to pursue PCMH in tandem with CAHPS PCMH survey administration rather than independently.

More than 70% of practices chose to change their patient experience survey during their PCMH transformation. Practices that previously administered the CAHPS PCMH survey were the most likely to change which survey they used. Many practices that previously implemented the CAHPS PCMH survey were part of programs or initiatives that determined which survey they would use. Once those programs ended, or once those programs required practices to pay for survey administration costs, these practices chose to drop the CAHPS PCMH survey or switch to a new survey altogether. One leader said:

We do our own surveys. We did do the CAHPS PCMH when we were still within the [state program], but we graduated from that group.…They were the ones to actually send those CAHPS surveys out.…[Now] we have our own patient surveys that we give to our patients, usually twice a year, with obviously different questions to see how we’re doing; some of those are similar to the CAHPS questions anyway. – Previous_4170

Many practices that currently administer the CAHPS PCMH survey reported that they chose to also administer a second patient experience survey. The most common reason for administering a second survey was the desire to obtain more frequent data to use in implementing and monitoring PCMH changes. Some practices also added a survey to target specific aspects of patient experience, such as probing more closely on wait times or laboratory test turnaround. One leader discussed administering a supplemental patient survey:

CAHPS PCMH is only done once a year. Our [supplemental survey] is done at every visit or procedure for all patients.…It’s more comprehensive, more timely, and provides data on a monthly basis. – Current_4064

Practices that were responsible for paying survey administration costs noted a desire to change from the CG-CAHPS or CAHPS PCMH survey because of the high cost of survey administration. Practices for which a state program or network paid survey administration costs did not report issues with the cost of administering CAHPS surveys. One respondent summarized the balance between costs and quality by noting:

We have done an internal patient satisfaction survey for 22 years. It’s fairly robust, but it wasn’t evidence-based or vetted through random-controlled studies, so we started Hospital CAHPS in the hospital and had our care coordination department participating in that. We actually hired a company to administer the HCAHPS survey and we liked that very much. Unfortunately, it was costing us $20,000 to $25,000 a year. – Never_4273

Theme 4: Practices administering the CAHPS PCMH survey focused on a diverse set of patient experience improvement areas and reported that the CAHPS data were actionable.

Practices currently administering the CAHPS PCMH survey focused QI efforts on provider-patient communication (30%), shared decision-making (30%), coordinating patient care (30%), and nurse-patient communication (29%) in addition to overall ratings of providers (27%) and the clinic (30%). Practices that previously used the CAHPS PCMH survey focused on 2 issues: nurse-patient communication (29%) and overall rating of providers (26%). Practices that never used the CAHPS PCMH survey focused on 4 aspects of care: provider-patient communication (29%), self-management support (24%), coordinating patient care (22%), and overall rating of the clinic (23%).

Among practices that had used CAHPS PCMH items, 88% of current users and 39% of previous users thought these items were actionable. When asked how these items were actionable, 30% said they helped standardize and formalize PCMH change processes, 18% said they helped in structural/environmental decisions, and another 15% said they were useful for monitoring specific areas of patient experience such as coordinating patient care and enhancing patient self-management support.

Forty-five percent of those currently using the CAHPS PCMH survey indicated that the PCMH-specific items helped them achieve recognition in specific PCMH standards or goals. These practices pointed to 2 specific items: “patients caring for their own health” (45% of practices currently administering the CAHPS PCMH survey) and “gaining access to care from other providers” (eg, via referrals) (30% of practices currently administering the CAHPS PCMH survey). One leader noted:

To confirm if there [are] any barriers that would keep the patient from meeting the goals that we need them to meet [to be] able to manage themselves, we specifically used the CAHPS PCMH items about taking care of [their] own health and talking about medication with the patient.…With the older population, who are on more medicines, it helps place priority on making sure they don’t have any barriers to taking medication…and making sure that they are on some type of regimen [for] exercise and diet. – Current_4356

Forty-seven percent of practices discussed how the CAHPS PCMH items added value to their PCMH transformation journey. For example, 41% of practices described how the CAHPS PCMH items helped improve provider access by highlighting issues related to scheduling, clinic hours, and questions being answered. One leader said:

One of the biggest areas of opportunity [we] found from using CAHPS PCMH was that a high percentage of our patients did not know how to receive care on the weekends or after hours.…[T]he clinic put together a QI plan to increase the awareness and really increase the percentage of patients [who] knew how to receive care after hours. We put together a PCMH brochure which was shared with all patients and was displayed in our exam rooms. As part of our Patient and Family Advisory Council, the patients decided to create a newsletter. So that served as a communication channel not just to patients on the council but to all patients here at the practice. – Current_11279

Similarly, practices reported using CAHPS PCMH items to identify a range of complex gaps in care and track progress in meeting patient needs for medication management and behavioral health integration. One provider noted:

The CAHPS PCMH medication question was fairly easy to deal with, simply because it involved our medical assistants.…They do medication reconciliations at every visit, but somehow that information wasn’t getting through to patients.…[W]e changed our scripting for how that works.…[T]here wasn’t some miraculous big change, but over time, there will be. – Current_1115

Furthermore, 11% of providers relayed how they improved their process of providing patient test results and follow-up. One provider said:

The CAHPS PCMH access question about after hours and weekends was especially useful for our practice, [as were] the medication question and the question under care coordination about patients receiving test results in a timely fashion. We talked about those test results data quite a bit and did scripting with staff.…Scripting helped give the patient a reference as to when they might know something. And that prevents the patient from undue worry [and] the staff from getting multiple calls.…When we use scripting, staff can say that X particular test is going to take 3 days, so that what we convey to the patient is specific and helpful. – Previous_4127

DISCUSSION

Our study extends previous research^19-23 by specifically examining use of patient experience data for a nationwide sample of practices. We found that practices engaged in PCMH transformation used patient experience data to standardize patient-centered care and to identify and assess improvements related to PCMH goals. The most common use of patient experience data across all practices was to monitor progress. Data were also shared widely within organizations, ranging from CEOs to advisory councils, and used for discussions about best practices with staff, providers, or other network practices. Han et al also found that one-third of PCMH practices engaged patients via advisory groups and patient experience survey feedback.³³ Patient experience data were also commonly used to specifically improve provider-patient interactions (eg, 1-on-1 counseling based on CAHPS performance).

Studies specific to health systems using CAHPS data found that practice leaders used CAHPS data to implement QI, develop a shared vision, and coach providers and staff on performance.^19,23 We extend this evidence by identifying that practices administering the CAHPS PCMH survey used these data as guideposts for providing patient-centered care and focused on a broader range of patient experience–related improvement projects than practices not using CAHPS data. The CAHPS PCMH items helped practices standardize how care is delivered, share best practices, provide care that is coordinated with other providers, and focus on supporting patients’ self-management needs. Practices also primarily used the CAHPS PCMH items during the critical transformation period that leads to the practice receiving PCMH Level 3 certification, and then many practices dropped the CAHPS PCMH items as their PCMH transformation status matured. Data from CAHPS surveys were considered actionable by practice leaders, particularly for provider-patient and nurse-patient communication, shared decision-making, coordinating patient care, and improving global ratings.

Limitations

Although we studied a large, varied national set of practices, the sample is not nationally representative. We did not include practices that sought or gained PCMH certification under different recognition programs. Also, because this study is not a survey of practices’ activities using patient experience data, it does not contain an exhaustive exploration of all uses of patient experience surveys from all stakeholders.

CONCLUSIONS

Practice leaders most often used their patient experience data to assess QI and PCMH-related improvements. Those who administered CAHPS surveys used the data to discuss best practices, share data with leaders and patient advisory councils, and improve provider-patient interactions. Practice leaders used both CAHPS and non-CAHPS patient experience data as part of QI activities, with practices administering the CAHPS PCMH survey focused on a more diverse set of patient experience improvement areas. Importantly, CAHPS PCMH patient experience data were reported as actionable in making improvements in patient-centeredness. Specifically, practice leaders indicated that CAHPS PCMH survey data helped them improve care coordination, self-management support, communication, and standardizing and coordinating the provision of care. CAHPS data were integral to the continuous QI needed for moving beyond formal PCMH recognition and maximizing primary care medical home transformation. This supports the recommendation of NCQA to administer CAHPS or CAHPS PCMH items as part of PCMH transformation.

Author Affiliations: RAND Corporation (DDQ, NQ, LA, RDH), Santa Monica, CA; Division of General Internal Medicine & Health Services Research, University of California, Los Angeles (RDH), Los Angeles, CA.

Source of Funding: This work was supported by a cooperative agreement from the Agency for Healthcare and Research Quality (contract No. U18-HS016980 and U18-HS025920).

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (DDQ); acquisition of data (DDQ, NQ, LA); analysis and interpretation of data (DDQ, NQ, LA, RDH); drafting of the manuscript (DDQ, LA, RDH); critical revision of the manuscript for important intellectual content (DDQ, LA, RDH); statistical analysis (DDQ, NQ, LA); provision of patients or study materials (DDQ); obtaining funding (DDQ, RDH); administrative, technical, or logistic support (DDQ); and supervision (DDQ).

Address Correspondence to: Denise D. Quigley, PhD, RAND Corporation, 1776 Main St, Santa Monica, CA 90407-2138. Email: quigley@rand.org.

REFERENCES

1. Perry RJ, McCall N, Wensky SG, Haber SG. Care Continuity in a Patient-Centered Medical Home Setting. RTI Press; 2016. Accessed April 16, 2020. https://www.ncbi.nlm.nih.gov/books/NBK532528/

2. Timbie JW, Hussey PS, Setodji CM, et al. Association between patient-centered medical home capabilities and outcomes for Medicare beneficiaries seeking care from federally qualified health centers. J Gen Intern Med. 2017;32(9):997-1004. doi:10.1007/s11606-017-4078-y

3. Pines JM, Keyes V, van Hasselt M, McCall N. Emergency department and inpatient hospital use by Medicare beneficiaries in patient-centered medical homes. Ann Emerg Med. 2015;65(6):652-660. doi:10.1016/j.annemergmed.2015.01.002

4. Fifield J, Forrest DD, Burleson JA, Martin-Peele M, Gillespie W. Quality and efficiency in small practices transitioning to patient centered medical homes: a randomized trial. J Gen Intern Med. 2013;28(6):778-786. doi:10.1007/s11606-013-2386-4

5. van Hasselt M, McCall N, Keyes V, Wensky SG, Smith KW. Total cost of care lower among Medicare fee-for-service beneficiaries receiving care from patient-centered medical homes. Health Serv Res. 2015;50(1):253-272. doi:10.1111/1475-6773.12217.

6. Accreditation with medical home. Accreditation Association for Ambulatory Health Care. Accessed April 16, 2020. https://www.aaahc.org/accreditation/medical-home/

7. Primary care medical home certification program. The Joint Commission. Accessed April 16, 2020. https://www.jointcommission.org/en/accreditation-and-certification/certification/certifications-by-setting/hospital-certifications/primary-care-medical-home-certification/

8. Internal Data From NCQA’s Monthly Progress Reports: Recognition Programs Operations. National Committee for Quality Assurance; 2015.

9. Report Cards: Practices. National Committee for Quality Assurance; 2017.

10. Wagner EH, Coleman K, Reid RJ, Phillips K, Abrams MK, Sugarman JR. The changes involved in patient-centered medical home transformation. Prim Care. 2012;39(2):241-259. doi:10.1016/j.pop.2012.03.002

11. Sugarman JR, Phillips KE, Wagner EH, Coleman K, Abrams MK. The safety net medical home initiative: transforming care for vulnerable populations. Med Care. 2014;52(11 suppl 4):S1-S10. doi:10.1097/MLR.0000000000000207

12. Qureshi N, Quigley DD, Hays RD. Nationwide qualitative study of practice leader perspectives on what it takes to transform into a patient-centered medical home. J Gen Intern Med. 2020;35(12):3501-3509. doi:10.1007/s11606-020-06052-1

13. Stout S, Weeg S. The practice perspective on transformation: experience and learning from the frontlines. Med Care. 2014;52(11 suppl 4):S23-S25. doi:10.1097/MLR.0000000000000239

14. Cronholm PF, Shea JA, Werner RM, et al. The patient centered medical home: mental models and practice culture driving the transformation process. J Gen Intern Med. 2013;28(9):1195-1201. doi:10.1007/s11606-013-2415-3

15. Friedberg MW, Coltin KL, Safran DG, Dresser M, Zaslavsky AM, Schneider EC. Associations between structural capabilities of primary care practices and performance on selected quality measures. Ann Intern Med. 2009;151(7):456-463. doi:10.7326/0003-4819-151-7-200910060-00006

16. Friedberg MW, Safran DG, Coltin KL, Dresser M, Schneider EC. Readiness for the patient-centered medical home: structural capabilities of Massachusetts primary care practices. J Gen Intern Med. 2009;24(2):162-169. doi:10.1007/s11606-008-0856-x

17. Friedberg MW, SteelFisher GK, Karp M, Schneider EC. Physician groups’ use of data from patient experience surveys. J Gen Intern Med. 2011;26(5):498-504. doi:10.1007/s11606-010-1597-1

18. Geissler KH, Friedberg MW, SteelFisher GK, Schneider EC. Motivators and barriers to using patient experience reports for performance improvement. Med Care Res Rev. 2013;70(6):621-635. doi:10.1177/1077558713496319

19. Quigley DD, Mendel PJ, Predmore ZS, Chen AY, Hays RD. Use of CAHPS patient experience survey data as part of a patient-centered medical home quality improvement initiative. J Healthc Leadersh. 2015;7:41-54. doi:10.2147/JHL.S69963

20. Case studies of leading primary care practice facilitation programs: lessons learned from leading models of practice facilitation. agency for healthcare research and quality. January 2013. Accessed September 10, 2020. https://pcmh.ahrq.gov/sites/default/files/attachments/LessonsLearned_022813comp_0.pdf

21. Improving patient outcomes with patient-centered care. GreenwayHealth. April 6, 2020. Accessed August 10, 2021. https://www.greenwayhealth.com/knowledge-center/greenway-blog/improving-patient-outcomes-patient-centered-care

22. Hudak RP, Julian R, Kugler J, et al. The patient-centered medical home: a case study in transforming the military health system. Mil Med. 2013;178(2):146-152. doi:10.7205/milmed-d-12-00170.

23. Maeng DD, Davis DE, Tomcavage J, Graf TR, Procopio KM. Improving patient experience by transforming primary care: evidence from Geisinger’s patient-centered medical homes. Popul Health Manag. 2013;16(3):157-163. doi:10.1089/pop.2012.0048

24. Bernard HR, Ryan GW. Code books and coding. In: Bernard HR, Ryan GW. Analyzing Qualitative Data: Systematic Approaches. SAGE Publications; 2010:75-105.

25. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;42(4):1758-1772. doi:10.1111/j.1475-6773.2006.00684.x

26. Charmaz K. Grounded theory: objectivist and constructivist methods. In: Denzin N, Lincoln Y, eds. Handbook of Qualitative Research. 2nd ed. SAGE Publications; 2000:509-536.

27. Glaser BG, Strauss AL. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Publishing Company; 1967.

28. Krippendorff K. Content Analysis: An Introduction to Its Methodology. 2nd ed. SAGE Publications; 2004.

29. Miller WL, Crabtree BF. The dance of interpretation. In: Crabtree BF, Miller WL, eds. Doing Qualitative Research in Primary Care: Multiple Strategies. 2nd ed. SAGE Publications; 1999:127-143.

30. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37-46. doi:10.1177/001316446002000104

31. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174.

32. NCQA PCMH standards and guidelines. National Committee for Quality Assurance. Accessed October 2, 2018. http://store.ncqa.org/index.php/catalog/product/view/id/2776/s/2017-pcmh-standards-and-guidelines-epub

33. Han E, Scholle SH, Morton S, Bechtel C, Kessler R. Survey shows that fewer than a third of patient-centered medical home practices engage patients in quality improvement. Health Aff (Millwood). 2013;32(2):368-375. doi:10.1377/hlthaff.2012.1183