Publication
Article
Supplements and Featured Publications
Improving Care Delivery at the Community Level: An Examination of the AF4Q Legacy
Author(s):
Objective: Aligning Forces for Quality (AF4Q) was the Robert Wood Johnson Foundation’s nearly 10-year, multicomponent initiative to create meaningful and sustainable change in 16 communities. Our purpose was to describe the likely legacy of the care delivery component of AF4Q among participating communities and the factors that influenced the legacy.
Methods: We used a multiple-case study approach. Our analysis relied on 3 key documents for each community, based on key informant interviews conducted between 2006 and 2015: (1) a summary of the community’s care delivery activities under AF4Q, (2) a summary of the community’s experience in the AF4Q program, and (3) a summary of the characteristics of each community and the multi-stakeholder alliance that led local efforts under AF4Q. We used a team-based consensual approach to analysis.
Results: We identified 3 types of legacies: (1) in 3 communities, there appear to be sustained infrastructures or wide-reaching activities attributable to AF4Q; (2) in 5 communities, AF4Q participation was used to advance preexisting activities; and (3) in 8 communities, the care delivery legacy is likely to be limited, because the local alliance focused on performance measurement instead of care delivery or the care delivery activities had limited reach and sustainability. Community contextual factors (eg, availability of other grant support) and alliance characteristics (eg, areas of expertise) greatly influenced the AF4Q care delivery legacy.
Conclusion: AF4Q appears to have created meaningful and sustained change in care delivery in half of the participating communities. Among the other communities, the considerable financial support and technical assistance provided by RWJF was not enough to overcome some of the contextual barriers that often hamper quality-improvement efforts.
Am J Manag Care. 2016;22:S393-S402
“Helping health professionals get better at improving care” was one of the long-standing pillars of the Robert Wood Johnson Foundation’s (RWJF’s) Aligning Forces for Quality (AF4Q) program, RWJF’s signature effort to help 16 diverse communities improve the strength, resilience, and quality of their healthcare systems.1 When pursued alongside performance measurement, public reporting, and consumer engagement, RWJF posited that improved care delivery would lead to meaningful and sustainable change within communities. To that end, RWJF provided multi-stakeholder alliances—collaborative groups of payers, purchasers, providers, and consumers—with funding and technical assistance to support improvements in care delivery. Alliances were given a fair amount of discretion in terms of how to pursue improved care delivery within their communities, but they were expected to meet certain requirements that evolved during the course of the program and report on their progress. The program was in operation from 2006 to 2015.2
RWJF’s multi-stakeholder approach under the AF4Q program followed the recognition that siloed, organizational-level attempts to improve quality resulted in only modest change3-5 and was aligned with statements from prominent healthcare institutions and leaders that multi-stakeholder approaches may be superior.6-8 Although conceptually appealing, evidence of the effectiveness of multi-stakeholder—led quality improvements (QIs) was (and remains) limited. A logic model, developed by the AF4Q evaluation team, depicts assumptions about how multi-stakeholder alliances under the AF4Q program could drive improvements in care delivery and health outcomes (Figure). Specifically, after assessing the needs of the community and developing a QI vision or strategy, the AF4Q alliance would create or build upon the QI infrastructure within the community (eg, procure and distribute QI resources, raise awareness about the need for QIs), leveraging the technical assistance and funding provided by RWJF. The infrastructure would provide a platform from which QI activities could be created or enhanced, aligned with AF4Q’s other programmatic areas (eg, consumer engagement, performance measurement), and spread throughout the community. The model also reflects that the alliances vary significantly in terms of history and market structure, and are influenced by factors in the external environment not directly related to the AF4Q initiative.
As part of the evaluation of AF4Q, we tracked alliances’ efforts to improve care delivery (herein referred to as QI) communitywide and reported our findings in a number of publications (Sidebar).9-17 In sum, we found that although alliances were slow to establish a QI infrastructure and launch QI activities, all alliances eventually implemented QI activities by establishing the activities themselves or delegating the task to close partners. There was considerable overlap in the focus of the QI activities across alliances (eg, most alliances encouraged adoption of patient-centered medical home [PCMH] processes) and the approaches employed (eg, most alliances established learning collaboratives). However, the quantity and quality of the activities pursued, or the “dose” of the AF4Q program QI interventions, varied considerably across communities. As a result, for a majority of patient care and patient outcomes prespecified at the start of the program and tracked under the evaluation, AF4Q communities did not experience greater improvement than non-AF4Q communities.
Given the large investments in multi-stakeholder alliance—led initiatives to improve quality in the public and private sectors,18 our goal in this paper was to provide additional insight about the AF4Q program’s QI legacy in the participating communities—insights that would be overlooked if one focused solely on the underwhelming quantitative results. Specifically, based on our qualitative data, we sought to describe the legacy that the QI component of the AF4Q program is likely to leave in each participating community and the factors that influenced the AF4Q program’s QI legacy. Findings from our analyses may provide RWJF staff and policy makers with insight about the cross-community results of RWJF’s investment in the area of QI and guide future efforts.
Methods
Design
We used a multiple-case study approach to examine the AF4Q program’s QI intervention within its real-life context across the 16 participating communities.19 Although multiple-case studies typically include fewer than 5 cases due to the complexity of the data,20 our analysis included all 16 AF4Q participating regions, as this analysis represents a component of the summative evaluation of the AF4Q program.
Data Collection and Coding
We conducted 1100 semi-structured key informant interviews during 4 rounds of site visits to AF4Q communities between 2006 and 2016, and 10 rounds of telephone interviews with alliance leaders between 2007 and 2014. Interviews were conducted with a number of individuals in each community during the site visits, including alliance directors, who oversee the work of the alliance; project directors, who are responsible for implementation of the AF4Q initiative; individuals who led or planned alliances’ activities in various programmatic areas (eg, QI, public reporting, and consumer engagement); and representatives from each of the targeted community stakeholder groups, typically consumers, physicians, hospital leaders, healthcare plans, and employers.21 The telephone interviews were typically conducted with 1 or more of the following individuals: the alliance director, project director, or individual identified by the alliance or project director as being responsible for the alliance’s QI activities (“QI leaders”).
During the interviews, we asked respondents about a number of topics, including the alliances’ structure, vision, goals, strategies, and decision making; details about specific QI activities; characteristics of the alliances’ markets and partnerships; and external factors that affected their decisions and activities. All interviews were audio-recorded, transcribed, and uploaded to Atlas.ti, a qualitative software package, for analysis.
Using a multistaged coding process, we first used deductive high-level (global) categories corresponding to the AF4Q initiative’s main programmatic areas and major concepts that are relevant across all communities (eg, alliance participation, resources, and structure). Next, all data that were globally coded were read for emerging themes. The transcripts were reviewed until no new themes emerged. This inductive process resulted in a final list of codes representing the key concepts and themes related to the QI programs. A more thorough description of our interviews and coding process can be found elsewhere.21
Analysis
Because of the large number of cases, our analysis relies on 3 key documents created for each AF4Q community, based on the interview data. First, we created a 2- to 3-page summary of the communities’ major QI activities under the AF4Q program. In 2013, the QI leaders verified and/or made corrections to our summaries. Second, we created a more comprehensive (14-20 page) summary of each community’s experience in the AF4Q program, which included a description of program governance, activities in the various programmatic areas, and contextual factors that influenced their activities, challenges, and successes. Third, we created a document that lists characteristics of each alliance (eg, the alliances’ history and origin, stakeholder dominance, and area[s] of expertise prior to joining the AF4Q program) and community (eg, health provider market competition and community size).
We used a team-based consensual approach to analysis.22,23 To ensure that the influence of the unique context of each case was sufficiently considered, each case was examined independently before any attempt was made to triangulate findings across cases. For each community, 2 authors reviewed the 3 key documents and drafted memos addressing 2 questions: (1) What is likely to be the legacy of the AF4Q program regarding improved care delivery? and (2) What factors influenced that legacy?
Patterns and themes across sites were identified through 4 weekly discussions. In several instances, the authors reviewed the interview transcripts to resolve disagreement, identify illustrative quotes, and further explore contextual factors that may have contributed to differences in legacy across AF4Q communities. Finally, we provided a profile of each of the AF4Q communities, including alliance and community characteristics, a measure of the “dose” of the AF4Q program’s QI intervention, and the percentage of patient experience measures that improved between 2008 and 2012.
Results
Our results revealed 3 different types of legacies that the AF4Q program is likely to leave across participating communities. We describe these legacies, highlighting specific AF4Q communities as examples, and the factors that may have contributed to those legacies. A profile of the communities can be found in Table 1.
New Infrastructure Legacy
In 3 communities, the AF4Q program’s QI legacy is a new infrastructure or advancement in care that has a broad reach in the community, and its existence or spread can be attributed specifically to the AF4Q program. Although the long-term sustainability of these infrastructures and advancements is uncertain, the alliances continued to engage in the QI work beyond the end of the AF4Q grant.
Under the AF4Q program, the alliance in Cincinnati began a series of multipayer PCMH pilot programs involving approximately 10 physician practices each. Partnering with TransforMED, an affiliate of the American Academy of Family Physicians, to build the curriculum, the alliance led the pilot program, recruited health plans to contribute a $20 per-member-per-month payment to participating practices, and created a multi-payer database to evaluate the pilots. This work was largely the impetus for the region being selected to participate in the Centers for Medicare & Medicaid Services’ (CMS) Comprehensive Primary Care initiative (CPCI), which expanded the number of payers and practices involved. This work, in turn, laid the groundwork for Ohio’s State Innovation Model award, which also focused on PCMH adoption. One respondent noted: “I see the evolution from Aligning Forces to CPCI to a statewide model for PCMH as one continuous flow.”
Respondents pointed to a number of factors that facilitated the alliance’s work: the alliance leader’s strong interest in QI, access to an executive on loan from GE Aviation who helped set goals around healthcare transformation from an employer perspective, long-standing commercial payer support for local initiatives, collegial relationships among the chief medical officers of the 3 dominant health plans and between the chief medical officers and the alliance, PCMH initiatives that coincided with plans’ announcements of national medical home initiatives, additional support from the CPCI and Beacon program funding from the Office of the National Coordinator for Health Information Technology to support the spread of PCMHs, and having more than 80% of physicians and hospitals connected through one of the largest and most advanced health information exchanges.
The alliance in Cleveland, established in 2007 specifically for the AF4Q program, took a different approach to community-level QI. The alliance created a QI Learning Collaborative modeled on the Institute for Healthcare Improvement’s Breakthrough Series Collaborative and the Chronic Care Model. The Collaborative was established to give providers an opportunity to learn from one another, primarily through best practices identified by analysis of the alliance’s measurement/reporting data and shared during the QI Learning Collaborative Summits. The Learning Collaborative Summits are held biannually, with up to 200 attendees at each event. The alliance uses a membership model and dues to support data analysis for the identification of best practices and the summits, which continued beyond the end of the AF4Q program.
The alliance’s strategy under the AF4Q program, to focus its efforts on a learning collaborative, stems from a previous public reporting effort for hospitals in Cleveland that heightened provider competitiveness and ultimately failed. Under the AF4Q program, the alliance chose a different direction. According to one respondent, the focus was to “raise all boats…rather than emphasize bad apples and say, ‘Here are people that aren’t providing quality.’”
The AF4Q program’s legacy in Western New York is not a new QI program, but rather the establishment of an alliance with QI credibility, experience, and grant support. The Western New York alliance was a nascent organization with little QI experience when it joined the AF4Q program. The alliance’s QI work—primarily focused on physician practice transformation—coupled with the state’s heavy investment in regional health improvement activities resulted in the alliance being awarded a number of additional grants focused on population health and practice transformation. Although the QI activities pursued by the alliance under the AF4Q program have not all been sustained, the alliance is well positioned to serve as a regional QI leader and secure additional funding to support local QIs.24-26
Further Faster Legacy
In 5 communities, the legacy of the AF4Q program is more difficult to detect. The alliances can point to new QI activities, new QI resources, and the adoption of improved care processes in the community since joining the program; however, attributing these changes specifically to the AF4Q program is difficult. The alliances or their partners were effectively engaging in QI activities prior to joining the AF4Q program, supported by other funding streams, and the AF4Q program helped to advance these activities further and faster. In sum, the requirements of the AF4Q program were well aligned with the ongoing QI work within the community.
Like in the new infrastructure legacy category, we identified considerable variation within the further faster legacy category. The alliances in Minnesota and Maine advanced QI statewide under the AF4Q program, largely because their QI partners had a statewide focus. In Maine, Maine Quality Counts, a membership-based, provider-led, multi-stakeholder group focused on QI, was one of the key alliance partners. Prior to the AF4Q program, Maine Quality Counts was engaged in QI activities in physician practices. During the AF4Q program, the organization established a number of new QI activities, including development of a PCMH program, a PCMH learning community, and community care teams focused on high-need patients. Through its efforts, the organization reached upwards of 40% of physician practices in the state; however, many of these activities were reported to be “not uniquely AF4Q” activities, but part of multiple efforts occurring simultaneously. Most notably, the alliance received $12 million as part of Maine’s State Innovation Model award from the CMS to assist practices in the adoption of PCMHs. According to one respondent, “[The] AF4Q [program] was a drop in the bucket” in comparison. Another respondent noted, “It’s easier to point to the indirect impact of [the] AF4Q [program] than it is the direct.”
In Humboldt County, California, and South Central Pennsylvania, the alliances’ catchment areas were relatively small and dominated by a large, local delivery system with considerable QI experience. As a result, in both communities, the planning and participation in AF4Q-related QI efforts were dominated by those systems. AF4Q program resources and support were used to push QI programs within those systems further. For example, in Humboldt County, California, the local independent practice association (IPA) had long participated in a pay-for-performance effort that funded QI initiatives and provider bonuses. Under the AF4Q program, the IPA partnered with large employers and an insurer to fund Priority Care, an effort to provide population-level management activities and intensive care coordination for patients with multiple chronic conditions. The creation and expansion of Priority Care may be attributable, in part, to the AF4Q program, although the building blocks were already in place.
The one alliance in the further faster category that lacked experience in QI prior to joining the AF4Q program was West Michigan. After a largely unsuccessful start trying to develop its own QI activities under the AF4Q program, the alliance partnered with local physicians who were developing a QI resource center, the Michigan Center for Clinical Systems Improvement (Mi-CCSI). Using the AF4Q program grant funding, the alliance provided critical early support to Mi-CCSI and exposure to the Institute for Clinical Systems Improvement in Minnesota, after which Mi-CCSI is modeled. Mi-CCSI continues to serve as a consortium of providers and payers collaborating to implement and evaluate clinical models in primary care to improve quality, even after the West Michigan alliance closed its doors in 2015. Although the AF4Q program provided important funding during Mi-CCSI’s formation, it is likely that Mi-CCSI would have been established in the absence of the AF4Q initiative.
Limited QI Legacy
In approximately half of the communities, the AF4Q program’s QI legacy is likely to be limited. In some cases, this limited QI legacy is a result of deliberate decision making by the alliances to pursue other work. Instead of devoting efforts to initiating QI activities, 3 alliances (Oregon, Wisconsin, and Washington) decided to use their data resources and analytical capabilities to drive change through performance measurement. Although RWJF encouraged alliances to engage in QIs along with other programmatic areas, as one respondent put it, “We don’t do QI.” Like many of the further faster communities, alliances in this group leveraged the work they were engaged in prior to the AF4Q program, and focused on performance measurement instead of QI.
For example, prior to joining the AF4Q program, the purchaser-led alliance in Washington used claims data from member health plans and self-insured employers to identify QI opportunities and highlight high-performing providers. Despite instruction from RWJF to undertake QI interventions, one respondent noted, “We knew from the get-go that we couldn’t do everything…We were pretty methodical in choosing which area within [the] AF4Q [program] that we were going to emphasize…For us, we’ve chosen to make that niche performance measurement and reporting.” The alliance leaders argued that they did not have the time or resources to engage in QI and that, as a purchaser-led organization, funding improvement processes for their suppliers (eg, hospitals and physician practices) were outside the scope of the organization’s mission. Nevertheless, the alliance’s role in performance measurement was highly valued within the state; the alliance received considerable funding under the State Innovation Model award, with a scope of work focused on coordination and implementation of a statewide performance measure set, data aggregation, and communication around the measures.
In the remaining AF4Q communities, the limited QI legacy is simply a result of various challenges encountered as the alliances planned, implemented, or attempted to spread and sustain their QI activities. These alliances initiated a number of QI activities, including practice coaching, learning collaboratives, developing QI tool kits for providers, and small pilot programs. However, the reach of the activities was rather limited, never spreading communitywide, and at the end of the AF4Q program grant period, the activities had terminated.
For example, one alliance in this category opted to run a pilot program to test a community-centered model for addressing the Triple Aim (better health, better care, and reducing costs). The alliance assembled a coalition of providers, public health agencies, community coalitions, public schools, faith-based organizations, health plans, consumers, and advocacy groups to leverage existing
activities, best practices, and community resources with the goal of making meaningful improvements in diabetes and childhood asthma. Between 2012 and 2015, the coalition met quarterly and, with AF4Q program funding, held listening sessions, developed an asset map of local resources, and led a month-long series of events to increase awareness about the prevention and management of diabetes. According to planners of the initiative, the program engaged more than 800 community residents, recruited new community partners, and brought attention to the issue of diabetes through distribution of brochures and coverage in the local press. The coalition ceased its regular meetings when the AF4Q program grant ended. Alliance staff developed a fundraising package in the hope of finding an organization willing to continue to convene the coalition, but there “wasn’t enough time to carry it through.”
Reflecting on the initiative at its end, participants from within and outside the alliance pointed to 3 major hindrances. First, the alliance’s leadership team oversaw the AF4Q program grant and conceived of the initiative after a strategic planning process, believing that there needed to be more community engagement around QI. However, the leadership team was not actively engaged in the initiative once it launched, and therefore, the coalition’s accountability was limited.
Second, the alliance and coalition encountered considerable pushback; for example, primary care physicians in the community stated, “We don’t have the resources to do anything more than we’re doing. We’re already doing this. We can’t take on any more.” There was also pushback from community leaders, skeptical of the alliance, which had little experience partnering with organizations in the community, “dictating how things should be managed.” The alliance had to hire a long-time community organizer to get all of the coalition partners to participate, and that process took years.
Third, this alliance was selected to join the AF4Q program in 2010, considerably later than other AF4Q communities. Respondents noted that the alliance’s AF4Q leadership team struggled to meet the requirements and goals established by RWJF, and was not able to focus on a single project until the last phase of the AF4Q program. According to one respondent, “We didn’t have enough time to…accomplish as much as I think we would have liked at the outset when we envisioned this.”
Among the other communities, respondents noted a number of challenges associated with their QI work. They are listed inTable 2.
Discussion
To describe the AF4Q program’s QI legacy in the 16 participating communities, we analyzed summaries of key informant interview data collected over nearly 10 years. Our findings suggest that the legacy is variable. In some communities, it is reasonable to conclude that the AF4Q program drove the creation of a new QI infrastructure that was sustained beyond the end of the program; in others, the program advanced QI initiatives already in place.
In half of the communities, we believe that the program is likely to have limited long-term impact on care delivery. Our results also suggest that contextual factors associated with the community and alliance played a considerable role in explaining the variation in program legacy. These findings are not necessarily surprising, considering that evaluations of other multi-site QI interventions have also reported variation in outcomes and the important influence of contextual factors.27 However, the AF4Q program represented the largest privately funded QI intervention to date; our results suggest that even with considerable financial support and technical assistance, many alliances could not overcome some of the contextual barriers that often hamper QI efforts.
Our categorization scheme was intended to highlight differences among legacies across the AF4Q communities. We cannot conclude that one type of legacy is superior to another. For example, it is not clear that creating a set of new QI activities within a community (ie, new legacy) is more impactful than advancing well-established ones (ie, further faster). Additionally, there’s little evidence to suggest that improving care delivery through direct interventions with physician practices, as encouraged by RWJF, is more effective than measuring and sharing performance, as several of the limited legacy communities opted to do.
Our findings raise important issues for future planners and investors in multi-stakeholder—led QI work as they consider site selection for similar QI initiatives. The first is the need to consider the type of legacy that program planners would like to establish. Some may prefer to push existing efforts further and faster, without having to invest in the time-consuming and expensive effort of developing new infrastructures. Other program planners might prefer to develop new infrastructures in communities that are lacking in QI investment. They may also prefer a more tangible product for their efforts.
The second issue is how to conduct site selection to maximize the chance of producing the preferred QI legacy. Based on the experience in the AF4Q program, it may be reasonable to assume that a community with a strong QI infrastructure may simply use the additional support to advance current efforts in the community. One approach to future site selection may be to look for characteristics and factors that were common among AF4Q alliances and communities with limited legacies (eg, leadership struggles or a track record of launching one-time QI programs). These characteristics may serve as disqualifiers to participation or, at a minimum, a warning signal to program planners. Alternatively, we can look at the characteristics common among the new infrastructure and further faster legacies. For example, our internal analyses show that all 5 alliances with QI interventions deemed “high dose” were also in the new infrastructure or further faster legacy categories. Future funders may want to select alliances that have the capacity to support QI activities that are strong in number, duration, scope, intensity, and reach (ie, elements that are hypothesized to contribute to a high intervention dose).11
An important limitation of the study is that the last round of interview data was collected near the end of the AF4Q program. Therefore, we cannot be certain about the long-term sustainability of QI activities or the multi-stakeholder alliances that support them. Second, there is no standard by which to judge the QI legacy in the AF4Q communities. As described elsewhere,2 RWJF’s direction to the alliances regarding their QI work shifted during the course of the AF4Q initiative. However, several directives to the alliances indicated that RWJF hoped that the alliances’ work would be sustainable beyond the life of the AF4Q program.1,28,29 In the absence of a clear goal for a legacy, we simply describe what we believe is the lasting impact in the community and compare AF4Q program participants with each other.
Assessing program legacy is one approach to evaluating the long-term impact of large, complex QI programs. Examining legacy is particularly important for grant-funded initiatives, like the AF4Q program, that aim for lasting, community-level effects that may not be identified by quantitative analyses conducted during the program period.30 The AF4Q program appears to have created meaningful and sustained change in care delivery in half of the participating communities. Although quantitative results from the AF4Q program have been underwhelming to date,14-16 future efforts to track outcome measures may suggest a greater AF4Q program effect.
Conclusion
The AF4Q program legacy in the area of care delivery varied across the 16 communities. Three alliances reported sustained infrastructures or wide-reaching activities attributable to the initiative, 5 utilized AF4Q program participation to advance preexisting activities, and 8 are likely to have a limited care delivery legacy. This legacy suggests that the AF4Q program may yield long-term improvements in patient care and patient outcomes in some communities; however, multi-stakeholder-alliance—led approaches to QI may not be a panacea for communitywide change. Our results also raise a number of issues for planners and supporters of large, multi-stakeholder–led QI efforts. Future funders may benefit from selecting alliances that have the capacity to support “high-dose” QI activities in terms of number, duration, scope, intensity, and reach.
Author affiliations: Center for Healthcare Studies, Northwestern University, Feinberg School of Medicine, Chicago, IL (JH, MM); Healthcare Leadership and Management, Medical University of South Carolina, Charleston, SC (JBH); Center for Health Care and Policy Research, and Health Policy and Administration, Penn State University, University Park, PA (DPS).
Funding source: This supplement was supported by the Robert Wood Johnson Foundation (RWJF). The Aligning Forces for Quality evaluation is funded by a grant from the RWJF.
Author disclosures: Ms Hamil, Ms Harvey, Dr McHugh, and Dr Scanlon report receipt of grants from RWJF. Dr Scanlon also reports meeting or conference attendance for RWJF.
Authorship information: Concept and design (MM, DPS); acquisition of data (JH, MM, DPS); analysis and interpretation of data (JH, JBH, MM, DPS); drafting of the manuscript (MM, DPS); critical revision of the manuscript for important intellectual content (JH, JBH, DPS); obtaining funding (DPS); and administrative, technical, or logistic support (JH, JBH).
Address correspondence to: megan-mchugh@northwestern.edu.
REFERENCES
1. Painter MW, Lavizzo-Mourey R. Aligning Forces for Quality: a program to improve health and health care in communities across the United States. Health Aff (Millwood). 2008;27(5):1461-1463. doi: 10.1377/hlthaff.27.5.1461.
2. Scanlon DP, Beich J, Alexander JA, et al. The Aligning Forces for Quality initiative: background and evolution from 2005 to 2012. Am J Manag Care. 2012;18(suppl 6):S115-S125.
3. US Department of Health and Human Services. 2010 national healthcare quality report. AHRQ Pub No 11-0004. http://archive.ahrq.gov/research/findings/nhqrdr/nhqr10/nhqr10.pdf. Published March 2011. Accessed May 8, 2016.
4. Dentzer S. Still crossing the quality chasm—or suspended over it? Health Aff (Millwood). 2011;30(4):554-555. doi: 10.1377/hlthaff.2011.0287.
5. Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ. Temporal trends in rates of patient harm resulting from medical care. New Engl J Med. 2010;363(22):2124-2134. doi: 10.1056/NEJMsa1004404.
6. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood). 2008;27(3):759-769. doi: 10.1377/hlthaff.27.3.759.
7. US Department of Health and Human Services. 2011 report to Congress: national strategy for quality improvement in health care. www.ahrq.gov/workingforquality/nqs/nqs2011annlrpt.htm. Published March 2011. Accessed May 8, 2016.
8. Institute of Medicine. Crossing the Quality Chasm. Washington, DC: National Academy Press; 2001.
9. Harvey JB, Beich J, Alexander JA, Scanlon D. Building the scaffold to improve health care quality in Western New York. Health Aff (Millwood). 2012;31(3):636-641. doi: 10.1377/hlthaff.2011.0761.
10. McHugh MC, Harvey JB, Aseyev D, Alexander JA, Beich J, Scanlon DP. Approaches to improving healthcare delivery by multi-stakeholder alliances. Am J Manag Care. 2012;18(suppl 6):S156-S162.
11. McHugh M, Harvey JB, Kang R, Shi Y, Scanlon DP. Measuring the dose of quality improvement initiatives. Med Care Res Rev. 2016;73(2):227-246. doi: 10.1177/1077558715603567.
12. Harvey J, McHugh M, Lunders C, Alexander J, Scanlon DP. Multi-Stakeholder Alliance Contributions to the Spread of Health Care Quality Improvement Programs. University Park, PA: Penn State; 2014.
13. McHugh M, Harvey JB, Hamil J, Verevkina N, Alexander JA, Scanlon DP. The impact of the Affordable Care Act on health care alliances’ quality improvement efforts in targeted communities: perceptions of health care alliance leaders. Jt Comm J Qual Patient Saf. 2016;42(3):137-145.
14. McHugh M, Shi Y, McClellan SR, et al. Using multi-stakeholder alliances to accelerate the adoption of health information technology. Healthcare. 2016;4(2):86-91. doi: /10.1016/j.hjdsi.2016.01.004.
15. McHugh M, Shi Y, Ramsay PP, et al. Patient-centered medical home adoption: results from Aligning Forces for Quality. Health Aff (Millwood). 2016;35(1):141-149. doi: 10.1377/hlthaff.2015.0495.
16. Duckett P, Kang R, Hamil J, Harvey JB, Goodman D, McHugh M. Investigating a multistakeholder alliance approach to reducing hospital readmissions [epub ahead of print May 5]. J Healthc Qual. 2016. doi: 10.1097/JHQ.0000000000000019.
17. McHugh M, Harvey JB, Kang R, Shi Y, Scanlon DP. Community-level quality improvement and the patient experience for chronic illness care. Health Serv Res. 2016;51(1):76-97. doi: 10.1111/1475-6773.12315.
18. Agency for Healthcare Research and Quality. EvidenceNOW: advancing heart health in primary care. AHRQ website. http://www.ahrq.gov/professionals/systems/primary-care/evidencenow.html. Accessed August 1, 2015.
19. Yin RK. Enhancing the quality of case studies in health services research. Health Serv Res. 1999;34(5 pt 2):1209-1224.
20. Yin R. Case Study Research. 3rd ed. Thousand Oaks, CA: Sage Publications, Inc; 2003.
21. Scanlon DP, Alexander JA, Beich J, et al. Evaluating a community-based program to improve healthcare quality: research design for the Aligning Forces for Quality initiative. Am J Manag Care. 2012;18(suppl 6):S165-S176.
22. Hill CE, Knox S, Thompson BJ, Williams EN, Hess SA, Ladany N. Consensual qualitative research: an update. J Couns Psychol. 2005;52(2):196-205.
23. Hill CE, Thompson BJ, Williams EN. A guide to conducting consensual qualitative research. Couns Psychol. 1997;25(4):517-572.
24. Power M, Tyrrell PJ, Rudd AG, et al. Did a quality improvement collaborative make stroke care better? A cluster randomized trial. Implement Sci. 2014;9(1):40. doi: 10.1186/1748-5908-9-40.
25. Landon BE, Wilson IB, McInnes K, et al. Effects of a quality improvement collaborative on the outcome of care of patients with HIV Infection: the EQHIV Study. Ann Intern Med. 2004;140(11):887-896.
26. Schouten LM, Hulscher ME, van Everdingen JJ, Huijsman R, Grol RP. Evidence for the impact of quality improvement collaboratives: systematic review. BMJ. 2008;336(7659):1491-1494. doi: 10.1136/bmj.39570.749884.BE.
27. Kaplan HC, Brady PW, Dritz MC, et al. The influence of context on quality improvement success in health care: a systematic review of the literature. Milbank Q. 2010;88(4):500-559. doi: 10.1111/j.1468-0009.2010.00611.x.
28. Robert Wood Johnson Foundation. Aligning Forces for Quality: the fourth phase. 2012 Request for Proposals. Princeton, NJ.
29. Robert Wood Johnson Foundation. Sustainability and long-term viability. Aligning Forces for Quality website. http://forces4quality.org/node/11989.html. Accessed May 8, 2016.
30. Beery WL, Senter S, Cheadle A, et al. Evaluating the legacy of community health initiatives: a conceptual framework and example from the California Wellness Foundation’s health improvement initiative. Am J Eval. 2005;26(2):150-165. doi: 10.1177/1098214005275627.
