The Centers for Medicare & Medicaid Services (CMS) Center for Medicare & Medicaid Innovation (Innovation Center) facilitates the development and testing of approaches to improved health care payment and service delivery through increased efficiency, quality, and reduced cost.
For over a decade across the United States, CMS has implemented and evaluated primary care transformation initiatives that aim to increase the comprehensiveness of care and strengthen coordination of care among primary care providers, other clinicians, and community-based social services.1 In particular, these innovations seek to improve care for complex patients, incentivize and support transitions from encounter-based payment to population-based payment, encourage multi-payer participation, and increase the use of measures that are meaningful to both providers and patients.1
This brief is one of a two-part series in which we present findings from a systematic review of independent evaluations commissioned by CMS to assess innovations focused on primary care transformation. In this brief, we explore selected organizational characteristics, as well as external or environmental factors, that influence implementation and sustainability. An organization’s external and internal environment provides the context in which an innovation is adopted.2,3 Context shapes the motivation for innovation, its design, and its implementation. It presents opportunities and challenges that must be leveraged and navigated to achieve success. Understanding pathways through which internal and external context affect innovation adoption provides insight to public and private decision-makers about features of models and transformation efforts that can improve health and health care delivery. In the other brief, we summarize key workforce characteristics, patient engagement strategies, applications of health information technology (health IT), and partnerships that influenced the implementation and possible outcomes of primary care transformation initiatives.*
* Skillman, M., M. Hertel, M. Dougherty, K. Soforic, E. Shapiro, S. Lee, and G.W. Torres. 2018. Lessons in Primary Care Transformation: Implementation and Adoption of Health Care Innovation Across Five CMS Initiatives. Research Brief. Chicago: NORC. https://reports.norc.org/issue_brief/primary-care-transformation-implementation/
Overview of Primary Care Transformation Initiatives
This study examined results across five primary care transformation initiatives with nearly 1,500 participating health care organizations. These initiatives ran for 3—12 years over a period spanning from 2002 to 2016; we draw our data from evaluation reports released between 2014 and 2017. Exhibit 1 introduces these initiatives.
Exhibit 1. Initiative Design and Evaluation Features
Understanding Innovation Adoption
We structured our approach to synthesizing evaluation findings across these diverse initiatives by adapting a conceptual framework from implementation science developed by Fisher, Shortell, and Savitz (2016), illustrated in Exhibit 2.2 This framework suggests that the implementation, adoption, and eventual outcomes of an innovation are informed by the external environment and the characteristics of the adopting organization, in addition to the specific innovation features.
Exhibit 2. Conceptual Framework for Analyzing Innovations
External characteristics include the policy and market environment, characteristics of the patient population, and the supply of providers. Organizational characteristics include the type of organization, resources available (e.g., staff, infrastructure, and finances), leadership, history, and culture. The interaction of organizational and external characteristics influences the choice of an organization (or individual provider) to pursue an innovation with particular innovation components such as aims, forms of payment and risk, and care delivery features. These characteristics also affect the process by which innovations are implemented and ultimately can affect the successes and challenges encountered during the implementation process.
Assessing variability of the external and organizational characteristics, innovations, and implementation across models and across participants in any one model is an important part of understanding the nature and range of primary care transformation outcomes as well as implications for sustainability. We developed and refined the conceptual framework by reviewing features of each initiative (e.g., purpose, innovation type, environmental and policy context) to develop high-level domains and research questions that could be applied across initiatives.
Data and Methods
Data Sources. We reviewed the final evaluation reports for five initiatives that focused on primary care transformation. The reports in our analysis were publicly available on the CMS Innovation Center website in Spring 2017. Four initiatives were categorized on the website as “primary care transformation.” We included six additional reports from the first-round Health Care Innovation Award (HCIA) portfolios that incorporated components of primary care transformation (e.g., through care coordination, intervention setting). We excluded two HCIA portfolios because they featured programs largely implemented in hospitals or health systems and settings other than primary care facilities (e.g., emergency facilities).‡ In terms of data gathering, we collected both qualitative and quantitative results at the initiative level, although in some cases evaluators mentioned specific awardees or practices, which we include in our final results. We only analyzed initiatives that were completed at the time of analysis. When final evaluation reports were not available, we analyzed the most recent interim report.§ We included one final report in our analysis that became publicly available while our study was underway.**
‡ Community Resource Planning and Prevention and Hospital Setting HCIA portfolios
§ The Comprehensive Primary Care Third Evaluation Report was available at the time of review.
** The Multi-Payer Advanced Primary Care Practice Third Evaluation Report was available at the time of initial review. Updates to the data were made following release of the final report.
Analysis. We developed a codebook using domains from Fisher, Shortell, and Savitz’s conceptual framework, described above. Before coding, we collected basic information from each initiative on program focus, payer, awardee number and type, intervention setting, conditions addressed, care or payment innovations, and major outcomes to add a level of inductive refinement to the codebook. The final codebook included 47 codes organized across nine code “families” that were based on the conceptual framework and research questions: care innovation; payment innovation; program features; staffing and workforce; policy and market environment; organizational context; outcomes; sustainability, replicability, and scalability; and implementation and evaluation context (e.g., challenges, facilitators, limitations, unintended consequences).
Six analysts participated in the coding process in March and April of 2017 using NVivo software (QSR International Pty Ltd., version 10, 2012). Coders achieved at least 87percent interrater reliability with at least two other team members. Coders met and held targeted consensus-building discussions to answer specific questions and discuss discordant themes and codes. Senior researchers conducted a random spot-check of final reports to check their agreement with coding decisions.
The team used NVivo’s querying function to retrieve coded data relevant to our research questions. Following this, we organized these data into subthemes that emerged inductively from the coded data and developed cross-initiative findings related to external and internal features.
Limitations. Synthesizing results across programs with varying evaluation designs, strengths of comparison group, units of analysis, and level of detail presented several challenges. We accommodated variation in unit of analysis across programs by consolidating specific quantitative measures reported for various programs into three broader categories of cost, utilization, and quality. We identified common units of analysis where the evaluation measures aligned and supplemented the findings with qualitative information where outcome measures did not exist. Additionally, reports varied considerably in the level of detail offered on implementation challenges, successes and lessons learned, and outcomes. We addressed these challenges in part by comparing findings at the initiative level as often as possible, making our units of analysis more similar. We also revisited reports when necessary to capture additional context around a theme of interest to inform findings at a higher level. However, the lack of data in some reports remains a limitation of our analysis and a consideration for what should be contained in future evaluation reports.
We identified several organizational characteristics that were important aspects of innovations in primary care transformation. These included organizational size, prior experience with innovation, and leadership commitment. Exhibit 3 provides an overview of these findings.
Exhibit 3. Organizational Factors in Primary Care Transformation Initiatives
Organizational size affected initial investment in models as well as the organization’s ability to change processes or practice and respond to feedback. Large, integrated primary care practices were more likely than smaller practices to have the resources needed to scale health IT systems and hire staff and consultants, including health IT, care management, and other health care providers and support staff.4 Larger practices also showed the capacity to participate in formal, centralized quality improvement initiatives (e.g., reviewing feedback reports, having staff project managers across multiple practices to standardize care management).†† Smaller practices and practices unaffiliated with specialists or hospitals were unaware of external resources to access comprehensive patient data, which compromised their ability to identify high-risk, high-cost patients and coordinate care.4,13,14 Creating change within larger organizations, however, also created challenges with navigating through complex administrative systems, which, in turn, complicated efforts to adjust clinical workflows or ensure consistent practices across multiple sites.4,6–11
†† Comprehensive Primary Care practice size, ownership, urban vs. rural status, and physician compensation were not significantly associated with reduced Medicare expenditures or improvements in patient-centered care, though some primary care providers in the initiative had little to no control over or information about how care was provided to their patients by providers outside their practice, making it difficult to realize shared savings.
Sustainability Findings: Organization Size
Sustaining care transformation was more feasible for larger practices and health systems.
Smaller practices and rural clinics struggled to cover the cost of full-time care managers, establish health IT systems, or offer extended hours.4,5 Usually, larger organizations’ inherently had more resources to draw from (e.g., larger workforce pool, existing health IT structures), which may explain this difference.
Prior organizational experience with innovation positively affected adoption of innovations. Organizations attributed implementation success to a combination of prior experience with care transformation and innovation in addition to having flexibility to tailor their models to population types and local contexts, among other site-specific factors.6–11 Some organizations that implemented novel innovations without evidence of prior success experienced more challenges in recruiting providers and establishing workflow changes than those that scaled existing programs or expanded to new sites.8
Implementation was smoother when organizations built on care coordination and care management endeavors that had been established prior to the CMS initiative (e.g., care transitions programs, accountable care organizations [ACOs], and patient-centered medical homes [PCMHs]); however, organizations found it challenging to attribute outcomes to specific models when organizations implemented other care transformation innovations. Further, these concurrent programs may have diminished the measurable impacts of CMS primary care transformation initiatives.4,9,12 In addition, other models such as a PCMH primary care transformation effort created competing organizational priorities and contributed to “change fatigue” among participating providers.13,14
Leadership and Management Style
Across initiatives, strong, invested leadership encouraged success. Positive leadership attributes included the following:
- Having a genuine desire to listen and elicit feedback4
- Being recognized by internal and external stakeholders as a respected leader in the field10
- Possessing a consistent vision and sufficient expertise to ensure fidelity to models, especially across sites10
- Identifying or acting as a champion to consistently communicate the program’s goals and obtain staff buy-in4,9
Though these attributes may apply to a variety of implementation settings, practice champions who were either a program manager or physician successfully obtained physician buy-in among practices. When a practice’s champion did not make efforts to elicit buy-in and communicate vision and purpose, clinicians and staff did not understand the initiative’s goals.4
In terms of managing care teams composed of clinicians and/or non-clinicians, effective leaders set a tone of a non-punitive collaboration and mutual understanding that was critical to coordinating care and efficiently refining workflows.6–11,12 Such leadership required significant time and investment; practice champions and care managers implementing some programs reported lacking time to invest in practice transformation and intensive care management activities.9 This barrier, in combination with other factors, may have contributed to a lack of quantitative impacts on quality of care, utilization, and cost.
Sustainability Findings: Leadership
The connection between organizational capacity, commitment, program success, and sustainability is complex.
Large organizations typically had extensive internal management and capital resources to operate complex interventions in changing or uncertain environments, which was highly advantageous for sustaining and scaling some initiatives.10 In terms of practice-level care transformation, PCMH features that were well integrated into care delivery workflows reflected a commitment to maintaining changes over time, even if the entire intervention was not formally sustained.13,14
In some cases, leadership commitment rather than quantitative evidence of success drove sustainability efforts.10,11 This finding may be related to commitments to serve targeted conditions, intervention design, timing and availability of quantitative results, or evaluation design that required a multiyear investment. Sometimes, programs were sustained when health improvements or savings were not expected in the immediate term (e.g., interventions targeting patients with cardiovascular disease), when data were not available to rigorously evaluate the program, or when qualitative findings indicated success and quantitative findings were not yet available.11 As may be true for other complex interventions involving multiple transformation components and stakeholders, practices implementing a multi-payer PCMH initiative required multiyear investment to build infrastructure and refine operations; by year three, practices felt they were finally in a position to achieve measurable outcomes.13,14
The external environment plays a large role in the implementation and success of a program, often referred to as environmental support. Federal and state policies, concurrent novel and continuing programs, and marketplace trends and regulations had substantial effects on initiative implementation and sustainability. Exhibit 4 provides an overview of these findings.
Exhibit 4. External Environment Factors in Primary Care Transformation Initiatives
Federal Payment Reform
Organizations leveraged changes in reimbursement policies to enhance financial support for some initiatives. With the introduction of the CMS Chronic Care Management Fee Schedule in 2015, one organization was able to bill for diabetes and congestive heart failure health coaching provided by qualifying clinical staff, enhancing financial support for program features.6 Another policy change in 2015 allowed Medicare Advantage (MA) plans to offer beneficiary incentives for health improvement programs, which facilitated another organization’s partnership with new MA plans, enabling the awardee to scale its intervention to 600,000 additional MA beneficiaries by the end of the implementation period.6
A number of organizations tailored their interventions to include staffing models and services that were reimbursable under Medicare and Medicaid. State Medicaid reimbursement policies appeared to drive these decisions to use lay health workers to fill peer educator, care coordinator, and patient navigator roles.10,11 In other cases, reimbursement rules inhibited sustainability. Lack of recognition for pharmacists as Medicare Part B health care providers was the primary barrier to sustaining some medication management interventions.7 Similarly, lack of reimbursement for psychiatric services in primary care clinics delivered by certain organizations complicated the financial sustainability of some interventions.10
Ongoing CMS demonstrations influenced tests of new models and initiatives at the state level. A number of Medicare and Medicaid initiatives and new model tests occurred concurrently with CMS primary care transformation initiatives, including delivery system reforms for dual eligible beneficiaries (e.g., financial alignment initiatives, state initiatives for dual eligible beneficiaries) and other value-based payment models (e.g., pay for performance, ACOs, State Innovation Models [SIMs]). Reforms under Medicaid and the Children’s Health Insurance Program included various payment models (e.g., per patient per month under PCMHs, population-based payments under ACOs) and waivers for home- and community-based services and for long-term services and supports.
Some organizations leveraged new payment models to continue or expand part or all of an initiative.9 The emergence of ACOs both contributed to and detracted from initiative sustainability, while SIM cooperative agreements enabled promising program features to expand under a different funding mechanism. Some practices in a multi-payer demonstration intended to replace CMS funding with shared savings or capitated payments through ACO arrangements, participation in value-based models, or the new monthly Medicare Chronic Care Management fee.13,14 Seven out of eight states in one multi-payer PCMH initiative received either a SIM Model Test or SIM Model Design award, and in 2014, 34 new ACOs across seven of the states in the initiative began participating in the Medicare Shared Savings Program.13,14 The impact of states’ participation in other initiatives varied by state context, and practices’ participation in these alternative payment models could generate support for or divert attention from the Innovation Center initiative.
State reforms influenced initiatives in varying and sometimes conflicting ways. A payer for a practice-based initiative added components of the initiative into SIM requirements, enabling them to continue funding for implemented changes.4 One organization noted that a multi-payer demonstration prepared practices to participate in ACOs and several practices joined ACOs.13,14 However, state officials expressed concern about the possibility of sustaining PCMHs with expanding ACOs and payment reforms under SIM grants.13,14 Additionally, the draw of the ACOs may have detracted from ongoing participation in another initiative—several of the practices that left one initiative early did so to join ACOs.4
Multiple local marketplace factors influenced the scale-up and sustainability of CMS initiatives, including divergent policy or market demands for organizations operating across multiple states, the conditions of local labor markets, and corporate mergers among organizations and market counterparts.10 One organization noted that the Rhode Island Affordability Standards, which required commercial payer investments in primary care, would support program sustainability after a multi-payer PCMH initiative ended.13,14
State Medicaid reforms that increased administrative burden on state Medicaid agencies and health plans along with Medicaid expansion created short-term uncertainties and delays. As a result, sustainability planning became more of a challenge for both state-level initiatives and organization-level interventions.10,13,14 While certain practices leveraged state Medicaid expansions to add new Medicaid patients to their panels during the multi-payer PCMH initiative, concerns persisted about whether the state could support enhanced medical home payments for a higher volume of enrollees, calling into question the capacity and, by extension, the long-term sustainability of the initiative.13,14
Sustainability Findings: Funding Stability
CMS payments or payment designs did not always cover the full cost of transforming primary care; this type of funding instability may have influenced decisions to sustain initiatives.
The upfront investments required for transformation such as infrastructure (e.g., health IT systems) and staffing (e.g., hiring care coordinators) was challenging for some organizations. Among practice-based initiatives, care transformation proved too costly for some practices to sustain or scale changes.
During the demonstration period, practices in a PCMH initiative leveraged funding from sources other than participating payers to fund portions of PCMH initiatives or complementary programs. For example, one awardee supported its health information exchange and clinical registry using the proceeds from a tax on medical claims. Moreover, among practices participating in other federal initiatives, all pursued opportunities to leverage those resources to improve their delivery systems.13,14
Similarly, the majority of practices (seven out of eight) in one demonstration agreed that CMS payments were insufficient to cover the extent of PCMH practice transformation.13,14 By design, this multi-payer demonstration created relationships between the number of participating payers and monthly Medicare care management fees received by practices. Full payer participation ensured there were enough eligible patients for which practices received care management fees; full payer participation also decreased the burden of differentiating services offered to patients covered by participating and nonparticipating payers.13,14
The structure and source of payments as well as expectations of organizations also impacted the degree to which primary care transformation could take place. For one initiative, incentives were shared across regions rather than mapped to individual practices; because financial rewards were diffused in this way, the link between lower service use by individual providers and practices and financial incentives was weakened.4 A number of practices participating in a multi-payer primary care initiative noted that funding was insufficient to incentivize providers to substantially change their practice style; these practices ultimately exited the initiative.13,14
This systematic review of five primary care transformation initiatives highlights the influence of organizational characteristics and external factors on innovation implementation. Specifically, organizational size, organizational experience, and team leadership style had significant bearing on implementation success. Additionally, federal policies and concurrent transformation initiatives as well as state environments further influenced innovation adoption and, ultimately, sustainability. While the contextual features we discuss clearly influenced implementation processes, few evaluations connected these features to impacts such as cost and utilization. In future evaluations, exploring and articulating linkages between contextual factors and outcomes will help policy makers better understand models and their bearing upon outcomes like cost, utilization, and quality of care.
1) Centers for Medicare & Medicaid Services. 2017. Advanced Primary Care Initiatives. https://innovation.cms.gov/initiatives/index.html#views=models (accessed July 5, 2017).
2) Fisher, E. S., S. M. Shortell, and L. A. Savitz. 2016. Implementation science: A potential catalyst for delivery system reform. JAMA 315 (4): 339–340.
3) Damschroder, L. J., D. C. Aron, R. E. Keith, S. R. Kirsh, J. A. Alexander, and J. C. Lowery. 2009. Fostering implementation of health services research findings into practice. Implement Sci 4: 50.
4) Mathematica Policy Research. 2016. Evaluation of the Comprehensive Primary Care Initiative: Third annual report. https://innovation.cms.gov/Files/reports/cpci-evalrpt3.pdf.
5) Burwell, S. M. 2014. Evaluation of the Medicare Frontier Extended Stay Clinic Demonstration. Report to Congress. https://innovation.cms.gov/Files/reports/MFESCD-RTC.pdf.
6) Acumen. 2017. Evaluation of the Shared Decision Making (SDM) Health Care Innovation Awardees third annual report. https://downloads.cms.gov/files/cmmi/hcia-shareddecisionmaking-thirdannualrpt.pdf.
7) Acumen. 2016. Evaluation of the Medication Management (MM) Health Care Innovation Awardees third annual report. https://downloads.cms.gov/files/cmmi/hcia-medicationmanagement-thirdannualrpt.pdf.
8) Mathematica Policy Research. 2017. Evaluating the HCIA—behavioral health/substance abuse awards: Third annual report. https://downloads.cms.gov/files/cmmi/hcia-bhsa-thirdannualrpt.pdf.
9) Mathematica Policy Research. 2017. Evaluation of Health Care Innovation Awards (HCIA): Primary care redesign programs, third annual report. https://downloads.cms.gov/files/cmmi/hcia-primarycareredesign-thirdannualrpt.pdf.
10) NORC at the University of Chicago. 2017. HCIA complex/high-risk patient targeting: Third annual report. https://downloads.cms.gov/files/cmmi/hcia-chspt-thirdannualrpt.pdf.
11) NORC at the University of Chicago. 2016. HCIA disease-specific evaluation: Third annual report. https://downloads.cms.gov/files/cmmi/hcia-diseasespecific-thirdannualrpt.pdf.
12) Mathematica Policy Research. 2015. Evaluation of the Medicare Coordinated Care Demonstration: Final report for the Health Quality Partners’ Program. https://innovation.cms.gov/Files/reports/mccd-hqp-finaleval.pdf.
13) RTI International. 2016. Evaluation of the Multi-Payer Advanced Primary Care Practice (MAPCP) Demonstration: Third annual report. https://downloads.cms.gov/files/cmmi/mapcp-thirdevalrpt.pdf.
14) RTI International. 2017. Evaluation of the Multi-Payer Advanced Primary Care Practice (MAPCP) Demonstration: Final report. https://downloads.cms.gov/files/cmmi/mapcp-finalevalrpt.pdf.