Research Brief: Lessons in Primary Care Transformation: Implementation and Adoption of Health Care Innovation Across Five CMS Initiatives

Lessons in Primary Care Transformation: Implementation and Adoption of Health Care Innovation Across Five CMS Initiatives


The Centers for Medicare & Medicaid Services (CMS) Center for Medicare & Medicaid Innovation (Innovation Center) facilitates the development and testing of approaches to improved health care payment and service delivery through increased efficiency, quality, and reduced cost.

For over a decade across the United States, CMS has implemented and evaluated primary care transformation initiatives that aim to increase the comprehensiveness of care and strengthen coordination of care among primary care providers, other clinicians, and community-based social services.1 In particular, these innovations seek to improve care for complex patients, incentivize and support transitions from encounter-based payment to population-based payment, encourage multi-payer participation, and increase the use of measures that are meaningful to both providers and patients.

This brief is one of a two-part series in which we present findings from a systematic review of independent evaluations commissioned by CMS of innovations focused on primary care transformation. In this brief, we summarize key workforce characteristics, patient engagement strategies, applications of health information technology (health IT), involvement of partners, and considerations for sustainability that influenced the implementation and, potentially, outcomes of primary care transformation initiatives; the second brief describes the organizational and external factors that impacted implementation.*

* Hertel, M., M. Skillman, K. Soforic, E. Shapiro, and G.W. Torres. 2018. Lessons in Primary Care Transformation: The Role of Organizational and External Features in Adopting Health Care Innovation Across Five CMS Initiatives. Research Brief. Chicago: NORC.

Providing insight into models and transformation efforts that can improve health and health care delivery can help public and private decision-makers better understand how to successfully implement innovations. To add to this ongoing policy conversation, we begin by describing the roles of care coordinators/care managers, lay health workers, and physicians who comprise the critical workforce necessary to transform primary care. In conjunction with specifying “who” carries out innovations, we outline strategies to recruit and sustain patient engagement, as well as effective patient identification through risk stratification, enhanced communication, and quality improvements facilitated by health IT. Our findings also describe the best approaches to forming partnerships with other organizations, such as creating formal agreements or relying on past relationships to pave the way for new collaborations. Lastly, we synthesize cost and utilization outcomes across initiatives. Though outcomes findings are mixed, they provide additional context for interpreting the relative success of the primary care transformation initiatives as related to initiative implementation, individually, and as a whole. Together, our qualitative and quantitative findings present a comprehensive picture of levers and conditions under which future initiatives may improve cost, utilization, and quality.

Overview of Primary Care Transformation Initiatives

This study examines results across five primary care transformation initiatives with nearly 1,500 participating health care organizations. These initiatives ran for 3–12 years over a period spanning from 2002 to 2016; we draw our data from evaluation reports released between 2014 and 2017. Exhibit 1 describes the scope and various characteristics of these initiatives.

Exhibit 1. Initiative Design and Evaluation Features

Information about the HCIA initiative also gleaned from the Innovation Center website:

Understanding Innovation Adoption

We structured our approach to synthesizing evaluation findings across these diverse initiatives by adapting a framework from implementation science developed by Fisher, Shortell, and Savitz (2016), illustrated in Exhibit 2.13 This framework suggests that the implementation, adoption, and eventual outcomes of an innovation are informed by the external environment and the characteristics of the adopting organization, in addition to the specific innovation features.

Exhibit 2. Conceptual Framework for Analyzing Innovations

SOURCE: Adapted from Fisher et al., 2016.

External characteristics include the policy and market environment, characteristics of the patient population, and the supply of providers. Organizational characteristics include the type of organization, resources available (e.g., staff, infrastructure, and finances), leadership, history, and culture. The interaction of organizational and external characteristics influences the choice of an organization (or individual provider) to pursue an innovation with particular innovation components such as aims, forms of payment and risk, and care delivery features. These characteristics also affect the process by which innovations are implemented and ultimately can affect the successes and challenges encountered during the implementation process.

Assessing variability of the external and organizational characteristics, innovations, and implementation across models and across participants in any one model is an important part of understanding the nature and range of primary care transformation outcomes as well as implications for sustainability. We developed and refined the conceptual framework by reviewing features of each initiative (e.g., purpose, innovation type, environmental and policy context) to develop high-level domains and research questions that could be applied across initiatives.

This brief explores findings and themes related to implementation processes of initiatives, specifically as they relate to innovation components including workforce, patient engagement, use of health IT, and partnerships with other groups or organizations to carry out interventions. These relationships reveal best practices and challenges to successfully adopting innovations and, in turn, impacting outcomes.

Data and Methods

Data Sources. We reviewed the final evaluation reports for five initiatives that focused on primary care transformation. The reports in our analysis were publicly available on the CMS Innovation Center website in Spring 2017. Four initiatives were categorized on the website as “primary care transformation.” We included six additional reports from the first-round Health Care Innovation Award (HCIA) portfolios that incorporated components of primary care transformation (e.g., through care coordination, intervention setting). We excluded two HCIA portfolios because they featured programs largely implemented in hospitals or health systems and settings other than primary care facilities (e.g., emergency facilities). In terms of data gathering, we collected both qualitative and quantitative results at the initiative level, although in some cases evaluators mentioned specific awardees or practices, which we include in our final results. We only analyzed initiatives that were completed at the time of analysis. When final evaluation reports were not available, we analyzed the most recent interim report.§ We included one final report in our analysis that became publicly available while our study was underway.**

‡ Community Resource Planning and Prevention and Hospital Setting HCIA portfolios

§ The Comprehensive Primary Care Third Evaluation Report was available at the time of review.

** The Multi-Payer Advanced Primary Care Practice Third Evaluation Report was available at the time of initial review. Updates to the data were made following release of the final report.

Analysis. We developed a codebook using domains from Fisher, Shortell, and Savitz’s conceptual framework, described above. Before coding, we collected basic information from each initiative on program focus, payer, awardee number and type, intervention setting, conditions addressed, care or payment innovations, and major outcomes to add a level of inductive refinement to the codebook. The final codebook included 47 codes organized across nine code “families” that were based on the conceptual framework and research questions: care innovation; payment innovation; program features; staffing and workforce; policy and market environment; organizational context; outcomes; sustainability, replicability, and scalability; and implementation and evaluation context (e.g., challenges, facilitators, limitations, unintended consequences).

Six analysts participated in the coding process in March and April of 2017 using NVivo software (QSR International Pty Ltd., version 10, 2012). Coders achieved at least 87-percent interrater reliability with at least two other team members. Coders met and held targeted consensus-building discussions to answer specific questions and discuss discordant themes and codes. Senior researchers conducted a random spot-check of final reports to check their agreement with coding decisions.

The team used NVivo’s querying function to retrieve coded data relevant to our research questions. Following this, we organized these data into subthemes that emerged inductively from the coded data and developed cross-initiative findings related to external and internal features.

For the quantitative outcomes data, we descriptively assessed program impacts for five primary care transformation initiatives using aggregate results, where available. In order to compare the impacts across the initiatives, we applied a crude calculation of per beneficiary per month for financial impact (spending) and per 1000 beneficiaries per quarter for utilization measures (hospitalization, 30-day readmission rate, and ED visits).

Limitations. Synthesizing results across programs with varying evaluation designs, strengths of comparison group, units of analysis, and level of detail presented several challenges. We accommodated variation in unit of analysis across programs by consolidating specific quantitative measures reported for various programs into three broader categories of cost, utilization, and quality. We identified common units of analysis where the evaluation measures aligned and supplemented the findings with qualitative information where outcome measures did not exist. Due to variations in type of quantitative analyses, making comparisons across initiatives was challenging. Furthermore, extrapolating outcomes to specific initiative features was confounded by internal and external factors, implementation fidelity, and ongoing federal and state policies and initiatives.

Additionally, reports varied considerably in the level of detail offered on implementation challenges, successes and lessons learned, and outcomes. We addressed these challenges in part by comparing findings at the initiative level as often as possible, making our units of analysis more similar. We also revisited reports when necessary to capture additional context around a theme of interest to inform findings at a higher level. However, the lack of data in some reports remains a limitation of our analysis and a consideration for what should be contained in future evaluation reports.


Team-based care characterized multiple initiatives focused on improving care coordination and care management. Delegating certain tasks to midlevel clinicians, nurses, social workers, and other support staff allowed clinicians to work at the top of their licenses. Teams that effectively shared information between clinical and support staff, and clearly defined team roles and responsibilities, also tended to report more successful collaborations with higher value on individual staff. For instance, primary care practices that delivered team-based care were more successful at achieving goals related to care management for high-risk patients, quality improvement, shared decision-making, and care coordination compared with practices that placed less emphasis on care teams.2

Explaining the value of new roles helped garner clinical team buy-in and facilitate team integration. Toolkits or informational handouts for care teams explained and delineated staff roles, mitigating possible confusion.7,9 Conversely, a lack of understanding about care managers’ responsibilities among other staff meant that care managers were often asked to perform duties outside of their role, in turn, eroding their specific function.2 Care teams were comprised of individuals with varying levels of clinical expertise, disciplines (e.g., behavioral health, pharmacy, or social work), and training, however, three staff types— care coordinators/managers (usually nurses), lay health workers, and physicians—emerged as especially critical to transforming care in the tested initiatives.

Care Coordinators and Care Managers

Most initiatives incorporated a centralized care manager or care coordinator in the team. This staff member managed communication between care teams, conducted assessments, facilitated transitions, followed up with patients, scheduled appointments, created care plans, or taught self-management, among other responsibilities. Registered nurses (RNs) and nurse practitioners (NPs) often managed centralized hubs for care coordination efforts. At times, NPs assumed physicians’ responsibilities or concentrated their efforts on highest-risk patients so that their clinical skills and judgment were put to the best use. In an expanded care manager/coordinator role, clinicians could spend more time with high-risk patients than physicians could; these personal relationships enabled them to gather information on patients’ clinical and social needs, and, in some cases, increased patients’ receptivity to the program (e.g., telehealth, home visits).

In one initiative, practices reported that care coordinators successfully helped patients use care more appropriately and reduced emergency department visits.2 Similarly, primary care practices participating in a patient-centered medical home (PCMH) initiative described care managers as the “most transformative aspect” of the model, which was demonstrated in the outcomes—practices that engaged in care management activities (at a high level of overall adoption within the practice) showed slower growth in acute-care expenditures compared with other practices as well as slower growth in emergency department (ED) visits not leading to hospitalization among high-risk patients, compared with other practices.11,12 These care managers, however, typically interacted with only a small percentage of a practice’s patients, and some states expressed skepticism about the initiative’s impact on the broader population.11,12

Lay Health Workers (LHWs)

LHWs (e.g., community health workers, peer educators, and patient navigators) acted as primary interfaces between initiatives and patients. They often acted as the primary interface between programs and patients and helped deliver home-based care, educated patients about disease management, or made referrals to community-based services. Often, LHWs had similar demographic or language backgrounds as the patients they served. Generally, LHWs were a cost-effective alternative to licensed clinicians and conducted patient outreach activities targeting Medicaid populations, who may be younger, healthier, and may not require intensive intervention from a licensed clinician or physician. Integrating LHWs into clinical care delivery, however, seemed less common; initiatives that successfully did so provided robust training and substantial team support to LHWs.4–9 In some cases, LHWs were viewed as a key determinant of program success due to their ability to engage high-risk, vulnerable patients as part of multidisciplinary care teams.8


Physician buy-in was crucial to successfully implementing initiatives. Although physicians were usually not central to intervention efforts and care teams, they played critical roles as program champions, experts (e.g., consulting on medication changes), and sources of referrals. A host of implementation challenges stemmed from a lack of physician involvement. For instance, physicians who did not buy into primary care transformation initiatives were usually not valuable sources of referrals into programs, making it harder to meet recruitment targets. Maintaining physician buy-in to initiatives waned when the level of financial gain was not as great as anticipated, such as when shared savings payments diminished over the course of the program.11,12 Further, the fee-for-service payment system incentivized multiple, brief visits, which conflicted with some physicians’ ability to complete the more intensive care management activities of some initiatives.2

Workforce Training

Sufficient prior experience and training was key for staff success. Multiple initiatives found that hiring intervention staff with previous experience in a specific care setting or disease condition as well as providing adequate on-the-job training facilitated implementation success.4–9 Initiatives that aimed at transforming primary care engaged a wide array of staff training, which varied by staff level, intervention, and site. Modes of training ranged from didactic to experiential learning. Typically, experiential learning (e.g., shadowing), ongoing training throughout the implementation period, motivational interviewing training, and tailored technical assistance (e.g., technical assistance related to problem-solving and onsite visits) were most useful to staff).11,12 Across multiple awardees, robust training was key to ensuring that initiative staff, especially LHWs or individuals filling new care team roles, felt prepared to carry out their responsibilities.4-9

Patient Engagement

Engaging patients with the goal of improving care coordination and patients’ health care experiences, along with empowering them to self-manage conditions, can improve quality and lower the cost of care. Successful patient engagement melds patient activation (a patient’s knowledge, skills, and ability to manage his or her own health) with interventions designed to promote positive behavior.14 Successful patient engagement occurs when patients or their representatives partner with health care providers to improve health, which depends on effective recruitment and sustained engagement. Reductions in acute care Medicare expenditures in one initiative were associated with the identification and incorporation of patient preferences into care planning as well as self-management support for chronic conditions with care team members trained in patient education, empowerment, and problem-solving.11,12 Evaluators detected this association despite a relatively low proportion of practices that conducted patient engagement activities (57 percent).12

Our findings fall across two major categories, or temporal phases—initial recruitment, meaning facilitating patients’ entry into interventions, and sustained or increased engagement over time, “staying the course even under stress.”14 Exhibit 3 provides an overview of promising patient engagement strategies reported by organizations participating in primary care transformation initiatives.

Exhibit 3. Patient Engagement in Primary Care Transformation Initiatives

Initiatives leveraged diverse staff to reach patients, including care managers, care coordinators, social workers, NPs, LHWs, and pharmacists. Although effective engagement strategies differed by intervention aim, structure, target population, and context, many initiatives recognized that patient engagement meant addressing a spectrum of clinical and nonclinical needs. Social workers or LHWs were often employed to deliver health education as well as social services and supports; however, physicians were sometimes preferred when discussing sensitive matters such as advanced care planning or conveying technical information about diseases and chronic conditions. We note that despite in-depth training and investments from dedicated staff, it remained difficult to meet all of patients’ social needs, especially if those needs existed alongside complex medical conditions.4–9 Throughout program implementation, it was common for organizations and practices to adapt their recruitment and engagement techniques, such as adjusting disease categories and intervention eligibility criteria.


Several recruitment strategies facilitated enrollment, and organizations reported success when using primary care physician referrals and identifying the correct time to introduce and enroll patients into the program. Generally, in-person interactions and patients’ familiarity with programs or the providers recommending programs facilitated enrollment.

Primary Care Referrals

Primary care providers were effective at referring patients into programs. Specifically, providers were well situated to refer patients into programs given their trusted relationships with patients and knowledge about patients’ clinical and social needs. Referrals from a trusted provider likely primed patients to be more accepting of programs, compared with cold calls or mailings from programs. For example, primary care practices with an opt-in strategy that reached out via mass communications (e.g., letters) were largely unsuccessful in prompting patients to make appointments to discuss screenings.2 Many community-based programs relied heavily on provider referrals, even as a supplement to reviewing hospital lists or risk-stratifying.4–9 Given their ongoing and often personal relationships with patients, physicians seemed better situated to provide in-depth clinical and nonclinical information than program staff.

The success of physician recruitment may depend on the type of program. For medication management programs, automatic enrollment was successful. For example, one organization used automatic enrollment using an opt-out recruitment model and successfully enrolled patients into medication management programs.5 Patients were particularly likely to engage when programs automatically scheduled medication management appointments for them.5


Successful engagement efforts balanced contacting patients upstream of an acute event against reaching out in the immediate aftermath of an acute event. Choosing the point at which to recruit patients and intervene could be determined by the specific patient populations or condition of focus. Among populations with chronic conditions, patients were generally more ready to engage with programs soon after an acute disease-related event, when their perceived need for information about care choices was highest.9 At the same time, it was important to assess whether approaching patients in crisis would overwhelm them; for example, calling patients after a hospitalization proved more effective than trying to recruit them while they were still in the hospital, when they may have been overwhelmed.7 Patients were also more engaged with their care if programs initiated outreach well before a treatment decision was necessary.4 For instance, one organization provided support around surgical decision-making during a primary care clinical visit instead of during a later specialty care visit.4

Building and Sustaining Engagement

Practice-based and community-based programs recognized the importance of sustaining patient engagement. Strategies ranged from workflow redesign and hiring culturally aligned staff to thinking creatively about how to effectively interact with patients. Overall, multiple modes or combinations of outreach and interaction may contribute to positive outcomes; primary care practices that used alternative outreach methods (e.g., e-mail, text messages, web portals) experienced slower growth in the rate of ED visits not leading to hospitalization, compared with other primary care practices in one initiative.11,12

Workflow Redesign

Interventions refined workflows and adapted technology to enhance health care access and communication with patients. Strategies included:

  • Scheduling same-day services (e.g., laboratory services, medication reviews, primary care visits) to mitigate scheduling and transportation challenges5,9
  • Extending walk-in hours2
  • Adding telephone lines and/or expanding hours patients can contact practices2
  • Creating patient portals to streamline communications2,10–12
  • Creating efficient check-in and scheduling procedures (e.g., automated processes for tracking available appointments)2,10


Multiple initiatives reported that cultural fluency helped reach patients.2,5-9,11  As previously noted, culturally or linguistically aligned staff who were either from or intimately familiar with targeted communities or who were provided with training on cultural competency tended to engage patients more easily. In particular, LHWs who mirrored patients’ background, language, or culture facilitated patient trust-building. Furthermore, giving programs flexibility to translate and tailor materials to reflect different cultural norms (e.g., dietary guidelines, personalized decision aids, health education) helped meet the needs of diverse populations.

Mode of Interactions

Both practice- and community-based initiatives noted the vital importance of continuous face-to-face interactions. This mode of direct contact was useful in building relationships and trust so that patients felt comfortable communicating with initiative staff, who, in turn, could tailor education as needed. Messaging also proved important; co-branding materials with logos that patients were familiar with facilitated engagement.5

Some initiatives reported that in-person interactions may have had a direct impact on outcomes.4–10 For instance, in-depth medication reviews involving complex information were more successful when conducted in-person.5 In-person interactions may also have implications for job satisfaction; staff who had face-to-face interactions with patients reported higher levels of satisfaction compared with those who did not.5,8 In contrast, patient-facing staff in one initiative had lower levels of satisfaction compared with staff in other roles, though these concerns may be implementation-related.4

Some programs under some initiatives engaged patients via remote interactions or home visits to conveniently reach patients, particularly those with limited mobility or complex social needs or those in rural areas who had difficulty accessing care.2,4-9,11 Telemedicine and remote monitoring enabled patients to consult with specialists and receive decision supports, while home visits gave patients access to education and social support from clinicians and LHWs.

No one mode of follow up emerged as a top follow-up strategy; rather, programs delivered a variety of services, such as home visits, telephone calls, and mailings, to conveniently bring education and care to patients depending on patients’ condition or health status.4–9 Follow-up phone calls after care transitions increased patient satisfaction, and practices believed this helped reduce readmissions.2 Follow up after an in-person medication management interaction also seemed effective.5

Health Information Technology

Three key indicators of health information technology (health IT) implementation success included (1) the extent to which interoperable health IT infrastructures were already in place, (2) the degree of integration with larger systems to access the “right” data, and (3) the level of organizational commitment to integrate health IT into clinical workflows. Exhibit 4 illustrates the various uses of health IT by organizations across demonstrations and notes some particular challenges that were highlighted.

Exhibit 4. Uses of and Challenges with Health IT in Primary Care Transformation Initiatives


Several initiatives used electronic health record (EHR) data to identify gaps in care, identify high-risk patients, and improve population health management through administrative data or health registries.2,4–9,11,12 Two critical factors to implementing successful patient identification systems included: 1) continually reviewed and refined algorithms that contribute to systematic EHR reviews, and 2) presence of a staff member who could review patient lists to make sure that the appropriate patients were identified as eligible for the program. In one initiative, organizations that reported high-level registry use for high-risk patients experienced slower growth in acute-care expenditures, compared with other practices.12 In cases where data report lags prevented practices from immediately identifying effective improvement strategies, patient-level data were still useful for identifying longer-term gaps in care and care management approaches.11 Participating practices in another initiative reported that risk-stratified care management was the biggest improvement related to implementing the model.2

For risk stratification to be successful, scoring criteria had to be consistent, capable of accurately identifying the right patients, and easy to understand. For some organizations, risk stratification (including claims-based risk stratification) resulted in challenges due to the complexity of the data and variance in data quality as well as how providers and clinicians applied scoring criteria.2,7 In a few cases, organizations and practices relied on real-time and in-person assessments when risk algorithms could not accurately identify eligible patients, but some organizations felt that staff (e.g., care managers) spent too much time screening algorithm-identified patients to see whether they were truly eligible for the program.6,11,12 To remedy this, organizations supplemented algorithm-based EHR enrollment with referrals from providers.6 Practices also moved toward more reliance on real-time assessments when they noticed discrepancies between risk scores in historical claims data and data based on clinical interactions and practices’ EHR data.11,12


Health IT also played a major role in facilitating communication among providers and between providers and patients. Some practices reported that use of patient portals improved patient engagement, although accessing portals was more challenging for older adults who lacked comfort with technology. For these practices, portals translated to time-savings because they allowed support staff to triage patient questions and communicate with patients more efficiently (e.g., not “playing phone tag”).2

Overall, the following activities enhanced care team and provider communications:

  • Using EHR-based messaging to enhance efficiency and establish provider-to-provider trust2
  • Operationalizing portable information (e.g., discharge documents)8
  • Sharing web-based resources in the field8
  • Using EHR data from partnering health care systems (e.g., information from a statewide health information exchange provided by pharmacists with patients’ medical and pharmacy fill history)5

Quality Improvements

Some practice-based initiatives connected quality improvements with the following EHR uses:2,11,12

  • Generating timely quality reports2,11,12
  • Tracking follow-up and preventive services2
  • Informing care management activities (e.g., identify gaps in care)7

Practices that reported using data reports from their EHRs to guide quality initiatives showed the greatest improvements in care delivery.2 Some practices leveraged EHRs to track electronic clinical quality measures (eCQMs), which meant that providers could track follow up related to preventive services to organize care for high-risk patients.2 In terms of supporting team-based care, EHR data and medical records informed team huddles and enabled some organizations to identify gaps in care and prepare for scheduled patient visits.7

Quality reports were most useful when they summarized data from current sources and arrived in a timely manner. Several practices in one initiative had access to a demonstration web portal that could generate quarterly practice-level reports as well as beneficiary utilization and assignment files, yet some practices found reports outdated and less salient than reports practices received from Medicaid or commercial payers based on claims and clinical data.11,12

Other Challenges

Lack of interoperability and inability to share data emerged as primary concerns across initiatives.4–9 This challenge was two-sided—some organizations lacked ability to share important data with providers and/or lacked capacity to receive information from other providers to inform care management. This was a particular challenge related to managing high-risk populations.7–9 Some organizations also reported encountering legal and regulatory obstacles related to data sharing, and awardees or partner organizations often lacked capacity to train staff to use new health IT features.8

Tailoring technology to meet demonstration or program needs required sufficient funding and IT support. Some organizations reported that adapting technology was more labor- and resource-intensive than they anticipated (e.g., creating a new platform for EHRs or telehealth devices).6 When software to support interventions was not readily available, a number of organizations collaborated with internal staff or vendors to develop program-specific platforms.4–9 Similarly, a few practices in another initiative used care management software to supplement practices’ EHR systems, which were constructed to support encounter-based billing rather than comprehensive care management. Nevertheless, many practices could not use EHRs to document care management encounters and could not offer care plans to their patients. These practices reported that additional funds would have enabled them to hire IT experts to improve EHR documentation and meet reporting requirements.2


It was common for organizations to involve partners when implementing initiatives. Although the conditions for successful partnerships varied by organization, intervention, or care setting, practice-based initiatives identified effective partnership and stakeholder engagement strategies and challenges that are applicable to multiple contexts, as illustrated in Exhibit 5.4-9 Partnerships were particularly important for interventions that linked different types of settings. Coordination across home-based and clinical settings, including unaffiliated health systems, required collaboration between providers or organizations through data or information exchange and referrals to services.

Exhibit 5. Creating Partnerships in Primary Care Transformation Initiatives

Aligning initiative goals with partner goals facilitated relationship-building. 6,7 The following strategies helped strengthen or establish collaborative relationships with other organizations:

  • Building on existing past relationships with partners2,8
  • Forming relationships with agencies that serve the same target populations8
  • Planning for and dedicating sufficient time to developing partnerships2,5,8

Experience also mattered to stakeholders; organizations in one initiative noted that successful partnerships were easier to form when medication management interventions were well established, or “mature,” because building trust requires time and effort.5

Generally, it was difficult to engage stakeholders that were not directly involved in implementing initiatives because they lacked incentives to participate or organizational priorities were different from participating organizations; when stakeholders experienced staff turnover, especially among government entities such as health departments, it was also difficult to sustain partner engagement.6,7

Formal agreements legitimized working relationships. Organizations in a primary-care based initiative engaged partners by forming formal financial arrangements.7 Collaborative practices agreements between physicians and pharmacists in medication management programs similarly had the potential to improve care coordination.5 In one initiative, practices that formalized practice agreements and had referral protocols with “commonly referred-to practices” (i.e., practices with preexisting relationships) had slower ED visit growth rates compared with practices that did not adopt formalized agreements.11,12 Similarly, in another initiative, practices were more likely to attribute patient follow up to formalized agreements with EDs and hospitals to track patients; however, care compacts or collaborative agreements were uncommon among practices (19 percent).2 Practices using the same EHR across specialists were less likely to establish care compacts with specialists as they could share information electronically.2

Sustainability Findings: Partnerships

CMS and payers were noted as essential to ensuring long term sustainability for large-scale initiatives.2,11,12

Specifically, CMS could provide valuable guidance to payers about sustaining practice support.2,11,12 One state participating in a multi-payer demonstration partnered with an independent regional nonprofit that monitored performance, aggregated data, and created a sustainability plan for the state’s intervention.11,12 Importantly, a committee composed of participating payers, government officials, and providers oversaw this partner’s work.11,12 In addition to financially supporting a multi-payer state-sponsored PCMH-based model, payer participation institutionalized changes and encouraged payers to invest long term in health care innovation.11,12

Cost and Utilization

The evaluations most commonly reported outcomes related to expenditures, utilization, and quality; however, there was considerable variability in analysis methods—use of a comparison group, target population, units of analysis—that hindered direct comparison. The following is a crude calculation and standardization across several initiatives to provide spending in units of per beneficiary per month, and utilization in units of per 1000 per quarter. Quality measures varied widely and often depended on the health outcomes relative to a target disease or condition and consequently could not be collapsed.

Variations in spending and health outcomes were observed across and within the initiatives. Cost savings were not always associated with decreases in hospitalizations, readmissions, or ED visits. For many smaller awardee-specific programs, the inability to identify a credible treatment group, a lack of comparison group in the pre-intervention and post-intervention period, unavailability of outcome measures, and low statistical power to detect effects on core measures posed challenges to assess program impacts. Exhibit 6 shows the adjusted quantitative outcomes for one HCIA portfolio and three other primary care transformation initiatives. There were few significant findings (indicated by the asterisks), especially at or above the p<0.05 level.

Exhibit 6. Abbreviated Summary of Primary Care Transformation Initiatives

††Frontier Extended Stay Clinic Demonstration was excluded due to lack of comparison data. We included outcomes from those initiatives whose main focus was on primary care transformation. While some innovations in the behavioral health, complex/high-risk, disease specific, medication management, and shared-decision making HCIA portfolios had elements of primary care transformation, their main outcomes could not be reliably attributed to changes in primary care delivery. ‡‡ Extension impacts. §§ Cumulative years 1-3, without CPC care management fee.

Two initiatives that showed positive cost outcomes provided comprehensive coordinated care in a PCMH environment or targeted patients that had experienced adverse health events and could not be transferred to acute care facilities. Achieving net savings of Medicare payments, however, was challenging across multiple initiatives.3,11,12

Expenditures related to implementation of a primary care transformation initiative may negate cost savings to claims achieved through the initiative. For example, an initiative designed to offer extended stay services at remote clinics (not included in Exhibit 5) averted 26 transfers and 26 hospitalizations per year, which translated into an estimated $285,558 and $423,904 in total Medicare savings per year, respectively.3 Despite these savings, continuing as an extended stay facility (e.g., cost of equipment, staffing) may not make the program budget-neutral to Medicare. Further, the volume of extended stay patients needed to cover costs was projected to fall far short of actual demand for extended stay services in remote or rural areas served by demonstration clinics.3


This review of initiative evaluation reports yielded valuable insights into initiative implementation that may apply to other models of care:

  • Staffing. Choosing not only the right staffing model but the right mix of staff types enabled providers to work at the top of their licenses and efficiently coordinate care. A clear and shared understanding of staff roles typically generated team cohesion and buy-in.
  • Patient identification and engagement. Determining the right time to introduce patients to interventions depending on their disease, severity, or condition improved patient engagement. Identifying the “right” staff who are best situated to engage patients built trust more effectively. Focusing on sustaining patient engagement through personal relationships and conveniently delivering care emerged as important considerations for future initiatives.
  • Health IT. Using health IT enabled the identification of high-risk patients, which enabled cross-provider communication and coordinated care management. Implementing health IT successfully depended upon existing health IT infrastructure and practices’ ability to integrate new components within larger systems. Strong partnerships enabled data exchange and referrals.
  • Outcomes. Overall, the evaluations did not directly tie implementation processes to cost and utilization outcomes; however, generally, outcomes were mixed. Where Medicare savings were demonstrated, savings in practice payments, acute care expenditures, and reductions in hospitalizations and ED visits emerged as primary drivers. Future evaluations should consider the ties between implementation success and features and ensuing outcomes.

While we have found that independent evaluations investigate to some extent the relationships between innovation components and implementation processes, further clarifying these linkages and their ultimate impact on innovation adoption and outcomes will provide critical insight into the conditions under which care transformation initiatives can be successful.


1) Centers for Medicare & Medicaid Services. 2017. Advanced Primary Care Initiatives. (accessed July 5, 2017).

2) Mathematica Policy Research. 2016. Evaluation of the Comprehensive Primary Care Initiative: Third annual report.

3) Burwell, S. M. 2014. Evaluation of the Medicare Frontier Extended Stay Clinic Demonstration. Report to Congress.

4) Acumen. 2017. Evaluation of the Shared Decision Making (SDM) Health Care Innovation Awardees third annual report.

5) Acumen. 2016. Evaluation of the Medication Management (MM) Health Care Innovation Awardees third annual report.

6) Mathematica Policy Research. 2017. Evaluating the HCIA—behavioral health/substance abuse awards: Third annual report.

7) Mathematica Policy Research. 2017. Evaluation of Health Care Innovation Awards (HCIA): Primary care redesign programs, third annual report.

8) NORC at the University of Chicago. 2017. HCIA complex/high-risk patient targeting: Third annual report.

9) NORC at the University of Chicago. 2016. HCIA disease-specific evaluation: Third annual report.

10) Mathematica Policy Research. 2015. Evaluation of the Medicare Coordinated Care Demonstration: Final report for the Health Quality Partners’ Program.

11) RTI International. 2016. Evaluation of the Multi-Payer Advanced Primary Care Practice (MAPCP) Demonstration: Third annual report.

12) RTI International. 2017. Evaluation of the Multi-Payer Advanced Primary Care Practice (MAPCP) Demonstration: Final report.

13) Fisher, E. S., S. M. Shortell, and L. A. Savitz. 2016. Implementation science: A potential catalyst for delivery system reform. JAMA 315 (4): 339–340.

14) Hibbard, J. H., J. Stockard, E. R. Mahoney, and M. Tusler. 2004. Development of the Patient Activation Measure (PAM): Conceptualizing and measuring activation in patients and consumers. Health Serv Res, 39 (4 Pt 1): 1005–1026.

15) Carman, K. L., P. Dardess, M. Maurer, M. Tusler, S. Sofaer, K. Adams, C. Bechtel, and J. Sweeney. 2013. Patient and Family Engagement: A Framework for Understanding the Elements and Developing Interventions and Policies. Health Affairs, 32 (2): 223–231.

16) Acumen. 2017. Evaluation of the Shared Decision Making (SDM) and Medication Management (MM) Health Care Innovation Awardees third annual report addendum.

17) Mathematica Policy Research. 2017. Evaluating the HCIA—behavioral health/substance abuse awards: Addendum to the Third annual report.

18) Mathematica Policy Research. 2017. Evaluation of Health Care Innovation Awards (HCIA): Primary care redesign programs, addendum to third annual report.

19) NORC at the University of Chicago. 2017. HCIA complex/high-risk patient targeting: Third annual report addendum.

20) NORC at the University of Chicago. 2016. HCIA disease-specific evaluation: Third annual report addendum.

This research was supported by the Laura and John Arnold Foundation. The views expressed in this brief are those of the authors and do not necessarily reflect the views of the funder.

About NORC

NORC at the University of Chicago is an objective and non-partisan research institution that delivers reliable data and rigorous analysis to guide critical programmatic, business, and policy decisions. Since 1941, NORC has conducted groundbreaking studies, created and applied innovative methods and tools, and advanced principles of scientific integrity and collaboration. Today, government, corporate, and nonprofit clients around the world partner with NORC to transform increasingly complex information into useful knowledge.

© NORC 2018