DP11-1111 Evaluation: March 2014 Project Update

CDC• Temeika Fairley, PhD, Lead Health Scientist• Angela R. Moore, MPH, Program Evaluation and Partnership Team Lead

ICF• Sarah O’Dell, Project Manager•Mary Ann Hall, Senior Associate/Acting Project Manager•Marnie House, Senior Technical Specialist• Dara Schlueter, Senior Associate• Shelby Cash, Associate• Revae Downey, Analyst• Jenn Mezzo, Analyst

DP11-1111 Evaluation Project Team

To determine:• the strategies programs* are

using to support young breast cancer survivors (YBCS), • the extent to which these

strategies used are effective, and • the extent to which these

strategies are sustainable.

Evaluation Purpose

☑ Conduct evaluation planning activities☑ Develop evaluation questions☑ Develop evaluation design and methodology☐ Implement evaluation☐ Analyze data to answer evaluation questions☐ Disseminate evaluation findings

Evaluation Plan And Approach

• Conducted Evaluation Planning Activities– Reviewed interim and annual progress reports– Conducted key informant interviews with

grantee program staff– Developed grantee and DP11-1111 logic models– Inventoried and assessed existing evaluation

efforts– Summarized preliminary findings

• Develop Evaluation Questions– Engaged relevant stakeholders

Evaluation Steps 1 & 2

1. What are the core components of the DP11-1111 cooperative agreement?

2. What are the factors that affect the implementation of DP11-1111 programmatic activities?

3. What support services and educational resources have organizations developed and/or implemented as a part of their intervention targeting YBCS?

4. What are the factors that affect the implementation of support services and educational resources?

Evaluation Questions

5. How have CDC’s technical assistance (TA) and support activities contributed to grantees’ capacity and sustainability efforts?

6. How have organization’s interventions/strategies affected awareness for, access to, and utilization of support services and educational resources among YBCS, health care providers, and families/caregivers?

7. How have DP11-1111 grantees’ interventions/strategies affected knowledge, attitudes and behaviors among YBCS, health care providers, and families/caregivers related to the risks for breast cancer in young women?

Evaluation Questions, continued

• Providers and staff to offer structured support services

• Infrastructure

• Partners

• Access to YBCS

• Subject matter expertise

• Equipment and supplies

• Funding and support to sustain intervention beyond CA

• Grantee funding

• Partnership building

• Organization infrastructure

• Technical Assistance and Guidance

• Topic area experts(public health translation research, program evaluation, communication, and training)

• Tools and resources Business Calendar SharePoint Listserv Program Update

Template Work Plan Template Performance Measure

Guidance Document CCCB Evaluation Toolkit Suitability Plan

Template Recipient Activity

Guidance Document

Grantee Inputs

CDC Inputs

CDC’s DP11-1111 Programmatic Activities (EQ 1)

• Collaborate with grantees to plan and implement evidence based strategies/interventions for YBCS Support grantees to develop programs designed to provide structured support

services Assist grantees in the development of educational and awareness resources Support grantees in the development of publications, modules, manuals, etc. that

relate to the program

• Facilitate the development of partnerships and exchange of information Link grantees to federal, community, and local partners Assist grantees in partnership support activities, including the creation of meeting

agendas and speakers. Work with established federal agencies , community based programs, and/or

national health organizations to help implement program activities

• Provide ongoing guidance and technical assistance to granteesConduct monthly with each grantee to provide TA and discuss progress Conduct quarterly calls with all grantees to discuss progress and share bet practices. Provide grantees with current research and recommendations for YBCS.

• Develop and support a program monitoring system Provide grantees with performance measure logbook. Require reporting of performance measures through interim progress reports.

• Develop and conduct an evaluation for the DP11-1111 CA.

Grantees’ DP11-1111 Programmatic Activities related to Capacity and Infrastructure (EQ1,2)

• Conduct Program Readiness, Oversight and Management Hire and retain staff Develop management plan Develop communication plan Participate in CDC meetings

• Establish New or Expand Existing Partnerships Build and maintain collaborative partnerships that support the development of materials and structural support for YBCS Work with established federal agencies , community based programs, and/or national health organizations to help

implement program activities

• Develop a Sustainability Plan Plan for resources needed to sustain strategies over time Use data to drive plans for sustainability

Grantees’ DP11-1111 Programmatic Activities related to Intervention Implementation (EQ1,2)

• Conduct Needs and Resource Assessment Identify gaps and take action to fill gaps identified

• Set Priorities Based on Needs Assessment Determine goals and objectives to be included in program plan

• Develop a Program Plan Define target population and recruitment plan. Establish priority goals, objectives, activities, and outcomes for the three year funding period Consider ways to collaborate with organizations to enhance implementation Determine methods for collecting baseline and follow-up data

• Implement a Program Plan Link program strategy to goals , outcomes, and impact related to the reduction of cancer burden Create action items to conduct implementation of strategy Describe how target population will be reached through strategy

• Evaluation and Monitor Progress Establish specific, measurable, and realistic short and long term outcomes. Determine both process and outcome evaluation priorities Determine a plan for dissemination of findings to key stakeholders Establish quarterly program milestones to assist with program monitoring

Process

Case Study Design: DP11-1111 Conceptual Framework

Implementation Strategies

Support Services (EQ3 ,4)• Conduct patient coaching • and/or patient navigation to assess/refer patients• Offer specialized support groups for YBCS and

caregivers• Implement YBCS telephone helplines• Develop and maintain partnerships to enhance the

deliver of support services• Convene advisory groups• Conduct healthcare systems redesign to increase

utilization of support services

Educational and Awareness Resources (EQ3, 4)• Create and offer presentations about issues related to

YBCS • Convene conferences targeting YBCS and providers• Offer workshops for YBCS, caregivers, and providers• Create website content related to YBCS• Use social media to raise awareness to YBCS issues

and existence of resources and services• Create videos, publications, workbooks, print media ,

and other materials tailored to the needs of YBCS, caregivers, and providers

• House and maintain an online resource library

Increased intervention sustainability (EQ 5)

• Increased intervention sustainability among grantees to ensure continued implementation after CA

Long-term outcomes(3-6 years)

Improved awareness knowledge, behaviors, and attitudes related to

YBCS (EQ 7)• Increased awareness and knowledge

among YBCS for strategies that reduce the risk of reoccurrence and improve quality of life

• Increased awareness and knowledge among providers for strategies that reduce the risk of reoccurrence and improve quality of life

• Improved behaviors that reduce the risk of reoccurrence and improve quality of life for YBCS

Intermediate outcomes (2-3 years)

Increased access to support services and educational resources

for YBCS (EQ 6)• Increased availability of improved

support services and educational resources for YBCS, caregivers, and providers

• Increased utilization of support services and educational resources among YBCS, caregivers, and providers

Healthy People

2020 Goals

• Increased the proportion of cancer survivors who report physical health-related quality of life similar to the general population

• Reduced overall cancer death rate

• Reduced female breast cancer death rate

Short-term outcomes (~1-2 year)

Increased awareness of the availability of support services

and educational resources (EQ 6)• Increased YBCS awareness of existing

support services and educational resources

• Increased provider awareness of existing support services and educational resources

• Increased YBCS caregiver awareness of existing support services and educational resources

Outcomes

Program monitoring, evaluation, and improvement

Increased grantee capacity (EQ 5)• Increased organizational capacity to plan and

implement intervention strategies to facilitate support services and educational resource development for YBCS

January 2013 DRAFT Version 2

• Developed Evaluation Design and Methodology– Developed multiple case study protocol

and design– Conducted in-depth interviews with CDC

program staff– Developed data collection instruments,

evaluation plan and clearance packages

Evaluation Step 3

• Case studies will be conducted with 12 selected organizations

– 7 DP11-1111 cooperative agreement awardees

– 5 organizations currently developing and/or implementing YBCS interventions/strategies (not DP11-1111 grantees)

• Similar target audiences

• Similar tools and strategies to reach their target audiences, and plan and implement key intervention activities.

• Unfunded organizations have been included to provide a broader understanding of YBCS interventions across various types of organizations and create an “inventory” of existing efforts across the U. S. aimed at providing education and support to YBCS.

Case Study Design: Sample

10

Case Study Design: Site Selection Process

• All 7 DP11-1111 grantees are included in the case studies

• In collaboration with CDC, we used a systematic selection process to find and assess unfunded organizations implementing YBCS interventions/strategies

• Unfunded organizations are not intended to “match” grantee organizations or to determine the effect of CDC funding; rather, inclusion of unfunded organizations will allow CDC to explore similarities and differences between two case types with respect to specific organizational attributes.

Case Study Design: Non-Grantee Site Selection Matrix

Domain Components

Target Population YBCS

Families and/or Caregivers

Healthcare Providers

Underserved Populations

Intervention Setting Clinical

Non-Clinical

Communication Channels

Mass Media

Small Media

Social Media

Interpersonal Communication

Intervention Strategies Educational Resources

Support Services

• John C. Lincoln Health Foundation (Phoenix, AZ)• Living Beyond Breast Cancer (LBBC) (Haverford,

PA)• Louisiana State University and Health Services

Center (New Orleans, LA)• Sharsheret (Teaneck, NJ)• University of California at Los Angeles (Los

Angeles, CA)• University of North Carolina at Chapel Hill

(Chapel Hill, NC)• Washington University at St. Louis (St Louis,

MO)

Funded YBCS Programs (DP11-1111 Grantees)

13

• Knight Cancer Institute Adolescent and Young Adult Oncology Program at Oregon Health and Science University (Portland, OR)

• Program for Young Women with Breast Cancer at Dana Farber (Boston, MA)

• Tigerlily Foundation (Reston, VA)• Young Survival Coalition (New York, NY)• University of Colorado at Denver

(Denver, CO)

Unfunded YBCS Programs

14

• Implement Evaluation– Obtained Institutional Review Board

(IRB) approval– Currently under review for Office of

Management and Budget clearance– Conduct of case studies• Document review• Site visits

Evaluation Step 4

• Each site visit will include:– In-depth interviews – On-site observation of key programmatic

activities– Development/refinement of program logic

model

• Each site will receive an in-depth case study after completion of the visit– Sites will review case studies before they are

finalized– Technical assistance calls will be offered to

review the case study report

Case Study Site Visits

• Analyze Data to Answer Evaluation Questions– Qualitative data analysis with Atlas.ti software

• Disseminate Evaluation Findings– Create case study topline summary reports for

each program– Synthesize findings to create a final evaluation

summary report– Develop manuscript and conference

presentation to disseminate findings

Evaluation Steps 5 & 6

Questions?

Points of contact

Mary Ann Hall, Acting Project ManagerMaryAnn.Hall@icfi.com

Marnie House, Senior Technical SpecialistMarnie.House@icfi.com

Thank you!