FY 2017 MDHHS Deliverable Submission Form · The formative evaluation described the status of...

FY 2017 MDHHS Deliverable Submission Form

Name of Project: MI-APPP

Document Name: MI-APPP Formative Evaluation Report (Finalized in FY 2017)

Project Number: X- 64898

Program Name: Center for Healthy Communities

Director or Coordinator Name:

Julia Heany, Ph.D.

Submitted By: Julia Heany, Ph.D.

Type of Deliverable: (Please check one)

Final report

Specialized report

Publication

Articles published

Brochures

Software application

Web screen shot

No report, publication, or study

Deliverable pending

OTHER material

If OTHER, please describe:

FORMATIVE EVALUATION REPORT

July 29, 2016

FORMATIVE EVALUATION

REPORTJuly 29, 2016

Created by Michigan Public Health Institute for the Michigan Department of Health and Human Services

Sara McGirr, MA Chiharu Kato, PhD Bianca Burch, MSW, MA Tara Dasigi, MPH Chelsea Walker, MPH Cordelia Martin‐Ikpe, MPH Julia Heany, PhD MPHI

Table of Contents

Introduction…………………………………………………………………………………………………… 3

Methods………………………………………………………………………………………………………… 4

Evaluation Objectives…………………………………………………………………………… 4

Data Collection Activities and Sample…………………………………………………… 4

Analysis………………………………………………………………………………………………… 5

Results……………………………………………………………………………………………………………. 6

Objective 1…………………………………………………………………………………………… 6

Objective 2………………………………………………………………………………………… 15

Objective 3………………………………………………………………………………………… 19

Objective 4………………………………………………………………………………………… 24

Objective 5………………………………………………………………………………………… 26

Discussion……………………………………………………………………………………………………. 32

Summary of Findings…………………………………………………………………………. 32

Limitations…………………………………………………………………………………………. 33

Recommendations…………………………………………………………………………….. 33

Conclusions……………………………………………………………………………………….. 36

3

Introduction The Child Adolescent School Health Section within the Michigan Department of Health and Human Services (MDHHS) is working with six high‐need communities across the state to implement the Michigan Adolescent Pregnancy and Parenting Program (MI‐APPP). MI‐APPP aims to engage pregnant and parenting adolescents 15‐19 years and their families in a system of care to reduce repeat pregnancies, strengthen access to and completion of secondary education, improve maternal and child health outcomes, and strengthen connections between youth and their support networks. Funding for MI‐APPP is provided through the Pregnancy Assistance Fund (PAF) administered by the US Department of Health and Human Services, Office of Adolescent Health.

MI‐APPP sites must use a Positive Youth Development (PYD) approach to serving pregnant and parenting youth. PYD is a prevention‐based approach to working with youth that aims to develop their resiliency through strengthening developmental assets such as social competence, problem solving, autonomy, and sense of purpose.1,2 All MI‐APPP sites must have the following programming components:

1) The Adolescent Family Life Program‐Positive Youth Development (AFLP‐PYD) Case Management Program: Communities serve at least 25 pregnant and parenting adolescents between the ages of 15‐19 per full‐time case manager.

2) Supplemental Services: Communities develop additional services designed to meet the unique needs of local pregnant and parenting adolescents and/or their families as identified through a needs‐assessment process. Pregnant and parenting youth and/or their families may participate in supplemental services regardless of whether they participate in AFLP‐PYD case management; however, each MI‐APPP‐funded community must serve at least 25 “supplemental service participants.”

3) Community‐level Steering Committee: The Steering Committee supports program planning, implementation, and evaluation, and must be representative of the community’s diversity, include pregnant and parenting youth, and ensure the program is responsive to and meets the needs of the community.

MDHHS contracted with the Michigan Public Health Institute (MPHI) to conduct a statewide evaluation of MI‐APPP. The evaluation is comprised of two parts: a formative evaluation3 which examined how AFLP‐PYD and other supplemental services were implemented across the state, and an outcome evaluation, which will examine what happened as a result of implementing these strategies. The following report focuses on the results of the formative evaluation. After an introduction to the methods used, the report will detail the findings of the evaluation and conclude with recommendations for improving implementation.

1 Development Services Group, Inc. 2014. “Positive Youth Development.” Literature Review. Washington, DC.: Office of Juvenile Justice and Delinquency Prevention. http://www.ojjdp.gov/mpg/litreviews/PositiveYouth Development.pdf 2 Search Institute (2016) Developmental Assets: Preparing Young People for Success. http://www.search‐institute.org/what‐we‐study/developmental‐assets 3 Scriven, M. (1991). Evaluation thesaurus. Sage.

4

Methods Evaluation Objectives

The formative evaluation described the status of implementation of AFLP‐PYD and supplemental services and identified factors that may influence effectiveness. The purpose of this evaluation is to optimize the potential for program success by identifying barriers and potential levers for change. Additionally, this evaluation will aid in the interpretation of outcome evaluation results. The objectives of the formative evaluation are as follows:

1) Assess the extent to which AFLP‐PYD was implemented with fidelity across MI‐APPP sites and the extent to which it was adapted.

2) Assess the extent to which supplemental services a) meet a need identified through the needs assessment and b) were implemented as planned across the MI‐APPP sites.

3) Identify barriers and supports that influenced the implementation of AFLP‐PYD and supplemental services and determine their impact on implementation processes across the MI‐APPP sites.

4) Assess the extent to which AFLP‐PYD and supplemental services are satisfactory to AFLP‐PYD and supplemental service participants and MI‐APPP site staff across the MI‐APPP sites.

5) Assess the extent to which AFLP‐PYD participants’ and supplemental service participants’ connection to needed supports and services changed over time across MI‐APPP sites.

The results of the evaluation are presented in light of these objectives.

Data Collection Activities and Sample

Staff Key Informant Interviews Key informant interviews were conducted with staff from each of the six MI‐APPP sites. All current MI‐APPP staff members, as of January 2016, were invited to participate. The interviews lasted between 30 and 75 minutes, were completed over the phone by trained interviewers from MPHI, and were audio recorded to aid in analysis. The protocol consisted of open‐ended questions about activities implemented by MI‐APPP sites, intended and unintended adaptations, implementation barriers and supports, perceptions of the intervention, and the program’s fit with participant needs. A total of 15 MI‐APPP site staff were interviewed, including project coordinators, case managers, recruitment specialists, program assistants, and data entry specialists.

Participant Key Informant Interviews Key informant interviews were conducted with a sample of AFLP‐PYD participants. Participant key informant interviews took place during scheduled site visits to each of the MI‐APPP sites. MI‐APPP staff at each site arranged interviews with at least five AFLP‐PYD youth during their site visit. Sites were asked to recruit from each case manager’s caseload participants who had been in the program for at least six months and to represent both adolescent mothers and fathers, when possible. Youth who chose to participate received a $25 gift card upon completion of the interview. The interviews lasted between 10 and 30 minutes, were completed in‐person by trained interviewers from MPHI staff, and were recorded to aid in analysis. The semi‐structured interview protocol consisted of open‐ended questions intended to capture AFLP‐PYD participants’ experiences in and perceptions of the intervention, as well as the extent to which the program fit their needs. A total of 32 AFLP‐PYD participants, including 26 young women and 6 young men (between two to seven per community) were interviewed.

5

Participant Forms A set of forms developed by the AFLP‐PYD developers, MDHHS, MPHI, and other external partners were used to collect participant information. Data captured in these forms for the formative evaluation included the number and timing of case manager contacts and AFLP‐PYD “My Life Plan” activities, referrals to other agencies, supplemental services received, and other implementation indicators. All of the forms were administered by MI‐APPP site staff as part of the normal course of case management and/or supplemental service provision. Each piece of data was entered into one of two secure data systems (LodeStar or the MI‐APPP Qualtrics System) by MI‐APPP site staff and was cleaned by MPHI staff to ensure accuracy and completion.

A total of 350 participants completed participant forms in the 18 months of implementation between August 1, 2014, and January 31, 2016. This included 174 AFLP‐PYD participants, 124 pregnant and parenting youth supplemental service‐only participants, and 52 supportive supplemental service participants (e.g. family members). Unless otherwise stated, these three types of participants will be reported on separately. Demographic information for AFLP‐PYD participant demographics is presented in Figure 1, for pregnant and parenting supplemental service‐only participants in Figure 10, and supportive supplemental service participants in Figure 11.

Analysis Analyses were designed to address evaluation objectives and maximize the utility of qualitative and quantitative data for understanding MI‐APPP implementation.

Staff Key Informant Interviews A coding scheme was developed through a consensus process and entered into a software program called NVivo for qualitative analysis. MPHI staff coded one MI‐APPP site staff interview and then met with the MPHI Analysis Lead to reach consensus about coding choices and to adjust the coding scheme. Once consensus on the first interview was complete, each MPHI staff member coded a set of MI‐APPP site staff interviews line‐by‐line using NVivo. Three MPHI staff members reviewed the coded transcripts to identify excerpts of text which they felt were coded incorrectly and codes they felt did not apply. Discrepancies were addressed with the original coder. Once all interview coding was finalized, MPHI staff created memos for codes which captured the main ideas in each evaluation area. Finally, MPHI staff developed propositions that presented the general findings by connecting multiple codes and memos. Participant Key Informant Interviews Based on the short and more structured nature of the youth participant interviews, rapid evaluation and assessment methods (REAM) were used for analysis (McNall and Foster‐Fishman, 2007)4. A preliminary coding instrument was created using Qualtrics and was tested. Once adjustments were made, a final Qualtrics instrument was created and used by MPHI staff to code 4‐10 interviews each. Once coding was completed, a matrix of coding within categories was created using Qualtrics, then exported to Microsoft Excel, and used to consolidate interview responses based on relevance and frequency of themes within responses to create memos. Two MPHI staff independently consolidated interview responses, then compared memos for consistency to produce the final results.

Participant Forms Participant form data were entered by MI‐APPP site staff into LodeStar, a database managed by Branagh Information Group, and into Qualtrics, a database managed by MPHI. Every month, MPHI received 4 McNall, M. & Foster‐Fishman, P.G. (2007) Methods of rapid evaluation, assessment, and appraisal. American Journal of Evaluation, 28(2), 151‐168.

6

cumulative LodeStar datasets from Branagh Information Group and downloaded datasets from Qualtrics. Variables needed for analysis from each dataset were extracted and merged to create a master SPSS data file containing all analysis variables. Those variables were then recoded, and composite variables were created to provide descriptive information on each measure.

Results The results of the evaluation are organized by the five formative evaluation objectives. Results from the participant key informant interviews, staff key informant interviews, and participant forms are drawn on to speak to each objective and are interspersed accordingly.

Objective 1: Assess the extent to which AFLP‐PYD was implemented with fidelity across MI‐APPP sites and the extent to which it was adapted.

Recruitment MI‐APPP site staff reported using a variety of recruitment techniques to find and connect with potential participants. The most commonly reported technique was spreading the word about the program to other agencies and organizations serving youth. Site staff shared this information largely via paper and electronic flyers, announcements at coalition or steering committee meetings, and presentations to outside groups or other programs within their sites. In addition to traditional referral networks, MI‐APPP site staff reported that social media and word of mouth from current youth participants were very effective recruitment techniques. Existing youth recruited other participants by sharing information about the program with their young pregnant and parenting friends and/or by bringing them to a supplemental service activity. One MI‐APPP site mentioned that it incentivized this word of mouth process by providing gift cards to existing participants who brought a friend who signed up for case management. AFLP‐PYD youth participants corroborated these findings, reporting that they were introduced to their case managers through other professionals in their life (hospital, doctor, and school personnel), family, friends, or mentors at their high school, or by receiving a MI‐APPP flyer.

Initial Engagement According to the Michigan AFLP‐PYD Case Management Protocol, once a referral has been assigned to a case manager, the assigned case manager should initiate a face‐to‐face contact with the prospective participant within 30 days to determine whether the youth will accept AFLP‐PYD services. After completion of the comprehensive baseline assessment (CBA) and consent (which may or may not occur at first meeting), the enrollment of the pregnant and parenting youth is complete.

Once youth made initial contact with the program, MI‐APPP site staff reported that a case manager or recruitment specialist would generally have a one‐on‐one discussion with the potential participants. The specifics of these meetings varied as case managers customized their approach to the needs and desires of the youth. The focus was largely on building rapport and making pregnant and parenting youth comfortable. Case managers often used these initial conversations to highlight their role as a support person to help participants work toward meeting their needs and goals. Case managers also talked about making sure youth were getting paired up with a staff member that would best meet their needs. For example, a male case manager talked about being sensitive when recruiting young mothers: “Normally I ask if they’re feeling timid or don’t feel comfortable talking to me…I have a female coworker who can initiate the process, and I’ll just be her wing person.” ‐MI‐APPP Site Staff

7

Reach The Michigan AFLP‐PYD Case Management Protocol specifies that programs should enroll female and male expectant and parenting participants age 21 or younger. Prospective pregnant and parenting participants who have more risk factors are to be prioritized for services. Figure 1 provides demographic information on the pregnant and parenting youth enrolled in AFLP‐PYD who completed a CBA. Generally speaking, Michigan ALFP‐PYD participants were mostly school‐aged, female parents ranging in age from 13 to 22 years old with an average age of 17 years old.

FIGURE 1. AFLP‐PYD PARTICIPANT DEMOGRAPHICS

*n = 174 AFLP‐PYD participants who completed the CBA **Demographics are listed in graph in following order: Pregnancy Status, Gender, Ethnicity, Race, and Age.

Figure 2 provides information on active youth participant caseload by quarter. Each quarter lasts three months, with the first quarter (Q1) spanning from August 2014 to October 2014 and so on. Each year includes four quarters, with Year 1 ranging from Q1 to Q4. Quarter 5 (Q5) was the beginning of Year 2 of MI‐APPP implementation, and contained data between August 2015 and October 2015. Figure 2 shows that from August 1, 2014, to January 31, 2016, MI‐APPP sites experienced a gradual increase in program reach. In the first year of implementation, the numbers of active pregnant and parenting participants more than tripled in size (from 29 to 97). This suggests that MI‐APPP sites have improved both their ability to recruit potential youth participants and their capacity to serve more participants. Both points are consistent with information obtained from the MI‐APPP site staff interviews.

≤15(9)

African American(116)

Hispanic(16)

Female(142)

Pregnant(49)

16(28)

White(30)

Not Hispanic(153)

Male(32)

Parenting(112)

17(39)

InterRacial(19)

Pregnant & Parenting (13)

18(40)

Other (8)

19(35)

20 +(23)

0% 20% 40% 60% 80% 100%

AGE

RACE

ETHNICITY

GENDER

STATUS

8

FIGURE 2. AFLP‐PYD PARTICIPANTS BY QUARTER

*n = 174 AFLP‐PYD participants who completed the CBA

**Graph shows the total number of participants in each quarter who had completed the CBA and were active or waitlisted in AFLP‐PYD.

Case Management Activities One of the initial steps in the AFLP‐PYD case management approach is completion of the comprehensive baseline assessment (CBA) as a way to systematically collect, record, and analyze pregnant and parenting youth information which serves as a basis for planning appropriate interventions and referrals. The Michigan AFLP‐PYD Case Management Protocol specifies that the case manager must complete the CBA with the youth participant within 60 days of enrollment. Figure 3 shows the number of CBAs completed across sites each quarter. CBA completion rates stayed relatively steady across most quarters, and as of January 31, 2016, 174 CBAs 5 were completed. Figure 4 details how many CBAs were completed each quarter at each MI‐APPP site.

FIGURE 3. CBA COMPLETION BY QUARTER ACROSS MICHIGAN

*n = 174 AFLP‐PYD participants who completed the CBA **CBA completion by quarter across the entire state of Michigan

5 Raw dataset contains 187 cases with a CBA. Of these, 13 cases were excluded from this analysis due to incompletion and double entries, resulting in 174 participants with completed CBAs. In order to calculate the rate and duration of time of CBA assessment, each participants’ date of activation was needed. There are some cases where participants’ activation dates was registered later than their CBA dates. In order to calculate the timing of CBA, those data were excluded from the analysis, resulting in 163 valid CBAs with valid activation dates.

29

5571

97

133 139

Q1 Q2 Q3 Q4 Q5 Q6

Year 1 Year 2 through January

29 29

19

31

43

23

Q1 Q2 Q3 Q4 Q5 Q6


9

FIGURE 4. CBA COMPLETION BY QUARTER ACROSS MI‐APPP SITES

* AFLP‐PYD participants who completed the CBA: BCHD n= 35; ERESA n = 37; Saginaw n = 35; Starfish n = 30; Pathways n = 29; CCPHD n = 8 **CBA completion by quarter across six MI‐APPP agencies

Of the AFLP‐PYD participants who were expected to have completed their CBA (N=206), the majority (69%) completed a CBA within 60 days of the youth becoming an active AFLP‐PYD participant, with an average of approximately 20 days between “Active” status and CBA administration. Figure 5 details the breakdown of the timeliness of assessment. The length of time between participants’ activation and CBA completion ranged from 0 to 217 days, with a median of 0 days. Out of the 143 CBAs administered within 60 days of being made active, 58% (N = 94) of participants’ CBAs were completed on the same day they became active. These figures suggest that sites may be waiting to make their AFLP‐PYD participants active until they finish their CBA.

Out of the case management participants with a completed CBA and a valid active status, twenty cases (10%) had a CBA completion date that was more than 60 days after the participant became “Active.” These cases were considered “late,” as per the instructions given in the MI‐APPP guidance. These CBAs were completed between 61‐217 days after the participant’s active date, with an average completion date of 100 days following a participant becoming active and a median completion date of 89 days after active status. It is the case that some of the late completion cases may include clients who reentered the program after a period of inactivity; however, the dispersion of these dates was likely most grossly impacted by the inconsistent use of the status change form across MI‐APPP sites addressed above. In some cases, sites may be opting to make almost any new participant “active” from the time they meet the client, regardless of readiness to begin case management. This could result in lengthy amounts of time between this status and CBA completion. This stands in contrast to other sites, who appear to start most of their clients with the “waitlist” status and, in some cases, begin working on the CBA before clients become “active.” A site‐by‐site examination of the CBA completion data supports the assertion that sites appear to be using these statuses differently. For example, the Saginaw and Starfish sites

0

2

4

6

8

10

12

14

Q1 Q2 Q3 Q4 Q5 Q6

BCHD

ERESA

Saginaw

Starfish

Pathways

CCPHD


10

(who liberally use the “waitlist” status) had no “late” CBA forms, but the BCHD and Pathways sites (who start most clients as “active” status) had several. Working to have clear guidelines around using the status change form will improve our ability to provide accurate, meaningful data on the timing of CBA completion.

FIGURE 5. TIMING OF CBA COMPLETION

*n = 206 AFLP‐PYD participants who were expected to have completed their CBA **CBA: Comprehensive Baseline Assessment ***Graph depicts how many AFLP‐PYD participants completed CBA in a timely manner

MI‐APPP case managers provide services and referrals to aid participants in completing their goals. AFLP‐PYD youth participants received help with activities, such as obtaining baby‐related goods, finding apartments or low‐income housing, securing transportation, attending counseling, receiving tutoring, and identifying resources that may be needed after leaving MI‐APPP. Pregnant and parenting participants also reported working with their MI‐APPP site case managers to set and accomplish a variety of goals, such as fixing their credit, enrolling in or completing their education, and finding a job. Several site staff reported taking an active part in helping youth access resources and learn skills. Staff often accompanied youth participants to find needed resources ‐ such as housing, furniture, food, or baby items – and to attend appointments with the adolescent’s doctor, court, or state social service programs.

The AFLP‐PYD Framework requires MI‐APPP site case managers to use a PYD perspective. MI‐APPP site staff often described engaging in activities founded in PYD principles and prioritized empowering youth and encouraging them to take charge of their lives. This process took many shapes, including using motivational interviewing techniques, engaging the adolescents’ voices in determining what activities to offer as a program, and focusing on providing youth with opportunities rather than on telling them what to do: “We don’t provide advice; we provide options.” ‐MI‐APPP Staff Adaptations AFLP‐PYD allows for flexibility in the timing of case management activities to allow staff to be responsive to the needs and preferences of their participants. Case managers took full advantage of this flexibility; the format and structure of case management activities varied greatly based on the youth’s needs and preferences as well as the style the case manager used to approach youth. One adaptation MI‐APPP staff described most often was allowing more time to build rapport before jumping into assessments or goal‐setting activities. Case managers felt that developing the relationship with the youth participants helped them feel more comfortable having open and honest communication.

69%

10%

21%

On time Late Needed, not yet done

11

“I might not always get right into the goals…It might just be more or less where I’m listening and letting them talk…And I’ll say that from the standpoint of working with the dads because boys/young men/men in general, it’s not as easy for one to really open up, and especially to another male…you’ve got to let them feel comfortable and be able to open up. Because, if not, you’re really going through the motions then, and they’re definitely not going to really gain anything from being in the program.” –MI‐APPP Staff

MI‐APPP staff said that they often quickly jumped into helping youth meet their immediate needs. MI‐APPP site staff noted that often the pregnant and parenting youth disliked waiting to begin meeting their needs and getting what they desired, and, as such, the program was in danger of participants disengaging if this element was delayed.

“And so for the teens that have been most successful with us, like we really are talking about goals right away. And even though the programming ‐ the way it’s set up ‐ is not to do that, it’s to wait, they really don’t want to wait. And if we try to follow the program the way it’s set out…most of the teens get pretty disinterested. They really want things done right away that meet their needs.” –MI‐APPP Staff

Staff also described making adjustments to their case management approach depending on the degree of involvement of the youth participant’s family. When families are present, conversations about needs or goals are influenced by the resources and perspectives of this support system. Case managers described navigating a delicate balance as they tried to ascertain how involved the youth wanted their parents to be as well as how much influence the family wanted to have.

Intervention Dosage AFLP‐PYD specifies that participants should receive a minimum of two face‐to‐face contacts with their case manager monthly at a location that is convenient for both parties; at least one of the face‐to‐face visits should take place at the participant’s home each quarter. In order to examine if this requirement was met, information captured in the contact log was analyzed for participants who had completed their CBA, had valid contact data, and had been in the program more than a month for face‐to‐face visit dosage (N = 144) or three months for home visit dosage (N = 126). 6 The lighter colored bars in Figure 6 demonstrate the extent to which MI‐APPP was meeting the requirement of two face‐to‐face visits per month. The majority (72%) of the youth participants had an average of one or fewer face‐to‐face meetings with their case manager each month, which was less than half of the recommended dosage. The average number of face‐to‐face meetings across participants was 1.14 meetings each month; however, AFLP‐PYD contacts ranged from 0 to 7 such meetings a month. The darker colored bars in Figure 6 demonstrate the extent to which MI‐APPP is meeting the requirement of one home visit per quarter. The majority (64%) of participants met with their case manager at home an average of one or more times per quarter, meeting the required dosage. Looking at each participants’ average number of home visits per quarter, the average is greatly dispersed, ranging from 0 to 18 per three month increment, with an overall participant average of 2.6 home visits per quarter.

6 In order to determine dosage, active number of days as of January 31 was calculated for each active participant. For exited participants, the active numbers of days were calculated by subtracting their CBA dates from their exit dates. Active days were converted into months (for Face‐to‐Face Visits) and quarters (for Home Visits).

12

FIGURE 6. PERCENTAGE OF RECOMMENDED FACE‐TO‐FACE VISITS AND HOME VISITS EACH AFLP‐PYD PARTICIPANT RECEIVED

Community ConnectionsMI‐APPP is intended to create an integrated system of care, including linkages to support for pregnant and parenting youth by connecting them with other programs and/or resources in the local community. Each MI‐APPP site’s staff reported building and leveraging connections with other organizations to varying degrees of success. MI‐APPP site staff talked about partnering with programs focusing on women’s and children’s health, such as family planning or teen health clinics; the Women, Infant, Child Nutrition program; and home visiting programs. Organizations and agencies that provide concrete supports, such as food banks and baby‐related supplies, were also common partners. Connections with early childhood supports and daycare or preschool centers were forged, as were partnerships with critical care agencies for participants who have experienced or suffered from sexual assault, domestic violence, or substance abuse. Finally, MI‐APPP sites connected with educational facilities, such as alternative high schools, community colleges and universities, and GED support programs, along with employment programs such as Michigan Works. MI‐APPP sites used these partnerships for a variety of purposes including leveraging community connections to recruit pregnant and parenting youth, providing supported referrals to participants, building successful steering committees to facilitate collaboration, and providing speakers and venues for supplemental services. Program Retention & Completion The AFLP‐PYD Framework suggests that several months or years may be required for participants to reach their goals and move through the program. However, MI‐APPP staff reported that keeping youth active in the program for this period of time was not always possible. One way to measure active retention in case management is by identifying the percent of AFLP‐PYD participants who remain engaged after six months and twelve months in the program. Figure 7 provides information on participant engagement at six months.

Of the 80 active participants who had completed a CBA 6 months or more prior to January 31, 2016, 58% (N=46) had completed their 6‐month follow‐up assessment. Of these 46 participants, 35 6‐month follow‐up assessments were administered and completed within 7 months (210 days, the recommended time frame) after the CBA, and 11 were completed late, meaning sometime after the 7‐month maximum

6% 6%

16%

72%

42%

7%

41%

10%

100% (All) 75‐99% 50‐74% < 50%

All Face to Face Visits (n = 144)

Home Visits (n= 126)

13

recommended time frame. As was identified among those participants who had “late” CBA completion dates, the timeframe for “late” 6‐month Follow Up completion varied. On average, late 6‐month Follow Ups were completed in 255 days (8 months after CBA, or 1 month later than our conceptualization of on‐time completion), with a median of 246 days. It should also be noted that some of the late completion dates may include cases with participants’ who reentered the program after a period of inactivity (exit and reentry). Taking into account all on‐time and late cases, 6‐month Follow Up forms were administered an average of 6.5 months (193 days) after the CBA assessment (with a range from 98 to 344 days). A site‐by‐site comparison revealed that the majority of 6‐month Follow Ups completed by Saginaw, BCHD and Eaton RESA were finished on time (Saginaw=5, 100% on time; BCHD =13, 93 %; ERESA = 8, 89%), whereas Starfish completed a majority of their 6‐month Follow Ups late (N = 6, 67 %).

FIGURE 7. 6‐MONTH FOLLOW‐UP COMPLETION AMONG AFLP‐PYD PARTICIPANTS

*n = 80 active AFLP‐PYD participants who were eligible to complete the 6‐month follow up **Graph depicts how many AFLP‐PYD participants completed 6‐month follow‐up form in a timely manner

Figure 8 provides information on participant engagement at 12 months. Of the 31 active participants who completed a CBA 12 months or more prior to January 31, 2016 and were retained at six months, 39% (N=12) had completed their 12‐month Follow Up assessment by January 31, 20167. Five of these assessments were completed late, meaning sometime after the maximum recommended time frame (370 days). As was the case with other forms, there was no noticeable pattern for late submission. “Late” 12‐month Follow Ups were completed in an average of 403 days, with a median of 391 days. As was the case with other form completion time frames, the late 12‐month Follow Up completion dates may include cases with participants who reentered the program after a period of inactivity (exit and reentry). ERESA and CCPHD did not have valid 12 month completion data for this analysis. Starfish and Pathways combined had the most 12‐month Follow Ups (N Starfish = 4, N Wayne = 4), followed by BCHD (N = 3) and Saginaw (N =1). Pathways had the highest proportion of on‐time completion of 12‐month Follow Up (N = 3, 75 %), whereas BCHD had the lowest proportion of on‐time completion (N =1, 33 %). Finally, as most participants had not yet been in the program for a year by this time, the majority (57%) of the 133 participants retained after 6 months were not yet due for their 12‐month Follow Up assessments.

7 There are two 12 month FU that are submitted without 6 month follow‐ups. There two cases were excluded for this analysis.

44%

13%

43%


14

FIGURE 8. 12‐MONTH FOLLOW‐UP COMPLETION AMONG AFLP‐PYD PARTICIPANTS

*n = 31 active AFLP‐PYD participants who were eligible to complete the 12‐month follow‐up ** Graph depicts how many AFLP‐PYD participants completed 12‐month follow‐up form in a timely manner

The Michigan AFLP‐PYD Case Management Protocol indicates that participants are allowed to remain in AFLP‐PYD until they have reached their goals or until they are no longer eligible, no longer wish to participate, or are unresponsive to contacts for three months. A total of 73 AFLP‐PYD participants with a completed CBA exited out of the program between August 1, 2014, and January 31, 2016. The number of days in the program before exiting varied from participant to participant, with an average of 202 days, a median of 165 days, and a range of 14days to 454 days after CBA completion. Notice that both the average and median number of days are less than 210 days, suggesting that exit from the program happens before, or around the 6‐month Follow Up. In fact, as of January 31, 2016, 24 % (N = 41) of the 174 participants who completed a CBA dropped out of the program sometime between CBA completion and their 6‐month Follow Up assessment. Eighteen percent (N = 24) of the 133 remaining participants exited the program between 6 months and 12 months after CBA completion.

Youth participant retention was also a challenge observed by MI‐APPP site staff. Figure 9 provides reasons for program exits. The two most common reasons provided for program exit were that participants were unable to be located (52%) or that they chose to exit the program voluntarily before completion (25%). Only 8% (n = 6) of participants exited because they were “self‐sufficient,” meaning they had accomplished what they needed in the program.

FIGURE 9. REASONS FOR PARTICIPANTS’ EXIT FROM AFLP‐PYD

*n = 73 AFLP‐PYD participants who exited the program sometime after completing the CBA.

23%16%

61%


1

2

2

5

6

18

38

Other

No Longer EligibleIndex Child Death

Moved

Self SufficientVoluntary Exit

Unable to Locate

15

When looking closely at the characteristics of participants who left the program, there were no major demographic differences between the general AFLP‐PYD population and exited participants. There were, however, slight differences observed. For example, 17‐year‐olds were slightly overrepresented among the exited group (31% vs. 22% in the overall participant pool). More data is needed to determine if this warrants a targeted intervention.

A site–by‐site examination of exited participants demonstrated that the number of exited participants was proportionate to the number of total participants at each site (BCHD=14 exited participants; ERESA=23; Saginaw=19; Starfish=9; Pathways=7; CCPHD=1). In other words, those sites with larger caseloads also generally reported a higher number of participant exits.

Objective 2: Assess the extent to which supplemental services a) meet a need identified through the needs assessment and b) were implemented as planned across the MI‐APPP sites.

Development of Supplemental Service Programming The ideas for supplemental service activities came from a variety of sources, but almost all were youth‐driven. Staff from most sites gathered ideas informally from conversations with case management youth participants and formally from surveys, focus groups, or discussions with adolescent advisors during a steering committee meeting.

Recruitment Supplemental service participants learned about upcoming activities in a variety of ways. The most frequently mentioned recruitment techniques included flyer distribution to youth‐serving agencies, case‐manager communication to pregnant and parenting youth, and word‐of‐mouth encouragement amongst pregnant and parenting youth. Other techniques used by several agencies included social media posts, local newspaper or billboard advertisements, and program newsletters. Most agencies reported offering incentives to encourage participation, including event transportation, food, prizes, free relevant services, and gift cards. Two sites arranged for youth participants to receive course credit for MI‐APPP activity participation. Staff reported that youth found it empowering to help other pregnant and parenting youth by bringing them to the program. Likewise, when they brought other youth to supplemental services, staff believed that the activity provided a good opportunity for youth participants to enhance their existing supportive relationships. Supplemental Service Activities MI‐APPP sites drew on staff capacity, community partners, and youth participant expertise and connections to offer pregnant and parenting youth and their families a variety of supplemental service opportunities. MI‐APPP staff offered ongoing group discussions, fun events or community activities, or some combination therein on a monthly or quarterly basis. When engaging folks in supplemental services, MI‐APPP sites generally aimed to incentivize and break down barriers to participation by offering food, transportation, childcare, and educational components. Sites covered educational topics including financial literacy (5 sites), healthy cooking and eating (4 sites), healthy relationships (3 sites), and parenting (3 sites). Entertainment‐focused activities were also offered which aimed to promote social connections and positive interactions among young parents and their children. Such activities included family fun days, picnics, holiday parties, and field trips. Finally, a minority of sites described their provision of material goods, such as diapers and clothing, as part of their supplemental services.

16

Reach Event turnout varied greatly from youth‐specific classes ‐ with as few as two participants ‐ to large public meetings (e.g. picnics, family fun days) ‐ with sometimes more than 60 people present. The majority of people attending supplemental service activities were also engaged in AFLP‐PYD case management at all but one site. Participants not in case management were usually family members supporting pregnant or parenting youth or, less frequently, other young parents. Demographic information on youth participants only receiving supplemental services (i.e. pregnant and parenting youth not enrolled in case management) is provided in Figure 108. Within the first 18 months of implementation, a total of 124 pregnant and parenting adolescents who were not active AFLP‐PYD participants received supplemental services9. The demographic composition of the youth receiving supplemental services was similar to that of AFLP‐PYD participants in terms of parenting status, gender, race, and age. Sixty‐five percent of supplemental service‐only pregnant and parenting teens were parents, 24% were expectant, and 11% were both pregnant and parenting. Similar to AFLP‐PYD participants, sixteen and seventeen‐year‐olds were the largest age group represented (46 % when combined). A site‐by‐site examination of pregnant and parenting youth supplemental service participants revealed that the MI‐APPP site in Saginaw served the largest number of supplemental service‐only participants (N=77).

Staff key informant interviews revealed that many of the sites had focused their supplemental services on meeting the needs of AFLP‐PYD participants rather than seeking additional non‐case management participants to serve. Likewise, many of the participants attended multiple events throughout the time‐frame. As such, the small number of supplemental service‐only participants served at each site may not reflect a lack of services offered or low attendance at supplemental services events. Rather, this may indicate that sites had a small but consistent contingent of non‐case management participants who attended multiple events or that events were largely attended by AFLP‐PYD participants.

8 These figures may include youth supplemental service participants who are also waitlisted or active AFLP‐PYD participants who have not yet completed the CBA as of January 1, 2016. 9 Of the 124 supplemental‐service only pregnant and parenting teen participants, there were 45 pregnant or parenting youth who attended supplemental services that were technically in the AFLP‐PYD LodeStar system, but who did not have an “Active” status. These participants were listed as active waitlist, inactive waitlist or outreach status, and as such they are not included in data on AFLP‐PYD participants reported on above. Data about these participants was pulled solely from the supplemental service forms they completed. Given the structure of the first version of the Supplemental Service Form, demographic information was not collected for those participants during these services. As such, demographic information was only collected and reported on for the 79 individuals who were not in this situation. Information on needs and services received, however, was collected and reported on for the total count of 124 participants, as this information was collected for all individuals who completed the Supplemental Service Form. A revised version of the supplemental service form is now in use to ensure quality of data for future analysis.

17

FIGURE 10. PREGNANT AND PARENTING ADOLESCENT SUPPLEMENTAL SERVICE PARTICIPANT DEMOGRAPHICS

*n = 79 pregnant or parenting youth who received supplemental services and had not completed a CBA for whom demographic information was available

In addition to pregnant and parenting youth, supplemental services were extended to family members or other supportive individuals connected to youth receiving supplemental services or AFLP‐PYD case management. MI‐APPP sites served 52 of these “supportive” participants from August 1, 2014, to January 31, 2016, including parents, grandparents, significant others, and other connections. These “other connections” included friends, a god‐mother, a non‐pregnant or parenting teen, a steering committee member, and a residential staff member for a group home. Information about the detailed nature of the relationships between supportive individuals and pregnant and parenting teen participants is beyond the scope of our formative evaluation; however, sites may not have a clear understanding of who is considered to be in this category of “supportive individual”. Providing local agencies with clear guidelines therein may help them better understand whose data is important to collect for federal reporting, which may improve the quality of the data for future analysis.

≤15(8)


Hispanic(16)

Female(111)

Pregnant(19)

16(17)

White(29)

Not Hispanic(104)

Male(13)

Parenting(9)

17(33)

InterRacial(4)

Pregnant & Parenting (51)

18(20)

Other (9)

19(17)

20 +(29)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

AGE

RACE

ETHNIC

ITY

GENDER

STATUS

18

FIGURE 11. SUPPORTIVE SUPPLEMENTAL SERVICE PARTICIPANT DEMOGRAPHICS

*n = 52 supportive supplemental service participants for whom demographic information was available Match with Needs Assessment Results MI‐APPP sites were instructed to tailor their supplemental services to meet community needs that were identified in the 2014 MI‐APPP Needs Assessment. An examination of the services offered and the needs identified in the statewide Needs Assessment report revealed that MI‐APPP sites have been able to offer services that impact a majority of these needs. Sites best addressed those needs regarding adult and peer connectedness promotion, youth education on sexual health and child health topics, and participant connection to services. Needs that were not yet addressed through supplemental services included prenatal and postnatal support for mothers; connection to affordable housing, transportation, and childcare; and educational supports. Interviews and participant forms suggested that many services not addressed in supplemental services were otherwise addressed through case management activities, which might have been more appropriate for particular needs. Table 1 splits the identified needs into two categories: “Addressed by Supplemental Services,” and “Not Yet Addressed by Supplemental Services.”

≤15(5)


Hispanic(8)

Female(33)

16(1)

White(15)

Not Hispanic(44)

Male(19)

17(1)

InterRacial(8)

18(1)

Other (6)

19(3)

20 +(41)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

AGE

RACE

ETHNIC

ITY

GENDER

19

TABLE 1. NEEDS IDENTIFIED IN THE 2014 MI‐APPP NEEDS ASSESSMENT REPORT

Addressed by Supplemental Services Not Yet Addressed by Supplemental ServicesSupport Navigating Service & Resource Systems Education on Pregnancy, Family Planning, and

Sexual Health Basic Family Wellness Services & Resources Peer Connectedness and Support Positive Parent‐Child Communication Strategies Maternal & Child Health Services & Programs Healthy Relationships Trusted Adult Connections Adult Mentors & Allies Motivation to Improve Familial Situation

Safe, Stable, & Affordable Housing Consistent & Reliable Transportation Affordable Quality Child Care Accommodations for Education Completion Motivation to Complete Education Educational Supports First Trimester Prenatal Care Mental Health Support Access to & Use of Postpartum Depression

Services

Objective 3: Identify barriers and supports that influenced the implementation of AFLP‐PYD and supplemental services, and determine their impact on implementation processes across the MI‐APPP sites.

Barriers & Supports to Effective Case Management Implementation

Recruitment Barriers & Supports MI‐APPP staff encountered a variety of factors that impacted their ability to recruit pregnant and parenting youth. Larger MI‐APPP sites took advantage of drawing on youth from within their agency, which was a boon to recruitment; those sites containing other programs serving youth or parents within the same organization usually had an easier time finding potential participants than those who were not similarly connected.

“Some of the other programs we’re aware of are housed in the Health Department so maybe all of their Health Department programs are feeding referrals to them, but we don’t really have that. [And] so in order to kind of continue to make sure that we’re meeting our numbers and continuing to work throughout the committee, we really have to regularly communicate with referral sources…to kind of keep it in the forefront of their minds.” –MI‐APPP Staff

Recruitment was also supported by current participants who reached out to tell other pregnant and parenting youth about MI‐APPP. When asked what their most successful recruitment method was, staff responded:

“Our best resource ended up being the teens that we got who liked the program, and then were willing to refer their friends, or circulate information about activities we were doing, or different resources in the community.” –MI‐APPP Staff

Some staff felt youth’s hesitancy to trust service programs was a major barrier to recruitment:

“I think sometimes it’s resistance, of course, with building a relationship with the case manager or divulging information…because a lot of these people have had bad experiences with systems and with different parts of human services. [And] so if I’m somebody else, ‘What is their intention? Are they really here to help me or are they here to harm me?’” –MI‐APPP Staff

20

Another critical challenge encountered during recruitment by several sites was perceived competition with existing home visiting services for pregnant and parenting youth. At several sites, MI‐APPP staff reported that there was confusion among local agencies about how to differentiate between the services offered by each agency and, accordingly, which potential participants should be referred to each type of program. The specific details of this service differentiation varied by community, with some sites reporting tension and others reporting issue resolution. Because of the unforeseen competition, MI‐APPP staff defined how their programs fit into existing service systems without duplicating service by “figuring out how we can be a niche in that system that enhances everybody’s work and benefits the teams.” In most cases, MI‐APPP staff spent some time discussing program distinction and clear goal definition and program foci with other agencies.

“We found that some of the case workers in the Health Department programs were not referring teens because they felt that it would be a duplication of services, and so it took time over the course of the first year with meeting with the Health Department supervisory staff and directors to kind of work out the fact…that we were really focusing in a different area than they were, and that we weren’t duplicating services.” –MI‐APPP Staff

Some MI‐APPP sites attempted overcoming this perceived competition by recruiting youth participants who were not connected to any human services system. This presented challenges to sites to change their recruitment approaches in order to engage potential participants.

Case Management Activities Barriers & Supports Prior experience in similar work facilitated MI‐APPP case managers’ implementation of AFLP‐PYD. Many case managers mentioned that their previous case management experience and/or experience working with youth provided a foundation for their current work. Likewise, previous knowledge of resources and people in the community was a huge implementation support. Staff members who had this advantage described “hitting the ground running,” while those who were without such connections engaged in time‐intensive bridge‐building. Site staff reported they and their coworkers varied in professionalism. Some staff struggled to keep up with the demands of case management activity documentation. Others highlighted struggles in communicating effectively with their coworkers, particularly when there was a lack of transparency due to mistrust or a lack of follow‐through in planning activities. When MI‐APPP staff did not engage in these processes, effective implementation of interventions were negatively impacted. While these cases were quite limited, supervisors described having to take action to help staff build these professional capacities or to let staff who could not meet these expectations go. In spite of the challenges of the job, most case managers were passionate, a quality which reportedly facilitated their ability to stay engaged despite difficulties: “I am so stinking happy; I literally will be with some of my clients or leave a group and be like, ‘I cannot believe I get to do this.’” ‐MI‐APPP Staff

Staff turnover affected almost every MI‐APPP site; five of the six programs reported at least one staff member leaving for another job or being terminated. Turnover affected all roles, which often created situations in which remaining staff temporarily or permanently took on new responsibilities. Several sites successfully replaced coordinators with existing case managers. Conversely, MI‐APPP sites faced various experiences when new employees replaced or filled new positions. Reactions ranged from difficulty adjusting to new expectations to gratitude for opportunities in sharing data entry and case management load as well as optimism regarding new staff member’s enthusiasm. All MI‐APPP sites were affiliated with larger host organizations, which were both a source of support and

21

frustration for case managers. Several sites mentioned that their host agency’s other youth programs and case management activities supported MI‐APPP implementation by aiding in recruitment and in supplemental service provision. However, barriers to activities included MI‐APPP staff’s limited access to supporting resources such as vehicles, social media communication restrictions, and computer security which impeded LodeStar installation and use. Community Connections Barriers & Supports MI‐APPP sites worked with various organizations. Some sites had success connecting with schools; a few programs even had permission for regular meetings in school campuses. These partnerships fostered efficient recruitment and assisted with participant retention, as connections at school were easier for some pregnant and parenting youth. Other schools did not welcome connections MI‐APPP agencies attempted to create. For example, one site experienced difficulty in enrolling participants in the local school district and said the schools appeared reluctant to take pregnant and parenting adolescents. Other challenges included a lack of necessary services in some MI‐APPP service areas and partners failing to follow through on planned collaborations. In contrast, remaining MI‐APPP sites provided resources more readily and forged partnerships easily. Overall, MI‐APPP sites aimed to expand and deepen their relationships with community partners, which they determined were a critical aspect of MI‐APPP implementation. Retention/Completion Barriers & Supports MI‐APPP sites reported various barriers and supports to participant retention. MI‐APPP staff were puzzled as to why participants became disengaged. Many site staff perceived this as a result of pregnant and parenting youth being interested in meeting their immediate needs rather than working toward future goals.

“So the example I have is when one of the teen dads gets a job; I don’t hear from ‘em for a while. It’s not because they’re doing bad or they don’t want to be in the program….They feel like, ‘Well, what’s next?’ They don’t understand; well, what’s next is you need to stay in the program because we’re going to help you with other stuff, and that’s ultimately going to help you become a better father.” –MI‐APPP Staff

Other staff talked about youth participants’ inconsistent follow‐through and lack of responsibility as a major hurdle to retention. Pregnant and parenting youth also described challenges with time management. When asked what prevented them from meeting their case managers in the past, participants often said they forgot about scheduled meetings, forgot to inform bosses of upcoming meetings, or felt too busy caring for their child to make it to their AFLP‐PYD appointments. Difficult life circumstances also contributed to inconsistent participation. Case managers frequently specified numerous phone numbers and housing changes as hurdles to staying in touch.

“We have a lot of clients who are, in all reality, poverty‐stricken. They move a lot. They have phones that get cut off, or they’re on a text line, or they’re on their brother’s line, or they’re on their grandma’s line. So as a case manager, that’s one of the biggest difficulties that I have engaging people is losing their contact....” –MI‐APPP Staff

Youth participants also described difficult life choices as barriers to connecting with their case managers. Scheduling conflicts, such as work schedules or doctor appointments conflicting with AFLP‐PYD appointments, were the most commonly mentioned reasons for missing meetings with case managers. Additionally, children’s, youth participants’, or case managers’ illnesses, youth’s depression or emotional issues, and transportation resulted in missed meetings.

22

Efforts to address these concerns and minimize missed meetings took many forms. Both MI‐APPP staff and pregnant and parenting youth mentioned transportation provision facilitated face‐to‐face meetings. Some sites even leveraged transportation to other non‐MI‐APPP activities as an incentive to obtain face‐time with youth participants.

“Some of our incentives have changed from like handing out a gift card to saying like, ‘Hey, I’ll pick you up from school and we’ll meet,’ or, ‘Hey, I’ll take you to work and we’ll meet at your job for a few minutes before you go to work.’” –MI‐APPP Staff

Other factors participants identified as facilitating these in‐person meetings included adolescents maintaining more stable work schedules and case managers’ willingness in their flexibility in scheduling meetings with youth. One case manager corroborated this finding and mentioned he had the most success when he allowed adolescents to choose their preferred schedule (i.e., schedule weeks in advance versus schedule week‐by‐week). A few staff members also encouraged retention by emphasizing youth’s ownership of experiences with MI‐APPP and by contacting pregnant and parenting adolescents regularly via texting and social media.

Several MI‐APPP sites were able to have a regular presence in participants’ schools. This was both a support and a barrier to retention. During the school year, MI‐APPP staff used their presence at school to maintain regular youth contact. Those who had space in schools benefited from youth participants dropping by; although less enthusiastic participants were less likely to drop by. Conversely, keeping participants engaged when school closed during the summer posed a great challenge.

“[…] School is out for the summer; they stopped communicating because they see us as part of the school system, and so we’re not—they don’t really need us. And then when school starts again, they pop back up again.” –MI‐APPP Staff

Finally, existing supports in youth participants’ lives presented either barriers or supports to retention. In some cases, participants who already received support from their family or other human service programs were difficult to retain as they already established sufficient systems of support. In other cases, however, parental support of youth’s participation played a critical role in keeping youth participants engaged; parents reinforced messages and encouraged responsibility. Likewise, pregnant and parenting youth’s participation was hindered if their families did not support their goals. One MI‐APPP staff member reported that long‐term engagement was challenging because participants heard from family members or others that they should capitalize on resources they can obtain initially and move on.

State Level Barriers & Supports Overall, MI‐APPP sites seemed satisfied with the support received from MDHHS and other state partners. A number of staff members noted it was relatively easy to reach out to MDHHS when technical assistance with reporting and documentation were needed.

“I think that we have a great amount of support from the state. The fact that we are able to just kind of call and kind of get that support from whomever….I have never had a problem getting in contact with anybody, and I have never had a problem with getting clarification on what the expectations are.” –MI‐APPP Staff

Additionally, staff expressed the biannual Learning Collaboratives provided useful information and opportunities to connect with other sites, all of which reportedly contributed to more effective program planning: “The Learning Collaboratives are wonderful. That’s huge. What we take away from those is so

23

beneficial and really helps us stay on track with our programming and enhance what we’re doing.” (MI‐APPP Staff)

However, sites reported challenges in learning and operating LodeStar and Qualtrics data entry and management systems. This was more difficult when sites did not designate a data entry staff member for assistance. Another challenge included understanding details around program setup. Some MI‐APPP sites struggled to manage competing expectations of their agency and the state during program start‐up. This process was complicated by the flexibility of what could be offered for supplemental services. Some sites expressed initial discomfort with not being provided a specific intervention or curriculum for supplemental services, which was viewed as unconventional in that it required use of a needs assessment in order to tailor the program to their community needs. Likewise, there were frustrations with communication delays regarding trainings and rollout of intervention materials during the setup phase.

Finally, MI‐APPP staff reported some grant requirements had felt overwhelming, such as the case management documentation, the caseload minimums, and the supplemental services requirement in addition to case management. However, now that MI‐APPP sites are up and running, the structure and expectations were largely embraced by the sites.

Barriers & Supports to Effective Supplemental Service Implementation

The barriers and supports for supplemental service implementation identified from the interviews draw on similar themes as those identified for case management. Supplemental services were most successful when sites eliminated potential barriers by providing transportation and childcare for pregnant and parenting youth. Providing additional incentives for youth participants, such as gift cards, food, and recognition of a friend also attending, boosted attendance. Event success was supported by the speakers’ ability to keep youth engaged; if the participants were drawn in by the presenter, the message was more effective. One MI‐APPP site received additional funding from a foundation for their supplemental services which allowed them to guarantee the regular services of larger‐scale events.

Barriers to supplemental service implementation included capacity constraints, such as the cost of providing transportation for adolescents, and not having a large space at their agency for all participants. Site staff were also limited on the activity types they could offer based on their agency’s liability policies. For example, one MI‐APPP site’s staff could not take participants to the local swimming pool because transportation could be provided due to liability concerns. Determining when youth could attend often proved difficult and required much trial‐and‐error to determine what works best. Multi‐session events proved particularly difficult for youth to stay engaged; participants would often miss one or more of the presentations in the series. Sites also reported challenges with requesting participants to fill out the Supplemental Services Form, particularly at large scale events or when they had previously filled it out multiple times. Overall, the sites reported mixed feelings as to whether their supplemental services were being implemented as planned. MI‐APPP staff reported learning more about what their youth want and need, and using this information to tailor their supplemental services. MI‐APPP sites also engaged staff members’ creativity, resulting in innovative activities that drew in more new participants and produced more quality programming. Other changes included emphasizing quality over quantity; two sites planned to have one supplemental service activity per month but have since cut back to one per quarter due to strain on staff time and budgetary constraints.

24

Objective 4: Assess the extent to which AFLP‐PYD and supplemental services are satisfactory to AFLP‐PYD and supplemental service participants and MI‐APPP agency staff across the MI‐APPP sites.

Satisfaction with AFLP‐PYD

Participants AFLP‐PYD participants were largely satisfied with their work with case managers and expressed great pride in what they had accomplished together. Case managers’ supportive personality traits were often mentioned as one of the best things about being part of AFLP‐PYD. These traits included being easy to talk to, being good listeners, remaining non‐judgmental, being trustworthy, being dependable, and maintaining an uplifting attitude. Other positive feelings associated with case managers were their ability in connecting participants with resources, keeping in touch, and getting along with the participants’ children.

“(Case manager) has helped me a lot. I probably wouldn't have gotten half of the things done that I have without her. She knows all the resources. I didn't know there were that many resources here in Michigan.” ‐ MI‐APPP Participant

The desire to see their case manager more often was the most common concern mentioned. Other concerns included feeling a slow progression in their youth‐case manager relationship, wanting to have one‐on‐one meetings with their case managers, and wanting to be connected to more community resources and more frequent local activities. A small number of participants reported feeling too crowded by their case manager’s level of interaction and wanted fewer meetings with their case manager.

Staff Staff had varying opinions about the effectiveness and appropriateness of the AFLP‐PYD intervention. Some staff appreciated the structure that it provided to their case management relationships. “AFLP…gives you more structure but at the same time flexibility. It gives you a roadmap to follow and you don’t have to go in and try and wing it.” ‐MI‐APPP Staff

Most case managers reported that the strength‐based and participant‐led elements of AFLP‐PYD were the core strengths of the approach, particularly when these concepts were applied to goal setting. They enjoyed the opportunity to help youth participants build skills to become independent rather than simply giving them flyers.

“[Participants] really appreciate you talking to them as a parent and as a person, and “What is your goal?” And, “I’m not going to tell you what to do. I’m here to help you with what you want to do.” So I think that they also appreciate the format of the program.” – MI‐APPP Staff

Staff also perceived that participants appreciated th e ongoing support aspect of the intervention.

“I think the case management works really well. I think they like—a lot of them don’t have supportive adults…So just being there, being somebody who is…consistent with them I think that works really well.” – MI‐APPP Staff

Staff recognized the potentially critical gaps between the program’s required and recommended communication (e.g. face to face, home visit) and participants’ preference for means of communication (texting, using free apps for texting). Simply put, MI‐APPP staff are digital immigrants, and participants are digital natives. In addition to impacting preferred ways of staying in touch and showing support, this

25

also presented challenges with the materials themselves not being digitized. This felt outdated to some staff and youth. Although few MI‐APPP staff mentioned specific intervention materials’ elements, those that did highlighted the “Me and My Life Activities” and the “My Life Plan.”

“I think ‘Me and My Life Activities’ and ‘My Life Plan’ is very helpful to them, as well as it’s helped guided our home visits along as well….Those two pieces have been very helpful with goal‐setting and envisioning their life.” – MI‐APPP Staff

However, one MI‐APPP staff member felt these intervention materials were not as appropriate for older youth participants.

“I don’t see it being a good fit for 20 and 21 year olds….The experience has been the older they are, and they have kind of moved beyond what’s being covered in ‘My Life Plan’ and ‘My Life Activities.’ They are more difficult to engage because they’re beyond that.” – MI‐APPP Staff

Several staff were less satisfied with the amount of “paperwork” the intervention required. They mentioned that the intake process and the associated forms (i.e. CBA and Auxiliary Assessment Packet) were too lengthy and could interfere with participants’ interest in continuing to receive services. A MI‐APPP staff member stated, “They just are completely thrown off by all of the documentation.” Case managers described using various techniques to make the process less burdensome or suspicious for participants:

“I do my very, very, very best to retain the information as we’re talking so I can come back to the office and fill things out. Because I feel like if I sit down with a clipboard on my first or second visit, it’s likely to scare any client away, but especially young parents who think that you are monitoring them.” – MI‐APPP Staff

They also expressed frustration with the perceived repetition in participant form questions, saying that this added additional time onto what is already an intensive case management process.

“More frustrating is when I have to ask some of the questions over again. Like when you’re doing the [Auxiliary Assessment Packet] and the CBA, some of the questions you’re going to have to ask over again and they already answered.” –MI‐APPP Staff

Despite their frustrations, MI‐APPP case managers reported they saw themselves and the MI‐APPP program as important agents in participants’ lives.

“I was the case manager for a little over a year, and it’s just been really exciting to watch the program grow, and just seeing some of the kids reach their goals, and do the things that they set out to do. And it’s a really exciting program, it’s a really good program.” –MI‐APPP Staff

Satisfaction with Supplemental Services

Participants Participants’ satisfaction with supplemental services varied across MI‐APPP sites and activity type. While they expressed enjoying the activities MI‐APPP held, there were mixed opinions regarding the specific aspects of the events. Some participants reported they enjoyed group discussions or presentation‐based activities because of the opportunities to interact with other young parents, to learn from more experienced parents, and to discuss important information relevant to young parents. The few who disliked group discussions were dissatisfied with the topic redundancy for returning youth participants

26

and the uncomfortable topics, such as safe sex workshops. Attending and occasionally speaking at workshops, conferences, and classes afforded youth to visit new places, learn new skills, gain new information, and find new outlets of self‐expression. Overall, participants believed the supplemental services were beneficial and stated they wanted to see attendance improve.

Staff MI‐APPP staff used supplemental services as a platform for developing knowledge and skills, supporting peer‐to‐peer and parent‐child interaction, and offering connections to resources. The flexibility MI‐APPP afforded around supplemental services meant staff had relative freedom to create responsive and specific programming to youth’s and their support’s needs and their local context. In doing so, youth participants could capitalize on local resources and staff connections. However, along with the flexibility came some challenges, particularly around MI‐APPP sites assessing their likelihood of success with their supplemental services. Some MI‐APPP staff questioned if they were focusing on the appropriate concerns and the most effective strategies to target the concerns in question. A few sites were interested in additional guidance around “best practices” for supplemental services and opportunities to learn from other MI‐APPP sites about what worked for them.

Site staff reported the well‐received supplemental services were those less formal opportunities focusing on “fun” activities that provided participants opportunities for socialization with their children and other pregnant and parenting youth.

“Some of those activities are fun because the teens don’t get to do anything, and they talk about that: ‘There’s nothing for us to do. We can’t go out with our friends. We can’t do these things.’ So having those moments in time where they can kind of relax and do something fun is a priority for them.” –MI‐APPP Staff

MI‐APPP site staff reported that youth participants also enjoyed those supplemental services focusing on learning new information and skills related to parenting. MI‐APPP staff also believed that participants appreciated the flexibility supplemental services afforded.

“I think they like…the freedom of knowing…they don’t have to come to a certain amount. If it sounds interesting to them, they’re going to show up, and I think they like the fact that they can bring positive support persons. That makes it a little bit more comfortable for them, and they are more likely to come.” –MI‐APPP Staff

The services site staff believed were less well‐received were those that covered information youth were already were familiar with. For example, a MI‐APPP staff member sensed a series based on healthy sexual behavior was uninteresting for adolescents; many of them had previously heard these messages. Interestingly, MI‐APPP staff members from one site were split on their opinions about pregnant and parenting youth’s reaction to a financial literacy class; one staff member thought youth were uninterested, while another thought they were enthralled.

Objective 5: Assess the extent to which AFLP‐PYD participants’ and supplemental service participants’ connection to needed supports and services changed over time across MI‐APPP sites. Participants presented with various needs upon beginning case management. The most common and least common needs identified at intake by AFLP‐PYD participants are presented in Figures 13 and 14 below.

27

FIGURE 13. MOST COMMON AFLP‐PYD PARTICIPANT NEEDS FROM MI‐APPP

*n = 106 AFLP‐PYD participants with a completed CBA and Auxiliary Assessment Packet

FIGURE 14. LEAST COMMON AFLP‐PYD PARTICIPANT NEEDS FROM MI‐APPP

*n = 106 AFLP‐PYD participants with a completed CBA and Auxiliary Assessment Packet

Pregnant and parenting supplemental service participants also presented with various needs. The most common and least common needs identified by these youth supplemental service participants are presented in Figures 15 and 16 below.

58%53% 52% 51%

45%

Employment Parenting Skills Healthy RelationshipSkills

Transportation Housing

15%13%

12%11%

8%

Partner Violence Primary Care PostnatalCounseling

Prenatal Care Pediatric Care

28

FIGURE 15. MOST COMMON PREGNANT AND PARENTING ADOLESCENT SUPPLEMENTAL SERVICE PARTICIPANT NEEDS FROM MI‐APPP

*n = 124 pregnant and parenting supplemental service‐only participants

FIGURE 16. LEAST COMMON PREGNANT AND PARENTING ADOLESCENT

SUPPLEMENTAL SERVICE PARTICIPANT NEEDS FROM MI‐APPP

*n = 124 pregnant and parenting supplemental service‐only participants

Finally, supportive individuals who received supplemental services from sites presented with various needs upon participating in MI‐APPP. The most common and least common needs identified by supportive supplemental service participants are presented in Figures 17 and 18 below.

33% 31%28%

26% 24%

Transportation Parenting Skills Employment Childcare Healthy RelationshipInfo

5% 5% 5%

2% 2%

Mental Health Prenatal Care Pediatric Care Intimate PartnerViolence Services

Postnatal Care

29

FIGURE 17. MOST COMMON SUPPORTIVE SUPPLEMENTAL SERVICE PARTICIPANT NEEDS FROM MI‐APPP

*n = 52 supportive supplemental service participants

FIGURE 18. LEAST COMMON SUPPORTIVE SUPPLEMENTAL

SERVICE PARTICIPANT NEEDS FROM MI‐APPP

*n = 52 supportive supplemental service participants

The Michigan AFLP‐PYD Case Management Protocol instructs staff to provide services and referrals to case management and supplemental service participants in order to meet their needs. Tables 2, 3 and 4 provide data on the most common services and referrals received between August 1, 2014, and January 31, 2016 by AFLP‐PYD participants, Supplemental Service‐only pregnant and parenting teen participants, and Support Supplemental Service participants. These numbers reflect the overall experiences of MI‐APPP participants who received services during this time‐frame and excludes those participants or time‐points for which data is unavailable.

35%

29%25% 25% 25%

Transportation Healthy RelationshipServices

Housing Food Assistance Employment

8%

6% 6% 6% 6%

Home Visiting Primary Care Postnatal Care Pediatric Care Family Planning

30

TABLE 2. Percentage of AFLP‐PYD Participants with Needs who Received Related MI‐APPP Services and/or Referrals

Need MI‐APPP Services Referrals

Child Care Services 22% 60%

Concrete Support Services 82% 93%

Educational Support Services 92% 89%

Healthcare Services 98% 98%

Healthy Relationship Services 80% 22%

Home Visiting Services 86% 20%

Intimate Partner Violence Services 64% 28%

Parenting Skills Services 76% 14%

Transportation Services 55% 33%

Vocational Services 50% 62%

TABLE 3. Percentage of Pregnant and Parenting AdolescentSupplemental Service Participants with Needs who Received Related MI‐APPP Services and/or Referrals












31

TABLE 4. Percentage of Supportive Supplemental Service Participants with Needs who Received Related MI‐APPP Services and/or Referrals












In addition to increased support and services connection, initial anecdotal evidence suggests that participants may be experiencing positive outcomes while in MI‐APPP. Site staff reported they witnessed youth becoming more assertive regarding their future because of the program. Case managers reported having supported participants through milestones such as finding a job, securing stable housing, finishing high school or getting their GEDs, and signing up for college. For example, 21 participants graduated from high school as a result of the program. While this does not necessarily indicate this milestone was a direct result of program participation, case managers may have played a role in helping participants take steps toward this goal. MI‐APPP staff also described finding ways to increase supplemental service participants’ knowledge, skills, and social and emotional connection to peers, family members, and communities through their participation.

“To my knowledge, no other program in the community allows or encourages support persons to attend events with pregnant and parenting teens, and our view is it’s extremely important to include those folks because it does take more than just one person to raise a child.” –MI‐APPP Staff

Youth participants also reported seeing positive changes in both their lives as well as their families’ well‐being as a result of their work with their case managers and their involvement in supplemental services. These positive changes included forming relationships with other young parents and case managers, restoring relationships with family, gaining self‐confidence and maturity, reaching goals, achieving personal growth, and feeling more stable and independent. The valuable knowledge and skills gained covered topics such as parenting, communication, financial literacy, life management, safe sex, and safe alcohol use.

“I know a lot more now when it comes to parenting and involvement with my child...They gave me information about stuff I wasn't even thinking about at the time. I was scared‐ I got a baby, I'm young....They just helped with my confidence in being a mother. At first I didn't think I was a good parent, and they would tell me ‘You doing good.’ So I thought ok I'm doing something right.” – MI‐APPP Participant

32

Pregnant and parenting youth expressed great appreciation for their case managers’ hard work and support that led to their personal growth in MI‐APPP.

“When you have a support system, it's hard for you to give up….It's really helpful to have her on my team, I guess. I am not a quitter anyways, but sometimes people need an extra motivation.” ‐MI‐APPP Participant

Discussion Summary of Findings

During the first eighteen months of implementation from August 1, 2014, to January 31, 2016, MI‐APPP provided AFLP‐PYD case management services and a variety of supplemental services to pregnant and parenting youth and their families in six high‐need communities in Michigan. This formative evaluation report aimed to describe the status of implementation of these services and to identify influences on progress and effectiveness in order to promote future program success.

The results of the evaluation suggest case managers have taken the positive youth development ethos of AFLP‐PYD to heart and they appreciate the balance of structure and flexibility afforded by the intervention. However, the case managers regularly adapted the intervention to fit the needs and preferences of the participants, citing difficulty with retention as one reason to prioritize goal‐setting and rapport‐building over assessment and intervention materials. Contact log data revealed that MI‐APPP sites were, on average, exceeding the requirement for number of home visits per quarter, but struggling to meet the requirement for number of face‐to‐face visits per month on average. The wide range of average visits per month suggests case managers may prioritize participants with more needs for this type of contact or some participants may be more successful in making meetings than others. In light of the high incidence of youth exiting because they were unable to be reached, this finding is likely connected to factors that make scheduling and keeping meetings with adolescents challenging, such as shifting schedules, unstable housing, and varying phone numbers.

Supplemental services based on participant input offered activities that addressed the majority of the needs identified through the 2014 MI‐APPP Needs Assessment. While some supportive individuals were served, the majority of attendees were AFLP‐PYD case management participants. Activities were most successful when they were interactive, youth‐focused, and highly incentivized and least successful when participants viewed the information as redundant or the speaker as boring.

Several sites mentioned being part of a host agency connected to other teen‐serving programs made their job easier, however agency policies also occasionally complicated programming. Staff turnover affected almost every MI‐APPP site and all roles, creating disruptions as new staff learned the intervention and developed relationships with participants and community partners. Support from MDHHS and state partners was highly regarded by MI‐APPP staff, as were professional development opportunities such as the Learning Collaboratives. The initial learning curves around program setup and data collection and entry were difficult, but sites have adjusted.

Sites have formed a variety of community connections to support participant recruitment, referral, and supplemental service offerings; however, perceived competition with home visiting programs made integration into the existing service system challenging for some sites. Partnerships with schools (when

33

they could be formed) were largely beneficial for recruitment and retention but posed challenges in the summer.

Overall, staff and AFLP‐PYD participants had very positive perceptions of MI‐APPP and believed it was positively impacting the lives of pregnant and parenting youth in Michigan. They expressed mutual appreciation; participants were grateful for what the program had done for them, and staff were often inspired by the youths’ grit.

“Teen parents are the most resourceful, resilient people in the world. [And] some folks would think resilience is a bad thing, but they’ve got to flip that. They look at teen parents as such a waste, a mistake. No! I look at them as resourceful, resilient, beautiful, knowledgeable.” – MI‐APPP Staff

Limitations

This project relies on a collaborative approach, and as such, the success of the evaluation depended on the cooperative efforts and participation of the MDHHS, MI‐APPP sites, and additional stakeholders. MI‐APPP sites took a lead role in collecting and entering certain participant form data. The ability to produce meaningful results therein rests on the quality of the data collected in the field. Furthermore, MI‐APPP sites also took a lead role in recruiting AFLP‐PYD participants to speak with our interviewers. As such, the participants interviewed may have had a uniquely positive relationship with their case manager, a fact which may bias the results. Finally, no conclusions about outcomes of the program can be made from the data included in this report.

Recommendations

Recruitment For sites that are struggling with recruitment, provide suggestions on how to connect with hard‐to‐

reach adolescents to engage them in the program and avoid competition with other programs. For example, staff could consider offering incentives to existing participants to refer their pregnant and parenting friends or classmates or building a close partnership with an alternative school to offer services to their youth.

Recommend agencies that are not closely connected to other young parent‐serving programs prioritize building relationships with referral sources. Yearly presentations about service offerings, monthly check‐in calls, and occasional drop‐bys at their offices may facilitate these connections. This is particularly important given staff turnover at both MI‐APPP sites and at other organizations.

Case management Reinforce the need for case managers to meet the requirement of two face‐to‐face visits each

month with participants to improve program fidelity and offer suggestions for supporting participants in scheduling and keeping face‐to‐face meetings.

Provide additional guidance on how case managers can use the intervention materials, such as “Me and My Life Activities” and “My Life Plan,” with participants earlier on in their case management relationship. For example, demonstrate how these activities can be used to build rapport and trust or how to incorporate these materials into discussions about meeting youth’s immediate needs.

34

Consider digitizing AFLP‐PYD materials, such as forms and intervention materials, to make them more accessible and interesting to tech‐savvy participants.

Support case managers in working with adolescents in the context of their family system. Consider providing a professional development opportunity where case managers can learn about balancing family opinions with those of the participant.

Staff Further explore why staff turnover at the sites is high and what can be done to retain staff. Other

case management programs may have suggestions for how they have been able to keep case managers engaged despite the high burnout potential of the job.

Build on the current training opportunities to create a comprehensive training system with the dual goals of 1) orienting new staff and 2) helping existing staff grow and build their professional capacity. Professional development may positively contribute to staff retention by improving participants’ confidence and competence in their work.

Provide guidance to sites hiring new staff about factors that heavily contribute to case manager success. For example, sites should attempt to hire candidates who are connected in the community and to other services, are comfortable with the population, and are experienced working with adolescents in order to get the best results.

Participant Retention Provide practice suggestions to support sites that are having trouble retaining participants. For

those that are losing youth once they move past the stage of meeting immediate needs, this may include techniques such as emphasizing the intervention tools earlier in the process to get participants thinking about long‐term concerns sooner.

Fostering an adolescent’s identity as part of a community of support for pregnant and parenting youth may also help them remain engaged. Activities that support this goal include participation in the steering or advisory committee, inclusion on the program website or in a local newspaper article about MI‐APPP, or contribution to supplemental service activity planning.

For sites struggling to maintain connections to youth due to unstable housing or changing phone numbers, consider getting contact information for other people the participant is always in touch with. These people can help track down a youth even if their current number changes. Contact through internet‐based social media apps may also be more effective than phone texting if the youth has regular access to the internet. This type of messaging does not require cell minutes, and can be done using free Wi‐Fi connections at schools, libraries, malls, or restaurants.

Present ideas for ways to keep youth engaged during summer vacation from school. Case managers should begin emphasizing the need for their continued involvement over the summer at least a month before school ends. They might consider offering participants transportation, such as a ride to regular doctors’ appointments or to work once a week, in order to parlay this contact into a face‐to‐face visit to discuss goal progress. Sites could also consider offering supplemental services that involve fun summer activities for youth and their children, such as outdoor activities, field trips to go fruit picking, or visits to a hands‐on science museum.

35

Community Connections Continue building bridges with home visiting programs at the state level, and further develop

strategies to help sites navigate perceived competition with home visiting programs at the local level. Encourage sites to engage in in‐person conversations or a group discussion with the leadership from all home visiting programs in their area to determine what sets each program apart and how they can best work together to ensure a fit between participant needs and available programs. Consider formalizing these referral decisions through a community‐wide service continuum and/or referral hub.

Supplemental Services Continue to provide new opportunities at the state and local levels for participants to speak out

about their needs and their experiences. Opportunities like attending the teen‐focused Learning Collaboratives, participating in steering or advisory committees, and presenting on panels at conferences build confidence and foster leadership among participants.

Provide guidance on best practices for working with adolescents, such as how to make events attractive for this age level. For example, sites should consider using teen‐focused marketing via social media or word‐of‐mouth to get the word out about larger events. When offering an educational event, make sure the topic is interesting to youth (not just “good for them”) and the speaker is engaging; hands‐on activities are a huge plus. Youth also appreciate having good food, having opportunities to be social with other teen parents and to interact with their children, and having something to take home with them (e.g. door prizes, crafts, gift cards). Asking for feedback following activities can help sites better understand what is and is not appealing to their youth.

Suggest evidence‐informed practices for teaching participants about common issues that sites often wish to address with supplemental services (e.g. parenting, healthy relationships, etc.). Consider offering a professional development opportunity where staff can learn about how to access evidence‐informed programming ideas and curricula online.

Encourage sharing among sites about what has or has not worked with their supplemental service offerings, either informally or through formal cooperative learning communities. This could take a variety of formats, including a MI‐APPP staff list serve or Facebook group, small group sharing at Learning Collaboratives, or quarterly calls devoted strictly to discussing challenges.

Build on momentum related to serving young dads and encourage sites to offer fatherhood‐specific programming. One community has had great success with hosting monthly gatherings for fathers of all ages where teen dads can connect with role models and learn about parenting. Other communities should consider how they could create opportunities for dads to connect with each other.

State Support / Data Consider holding a data refresher training with MPHI, LodeStar, and MDHHS to clarify guidance

around data issues.

Develop a process for holding sites accountable for completing contact log data. If the data quality improves, it will provide a useful source of information about program fidelity moving forward.

36

Work with case managers to streamline the required participant forms to alleviate the burden of this data collection process. For example, staff have expressed interest in participating in a workgroup to revisit the CBA to determine questions that could be eliminated to make the process shorter.

Develop and maintain a record of practice recommendations. Currently sites receive ongoing guidance as needed on particular case management or supplemental service issues from a variety of sources. Keeping a centralized, up‐to‐date record of these recommendations would ensure that all state partners are aware of guidance given and are able to vet its impact on the data.

Conclusions

MI‐APPP has experienced a number of successes during the first eighteen months of implementation. Both staff and participants are enthusiastic about the program and are largely satisfied with its initial influence on teens’ lives. Staff have embraced the Positive Youth Development core of the intervention and built community partnerships to support pregnant and parenting youth’s success. Sites should take advantage of opportunities to learn from each other’s successes in partnering with local schools, hosting appealing supplemental services, and engaging young fathers. Other opportunities for improvement include finding ways to better maintain contact with less‐connected participants to improve face‐to‐face visit averages and participant retention. The program would likewise benefit from additional guidance around supplemental service offerings and a comprehensive approach to retaining site staff. The implementation challenges encountered by MI‐APPP are common among new programs, and addressing these challenges will increase MI‐APPP’s potential for promoting the health and well‐being of pregnant and parenting adolescents.

FY 2017 MDHHS Deliverable Submission Form · The formative evaluation described the status of...

Documents

Transcript of FY 2017 MDHHS Deliverable Submission Form · The formative evaluation described the status of...