Evaluation of Michigan Child Care Evaluation of Michigan Child Care Expulsion Prevention Program Expulsion Prevention Program
(CCEP), (CCEP), 2007-20102007-2010
Michigan State UniversityMichigan State University
October 27, 2010October 27, 2010
Rosalind H. KirkRosalind H. Kirkaa
John S. CarlsonJohn S. Carlsonaa
Laurie A. Van EgerenLaurie A. Van Egerenaa
Holly Brophy-HerbHolly Brophy-Herbaa
Stacy L. BenderStacy L. Benderaa
Betty TablemanBetty Tablemanaa
Mary A. MackrainMary A. Mackrainbb
Deb MarciniakDeb Marciniakcc
Sheri FalvaySheri Falvaydd
aaMichigan State UniversityMichigan State UniversitybbMichigan Child Care Enhancement ProgramMichigan Child Care Enhancement ProgramccMichigan Public Health InstituteMichigan Public Health InstituteddMichigan Department of Community HealthMichigan Department of Community Health
AgendaAgenda
CCEP’s research questions (child, CCEP’s research questions (child, provider, program, family, CCEP provider, program, family, CCEP process & fidelity)process & fidelity)
Evaluation approachEvaluation approach Evaluation strategiesEvaluation strategies Strategies – strengths and challengesStrategies – strengths and challenges Use of CCEP evaluation resultsUse of CCEP evaluation results
Child Care Expulsion Prevention Child Care Expulsion Prevention Program (CCEP), MichiganProgram (CCEP), Michigan
Began in late ‘90sBegan in late ‘90s Initiated by MDCH, supported with funding from MDHSInitiated by MDCH, supported with funding from MDHS Plans for state-wide coveragePlans for state-wide coverage At time of evaluation, 16 programs covering 31 out of At time of evaluation, 16 programs covering 31 out of
83 counties83 counties Approx. 500 - 600 children per year.Approx. 500 - 600 children per year. Programmatic consultation also provided.Programmatic consultation also provided. After T1 data collection ended in 2009, focus of CCEP After T1 data collection ended in 2009, focus of CCEP
changed to 0-3 yrs.changed to 0-3 yrs. Along with many other MI programs funding ended on Along with many other MI programs funding ended on
30 September 201030 September 2010
Research questionsResearch questions Child Outcomes (John)Child Outcomes (John)
1.1. Does the severity of children’s challenging Does the severity of children’s challenging behavior decrease from the onset of CCEP behavior decrease from the onset of CCEP services to the conclusion of services?services to the conclusion of services?
2.2. Does children’s social and emotional Does children’s social and emotional health increase from the onset of CCEP health increase from the onset of CCEP services to the conclusion of services?services to the conclusion of services?
3.3. Does the impact of services on children’s Does the impact of services on children’s behavior last past services?behavior last past services?
4.4. Do children receiving CCEP services Do children receiving CCEP services successfully stay in child care vs. being successfully stay in child care vs. being expelled?expelled?
Research questionsResearch questions
Parent outcomes (Holly)Parent outcomes (Holly)
5. Do subjective feelings of parental 5. Do subjective feelings of parental competence in dealing with their child’s competence in dealing with their child’s challenging behavior increase as a result challenging behavior increase as a result of CCEP services?of CCEP services?
6. Are families able to consistently attend 6. Are families able to consistently attend work or school?work or school?
Research questionsResearch questions
Child Care Provider Child Care Provider outcomes(Laurie)outcomes(Laurie)
7. Is the childcare provider better able to 7. Is the childcare provider better able to recognize early warning signs of social and recognize early warning signs of social and emotional challenges in infants, toddlers, emotional challenges in infants, toddlers, and preschoolers?and preschoolers?
8. Is the child care provider better able to 8. Is the child care provider better able to manage challenging behavior in the child manage challenging behavior in the child care setting, with all children?care setting, with all children?
Research questionsResearch questions
Child Care Program outcome (Ros)Child Care Program outcome (Ros)
9. Has the social and emotional quality of 9. Has the social and emotional quality of the child care setting receiving CCEP the child care setting receiving CCEP services improved?services improved?
Research questionsResearch questions
Program Fidelity (Laurie)Program Fidelity (Laurie)
10. What is the fidelity of the child and 10. What is the fidelity of the child and family consultation process among CCEP family consultation process among CCEP programs? programs?
11. What is the fidelity of the programmatic 11. What is the fidelity of the programmatic consultation process among CCEP consultation process among CCEP programs? programs?
Evaluation approachEvaluation approach
Collaborative and consultativeCollaborative and consultative
Built upon existing systemsBuilt upon existing systems
Mixed method – mainly quantitative, Mixed method – mainly quantitative, some qualitativesome qualitative
Four overall strategiesFour overall strategies
1.1. Cross-sectional Cross-sectional (formative)(formative): : Consultant Consultant surveysurvey
2.2. Longitudinal study Longitudinal study (mainly summative)(mainly summative): : Pre-post data + 6 month follow-up from Pre-post data + 6 month follow-up from intervention group using measures of child, intervention group using measures of child, parent, provider outcomesparent, provider outcomes
3.3. Quasi-experimental comparison Quasi-experimental comparison studystudy (summative)(summative): : Comparison group with Comparison group with pre-post data matching longitudinal pre-post data matching longitudinal intervention groupintervention group
4.4. Case studies Case studies (formative): Perceptions of (formative): Perceptions of experiences with CCEP based on interviews.experiences with CCEP based on interviews.
1. Cross-sectional strategy: 1. Cross-sectional strategy: strengthsstrengths
On-line survey of consultants on participation in CCEP On-line survey of consultants on participation in CCEP and delivery of service, including compliance with six and delivery of service, including compliance with six CCEP cornerstonesCCEP cornerstones
‘‘Snap-shot’ of program and processes based on Snap-shot’ of program and processes based on perceptions of consultants and administratorsperceptions of consultants and administrators
Electronic surveys are accessible, flexible, user Electronic surveys are accessible, flexible, user friendly and can be quick to analyzefriendly and can be quick to analyze
Very collaborative with CCEP in design, data Very collaborative with CCEP in design, data collection, interpretationcollection, interpretation
Provided a wealth of information for program Provided a wealth of information for program improvement, etc.improvement, etc.
Collaboration provided opportunity to share expertise Collaboration provided opportunity to share expertise & help develop CCEP internal monitoring systems& help develop CCEP internal monitoring systems
Cross-sectional strategy: Cross-sectional strategy: potential challengespotential challenges
Potential factors affecting response rate: Potential factors affecting response rate: organizational change, personal views organizational change, personal views about evaluation, stress levels, vacations, about evaluation, stress levels, vacations, sickness, staff turnover, workload, length sickness, staff turnover, workload, length of survey etc, of survey etc,
Anonymity can mean that survey data Anonymity can mean that survey data more likely to be accurate but non-more likely to be accurate but non-respondents cannot be targeted to respondents cannot be targeted to increase response rate. increase response rate.
Cross-sectional strategy: Cross-sectional strategy: survey of consultants, 2008 (N =29)survey of consultants, 2008 (N =29)
Gender
Female 100%
Age
Mean (yrs) 43
Range 27-60
Race/ethnicity
White 76%
Af-Amer 21%
Asian 3%
Endorsement MI AIMH
Level 2 24%
Level 3 72%
Experience
Child MH 10 years
CCEP (yrs) 4
Status with CCEP
Full-time 59%
Part-time 41%
PT Mean 20 hrs
Educational Level
Master’s 83%
Bachelor’s 17%
Degree/Major
Social work 59%
Psychology 17%
Other 24%
State licensure
Yes 83%
No 17%
Cross-sectional strategy:Cross-sectional strategy:survey summaries/ research briefssurvey summaries/ research briefs
1. Informing Providers About CCEP Services1. Informing Providers About CCEP Services 2. Child and Family Consultation Processes2. Child and Family Consultation Processes 3. Programmatic Consultation Processes3. Programmatic Consultation Processes 4. Reflective Supervision4. Reflective Supervision 5. Group Training and Individual Coaching 5. Group Training and Individual Coaching
of Providers and Parentsof Providers and Parents 6. Consultants: Experience, Job Satisfaction, and 6. Consultants: Experience, Job Satisfaction, and
Organizational SupportOrganizational Support 7. The Most Important Things Consultants Do7. The Most Important Things Consultants Do 8. Collaboration with Michigan Child Care Coordinating 8. Collaboration with Michigan Child Care Coordinating
Council, MSU Extension, and the Great Start CollaborativeCouncil, MSU Extension, and the Great Start Collaborative 9. State-Level Training and Technical Assistance 9. State-Level Training and Technical Assistance Available at Available at http://outreach.msu.edu/cerc/research/ccep.aspxhttp://outreach.msu.edu/cerc/research/ccep.aspx
Cross-sectional strategy: Cross-sectional strategy: other survey resultsother survey results
Preventing Children’s Expulsion from Preventing Children’s Expulsion from Childcare: Variations in Consultation Childcare: Variations in Consultation Processes in a Statewide ProgramProcesses in a Statewide Program
Poster and Survey summaries/research briefs at SRCD Poster and Survey summaries/research briefs at SRCD conference (2009) View at:conference (2009) View at: http://outreach.msu.edu/cerc/http://outreach.msu.edu/cerc/
2. Longitudinal strategy - strengths2. Longitudinal strategy - strengths
Able to assess child, parent, provider and Able to assess child, parent, provider and program outcomes pre (T1) and post (T2) and if program outcomes pre (T1) and post (T2) and if these were sustained over 6 months(T3). these were sustained over 6 months(T3).
Collaborative – state and local e.g. consultation Collaborative – state and local e.g. consultation on selection, organization and use of measures; on selection, organization and use of measures; attendance at monthly meetings; electronic Q & attendance at monthly meetings; electronic Q & A; personal contacts between consultant and MSU A; personal contacts between consultant and MSU team especially with new staff; collaborative team especially with new staff; collaborative troubleshooting at state level.troubleshooting at state level.
Built on existing systems so incorporated Built on existing systems so incorporated measures already used by consultants e.g. DECAmeasures already used by consultants e.g. DECA
Longitudinal study sample sizeLongitudinal study sample size
Time when cases received
Sample size
Child & family cases (T1)
432
Child & family cases (T2)
394
Child & family cases Follow-up (T3)
177
Programmatic cases(all)
55
Sample sizes included in analyses varied depending onthe quality of the data collected
Children & Families intervention sample Children & Families intervention sample (N=361)(N=361)
Child’s age-months
Mean (SD)43.2 (13.2)
0-35 25%
36-60+ 75%
Gender
Male 75%
Race/ethnicity
Afr- Amer 15%
White 77%
Other 8%
Hispanic 8%
Household income
Low 34%
Family
2-parent 60%
Provider
Center 86%
F. home 5%
Gp. home 7%
Relative 1%
In-home 1%
Previous expulsions
10%
3. Quasi-experimental strategy3. Quasi-experimental strategy
Includes collection of matching data from a sample of Includes collection of matching data from a sample of children exhibiting challenging behaviors but resident in a children exhibiting challenging behaviors but resident in a county where CCEP unavailable. Need to create a matched county where CCEP unavailable. Need to create a matched sample (N=86). sample (N=86).
Enables comparison with CCEP intervention group beyond Enables comparison with CCEP intervention group beyond maturation changesmaturation changes
Ongoing challenges (resources - time, staff, organization, Ongoing challenges (resources - time, staff, organization, incentives) for recruiting and participation of comparison incentives) for recruiting and participation of comparison group but not resident in county with CCEPgroup but not resident in county with CCEP
Limitations –missing data, multiple raters, reliance on self-Limitations –missing data, multiple raters, reliance on self-report measures and interviews, how representative was report measures and interviews, how representative was the intervention group who participated in the evaluation, the intervention group who participated in the evaluation, were comparison families enough like CCEP group even were comparison families enough like CCEP group even with matching? what other services, if any, were with matching? what other services, if any, were comparison families receiving in their own counties? Did comparison families receiving in their own counties? Did counties with CCEP differ from counties without?counties with CCEP differ from counties without?
Outcome Measure
Child 1. Devereux Early Childhood Assessment (DECA; LeBuffe & Naglieri, 1999).
2. DECA-Infant-Toddler Version (DECA-IT; Mackrain, LeBuffe & Powell, 2007)
3. Problem Coding Grid developed by Michigan CMH. 4. Subscales from the Behavior Assessment System for Children-
Second Edition (BASC-2; Reynolds & Kamphaus, 2004)5. Retention, placement, and expulsion.
Parent 1. Parenting Stress Index/Short Form (PSI/SF; Abidin, 1990)2. Skills and Knowledge subscale of the Psychological Empowerment
Scale (PES; Akey, 1996)3. Work productivity
Provider 1. Early Warning Signs (developed by MSU team)2. Goal Achievement Scale (GAS; Alkon, Ramler, & MacLennon, 2003)3. Teacher Opinion Survey (TOS; Geller & Lynch, 1999).
Consultation process, effectiveness, and acceptability
Adaptation and/or sub-scales of various instruments including:1.Parent-Teacher Relationship Scale, (PTRS; Vickers & Minke, 1995)2.Consultation Evaluation Form (CEF; Erchul, 1987).3.Behavioral Intervention Rating Scale (BIRS; Von Brock & Elliott, 1987).4.Benefits of Consultation (Sheridan, 1998, 2000a, 2000b) including other sub-scales from BIRS)5.Competence of Other (Sheridan 1998, 2000a, 2000b)
Does consultation make a difference to Does consultation make a difference to parents? parents?
Awaiting final results, (on child, parent, provider, Awaiting final results, (on child, parent, provider, program outcomes and perceptions of program outcomes and perceptions of effectiveness & relationships). effectiveness & relationships).
With qualifications, trends prior to the final analyses With qualifications, trends prior to the final analyses have indicated that:have indicated that:
Both parental competence increased and stress Both parental competence increased and stress reduced more among parents who used reduced more among parents who used consultation services. consultation services.
There was strong/high levels of satisfaction with There was strong/high levels of satisfaction with the perceived consultation process, its’ the perceived consultation process, its’ effectiveness, and acceptability by both parents effectiveness, and acceptability by both parents and providers.and providers.
Interim results (N=129) - Change in child outcomes Interim results (N=129) - Change in child outcomes after early childhood mental health consultation (see after early childhood mental health consultation (see
link to poster)link to poster)
Before taking dosage of CCEP into account, raw parent & provider data Before taking dosage of CCEP into account, raw parent & provider data showedshowed::
Both CCEP and comparison children showed significant improvements in behavior Both CCEP and comparison children showed significant improvements in behavior problems and positive behaviors over the study period.problems and positive behaviors over the study period.
For parent report in the CCEP group, attention problems and functional For parent report in the CCEP group, attention problems and functional communication continued to improve 6 months after consultation; most others communication continued to improve 6 months after consultation; most others remained level.remained level.
Are higher doses of consultation linked to greater improvement in child Are higher doses of consultation linked to greater improvement in child challenging and positive behaviors compared to lower doses?challenging and positive behaviors compared to lower doses?
After taking satisfaction with CCEP into account, more hours of consultation with After taking satisfaction with CCEP into account, more hours of consultation with providers (but not parents) predicted increases in provider reports of some positive providers (but not parents) predicted increases in provider reports of some positive behaviors.behaviors.
At 6-month follow-up, more hours of provider consultation was linked to continued At 6-month follow-up, more hours of provider consultation was linked to continued improvements in parent-reported attention problems. improvements in parent-reported attention problems.
Gains made in behavioral concerns and functional communication were not Gains made in behavioral concerns and functional communication were not sustained.sustained.
Do children with challenging behavior who receive consultation show more Do children with challenging behavior who receive consultation show more behavior improvement compared to children with challenging behavior who behavior improvement compared to children with challenging behavior who do not receive consultation?do not receive consultation?
While children in the intervention (N=129 and comparison (N=59) groups both While children in the intervention (N=129 and comparison (N=59) groups both improved over time, probably due to maturation, the CCEP group showed greater improved over time, probably due to maturation, the CCEP group showed greater improvements in behavior than the comparison group in almost all areas.improvements in behavior than the comparison group in almost all areas.
4. Case studies4. Case studies
Sample: (N=9 children) 2 programs, 3 Sample: (N=9 children) 2 programs, 3 consultantsconsultants
Method: Interviews in-person or phone Method: Interviews in-person or phone with parent, provider (s) and consultantwith parent, provider (s) and consultant
Analyses: Coded & content thematically Analyses: Coded & content thematically organized around process and outcomesorganized around process and outcomes
CASE STUDY SAMPLE
Name Sex Age
Reason for referral
Household # I Outcome
Dylan M 60m
Listless, withdrawn Mother, stepfather
5 Adjusted, kindergarten
Sophia F 40m
Defiant, aggressive Mother, boyfriend, sibling
2 Mom lost job, withdrawn from cc
Jason M 71m
Head-banging, tantrums
Single mother 3 Reduced intensity
Ryan M 51m
Tantrums, screaming
2 bio. parents, twins
3 Reduced intensity, moved on to schoo
Kayla F 41m
Defiant, hyperactive
2 adoptive parents, sibling
3 Parent & provider behavior adapted
Nathan M 49m
Developmental delay, aggressive
2 bio. Parents, sibling
3 Parent & provider behavior adapted
Madison F 60m
Tantrums, disruptive
2 bio. parents 1 Provider adaptedKindergarten
Hannah F 42m
Aggression Single mother 3 Incomplete consultation- moved out of state
Daniel M 48m
Aggression, sexualized behavior
Single mother 4 Expelled
#I=Number of interviewees
Case studies: strengthsCase studies: strengths
Combines quantitative and qualitative Combines quantitative and qualitative methods.methods.
Illustrates the variation and unique Illustrates the variation and unique relevance for individual children relevance for individual children
Adds depth to the understanding of the Adds depth to the understanding of the processes that underpin consultationprocesses that underpin consultation
Highlight the importance of context and Highlight the importance of context and relationships for interventionrelationships for intervention
Case studies: challengesCase studies: challenges
Balancing case study importance with a Balancing case study importance with a primarily outcome focused evaluation.primarily outcome focused evaluation.
Self-selection bias in sampleSelf-selection bias in sample Combining meaningfully with quantitative Combining meaningfully with quantitative
data- using quotes in body of report data- using quotes in body of report (outcomes), thematic table about process (outcomes), thematic table about process and ‘stories’ about children with and ‘stories’ about children with standardized scores compared to meanstandardized scores compared to mean
Program’s use of preliminary evaluation Program’s use of preliminary evaluation resultsresults
Accountability.Accountability. Was the money being spent as Was the money being spent as agreed? Was it being spent wisely?agreed? Was it being spent wisely?
Planning – program and community Planning – program and community Where to focus Where to focus limited resources? Was more needed? Help others limited resources? Was more needed? Help others understand the consultants’ role and perspective and understand the consultants’ role and perspective and the contribution it can make to community planning. the contribution it can make to community planning. Grant preparations.Grant preparations.
Quality improvement.Quality improvement. How could CCEP build on its How could CCEP build on its strengths? What could CCEP have done better? Ready strengths? What could CCEP have done better? Ready access to evaluator expertise offered more support. e.g. access to evaluator expertise offered more support. e.g. internal monitoring systems.internal monitoring systems.
Advocacy & Dissemination. Advocacy & Dissemination. Tell others about CCEP Tell others about CCEP successes and challenges. Politicians, potential funders, successes and challenges. Politicians, potential funders, academics-contribution to the ECMH knowledge base. academics-contribution to the ECMH knowledge base.
Closing commentsClosing commentsfrom Daniel’s momfrom Daniel’s mom
““I think it’s (CCEP) an awesome program, I really I think it’s (CCEP) an awesome program, I really do. There are a lot of daycares out there that if do. There are a lot of daycares out there that if they come across just the littlest behavior, and they come across just the littlest behavior, and the child becomes difficult to take care of, they the child becomes difficult to take care of, they just give up and say just give up and say ‘okay, well we can’t have ‘okay, well we can’t have him in the daycare’him in the daycare’. So someone like Julie . So someone like Julie (consultant) that could come out and talk to the (consultant) that could come out and talk to the caregivers and explain different ways of doing caregivers and explain different ways of doing things, I mean, I think that’s awesome because things, I mean, I think that’s awesome because then you know, the kid can stay in the daycare then you know, the kid can stay in the daycare and the mother can continue working. I mean, I and the mother can continue working. I mean, I think it’s a really good program.” think it’s a really good program.”
Further informationFurther informationPrincipal Investigators:Principal Investigators:
John Carlson, PhD, NCSP; John Carlson, PhD, NCSP; Asc. Professor, College of Education; Asc. Professor, College of Education; [email protected]@msu.edu
Holly E. Brophy-Herb, PhD, Holly E. Brophy-Herb, PhD, Associate Professor, Human Dev. & Family Studies;Associate Professor, Human Dev. & Family Studies;[email protected]@msu.edu
Laurie A. Van Egeren, PhD,Laurie A. Van Egeren, PhD,Director, Community Evaluation and Research Center Director, Community Evaluation and Research Center (CERC), University Outreach and Engagement(CERC), University Outreach and Engagement; ; [email protected]@msu.edu
Useful linksUseful links MSU CCEP EVALUATION RESULTS REFERRED TO HEREMSU CCEP EVALUATION RESULTS REFERRED TO HERE: :
BRIEFS AND POSTERS BRIEFS AND POSTERS http://outreach.msu.edu/cerc/research/ccep.aspxhttp://outreach.msu.edu/cerc/research/ccep.aspx
TECHNICAL ASSISTANCE CENTER FOR SOCIAL EMOTIONAL TECHNICAL ASSISTANCE CENTER FOR SOCIAL EMOTIONAL INTERVENTION: http://www.challengingbehavior.org/INTERVENTION: http://www.challengingbehavior.org/
UNIVERSITY OF WISCONSIN – EXTENSION: UNIVERSITY OF WISCONSIN – EXTENSION: http://www.uwex.edu/ces/pdande/evaluation/index.htmlhttp://www.uwex.edu/ces/pdande/evaluation/index.html
NSF Online Evaluation NSF Online Evaluation Resource Library: Resource Library:
http://www.oerl.sri.comhttp://www.oerl.sri.com
TROCHIM, W.M. THE RESEARCH METHODS KNOWLEDGE TROCHIM, W.M. THE RESEARCH METHODS KNOWLEDGE BASE, 2ND EDITION: BASE, 2ND EDITION: http://www.socialresearchmethods.net/kb/http://www.socialresearchmethods.net/kb/