Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human

Services and Entire School Populations to Investigate What Works and for Whom

45-Minute Paper Presented at University of South Florida’s 31st

Annual Research & Policy Conference: Child, Adolescent, and Young Adult Behavioral Health, 4-7 March, 2018, Tampa, Fl.

Mansoor A. F. Kazi, PhD, President, Realist Evaluation Inc., Director Program Evaluation Center, Fredonia SUNY;Rachel M Ludwig, LCSW, Chautauqua Tapestry Director;Susan Hoerter, DO, Rockland County SOC Expansion Grant PI;Janet Sliva, Rockland County SOC Expansion Grant Manager

Marie McLaughlin, Director, Manchester Youth Justice Service, England

m.mclaughlin1@manchester.gov.uk

Acknowledgements

• Dr. Bret Apthorpe, Superintendent; Jessie Joy, Director of Curriculum, Instruction & Assessment; Tina Sandstrom, Director of Schools, Jamestown Public Schools; John Panebianco, Director of Pupil Services

• Dr. Gary M. Blau, Chief, Child, Adolescent and Family Branch, Center for Mental Health Services, Substance Abuse and Mental Health Services Administration

• Patricia Brinkman, Commissioner; ChautauquaCounty Department of Mental Hygiene, Mayville

• Manchester City Council—Marie McLaughlin,Director Youth Justice Services

Acknowledgements

• Tom Zugibe, District Attorney

• Michael Leitzes, Mental Health Commissioner

• Mary Jean Marsico, Rockland BOCES

• Dr. Susan Hoerter, Rockland SOC Expansion PI

• Janet Sliva, Rockland SOC Expansion Grant Manager

• Barbara Gavin, Director, Child Welfare Services

• Carol Korngold, Safe Youth Center Executive Director

• Kathy Farber, Safe Youth Center Consulting Director

• Kathy Tower-Bernstein, Director of Probation

Tapestry of Chautauqua

County, New York

Mansoor A. F. Kazi, President, Realist Evaluation Inc.; Affiliate Research Fellow, University at Buffalo; Director, Program Evaluation Center, Fredonia State University of New York; Adjunct Fordham University

kazim@fredonia.edu mkazi@buffalo.edu• American Evaluation Association (eval.org) Co-Chair

Human Services Evaluation Topical Interest Group & Social Work Topical Interest Group.

Based on Kazi, M. A. F.

(2003) ‘Realist Evaluation

in Practice’, London: Sage

Realist Evaluation Partnerships

SAMHSA’s

Gold Award for

Outstanding

Local

Evaluation

2010 Training

InstituteSTAKES/THL

MANCHESTER CITY

COUNCIL, England

ROCKLAND COUNTY NY

New York State System of Care SAMHSA Funding

Funded by SAMHSA—Substance Abuse and Mental Health Services Administration, Center for Mental Health Services:

• ‘Tapestry of Chautauqua’ Systems of Care Grant (with Chautauqua County Department of Mental Hygiene, $9,000,000 2008-2015; $977,317 for evaluation, Funded by SAMHSA; Recipient of SAMHSA’s Gold Award for Outstanding Local Evaluation, July 2010).

• ‘Tapestry of Chautauqua’ Systems of Care Expansion Grant (with Chautauqua County Department of Mental Hygiene, $4,000,000 2015-2019)

• Rockland County (NY) Systems of Care Expansion Grant, with Rockland County, $4,000,000 2016-2020; $800,000 for evaluation, Funded by Substance Abuse and Mental Health Services Administration (SAMHSA).

Take the

Gary

blauchallenge

T= (V + B + A) x (CQI)2EBP’s & Clinical

Excellence

Cultural & Linguistic

Competence

Continuous Quality

Improvement

Family Driven

Youth Guided

Transforming Children’s

Mental Health in America!

System of Care: Focus on Values

Children’s Mental Health Initiative (CMHI):

Systems of Care

CMHI has impacted 22% of

the nations counties and 33

federally recognized Indian

Tribes

Since 2000, SAMHSA

administers an average of 61 system of care grants per year

CMHI sites have served over

100,000 children and youth

There are currently 88 CMHI funded

sites

Systems Of Care Work!

Reduced behavioral and emotional problems

Increased behavioral and emotional skills

Reduced suicidal ideation and attempts

Reduced substance use problems

Improved functioning in school and in the community

Improved ability to build relationships

View all of the KSOC-TV webisodes on the SAMHSA YouTube channel or by going to www.samhsa.gov/children

The Goals of a System of Care

• To improve access and expand the array of services and supports for children and youth with a serious emotional disturbance and their families.

• To promote the full potential of every child and youth by addressing their physical, emotional, intellectual, cultural and social needs in family, school & community

Who Comes to School?

• 21% of children ages 9-17 have a diagnosable mental health disorder (U.S. Surgeon General, 1999; NIMH, 20% currently)

• 20% of children with mental health disorders are identified and receive mental health services (U.S. Surgeon General, 2000; mentalhealth.gov—20% currently)

• 50% of students age 14+ who have a mental illness drop out of high school…the highest dropout rate of any disability group (U.S. Department of Education, 2001) (NIMH currently)

Chautauqua Tapestry

• Why “Tapestry”?

• Quick history

• Goals

• Partners

Chautauqua County, NY

SAMHSA’s Theory of Change

Expansion = System Change

System of care expansion is not a “project”

Goal is sustainable systemic changes in child-serving systems to improve services and outcomes

Occurs with or without federal grant

Grants are “venture capital” to achieve larger system change

Linked to other system reforms

Keys to Successful Expansion

1. Commitment of high-

level policy and decision

makers

2. Realistic goals

3. High-priority goals

4. Specific, concrete

strategies

5. Strong leadership to

manage implementation

6. Linkages with larger

system reforms in

environment

7. Cross-agency

partnerships

8. Commitment across key

stakeholders

9. Staff and resources

allocated to

implementation work

Evaluation as Sustainability Strategy

• Similar organizations collecting quality data, but uncoordinated and isolated

• Similar organizations collecting quality data for the same outcomes in a coordinated and collaborative network

• Similar organizations collecting quality data that align under similar outcomes for broad community impact

Collaboration Defined:

A mutually beneficial relationship between two or more parties who work toward common goals by sharing responsibility,authority and accountabilityfor achieving results.

Evaluation Examples

• Jamestown Public School

• North Rockland School District

• Manchester City Council, UK

Critical Realism and Outcomes

• Sayer (1984): Realism links theoretical abstraction with empirical research

• “Essentially eclectic, the realist approach seeks to move away from what may be now rather sterile epistemological debates surrounding scientific versus pluralistic and constructivist paradigms, using all methodologies as appropriate within the realist framework. The approach facilitates the interrogation of empirical data with a view to identifying the nature of specific structural influences on outcomes” (p.45).

• Realist evaluation embraces outcome measurement and also investigates the factors that influence the outcomes

Realist Evaluation

• Realist evaluation seeks to evaluate practice within the realities of society, based on Emergent and Symbiotic elements of natural law.

• All systems are emergent and constantly in a state of evolution, along with the reality that we are all symbiotically connected to nature and to each other.

• Practice takes place in an open system that consists of a constellation of inter-connected structures, mechanisms and contexts.

• Realism aims to address all the significant variables… through a realist effectiveness cycle which links the models of intervention with the circumstances in which practice takes place (Rom Harre’s models, analogous with reality)

• Explanation at any one time requires further investigation andfurther explanation (continuous testing & development of RomHarre’s models)

Purposes and Objectives of evaluation

• Research methods drawn from both epidemiology and efficacy research traditions in a realist evaluation

• Partnership with human service agencies & schools to investigate what works and for whom. Real live data mined from human service agencies is used to investigate the effectiveness of the human service interventions.

• Emphasis is on data naturally drawn from practice.

• Binary logistic regression as part of epidemiologic evidence based on association, environmental equivalence, and population equivalence.

• How evaluators and agencies can make the best use of the available data to inform practice.

Purposes and Objectives of evaluation

• Research indicates that the reform of the system of care and the use of initiatives such as wraparound are effective in improving mental health and functioning in school (Kutash, 2006; Reback, 2010; Goldenson, 2011)

• However, most studies have focused on at risk groups rather than the total school populations.

• The purpose of this study is to undertake a 100% evaluation of all school-based services, utilizing data on the entire school populations, in a longitudinal study.

Evaluation Strategy With Each Agency

• Evaluation resource and service for you and for each participating agency

• How to access and to use your own MIS data

• How to analyze this data repeatedly to inform practice

• Carried out with you and for you only—you decide who to share with and how to use the findings

Realist Evaluation: What Interventions work & in what circumstances

• A combination of efficacy research & epidemiology traditions

• Management Information System (MIS) Data routinely collected but typically not used for evaluation in agencies

• Investigate interrelationships between outcomes, client demographics, client circumstances, & services provided (interventions)

• Methods such as binary logistic regression can predict the likelihood of effectiveness of an intervention in given circumstances

• Use findings at regular intervals to better target and develop services

Education Partners in SOC

• Education plays a critical role in the development of children.

• Positive learning experiences help prevent emotional and behavioral problems.

.

School Examples of Realist Evaluation

• 17 NYS school districts have participated

• 100% data from school + participated agencies

• What works and for whom in achieving school,

agency and system of care outcomes

• Examples of school data

Demographics Intervention Outcome

• Ethnicity• Gender • Lunch status, • Special education

• School based interventions

• Pre-school programs

• Mental health • DSS

• GPA/ State tests• Behavior incidents• Attendance• Drop out rates

Graduation rates

Data analysis and utilization

• Single system & one group pre/posttest

• Comparison of outcomes between baseline andsubsequent periods and those receiving and notreceiving interventions

• Investigation of patterns between outcomes,demographics and interventions

• Binary logistic regression

• Data analysis in partnership with schools andagencies

• Utilization of findings to develop and improveservices for children and families

Outcomes: Jamestown Public Schools

• Strengths and Difficulties Questionnaire (SDQ) April 2015 and April 2016

• Recorded Incidents of Behaviour 2014/15 and 2015/16

• North West Evaluation Association (NWEA) Measures of Academic Progress (MAP) Fall 2015 , Spring 2016 & Fall 2016 (grade levels K to 10)

Universal Screening Tool: SDQ

• Strengths and Difficulties Questionnaire (SDQ)

• Brief measure of pro-social behaviour and psychopathology, 3-17 yr olds

• Goodman (2001)—reliability .73

• Five factors: emotional symptoms, conduct problems, hyperactivity, peer relationships and pro-social

• Grade levels K to 4 in April 2012, 2013, 2014, 2015 & 2016

SDQ 2012 Normal Scores/Ethnicity

SDQ 2012 Predictors for

abnormal/borderline total scores

Predictors for Change from 2012 to 2013

Total SDQ

Hyperactivity Level 2015 SDQ

Hyperactivity Level 2016 SDQ

Hyperactivity Level Change from 2015 to

2016 SDQ

Hyperactivity Level Change from 2015 to

2016 SDQ: Predictors for improvement

Hyperactivity Level Change from 2015 to

2016 SDQ: Striders Advocacy

Hyperactivity Level Change from 2015 to 2016

SDQ: Jamestown Behavioral Health Clinic

Tapestry Partners Again Predictors for

Improved Behavior in Schools• Recorded incidents of behavior Compared between

2014-15 and 2015-16 school years

• Overall mean was 0.96 in 2014/15 & 1.58 in 2015/16 recorded incidents per student ( n = 5134) in Jamestown Public Schools in both years

• Once again Tapestry partners’ services were predictors for improved behavior, this time several more!

Safari Improved Behavior

Improving Measurement of Academic Achievement

• Northwest Evaluation Association (NWEA)

• Measures of Academic Progress Normative Datafor Reading and Mathematics

• More reliable than GPA

• Used three times a year up to grade level 10

• Beginning from Baseline Year 2015/16

• First two times used Rasch Unit Scale (RIT) butfor the third assessment will use expectedgrowth

• Of those proficient at level 3 or 4, 51.8%demonstrated growth that was less thanexpected in the first two reading assessments

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

Predictors for Improving RIT Score at Median 9

or Higher

Plan for Chautauqua Tapestry 2.0(2015-2019)

– Chautauqua County the only one to receive SAMHSA’s Gold Award for Outstanding Local Evaluation.

– Tapestry 2.0 will deepen its evaluation with school districts and human service agencies in the county.

– Chautauqua County is a world-leader in realist evaluation, and to date 28 agencies (including 7 school districts) have collaborated

Outcomes: North Rockland Schools

• Recorded Incidents of Behaviour- 2016/17 (grade levels K-12th)

• Istation Universal Screener- Fall 2016 & Spring 2017 (grade levels 3rd to 6th)

• Developmental Reading Assessment 2 (DRA2)- Fall 2016 & Spring 2017 (grade levels 1st- 4th)

• North West Evaluation Association (NWEA) Measures of Academic Progress (MAP) Fall 2015 , Spring 2016 & Fall (grade levels K to 10) from 2017/18 School Year

North Rockland Schools 2017/18

Istation Universal Screener- Fall 2016 & Spring 2017 (grade levels 3rd to 6th)

Developmental Reading Assessment 2 (DRA2)- Fall 2016 & Spring 2017 (grade levels 1st- 4th)

Recorded Incidents of Behavior 2016/17 (8.9%)

to First Period 2017/18 (4%; N = 8026)

Plan for Transforming Rockland County(2016-2020)

– National Evaluation Requirements

– Local Evaluation Strategies

– Continuous realist evaluation of services utilizing entire Management Information Systems

– Department of Social Services

– County Mental Health Services

– Probation Service

– North Rockland and Nyack School Districts

– Continuous quality improvement

Plan for Rockland County(2016-2020)

– Rockland County Evaluation will expand from the largest North Rockland Schools to include all 8 school districts (n = 41,000)

– One of 37 in NY, Board of Cooperative Educational Services (BOCES) plays central role with Partnership for Safe Youth

– 100% data from County Mental Health, Department of Social Services, Probation Service and other human service agencies, and merged with whole school big data

Example: Manchester Youth Justice Service in England

• This interim report is based on electronic data

comprising of a 100% sample of 2,526 youth in

the 4-year period January 2011 to December

2014

• 76.3% were male, and 62.2% of the youth were

white

• The average age at date of arrest for all youth

was 15 years, ranging from 10 to 18 years.

Manchester Youth Justice Service in England

• Offence database recorded every episode of an

offence reported on the youth,

• Work out those that had offended after the first

recorded offence, and those that had more than

one, in the 4-year period from January 2011 to

December 2014.

• It was found that 66.1% had not offended again

after the first recorded offence (as compared

with 45.6% in the previous 10-year analysis

reported at AEA 2012).

Binary Logistic Regression Model

Predictors for returning

• Age at first order. Those older by every year were 1.5

times more likely not to return

• Gender. Females were more likely not to return than

males.

• Those who had completed the order were 7 times more

likely not to return.

• Youth living with parents or relatives were 2.6 times more

likely not to return.

• Those with higher baseline ASSET scores were more

likely to return

• Those on youth caution more likely not to come back

Manchester Youth Justice Service:Linking Data with Practice

• Using our data differently

• Clearer picture of our cohort

• Understanding what is working andwhat is not working

• Helping to target resources moreeffectively

• Identifying predictors of re-offending

Manchester Youth Justice Service:Looking Forward

• Improving the way we record data

• Working with partners to improve useof data

• Merging this data with a wider pool ofdata within the City of Manchestere.g. schools, police, health, benefitsagency

• Targeting resources more effectivelyto get better outcomes

Conclusion

• Realist evaluation strategies help to utilise theavailable whole school data and to conduct deeperanalysis

• Explanations for differences between those thatimprove on an outcome and those that do not

• Identifies patterns in the data where multiplefactors are influencing the outcome, and selectsthe main factor or factors responsible for theoutcome, with a prediction of the odds of achievinga given outcome in particular circumstances

• Repeatedly investigate changes in outcomes,intervention & the contexts