HeadStart Kent Knowledge Seminar 2

21st January 2015

Measuring Outcomes

Time Agenda Who

9.30 – 9.40 Introductions (around the table and presenters)

Florence/Angela

9.40 – 9.45 Recap of Learning from Seminar 1

Key Messages: Summary

Broader Strategic Issues For Headstart Project

Alex Hassett

9.45 – 9.55 Feedback on the Board and Shadow Youth Board

Angela Ford

9.55 – 10.05 Broader Evaluation Programme

Ugochi Nwulu

10.05 – 10.30 Activity: Challenges faced measuring resilience outcomes. Individual and

group exercise – feedback to the wider group

Alex Hassett

10.30 – 11.00 • Challenges and practical issues in measuring resilience

• Measures of resilience – some ideas

• A domains approach to measuring resilience

Mark Kerr

11.00 – 11.20 Coffee Break

11.20 – 12.00 Activity: Mapping where your service fits and what you measure Mark Kerr and Alex

Hassett

12.00 – 12.15 Feedback on learning

Alex Hassett

12.15 – 12.30 Way Forward

Florence / Angela

Resilience is not a trait but an interaction between risk and protective factors

We need to ensure we take an ecological and developmental view of resilience

It is useful to focus on resilience in terms of the areas or ‘domains’ of a person’s life that can be changed.

Negotiation and navigation

We need to consider what resources are available and how accessible are they

Key Messages: Summary

The following are the long term issues that need to be addressed:

Developing an overarching framework of resilience that the range of agencies can sign up to

Encouraging a long term interdependency between individuals, services, agencies on providing an holistic approach to young people

Providing a coherent system for evidence based evaluation ensuring that each element of the system is clear on how they evidence outcomes and impact

Broader Strategic Issues for Headstart Project

Reminder of Kent Activity C

ante

rbu

ry Penn State Resilience in

secondary schools

Penn State Resilience in primary schools.

Penn State Resilience in community and target workers

Family approach TBC

No

rth

We

st K

en

t Safe Spaces in schools

Safe Spaces in community hubs

Coping packs

Family Focus KS2 ACP

Peer mentors

Active listening mentors

Youth MH First Aid

Online counselling

Than

et Restorative approaches

in schools

Restorative approaches in the community

Target Restorative approaches in schools

Restorative Ambassadors

Restorative approaches families

Resilience Mentors: evidence based model of intensive support. FRIENDS

Digital World: full services directory , volunteering & mentoring opportunities, self -referral form and sign posting to social media

Social Marketing: skills roadshows, coping packs, phubbing.

Pa r t n e rs h i p P ro g ra m m e B o a rd , S h ad ow B o a rd , Kn ow l e d ge S e m i n ars

Coproduction throughout Young people and Families

Feedback from Young People

Young people found the domains resilience approach useful Identified areas of HeadStart Kent they felt would have most

impact Resilience mentors Coproduction social marketing Family resilience Safe Spaces Peer support Online directory

Ensure Kent’s Emotional Wellbeing Strategy is central to developments.

Contributing to service redesign

Connecting to the system and enabling change

Wider stakeholders already mapping and exploring system redesign

Outcomes

Early Help: improved emotional resilience and receive early support

Access: Receive timely, assessing and effective support

Whole Family: Recognises and strengthens and wider family relationships.

Recovery and Transition: Prepared for and experience positive transitions

Kent’s Emotional Wellbeing Strategy for Children, Young People and Young

Adults

Every one needs to consider how they contribute to building resilience, and what they could do enhance it further.

If our outcomes frameworks are to be guided by the domains (risk and protective factors), we need strategic cohesion across Kent including:

Workforce being prepared to work systemically.

Shared language

Less duplication

Easy moving and less transitions.

Workshop

There is a greater awareness of activity locally and countywide and a lot has happened over a short period of time

People fed back that the knowledge seminars have been useful and thought provoking.

Some of the challenges include: How to build coherence and ecological links when more than one

intervention is working in the same area.

How to involve more young people of greater diversity

How to get passion and buy in from professionals

How to increase the understanding and scale of social marketing

Workshop Messages

HeadStart Kent Knowledge Seminar

21 January 2015

For HeadStart information http://kelsi.org.uk/pupil_support_and_wellbeing/targeted_support/inclusion/kiass/headstart.aspx HeadStart Kent Twitter is: @HeadStartKent #headstartmatters

HeadStart Kent Knowledge Seminar

21st January 2015

Evaluation progress

Ugochi Nwulu Ugochi.Nwulu@kent.gov.uk

Ugochi Nwulu - KCC / University of Kent

Rob Comber - Education and YP’s Services, KCC

(Eileen McKibbin - Research and Evaluation, KCC)

Gabriela Sette - CHSS, University of Kent

(Prof Patricia Wilson - CHSS, University of Kent)

Evaluation team

Key evaluation questions:

1. What are the HeadStart interventions?

2. What is the theory of change across the programme?

3. How does each intervention contribute to the theory of change?

4. What is working well and not so well in the implementation and the delivery of HeadStart Kent?

5. What are the critical and effective elements of the programme which now need to be scaled up fora Kent wide approach to building emotional health and resilience

Process evaluation of HeadStart Kent

Kent area Intervention 1 Intervention 2 Intervention 3

Data sources:

HeadStart operational team / Community

practitioners / School staff / Resilience mentors

Young people

Evaluation methods

Case study

Focus group Questionnaires

Learning from participants and stakeholders

How we will measure the impact:

HeadStart Schools data

- demographics, risk factor profiles

- pupil absences, exclusions

- numbers accessing targeted support

- CYP who participate in the HeadStart programme

Baseline data collection

January to March:

National HeadStart conference

Synergies with the national evaluation - field work, surveys

Informal interviews and refinement of plans

March to May:

Focused data collection period

Data analysis and write up

Evaluation report

Will include plans for an impact evaluation of fully scaled up projects

Next steps

Take a few minutes to think about the challenges you face in measuring resilience outcomes in the work that you do or the work that you commission. Please note down your challenges and concerns

Now spend a few minutes discussing with your group what those challenges are. Please can each group decide on the 2 main challenges or concerns they face when considering measuring resilience outcomes?

Exercise One

Example outcome model

Educational Development

Results In

Increased Protective Factors

Decreased Risk

Factors

Producing

Personal Development

SocialDevelopment

Intrinsic Outcomes (individual well-being)

Extrinsic Outcomes (wider social good)

Source: Young Foundation, 2012

Can we measure resilience

The ‘what’ are we measuring – what is our Dependent Variable (DV)

Whether to use a global resilience scale or domain based measure

Self report versus third party rating

Age appropriateness of measure

Validity of measure

Requirements of analysis

Challenges in measuring

‘The development of a measurement instrument capable of assessing a range of protective mechanisms within multiple domains provides an approach to operationalising resilience as a dynamic process of adaptation to adversity (Olsson et al., 2003)

Ideally, measures of resilience should be able to reflect the complexity of the concept and the temporal dimension. Adapting to change is a dynamic process (Donoghue and Sturtevant, 2007)

Can we measure resilience?

Assessments of resilience need to consider:

I. a) the risk or adversity

II. b) assets/resources that might offset the effect of the risk

III.c) the outcome

Quantitative direct measurement – using a resilience measurement scale as an outcome measure

Quantitative indirect measurement – modelling a range of data with multivariate statistics

Qualitative – understanding individual experiences

Measuring Resilience

Several scales developed but not widely adopted and no clear preferred option

Definitional clarity needed which influences how we tackle this

Virtually no valid measures or children

Would need to measure availability of resources at all ecological levels to understand those that demonstrate

Only potential measure: California Healthy Kids Survey – The Resilience Scale of the Student Survey (Sun and Stuart, 2007)

Measuring Resilience cont’

Remember ‘resilience’ is the ability to overcome adversity due to the interaction of risk and protective factors

Cannot assess resilience until the child experiences adversity

Current Headstart project requires us to focus on the antecedents of resilience i.e. protective factors

The ‘DV’ problem

Headstart programme aims more suitable for a domain approach

The time limited (currently) nature of your work means for many children (hopefully) resilience will not be experienced due to the absence of significant adversity

The number of partner organisations, some with specific domains of focus, mean individually you are unlikely to provide all protective factors needed. However collectively you will

Resilience or Protective Factors?

Key domains What are you working on?

Secure Base

Education

Friendships

Talents and interests

Sociable Behaviour

Positive Values

Resilience Domains (Daniel & Wassell)

One of many decisions that must be taken alongside considerations such as age, ability, domain of interest and ecological level

Can be a combination – both self and others e.g. teacher, parent, carer or other professional and results combined

Research has indicated problems if incongruity between child, parent, and teacher reports

Self Report or Third Party Rating

Variable focused Link among measures of degree of risk/or adversity,

outcome, potential quality of individual or environment (to compensate/protect)

Person focused Compare people with different profiles (within or across

time) on sets of criteria to ascertain what differentiates resilient from non resilient children

Models / Approach to Resilience

Source: Masten, (2001)

Variable focused studies

Compensatory effects

Enough positive assets could offset the burden in child’s life from one or many risk influences

Three models

Main effect

Indirect

Interaction

Variable focused studies

Main effect models

Asset

Bipolar predictor

Risk

Outcome

-

+

- +

Variable focused studies

Indirect models (example)

Asset

Effective parenting

Risk

Outcome

-

+

+

Variable focused studies

Interaction models

Moderator

Risk activated moderator

Risk

Outcome

– Not found very often (difficult to detect)

Pros and Cons

Variable focused + Max. stats power, suitable for searching specific links

between predictor and outcome - Fail to capture striking patterns in lives of real people, risk

of losing sense of the whole

Person focused + variables assembled in naturally occurring configuration,

well suited for search for common and uncommon patterns in lives

- Can obscure specific linkages

Resilience-Based Practice

Three Main Principles: Inclusive, respectful and engaged

practice

Strengths-based practice

Solution focused approaches

Also, Fostering community and social

connectedness

Attachment theory and Circle of Security

Differences in ecological emphases: Australia vs. UK

Community

Family

Individual

Resilience in Practice - UK

Intervention Themes:

Improvement of self-esteem / to like self more

Improvement of peer relationships

Improvement in school experience / behaviour

Control of anger / managing disagreements

Naming feelings / emotional literacy

Target

Anger, aggression

Strategies

Anger control, emotional intelligence

Intended Outcomes

Increased self-esteem, improved peer relationships and school experience

Resilience in Practice- Australia Intervention Themes Subtypes Occurrence

1. Health a) Physical/Medical 11%

b) Mental/Behavioural 4%

2. Emotions & Attachment a) Parent-child dyads 14%

b) Broader family r/ships 3%

3. Parenting Skills & Confidence a) Expectations/boundaries 22%

b) Support: Peer/Playgroups 7%

4. Legal Issues 5%

5. Employment & Education or Training

8%

6. Finances & Housing 5%

7. External Supports for CHILDREN a) From the School 7%

b) From the Community 4%

8. Reduce Social Isolation 10%

Australian Practice… Chains of Support

NSWb1:

Address mother’s social isolation link mother with community supportive playgroup mother-child bonding and attachment is facilitated new social networks and connections with the community are created

NSWa1:

Address uncontrolled behaviour, aggression in children/poor attachment evident assist father in putting strong boundaries, routines and expectations in place at home children seen as having greatly improved emotional regulation, able to cope in new spaces or with new people father more competent and relaxed

Mapping interventions to Literature

1. Health

2. Emotions & attachment

3. Parenting confidence and skills

4. Legal issues

5.Employment/E&T/edn/training

6. Finances/Housing

7. External supports for children

8. Reduce Social Isolation

Res. Domains

Secure Base

Education

Social Competencies

Literature

Build Caring relationships

Establish and maintain Self-Efficacy

Mobilizing protective resources

Creating Opportunities

Foster Resilience Strings

Ecological

Resilience-based Practice – Implications…

Practice should aim to target all ecological levels to align to a comprehensive view of resilience

Policy: Ecological supports, especially community level investment

Building a common language around resilience to promote strategic change as disciplines and agencies endeavour to work in concert

Flexibility for assessment and practice frameworks – creativity and individualisation

Resilience-based Practice – Implications… cont’

Comparative research: Underlying processes vs. behaviours; impacts of various ecological levels

Impact and unique predictive value of attachment as central to intervention efforts

Family definitions of adversity and their priorities for help

Evaluate, evaluate, evaluate! Assessment of outcomes to see if resilience-based interventions work!!

Essentially, resilience-based practice needs to aim for consistency

in scope and application + flexibility

Outcome evaluation as critical for determining best practice

‘Absence of a conceptually sound and psychometrically robust measure of resilience for children under 12’ (Windle et al., 2011)

Reading ability

Problems with cognitive processing of Likert Scales

Distinguishing between how they feel now, and how they typically feel – ‘situational effect’

Developmental age versus chronological age

Issues with Age (Self Report)

Measuring Resilience - Outcomes

OUTCOMES:

Measured through informal channels and processes:

Feedback from parents and other agencies, observations, anecdotal evidence, children’s art

Positive (presence) and negative (absence) indicators of resilience:

Improved social skills, decreased anxiety, increased emotional regulation, better school performance, engagement in community activities

Measuring Resilience – Implications…

Consistency in outcome measurement to aid evaluation

Policy: Focused drive to incorporate sound outcome measurement

Research to examine breadth of assessment and measurement tools – recommend utility and relevance in the context of actual outcomes for clients

Again, the emphasis lies on consistency of usage: Assessment and outcome measurement procedures need to align to a resilience-based framework if this is the approach being explicitly espoused by the organization

To be a valid measure it must have been subject to a number of validation checks including: Content validity

Internal consistency

Criterion validity

Construct validity

Reproducibility

And others…

Should also go through peer review

So always try and use and existing measure!

Validity of Measure

Selection of Measures

What are you going to do with the data?

What questions do you want the data to answer?

What type of analysis do you need to do?

Do you want to be able to generalise your findings?

The type of data you collect influences the analysis you can do

Considering Analysis

Within group or between groups?

The need for ‘controls’

Score cut off points – defining expected levels, what does success look like?

Sample size and the impact on power and effect

Technical Considerations

Who are the focus of your activities?

What is their adversity?

Why do you think creative activities might build resilience? (your ‘theory of change’)

How might they benefit from a resilience promoting activity?

How will you know they have benefited?

How do you currently demonstrate this?

Your Work

Creative Activities

New skills/learning

Social connections/networks

Efficacy

Cognitive Health

You have 40 minutes for this exercise. As a group we would like you to think about each of your services or interventions that you or your service provides or you commission. Using the domains framework, and the chart provided, think about which domains you have an impact on and how you measure or could measure outcomes within that domain.

Once each group member has mapped their service and interventions look at the gaps and think about how they could be filled.

Spend the last 5 minutes of the exercise discussing what you feel your key learning has been from this exercise.

Activity