Development and Evaluation of the PCYC Catalyst Outdoor Adventure

Youth Intervention Program

Arron SullivanPCYC Bornhoffen

James NeillUniversity of Canberra

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

19th National Outdoor Education Conference, University of the Sunshine Coast, Sippy

Downs, Queensland, Australia, Wednesday 30 March, 2016

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Project plan – 2 components

Research and Evaluation

Program Delivery

• Work with 10 YP at risk• 15 day program duration • Partner with 6 education

and / youth centres • Work regionally and

locally• Collaborative practice• Evidenced-based

practice

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Key partners

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Levels of at-riskness (Greenaway, 1995: 3)

1. Temporary - Commit minor crimes in the company of others

2. Difficult and disturbed - Temporary involvement with crime, but whose offending is linked with wider problems like home conflicts, school based difficulties

3. Persistent offenders – Often of low intelligence and troublesome in school, with parents who exercise poor supervision and may themselves be involved in crime

4. One-off serious offenders – Crimes are rare and isolated, unexpected and not explained by social factors or environment. For these offenders psycho-therapeutic or behaviour modifying treatment in a secure setting is usually effective

5. Persistent and serious offenders - Offenders from category three and four for whom effective intervention is particularly difficult as there is a high likelihood of re-offending.

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Purpose and intent Project Purpose To provide a positive intervention that facilitates change for young people and their families using an interagency collaborative strategy

Program Purpose Working with young people (aged 13 to 16 years) who are at risk of adverse outcomes in their educational, vocational, and life-course pathways.

Intent To help young people to make positive life choices, experience a meaningful life, make a positive contribution to their community, and assist in the transition into young adulthood

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Purpose and intent

Project outcomes

1. Encouraging positive change in each young person by recognising and celebrating the observable outcomes

2. Partnering with Learning and Education centres that already have a behavioural management or learning engagement process in place

3. Engaging parents and other community organisations/members to support the change process for young people

4. Developing protective factors, positive self awareness and resilience in young people

5. Leading an interagency collaborative strategy for the development of community

6. Evaluating the short and long term outcomes of the project to develop its effectiveness

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Program specifics 1. Encouraging positive change in each young person by recognising and celebrating the observable outcomes• By helping young people to reflect positively on

experience and actions• Recognising and building on personal strengths• Managing personal behaviour and asking for help

reflecting on areas for improvement in behaviour• Encouraging participants and parents or guardians to

LEAD themselves first:- Learn to reflect positively on experience·Engage and manage their emotionsAdapt their thinking attitudes and behaviour to their situations and challenges· Demonstrate a willingness to:- work on yourself- participate positively in a group - contribute positively to the community • Learning to set realistic and achievable personal goals• Working as part of a team and taking personal

responsibility for the outcome• Raising awareness of all aspect of life resulting in a

positive and meaningful contribution to community

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Program model

Staff model Leadership development

-Two facilitators -Two teachers/school

councillors/youth worker

10 week program cycle • Partner selection • Participant selection • Parent commitment • 3-day Lead In• 9-day Expedition • 3-day Follow up• Participant mentoring N

ation

al O

utdo

or E

duca

tion

Conf

eren

ce 2

016

Program delivery partnersPCYC branches• Dalby • Mt Isa • Goondiwindi• Zillmere

Qld Department of Education State High Schools • Woodridge• Brackenridge • Helensvale • Goondiwindi • Dalby State High School• Spinifex College (Mt Isa)

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Background

• 2004 First Catalyst Program delivered • 2008 International Consortium for Experiential

Learning• 2009 Funding included seed funding for research• 2011 produced our own in-house evaluation • By 2012 we had worked with 24 partners and

invested $300,000 in programs • Maintained long-term partnership with strategic

view (2008 – 2013)

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Questions about the program model ? N

ation

al O

utdo

or E

duca

tion

Conf

eren

ce 2

016

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Research evaluation design• Mixed methods• Quantitative (questionnaires)

• Life effectiveness• Mental health• Behavioural conduct

• Qualitative (interviews)• Multiple perspectives• Youth self-report• Observer-report

• Longitudinal• Pre (1st day)• Post (last day)• Follow-up (6 to 12 months)

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Adventure therapy outcomes• Bowen & Neill (2013) synthesised the

outcomes of 197 adventure therapy studies.

• The overall standardised mean effect size was 0.47, a moderate effect.

• More specific benchmarks were suggested for adolescents (0.44) and specific types of outcomes.

• Useful for benchmarking evaluation of specific programs.

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Results: Life effectiveness• Adapted Youth at Risk Program Evaluation Tool

(YARPET; Neill, 2007) , 30 items measuring 10 factors:

Emotional ResilienceGoal SettingHealthy Risk-takingLocus of ControlSelf-Awareness

Self-EsteemSelf-ConfidenceCommunication SkillsCommunity EngagementCooperative Teamwork

• Short-term change: 0.16, N = 38; 60% of participants reported increases

• Longer-term change: 0.29, N = 20; 62% of participants reported increases

Results: Life effectiveness

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Goal Setting

Community Engagement

Locus of Control

Self Awareness

Emotional Resilience

Cooperative Teamwork

Self Esteem

Self Confidence

Healthy Risk Taking

Communication Skills

0.03

0.03

0.10

0.11

0.26

0.33

0.44

0.44

0.51

0.72

Longer-term changes

Results: Mental health

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

• Adapted General Well-Being (GWB; Heubeck & Neill,

2000; Veit & Ware, 1983) : 10 items measuring 2 factors:

Longer-term

Short-term

-0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2

0.95

0.07

-0.1

-0.33

Distress Well-being

Distress

Well-being

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Results: Behavioural conduct• Adapted Adolescent Behavioural Conduct

(Mak, 1993): 8 behaviours over past 6 months• Overall reduction (12.5 to 10.7 incidents/month):

Drug useCheatingVehiclesWagging

VandalisingStealingFighting

Harming

0.38

0.17

0.14

0.04

-0.27

-0.34

-0.47

-0.68

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Results: Youth interviewsYouth interviews revealed common themes that: • youths came from, but were seeking to overcome,

challenging backgrounds (personal, family, social)• faced challenges during the program (physical, social,

existential)• which fostered • personal and social development • motivation to work for change• a more optimistic outlook on the future

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Program recommendations1. Adventure therapy vs. therapeutic

adventure

2. Screening/Profiling/Selection

3. Physical challenge of initial hiking

4. Group storming and norming

5. High adventure activities

6. Reflective activities

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Key learningsProject & program delivery1. Multiple partners –

multiple masters2. Program integrity3. Evolving on the run4. Staff turnover

Research evaluation1. Program objectives2. Research vs.

evaluation vs. action-research

3. External vs. internal

Nati

onal

Out

door

Edu

catio

n Co

nfer

ence

201

6

Contacts for further info

James Neill Assistant Professor

Centre for Applied PsychologyFaculty of Health Mob 0432 925 211

james.neill@canberra.edu.au

Arron Sullivan Co Manager PCYC Bornhoffen

Leadership DevelopmentManager

Mob 0407 148 123arron.sullivan@pcyc.org.au

www.bornhoffenpcyc.org.au

References • Bowen, D. J. & Neill, J. T. (2013). A meta-analysis of adventure

therapy outcomes and moderators. The Open Psychology Journal, 6, 28-53. doi: 10.2174/1874350120130802001

• Bowen, D. J. & Neill, J. T. (2016). Effects of the PCYC Catalyst outdoor adventure intervention program on youths' life skills, mental health, and delinquent behaviour. International Journal of Adolescence and Youth, 21(1). doi: 10.1080/02673843.2015.1027716.

• Neill, J. T. & Bowen, D. J. (2014). Research evaluation of PCYC Bornhoffen Catalyst intervention programs for youth-at-risk [2012-2013]. Canberra, Australia: University of Canberra.

• PCYC Bornhoffen Adventure Development (2010). Catalyst Project 2009-2010 Leading change: Literature review. Retrieved from http://www.bornhoffenpcyc.org.au/CATALYST%20LIT%20REVIEW%202010.pdf

References • Neill, J. T. (2007). A measurement tool for assessing the effects of

adventure-based programs on outcomes for youth-at-risk participants. Retrieved from http://wilderdom.com/tools/leq/YouthDevelopmentLEQScalesPaper.htm