PROGRAMME REPORT 2013-2014

2

Since we launched in 2012, Project Oracle has already helped shape the agenda for the use of evidence in London’s children and youth sector. The projects at the heart of Project Oracle work to improve the lives of young people in London. So far, over 200 projects have embarked on training in evaluation approaches and over 150 have had the quality of their evidence validated.

However, Project Oracle is not concerned with pushing London’s children and youth sector to deliver evidence for evidence’s sake. What we aim to create and support is a culture where there is an appreciation for the use of solid evidence. Working towards that culture helps to build and improve existing projects and services whilst sharing what works across the sector. We have five Standards of Evidence to help make this process easier to understand.

This year we are excited to announce the establishment of Project Oracle as a not-for-profit Company Limited by Guarantee. It is our hope that moving Project Oracle from a “project” to an established “not-for-profit” will help ensure a sustainable future and the continued delivery of our services.

All of this would not have been possible without the valued support of our funders: the Greater London Authority, the Mayor’s Office for Policing and Crime and the Economic and Social Research Council, and the engagement and energy of youth service providers, partners, academics, and commissioners.

We are looking forward to working with all of you in the coming year!

GEORGIE PARRY-CROOKEProfessor of Social Research and Evaluation, London Metropolitan UniversityCo-Director, Project Oracle

STEPHEN BEDIAKODirector, The Social Innovation PartnershipCo-Director, Project Oracle

I am working hard to help every Londoner get a good start in life and to have the opportunity to participate fully in all that our great city has to offer. Our young people need to be seen as assets that we invest in rather than problems we need to solve. That is why I fully support Project Oracle’s on-going work to embed the use of evidence in our decision making for youth programmes.

It is imperative that we have a real understanding of what policies and programmes work, under what conditions they are effective and who they are benefiting. This type of impact measurement needs to be rigorous and consistent so everyone working and delivering in the sector has a similar understanding of what makes good evidence and how to use it to improve delivery of services to young people.

Project Oracle should not be in addition to our other work in this sector, rather the Project Oracle model should be the foundation of our work in this sector helping every programme, policy and decision be a better one. At the Greater London Authority we have also integrated the use of the Project Oracle model across our London Schools Excellence Fund to achieve these same results. In this annual report, you will see the great strides Project Oracle has made in the past few years and a vision for what it will become in the future.

BORIS JOHNSONMayor of London

WELCOME

3

ABOUT US

SUPPORT OFFERS

EVIDENCE COMPETITION

THOUGHT LEADERSHIP

CHILDREN AND YOUTH EVIDENCE HUB

CASE STUDIES

JOURNEY 2011-2014

VALIDATED YOUTH PROVIDERS

LOOKING AHEAD

4

6

8

10

12

14

18

20

22

CONTENTS

Copyright © Project Oracle • September 2014 • main photographs: Stephen Blunt, London Metropolitan University • design: Yogesh Taylor

4

ABOUT US

Project Oracle is a Children and Youth Evidence Hub which aims to improve outcomes for young people in London. We achieve this by promoting better evidence and the understanding of what really works across London’s children and youth sector. As we build the capacity of children and youth providers and funders to develop and commission evidence-based projects, we are creating an ecosystem in which evidence is widely gathered, used and shared.

Our unique model provides commissioners, practitioners and the public with confidence in project outcomes. This helps to ensure public money is directed to those activities that are most likely to deliver maximum positive impact for children and young people.

Using the Standards of Evidence

We have established a framework that describes what constitutes good evidence; it is made up of five Standards of Evidence that form the core of our Evidence Hub. Each Standard reflects how a project is currently gathering and using evidence. On the Project Oracle website we are compiling a growing resource of independently assessed and evaluated projects. We support these projects on their evidence journey, helping them to identify how they can move on to higher Standards of Evidence and working with them to develop rigorous evaluation systems. We do not assess the quality of the projects, but rather the quality of their evaluations.

Working with commissioners

Commissioners and funders are able to use the Standards to identify which organisations and projects have the greatest potential to make a difference. We work with large funders such as the London Schools Excellence Fund (LSEF) and Trust for London to help them embed evidence into their commissioning processes and decisions. With the Mayor’s Office for Policing and Crime (MOPAC) we have been developing an outcomes framework for the projects funded by the London Crime Prevention Fund.

5

4

3

2

1

We know why and how the change happened. This works everywhere.

We know why and how the change happened. This works elsewhere.

We believe there is a change caused by us. We can make this happen consistently.

We have seen there is a change.

KEY

Projects validations 2011-2013 Projects validations 2013-2014

We know what we want to acheive.

PROJECT MODEL & EVALUATION PLAN

52

74

12

3

4

SYSTEM READY

MODEL READY

EVIDENCE OF IMPACT

INDICATION OF IMPACT

STANDARDS OF EVIDENCE

5

5

4

3

2

1

We know why and how the change happened. This works everywhere.

We know why and how the change happened. This works elsewhere.

We believe there is a change caused by us. We can make this happen consistently.

We have seen there is a change.

KEY

Projects validations 2011-2013 Projects validations 2013-2014

We know what we want to acheive.

PROJECT MODEL & EVALUATION PLAN

52

74

12

3

4

SYSTEM READY

MODEL READY

EVIDENCE OF IMPACT

INDICATION OF IMPACT

5

4

3

2

1

We know why and how the change happened. This works everywhere.

We know why and how the change happened. This works elsewhere.

We believe there is a change caused by us. We can make this happen consistently.

We have seen there is a change.

KEY

Projects validations 2011-2013 Projects validations 2013-2014

We know what we want to acheive.

PROJECT MODEL & EVALUATION PLAN

52

74

12

3

4

SYSTEM READY

MODEL READY

EVIDENCE OF IMPACT

INDICATION OF IMPACT

STANDARDS OF EVIDENCE

Copyright © Project Oracle 2014

6

SUPPORT OFFERS YOUTH SERVICE PROVIDER SUPPORT

In order to make evidence work in practice, Project Oracle provides a variety of support services to key stakeholders in the children and youth sector. We work with youth service providers, commissioners and funders, and evidence experts to improve the way they engage with and share evidence and knowledge between them.

Children and young people’s services over the past decade have been calling out for change and innovation. Increased pressure on public budgets has amplified already persistent challenges in youth crime, employment and education. “Doing more with less” has become the expected norm. This underlines the need to track the impact of projects to effectively direct funds towards those that really work.

New methods for translating research and evaluation into effective, evidence-based practice need to be developed and adopted. These methods need to include mutual benefits for policymakers, service commissioners and researchers as well as for the providers of children and youth projects. This is where Project Oracle comes in.

Making evidence accessible

We are excited about evidence as a tool for understanding and communicating impact; and we are passionate about supporting the children and youth sector to strengthen, grow and innovate. We hope to support all stakeholders in making the evidence journey easier, more accessible and fun!

This, of course, doesn’t mean losing sight of the goal. Ultimately we want to boost the capacity of the sector by ensuring children and youth projects are able to track their outcomes, whilst also increasing the ability of commissioners to use this track record of impact to inform how public money is spent. We want to develop an evidence agenda that is practical and targeted at improving the lives of the capital’s children and young people.

Project Oracle delivers training and support to providers who want to better understand how to develop appropriate evaluation approaches to demonstrate their impact and improve the effectiveness of their projects.

We have developed a series of hands-on workshops, each focussed on building projects’ evidence in line with the Standards of Evidence. These include trainings on: Theory of Change, Evaluation Planning & Tools, Analysing & Reporting, and Understanding Comparisons in Evaluation.

We also offer on-going support to all projects working towards validation. Through bespoke support packages we recognise that the projects who come to Project Oracle differ in their knowledge and understanding of evidence and impact.

I really enjoyed how hands-on Project Oracle’s training is. Developing our Theory of Change and evaluation plan meant we could embed evidence into our project from the start. As a busy teacher and leader, I have appreciated the team’s support on the phone, by email and at training and networking sessions throughout the year. I look forward to working with them again in future.

ANUPAMEYA JAINTri-borough Alternative Provision Leader of

Modern Foreign Languages at The Bridge AP Academy

“

“

7

COMMISSIONER & FUNDER SUPPORT

EXPERT & ACADEMIC SUPPORT

Project Oracle offers a number of services to Commissioners and Funders. Our very popular Seminar Series runs twice a year and provides commissioners and funders with an in-depth understanding of how to build outcomes based commissioning processes.

We also work with Commissioners and Funders to design bespoke outcomes frameworks for their grantees. These frameworks improve accountability, inform commissioning, show the impact of funding and resources and further contribute to an overall evidence base.

Project Oracle has several approaches to engaging academics and experts. Our most well-known offer are the research placements in which we match up Masters and PhD students with children and youth providers who need assistance in building capacity for evidence and evaluation.

In addition, we host an expert panel in which accomplished academics can find several ways to work with Project Oracle. These include leading seminars in their areas of expertise, presenting their work at our events, joining in funding opportunities and reviewing the robustness of children and youth projects’ evidence at Standard 3 and above.

It was a really useful seminar series, facilitating us in sharing our experience and learning. I enjoyed it more than I thought I would! The seminars provided me with further ideas on how to take an evidence-based commissioning process forward within my organisation.

MONICA GOLDINGHead at Premier League Charitable Fund

““

Our researcher’s contributions have been vital to building the evidence of ourBuild-It project. Her time and expertise have allowed us to compile the evidence we need to expand our work.

DIMITRIOS TOUROUNTSISHead of Learning at London Youth

“ “

8

EVIDENCE COMPETITION

Our goal is to showcase the effort of providers in the children and youth sector in London. We seek to create a culture where “what works“ is shared and supported by a broad community of stakeholders including commissioners, youth providers, academics, policymakers and funding bodies.

Project Oracle and its supporters offer financial incentives to London’s children and youth projects each year as part of an Evidence Competition. The competition does not judge which projects are good or bad, only whether they are making good use of the evidence of their work. The main criteria for this is that they have a consistent and rigorous process to show and share their impact.

This year’s competition

This year, the categories were based on which Standard of Evidence projects were working towards. Over £20,000 is being awarded to children and youth projects which have excelled at moving through the Standards of Evidence to Standards 2 and 3. We were extremely pleased with the calibre of entries in the 2014 competition, which demonstrated that rigour can be obtained even on limited budgets.

We have seen the biggest movement across Standards since our inception with fourteen projects using the competition to work towards Standard 3 validation. This is a feat to be celebrated; each project has developed innovative ways of demonstrating the rigour and impact of their work. We are proud to highlight the impressive work of the seven winners on this page.

STANDARD

3

STANDARD

2

We believe there is a change caused by us. We

can make this happen consistently.

We have seen there is a change.

Best completed evaluation

Best submitted evaluation

plan

2014 COMPETITION WINNERS

9

TEENS & TODDLERS is the only youth development project in the UK designed to raise the aspirations of young people by pairing them as a mentor and role model to a child in a nursery who is in need of extra support. They measure the impact of the project on both the teenagers and the toddlers taking part. The project has been validated based on the evidence of impact on the toddlers. A matched comparison group study showed that those toddlers taking part showed significantly greater improvements in some of the key outcomes compared to their peers in the control group.

In 2012 a randomised control trial on the impact of the project on teenagers found no impact on outcomes around reducing teenage pregnancy, however it did indicate a positive impact on self-esteem. The Teens and Toddlers team have refined the project and are now undertaking a comparison group study on the impact on the teenagers’ attendance and attainment at school which will report in early 2016.Evidence summary: www.project-oracle.com/projects/info/teens-and-toddlers

MYBNK develop and deliver financial education for 11-25 year olds, in schools and youth organisations. Working collaboratively with young people, they have created high impact and high energy financial education programmes on saving, budgeting, survival money management, and what banks do with your money. The evaluation compared the knowledge, skills and attitudes around money of young people participating in MyBnk programmes versus those in a control group and found significant differences in all of these measures.Evidence summary: www.project-oracle.com/projects/info/mybnk

DANCE UNITED positively changes the lives of marginalised and hard-to-reach people through high-quality dance training and performance. Based on an indication of positive impact from a pilot programme, Dance United has partnered with King’s College Clinical Academic Group to undertake a randomised control trial which will test indicators related to improved self-confidence, self-efficacy, resilience and optimism; improved interpersonal relationships and friendships and improved physical health.Evidence summary: www.project-oracle.com/projects/info/dance-united-mental-wellbeing-project

CATCH 22’s goal is to deliver social benefit by turning chaotic lives around. The Engage in Education programme is an evidence-based, in-school intervention that addresses the risk factors associated with exclusion, including hidden communication needs, and aims to improve attendance and attainment. Catch 22 partnered with the University of Cambridge and is undertaking a cluster randomised control trial across 36 schools to test the impact of the intervention.Evidence summary: www.project-oracle.com/projects/info/the-london-education-and-inclusion-project

OASIS YOUTH SUPPORT is a youth violence intervention based at a London hospital Emergency Department which aims to reduce the number of young people returning to the Emergency Department as a result of violence and to increase the Emergency Department is contribution towards a public health approach to violence in two boroughs with considerably high levels of deprivation. Evidence summary: www.project-oracle.com/projects/info/oasis-youth-support

LONDON YOUTH’s Hindleap Warren Outdoor education programme challenges young people, encouraging and enabling them to reach out beyond that which is familiar and comfortable. The sense of adventure it instils and the team-work it inspires, help young people come together, win new confidence and gain invaluable education for life.  Evidence summary: www.project-oracle.com/projects/info/hindleap-warren-outdoors-education-centre

THE WINCH Promise Academy is a pilot project bringing together mental health, education, and social interventions to help young people aged 11-25 flourish and succeed despite poverty or disadvantage. The Academy supports young people to improve their life chances, to succeed within education and employment, to demonstrate enterprising behaviour and to have increased mental and physical wellbeing, aiming to build the resilience and character strength that young people need to succeed.Evidence summary: www.project-oracle.com/projects/info/promise-academy-pilot-at-the-winch

2014 COMPETITION WINNERS

10

THOUGHT LEADERSHIP

Project Oracle has contributed significantly to shaping the agenda on the use of evidence in London’s children and youth sector. This has helped to position us as a thought leader in all things impact related in the sector. We are proud to have made a meaningful impact in such a short time.

Synthesis studies

Part of what positions Project Oracle as a thought leader are our synthesis studies which summarise and share what evidence exists in a specific issue area. In an accessible way, these studies review evidence-based projects from inside and outside the Project Oracle structure to help the children and youth sector get an overview of effective approaches.

Together, the studies build a library of synthesised evidence to help providers in designing projects and commissioners in funding them. We have launched three new studies this year: “Gangs, Youth Violence and Crime”, “Sport and Reducing Youth Crime” and “Education, Employment and Training”. We have three more in the pipeline over the next year and look forward to sharing them.

In the news

Our achievements have not gone unnoticed. We have received extremely positive news coverage from mainstream press, including features in The Guardian and The Financial Times that highlight the achievements of both Project Oracle and the projects who have worked with us.

Recently we released four short films as part of our “How Can We Make Evidence Easier” series developed in partnership with New Philanthropy Capital (NPC). It was great to capture the evidence experiences of children and youth projects, funders and commissioners as well as experts. You can watch the films on Project Oracle’s youtube channel.

DR SIMON MCMAHON(PROJECT ORACLE)

DR SIMON MCMAHON (PROJECT ORACLE)AND DR JYOTI BELUR (UCL)

DR JYOTI BELUR (UCL)

Gangs, Youth Violence and Crime

Examines twelve programme evaluations aimed at reducing gang and youth violence in the London area.

Sport and Reducing Youth Crime

Considers evaluations of sports-based youth interventions in London and finds different types of evidence claiming that sport can indeed reduce youth violence and crime.

Education, Employment and Training

Reviews the evidence of education-based interventions working with young people who are not in education, employment or training in order to reduce violence.

11

PROJECT ORACLE IN THE NEWS

CAROLINE BOSWELLHead of Education and Youth

Greater London Authority

TIM HARFORDAuthor, Columnist for the Financial Times and presenter of Radio 4’s “More or Less”

“

““

“ Project Oracle’s approach can help inform how best commissioners target and shape programmes in an increasingly challenging funding environment.

The good news is that help is available. There’s Project Oracle, for example, a new London-focused outfit that aims to support programme providers in gathering useful evidence about what’s working, while also educating the people who commission such programmes that it’s important to ask for evidence.

David Pritchard, Head of Measurement and Evaluation at NPCBen Schiller, Copy Writer at Co.Exist

12

CHILDREN AND YOUTH EVIDENCE HUB

An Evidence Hub is a dynamic ecosystem which supports the use of research and evidence to provide insight into which initiatives and approaches are having the greatest impact on desired outcomes. The Project Oracle approach is built upon our understanding that to improve evidence, it is necessary to develop a system which not only recognises rigour, but is accessible and practical at the same time.

Making the ecosystem work

The Project Oracle Children and Youth Evidence Hub increases the generation of evidence by children and youth projects, whilst simultaneously increasing the usage of evidence by commissioners and funders. We act as a base through which evidence is shared across the three stakeholders in this system: children and youth service providers, commissioners and funders, and academics and experts. Each group contributes and gains from the system, making it sustainable and useful for all.

As each of the three stakeholders participates in the evidence ecosystem, it becomes more expansive and robust. The children and youth sector generates better evidence, uses better evidence, and shares this evidence widely. With access to this increasingly rigorous and large evidence base, projects can deliver better services, which leads to improved outcomes for children and young people.

Project Oracle’s role is to ensure that the ecosystem works smoothly, that evidence is effectively produced, reviewed by experts and put into practice on the ground. We truly enjoy and take pride in drawing this evidence together and are passionate about making evidence work in practice.

YOUTH PROVIDERS

GAINCONTRIBUTION

GAIN

GAIN

CONTRIBUTION

CONTRIBUTION

FUNDERS AND COMMISSIONERS

ACADEMICSAND EXPERTS

IMPROVED YOUTH

OUTCOMES

Support to providers to

articulate and understand their offer

better

A place to showcase their work

Better understanding of projects’ practical

limitations

Expert information on

improving services

Support to articulate and understand their offer

better

Expert information on

improving services

An understanding

of emerging and innovative work and the

evidencefor it

Support to grantees in

building their evidence and

evaluation capacity

Confidence in the effectiveness of

their work

A shared understanding

of expected outcomes with

grantees

Shared information on their project’s

impact

BETTER EVIDENCE

GENERATED

BETTER EVIDENCE

USED

SHARED EVIDENCE

BASE

Real data for researchers to

assess what works

PROJECT ORACLE ECOSYSTEM

13

YOUTH PROVIDERS

GAINCONTRIBUTION

GAIN

GAIN

CONTRIBUTION

CONTRIBUTION

FUNDERS AND COMMISSIONERS

ACADEMICSAND EXPERTS

IMPROVED YOUTH

OUTCOMES

Support to providers to

articulate and understand their offer

better

A place to showcase their work

Better understanding of projects’ practical

limitations

Expert information on

improving services

Support to articulate and understand their offer

better

Expert information on

improving services

An understanding

of emerging and innovative work and the

evidencefor it

Support to grantees in

building their evidence and

evaluation capacity

Confidence in the effectiveness of

their work

A shared understanding

of expected outcomes with

grantees

Shared information on their project’s

impact

BETTER EVIDENCE

GENERATED

BETTER EVIDENCE

USED

SHARED EVIDENCE

BASE

Real data for researchers to

assess what works

PROJECT ORACLE ECOSYSTEM

Copyright © Project Oracle 2014

14

An interview with Stefan Imeson, Healthy relationship Project Worker at Working with Men; Sandra Vacciana, Programme Manager at Tender Education and Arts; and Anita Bhardwaj, Programme Manager at Women and Girls Network

Why did you come to Project Oracle?The Bridge Project is born out of a consensus that the voluntary sector needs a gendered approach to tackle relationship violence amongst young people. It emerged as a partnership between Working with Men, Women and Girls Network and Tender. Project Oracle was commissioned by the Bridge Project’s Funder, Trust for London, to embed evidence into the project from the very start.

What has Project Oracle supported you to do?After attending Project Oracle’s Theory of Change training and being supported by a researcher for three months, we designed an accessible evaluation framework across the three organisations. We chose the key outcomes from our Theory of Change for which to measure distance travelled.

What are the benefits and challenges of using evidence?Incorporating evaluation into the project was at times challenging. School timetables meant that in some locations the intervention had to be shorter than initially hoped and one of the survey questions was misunderstood by the young people; but with structures and tools in place to track our impact, the Bridge Project will continue to learn and improve. It is great to see the record of the outcomes we achieve with young people, such as their attitudes to relationship violence and their ability to recognise emotional and psychological abuse.

What are the next evidence steps for your organisation?We hope to offer 9 more projects this school year, continuing our impact assessment as we go. We would also like to compare the results of different models of delivery.

Find out more: www.tender.org.uk

BRIDGE PROJECTStandard 1 Case Study: A youth partnership project promoting healthy relationships through drama & education to prevent domestic & sexual violence in the lives of young men and women aged 11-14.

REACH

560students reached and trained

50 teachers

IMPACT

83%of young people participating in the

project said that they now know where to get support or advice on

healthy relationships

Project Oracle’s support helped us to confront challenges. They encouraged us to be ambitious with our evaluation, whilst respecting our staff’s knowledge of the realities of delivering a project on the ground.

SALLY MEHTAOperations Director at

Working with Men

“ “

15

An interview with Zenobia Talati, Promise Worker atThe Winch

How did you get involved with Project Oracle?Promise Academy, a pilot project by The Winch, attended a Project Oracle workshop on Theory of Change. We were impressed that, through the workshop, we were able to map out our Theory of Change. Since being validated at Standard 1 in November 2013 we have put our evaluation plan into practice: in August 2014 we received Standard 2 validation for the evidence collected so far.

How has Project Oracle helped you to develop your evidence collection?By taking advantage of Project Oracle’s research placement offer, we were able to learn the key elements of data gathering and analysis. From this we were able to identify the right tools and models for our project.Our intervention is based on Adolescent Mentalisation-Based Integrative Treatment (AMBIT) – a model of intervention used for working with young people with complex needs. As a consequence, we decided to use the Adolescent Integrative Measure (AIM) survey as a tool of measurement as it is part of the AMBIT toolbox. We also track the young people’s character strengths.

What are the benefits and challenges of using evidence?Gathering evidence is not always easy, but we find submitting it to Project Oracle very straight forward. We wanted to find out what works through evidence anyway, and validation through Project Oracle means we receive more feedback on how to improve. External validation feels good, too – it is motivating to know we are meeting standardised criteria – and at Project Oracle’s events we see that we are not struggling alone.

What are the next steps for your project’s evidence journey?We want to continue our evidence journey and are assessing if we could set up a comparison group in order to submit for a Standard 3 validation. We are committed to showing others that this model works and to improving it along the way.

Find out more: www.thewinch.org

THE WINCHStandard 2 Case Study: The Promise Academy pilot programme brings together mental health, education, and social interventions to help young people, aged 11-25 flourish and succeed despite poverty or disadvantage.

Through Project Oracle we are part of a network of people who want their work to be evidence based - not just for money and funding but because we want to make a difference.

PAUL PERKINSCEO at The Winch

“ “REACH

The Winch works closely with

400children and young people in

Camden and bordering boroughs and reaches around 1,200 children

and young people each year

IMPACT

24young people receiving targeted support by the Promise Academy

pilot have shown substantial decreases in their AIM survey scores

16

CHANCE UKStandard 3 Case Study: An early intervention charity mentoring children aged 5-11 with behavioural difficulties who may be at risk of criminal offending behaviour later in life.

An interview with Christine Hatt, Quality and Evaluation Manager at Chance UK

How did you come to Project Oracle?Project Oracle first contacted us when they were designing their Standards of Evidence; we gave them feedback on the standards from a provider organisation’s point of view. In October 2013, Chance UK was one of the first organisations to be validated at Standard 2 and we won a prize in the 2013 Evidence Competition.

What support does Project Oracle offer you?Chance UK sees great value in Project Oracle as an organisation where small charities at similar stages in their evidence journey can exchange ideas and jointly problem solve.

What are the benefits and challenges of using evidence?For a relatively small charity like Chance UK, building a rigorous evidence base can be a challenge. Through the support and networking that Project Oracle has offered us, we have been able to develop our network within the evaluation sector. The technical support Chance UK has received from Social Research Unit at Dartington to help us set-up a sophisticated Randomised Controlled Trial (RCT) has been invaluable. Chance UK use the Strengths and Difficulties Questionnaire as our key evaluation tool. We consistently ask referrers, children, their parents and their Chance UK mentors to fill in the questionnaire in order to track outcomes. Our efforts have paid off and we find it valuable to be able to quote clear percentage improvements to funders and to be sure that what we are doing has a positive impact on the children we work with.

What are the next steps for your project’s evidence journey?Over the next two years Chance UK and Dartington will carry out our ongoing RCT, sponsored by a Realising Ambition grant from the Big Lottery Fund. Once the analysis is done we will submit it to Project Oracle for further stages of validation. The results of the trial will further help us to understand and build upon our areas of strength and help us to further improve our work with children.

Find out more: www.chanceuk.com

REACHOver

180children currently supported in

London across 5 boroughs – even more are reached through Chance

UK’s franchise model

IMPACT

70%of children in the Islington

programme finish mentoring on an SDQ score under 16 – meaning they no longer display an abnormal level

of behavioural difficulties

It is helpful to have something to measure yourself against. The Project Oracle Standards of Evidence act as a prompt to move forward.

ROSS ADAMSSenior Programme Manager -

Hackney, Islington & Girls Programme at Chance UK

““

17

An interview with John Milton, Director at the Archway Project

Why did you want a Research Placement?We had already gone through the process of developing a Theory of Change and a rough evaluation plan with Project Oracle. We wanted to continue developing our evaluation and evidence gathering journey. When we heard about the research placements we thought it was a unique and invaluable opportunity to dig deeper into what we were doing and how we were doing it! We were matched quickly with a researcher, Natalie, from UCL.

How has having a research placement helped you?When reviewing our evidence base, Natalie saw that our data was too incomplete to draw strong conclusions. With Natalie’s support, we developed a new evaluation plan to improve the consistency of our data. Now we track students’ confidence levels and grades to better understand if the organisation is delivering the right type of intervention. Qualitative interviews with service users provide additional insights to how our project benefits young people at risk.

How did you get matched with your researcher, Natalie?We came across Natalie through a Project Oracle research placement fair, where project heads and researchers could meet each other face to face to see if they would like to work together. Nathalie was a great fit for us and agreed a work plan and attended an induction event. Throughout the placement, Archway and Natalie have had access to Project Oracle for additional guidance. We have both found it useful to access resources on how to develop and roll out a new evaluation plan.

What’s next for your project? Archway is committed to keeping up our data collection and we hope to submit our evidence to Project Oracle for validation at Standard 2. We can see that our project has a range of positive impacts, such as reduced truancy, reduced social isolation and increased confidence. It will be great to show this impact more clearly in numbers.

Find out more: www.archwayproject.org

THE ARCHWAY PROJECTResearch Placement Case Study: A youth project providing accredited courses to divert young people aged 11-19 from involvement in the criminal justice system and to improve their attainment at school and in work.

Being removed from a highly structured university environment as well as academic language is a valuable experience for researchers who want to go into practice. The research placements are implementation-focused, designed in collaboration with the researcher, Project Oracle and the project: this means that the work plan is realistic and the organisation is able to communicate clearly what is feasible on the ground.

NATALIE ARNETTMSc student in Crime Science at UCL

““

REACH

Across

5London borough plus north Kent

Working with

350 young people per year

18

JOURNEY 2011-2014

167GUESTS AT ACTION DAY

EVENT IN CITY HALL2014

33RESEARCH PLACEMENTS

2011-2014

25EVIDENCE COMPETITION

ENTRIES2014

Over 70% of providers with a

research placement progressed to the next Standard of

Evidence

14for

Standard 311for

Standard 2

WORKING WITH

LONDONPROVIDERS

(Project delivery by borough)

19

145VALIDATED PROJECTS

2011-2014

481TRAINED PROVIDERS

2011-2014

126Validated at Standard 1 16

Validated at Standard 2

3Validated at Standard 3

16EVIDENCE CHAMPIONS

2014

32 funders and commissioners over the past 2 years, including representatives from the GLA, MOPAC, TFL,

the Met Police and local authorities including

Westminster, Hackney, Islington

an increase of

280% between 2013

and 2014

WORKING WITH

LONDONPROVIDERS

(Project delivery by borough) Projects validated by Project Oracle

20-29 Projects30-39 Projects40+ Projects

20

VALIDATED YOUTH PROVIDERSOrganisations with projects validated byProject Oracle

Beatbullying

Belong London

Brathay Trust London

Bridge Project

Bridging A Gap Ltd

British Film Institute

Broadford Primary

Brunel University

CalibreMinds

Capital Classics & Classics for All

Capital Conflict Management

Catch22

CfBT Education Trust

Coin Street Community Builders

Cricket Foundation

Cuckoo Hall Academies Trust

Dance United

Design Shed

Eastside Young Leaders’ Academy

Elklan Training Ltd

Enhancing Achievement

Family Action

Field Studies Council

Gladesmore Community School

Glenthorne High School

Grafton Primary Cluster P4C Project

Greater London Authority

Haringey Youth Offending Service (x2 projects)

HM YOI Feltham (x2 projects)

Holy Family Catholic School

Hope UK

Khulisa UK

Greenhouse

MyBnk

Teens and Toddlers

Active Communities Network

Ampersand Learning Ltd

Apps for Good, CDI Europe

Chance UK

Dance United

Fight for Peace

London Bubble Theatre Company

London Youth (x2 projects)

Growing Against Gangs and Violence

Oasis UK

Only Connect

St Giles Trust

The Boxing Academy

The School & Family Works

The Winch

Working With Men

198 Contemporary Arts and Learning

Achievement for All 3As

Action for Children

Apps for Good, CDI Europe

Arc Theatre

Archway Project

ARK Maths Mastery

Ashford Place

STANDARD 3

STANDARD 2

STANDARD 1

ACEd

ACEdACEdACEdEmACEdACEd

ACEd

ACEdACEd

ACEdACEd

ACEdACEd

ACEdACEdACEd

ACEdACEd

ACEd

ACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPS

AC

AC

ACAC

ACACAC

AC

ACACACAC

AC

AC

AC

AC

AC

AC

ACEdEmEFHACEd

ACEd

ACEdACEd

ACEdACEdACEdACEd

ACEdACEd

ACEdACEd

ACEdACEd

ACEdACEd

ACEdACEd

ACEdACEd

ACEdACEdACEdACEdACEdACEd

ACEd

ACEdEmEFHOPS

ACEdEmEFHOPS

A

A

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEd ACEdEmEFHOPSACEdEmE ACEdEmEFHOP

ACEdEmEFHOP

ACEdEmEFHOP

ACEdEmEFHOPACEdEmEFHOP

ACEdEmEFHOP

ACEdEmEF

ACEdEmEF

ACEdEmEFACEdEmEFACEdEmEF

ACEdEmEF

ACEdEmEFHOPS

ACEd ACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmE

ACEdEmE

ACEdEmE

ACEdEmE

ACEdEmE

ACEdEmEFH

ACEdEmEFH

ACEdEmEFHACEdEmEFH

ACEdEmEFH

ACEdEm

ACEdEmACEdEmE

ACEdEmE

A

AC

AC

AC

ACAC

AC

AC

ACAC

ACEdEmEF

ACEdEmE

ACEdEmEA

A

ACEdEmEFHO

ACEdEmEFH

ACEdEmEFHACEdEmEFH

ACEdEmEFH

ACEdEmEF

AC ACEdEmEFHOP

21

Royal Greenwich Teaching School Alliance (RGTSA)

Royal Society of Chemistry

Safer London Foundation (x2 projects)

Save the Children

Southbank Mosaics

Southwark Schools Learning Partnership (SSLP)

St James the Great

St Michael’s Catholic College

STEMNET

Street League

Tender Education & Arts (x2 projects)

The Archbishop Lanfranc School

The Boys Clubhouse

The Bridge AP Academy

The Complete Works Ltd

The Compton School

The Prince’s Trust

The School & Family Works

Tricycle Theatre

Ukie

University of Oxford

Voyage Youth

WeSayWePay

Young Hackney

Arts

Crime

KEY Education

Employment

Housing

Other

Enterprise

Family

Poverty

Sports

2014 Competition Entry

London Schools Excellence Fund project

King’s College London

Lambeth Teaching Schools’ Alliance

Lampton Academy

Leap Confronting Conflict

Let’s Think Forum

Lewisham Neighbourhood Community Safety Service

Lewisham Youth Offending Service

Leyton Sixth Form College

London Borough of Barking and Dagenham

London Borough of Camden

London Borough of Southwark

London Early Years Foundation

London Transport Museum (x2 projects)

London Youth

MAC-UK

Metropolitan Police (x4 projects)

Mulberry School for Girls

Museum of London

Myscience.co Limited

National Literacy Trust

New Horizons Federation

NSPCC

Off Centre

OLMEC

PFEG

Pickhurst Infant School

Present Moment

Primary Advantage

Principles in Finance

Queen Mary University of London

Ridgeway Primary School

A ACEdEmEACEd ACEdEmEFH ACEdEmEFHOP ACEdEmEFHOPS

ACEd

AC ACEdEm ACEdEmEFHOACEdEmEF ACEdEmEFHOPS ACEdEmEFHOPS

ACEdACEd

ACEdACEd

ACEd

ACEd

ACEdACEd

ACEdACEd

ACEd

ACEdACEdACEdACEdACEd

ACEd

ACEd

ACEd

AC

AC

AC

AC

ACAC

ACAC

AC

AC

ACEdEmEFHOP

ACEdEm

ACEdEm

ACEdEm

ACEdEm

ACEdEm

ACEdEmEFHO

ACEdEmEFHO

A ACEdEmEFH

ACEdEmEFHACEdEmEF

ACEdEmEFACEdEmEF

ACEdEmEF

ACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPS

ACEdEmEFHOPSACEdEmEFHOPSACEdEmEFHOPS

ACEdACEdACEd

ACEd

ACEd

ACEdACEdACEd

ACEd

ACEdACEdACEd

ACEdACEdACEd

ACEdACEd

ACEdACEd

ACEdACEd

ACEdACEdACEdACEd

ACEd

AC

AC

AC

ACACAC

AC

AC

AC

AC

ACEdEm

ACEdEm

ACEdEm

ACEdEm

ACEdEm

ACEdEm

ACEdEmEFHO

ACEdEmEFHOPSA

A

ACEdEmEFHOP

22

In the year ahead, we will continue to support an ecosystem in which children and youth projects of all sizes have the capability to design and deliver projects with proven outcomes. We aim for a children and youth sector in which commissioners and funders appreciate and reward evidence-based projects, and where academics and experts provide rigorous and practical evaluations that contribute to everyone’s understanding of what works, for whom and under what circumstances.

Our next step is to move Project Oracle from an exclusively grant funded project to a long-term sustainable initiative. To that end we have established Project Oracle as a not-for-profit company limited by guarantee, which will allow Project Oracle to build strong and lasting ties in our community. We want to set up a governance structure that is transparent and continues to give the children and youth sector a clear view of our aims and priorities.

Leading by exampleWe are pleased to present Project Oracle’s Theory of Change on this page and we look forward to developing it further. We have also received our first external process evaluation, commissioned by Nesta and carried out by the institute for Employment Studies, which is available on the Project Oracle website. It was useful to get an independent review of which services are working well, why this might be, and where we need to improve. Moving forward, we will be implementing the changes indicated in our evaluation as well as embarking on a new independent evaluation to assess the impact of our work across London. Our evaluation also helps us to consider and solve our various challenges including more meaningful outreach across London, improved engagement with academics and increased buy-in from funders and commissioners.

We look forward to working with all of you in the years to come as we use evidence and evaluation to build a better future for London’s children and young people.

AMY M. HOCHADELSenior Manager, The Social Innovation PartnershipProgramme Manager, Project Oracle

LOOKING AHEAD

GENERATING EVIDENCE

SHARING EVIDENCE

USING EVIDENCE

Evidence competition

for youth providers is run annually

Youth providers are

given evaluation

training and support

Researchers are placed with youth

providers to support their evaluations

Youth providers’

evidence is validated

against the Standards of Evidence and feedback on increasing

robustness of evidence is provided

Existing evidence is synthesised

into thematic studies

Validated evidence is

collated and synthesised in

one online Evidence Hub

Youth providers are incentivised

to gather evidence

Youth providers have an

increased capacity to

evaluate their projects

Youth providers have an increased understanding of the robustness of their evidence and

how they can improve this

There is a single point of reference

for evidence relating to

children and youth

interventions

Commissioners and youth providers

design projects based on

evidence of what works

IMPROVED OUTCOMES

FOR CHILDREN AND YOUNG

PEOPLE IN LONDON

PROJECTS DELIVER BETTER

VALUE FOR MONEY

Increased awareness of

which projects work and

which don’t

There is a proliferation of

projects that really work and a drop in those

that don’t

Youth providers are more able

and more driven to produce

rigorous evaluations

There is more high quality

evidence

There is a set of standards

for what constitutes

good evidence as a project

develops

Commisioners take part in training on

evidence-based commissioning

Commissioners are supported to

build evidence into their

commissioning processes

(e.g. Theory of Change, outcomes

frameworks)

Commissioners increasingly fund

evaluation to generate new

evidence where it doesn’t exist

Commissioners increasingly expect more

robust evidence from youth

providers they fund

Commissioners increasingly

draw on existing

evidence so commission

projects most likely to work

Commissioners are more able to use evidence in commissioning

Agreed Standards of Evidence are

published

Commissioners have increased conviction that

evidence is a vital part of the

commissioning process

1

1

2

2

3

3

4

4

5

5

6

6

7

7

8

8

9

9

Assumptions

Project Oracle Standards of Evidence are recognised as the leading means of assessing the quality of evidence in the children and youth sector.

Competition criteria are designed to incentivise rigorous evaluation.

Student researchers have appropriate expertise.

Validation process enables ongoing improvements in evidence gathering.

Commissioners currently struggle to use evidence in commissioning processes because they can’t compare evidence between different projects in a structured way.

Youth providers are able to access sufficient resources to undertake more rigorous evaluation.

There is buy in across the commissioning organisation. Staff at appropriate levels of seniority are targeted.

Project Oracle’s Evidence Hub is recognised as the leading source of evidence on children and youth interventions in London.

The format of the evidence is digestible, focuses on learning, and youth providers and commissioners are able to transfer this to their project design.

THEORY OF CHANGE

23

GENERATING EVIDENCE

SHARING EVIDENCE

USING EVIDENCE

Evidence competition

for youth providers is run annually

Youth providers are

given evaluation

training and support

Researchers are placed with youth

providers to support their evaluations

Youth providers’

evidence is validated

against the Standards of Evidence and feedback on increasing

robustness of evidence is provided

Existing evidence is synthesised

into thematic studies

Validated evidence is

collated and synthesised in

one online Evidence Hub

Youth providers are incentivised

to gather evidence

Youth providers have an

increased capacity to

evaluate their projects

Youth providers have an increased understanding of the robustness of their evidence and

how they can improve this

There is a single point of reference

for evidence relating to

children and youth

interventions

Commissioners and youth providers

design projects based on

evidence of what works

IMPROVED OUTCOMES

FOR CHILDREN AND YOUNG

PEOPLE IN LONDON

PROJECTS DELIVER BETTER

VALUE FOR MONEY

Increased awareness of

which projects work and

which don’t

There is a proliferation of

projects that really work and a drop in those

that don’t

Youth providers are more able

and more driven to produce

rigorous evaluations

There is more high quality

evidence

There is a set of standards

for what constitutes

good evidence as a project

develops

Commisioners take part in training on

evidence-based commissioning

Commissioners are supported to

build evidence into their

commissioning processes

(e.g. Theory of Change, outcomes

frameworks)

Commissioners increasingly fund

evaluation to generate new

evidence where it doesn’t exist

Commissioners increasingly expect more

robust evidence from youth

providers they fund

Commissioners increasingly

draw on existing

evidence so commission

projects most likely to work

Commissioners are more able to use evidence in commissioning

Agreed Standards of Evidence are

published

Commissioners have increased conviction that

evidence is a vital part of the

commissioning process

1

1

2

2

3

3

4

4

5

5

6

6

7

7

8

8

9

9

Assumptions

Project Oracle Standards of Evidence are recognised as the leading means of assessing the quality of evidence in the children and youth sector.

Competition criteria are designed to incentivise rigorous evaluation.

Student researchers have appropriate expertise.

Validation process enables ongoing improvements in evidence gathering.

Commissioners currently struggle to use evidence in commissioning processes because they can’t compare evidence between different projects in a structured way.

Youth providers are able to access sufficient resources to undertake more rigorous evaluation.

There is buy in across the commissioning organisation. Staff at appropriate levels of seniority are targeted.

Project Oracle’s Evidence Hub is recognised as the leading source of evidence on children and youth interventions in London.

The format of the evidence is digestible, focuses on learning, and youth providers and commissioners are able to transfer this to their project design.

THEORY OF CHANGE

KEY Activity

Assumption

Outcome

Core Outcome

Copyright © Project Oracle 2014

PROJECT ORACLE CHILDREN & YOUTH EVIDENCE HUB

WWW.PROJECT-ORACLE.COM@PROJECT_ORACLEINFO@PROJECT-ORACLE.COM

0207 148 6726

PROJECT ORACLE IS FUNDED BY

PROJECT ORACLE IS MANAGED BY

Copyright © Project Oracle 2014