Ongoing evaluation of the COASTAL project - Report 2

On-going evaluation of the COASTAL Project Report 2 – October 2011 (revised)

On-going evaluation of the COASTAL project

Report 2: October 2011

©Copyright Wavehill Ltd 2011 Page | 1

Acknowledgements

We would like to thank everybody who contributed to this report and the research upon which it is

based. In particular we would like to thank the COASTAL project staff and the many individuals who

were being supported by the project who took the time to speak to the Wavehill team. This

evaluation would not have been possible without each of those contributions.

Report prepared by: Richard Brooks, Director - [email protected]

Louise Petrie, Researcher

Quality assurance: Endaf Griffiths, Director

Client contact: Clive Prior, Regional Project Director - [email protected]

Version: Final (revised)




Contents

Summary of the report ................................................................................................. 3

1. Introduction and Context ....................................................................................... 9

1.1. The COASTAL project ......................................................................................................... 9

1.2. The evaluation ................................................................................................................. 11

1.3. Structure of the report ..................................................................................................... 11

2. Outputs, results, impacts and outcomes.................................................................12

2.1. Participation Outputs and Results .................................................................................... 12

2.2. Impacts and outcomes ..................................................................................................... 22

3. Views of Strategic Stakeholders; 2011 ....................................................................31

3.1. Work Programme............................................................................................................. 32

3.2. Progress ‘Work Star’ data is not being captured or used by all projects ............................ 32

3.3. An increased emphasis on the externalisation of social care services and independence

within participants ....................................................................................................................... 33

3.4. New Service Models – externalised Social Firms ............................................................... 34

3.5. The balance between social care services and employment services ................................ 34

3.6. Weak employment demand ............................................................................................. 38

3.7. Competition for ESF participants ...................................................................................... 38

3.8. Cross (Local Authority) border services ............................................................................ 39

3.9. Procurement of services and suppliers is complex ............................................................ 39

3.10. Collaboration and sharing of good practise................................................................... 39

3.11. A critical response to this report................................................................................... 40

Appendix 1: Updated baseline data ..............................................................................41

Appendix 2: Analysis of COASTAL project management data ..........................................43

Output and result definitions ....................................................................................................... 43

Analysis of Coastal participants.................................................................................................... 45




Summary of the report This is a summary of the October 2011 report of the on-going evaluation of the COASTAL project

which is being undertaken over a five year period and run parallel to the delivery of the project. The

evaluation is being carried out by the social research and evaluation specialists Wavehill.

Evaluation is an important component of publically funded projects, services and activities. It is used

to measure the efficiency and effectiveness of projects and programmes, to understand how policy

is being delivered to citizens, and to test if project teams are achieving what they are being funded

for. Evaluation identifies what is working well and provides a critique where strategy and delivery is

not effective.

The headline targets of the COASTAL project are currently under revision and awaiting WEFO

approval, but the proposed targets at the time of writing are (October 2011):

• 8,500 economically inactive participants to be engaged;

• 1,000 participants to move into sustainable employment (12%);

• 6,500 participants gaining a qualification (76%); and

• 8,100 participants gaining other positive outcomes (95%).

A possible funding risk to COASTAL and delivery agents

The evaluation has found that, in some cases, delivery agents have been slow to adapt into the

COASTAL focus on employability and employment outcomes, rather than the provision of a more

social care based model. This has quite significantly reduced the propensity of COASTAL to reach its

outcome and results targets. As a result funding may be at risk to the COASTAL project as a whole or

to individual delivery agents.

Outputs, results, impacts and outcomes to date

As of the end of June 2011, COASTAL has supported just over 2,194 (1,200; 2010) participants. This is

lower than the numbers we would have expected to have been engaged so far if the project is to

meet its participation targets of 8,500.

The lowest ‘results’ across COASTAL are in the number of participants progressing into employment,

only 37 from 2,194 participants. This is considered low as the original targets anticipated 32%

(2,870/9,020) finding employment and the proposed revised targets 11.8% (1,000/8,500). Clearly, at

the current rate of progress, far fewer participants will achieve employment outcomes than

anticipated when the COASTAL project was planned.

Essentially COASTAL must be able to demonstrate that participants increase their employability as a

result of inclusion and participation in the portfolio of COASTAL projects. This means that other

outcomes and results must be identified and recorded to demonstrate the benefits that COASTAL

brings to its participants. As things stand, it is hard to identify any progress in terms of participants

improving their employability or moving into employment outcomes based on the monitoring data

that is currently available. This is not to say there has been no progression, but there is little data

(evidence) to demonstrate the progress that has been made.




The 2010 evaluation report highlighted the lack of WEFO results and the need to provide evidence of

distance travelled, or soft outcomes. This is still the case in 2011 and, with so few WEFO results

being recorded, it is even more important to measure the softer benefits of COASTAL participation.

Work Star to record progression in COASTAL participants

The primary tool being used to measure progress towards ‘employability’ and employment is Work

Star. In 2010 this was selected by the COASTAL board as a regular way of capturing the change in

individual participants, and then grouping or aggregating the scores to show project and then

COASTAL progression.

The evaluation has found that the bedding in of this tool has not been particularly successful, in part

because there was not enough training and support in its use. A new focus should be placed on

using the Work Star in 2012, with suitable training to ensure correct and frequent use and Wavehill

providing the central COASTAL team with more support in analysing the data as it is collected.

The balance between social care and employment and employability in COASTAL projects

The balance between COASTAL as a social care project and COASTAL as an employment and

employability project needs to be reviewed.

ESF is primarily an economic programme that aims to raise the GDP of the convergence area in West

Wales through a range of economic interventions.

‘To create a high skill, knowledge driven economy, with full employment, a skilled,

adaptable workforce and responsive businesses, at the cutting edge of sustainable

development. The aim of Priority 2 being … Increasing employment and tackling economic

inactivity to raise levels of employment and economic activity, and secure higher

participation in the labour market.’ WEFO - West Wales and the Valleys Convergence

Programme Operational Programme for the European Social Fund 2007-2013

The COASTAL participants are in many cases a very long way from the labour market, and many have

a very low probability of finding mainstream employment opportunities in the short to medium

term. This places them in the category of not being able to work or not thinking about work. This

was illustrated using the graphic below in the 2010 report.

Figure 1: An illustration of the continuum of participants COASTAL will work with

Not thinking Self-reliance

about work In-work

Work ready

Participants at this point Participants at this point

need in-depth support require less support




Feedback from stakeholders in 2010 and 2011 suggests that the majority of those participants that

the COASTAL project is currently working with are towards the left side of the line in the graphic

Figure 1. This places a question about the propensity of the COASTAL participants to increase

economic activity and increase employment under the Priority 2 theme from which COASTAL is part

funded.

Further evidence for this comes from the proportion of COASTAL participants that are reaching the

ESF output and result targets of entering learning, gaining qualifications, finding employment and

gaining skills that increase employability, measured as ‘other outcomes’. The proportions of

participants reaching these results are very low, less than 3% progressing into qualifications, further

learning or employment as a proportion of all participants. This suggests that the large majority of

the COASTAL cohort still requires in-depth support and are probably attending some COASTAL

projects primarily for social care, rather than for employability.

Conclusion

COASTAL is an ambitious project that brings together partners and delivery agents from different

Local Authorities and silos of delivery. The ultimate aim is to create a paradigm shift in social care

services, where people with various disabilities can move towards more mainstream and

independent lives by becoming more employable and ultimately employed. There is much evidence

to suggest gaining employment includes personal therapeutic benefits as well as social and

economic status, but employment outcomes need to be made to realise these benefits. The

COASTAL project has many of the elements in place to be successful, but must now focus on

employability and employment. If it does not funding may be at risk as WEFO targets are unlikely to

be met.




Key findings and recommendations

This final section of the report will briefly draw together the key findings of the evaluation and

present the recommendations that are, accordingly, being made.

Recommendation 1

Issue

Employment outcomes are not being realised at a rate that will satisfy the WEFO contract. A greater

number of job opportunities need to be identified for COASTAL participants.

Evidence

• The employment rate within Coastal is below 2%.

Recommendation

It may be that COASTAL, as a collective or the individual delivery agents, need to extend and

emphasise the role of employment liaison officers, responsible for identifying businesses willing to

work with COASTAL participants. The role could be to work between the COASTAL teams and the

local employers to negotiate employment places and employment experience. This dialogue may be

increased initially through SETs (Specialist European Teams – WEFO).

Recommendation 2

Issue

Whilst there are differences between delivery agents, taken as a whole, COASTAL is not focussed

enough on employability and employment outcomes. This creates a funding risk to COASTAL as a

whole and also to individual delivery agents.

Evidence

• Analysis of outputs and results.

• Discussions with stakeholders.

Recommendation

The correct balance between COASTAL as an employment project and COASTAL as a social care

project needs to be agreed on between the partners as there are some significant differences in

opinion and potentially a risk to funding.




Recommendation 3

Issue

Work Star should be used pervasively across COASTAL to demonstrate soft outcomes from delivery

agents and progress towards employability to funders and partners.

Evidence

• Work Star is not being used as a regular progress measure by delivery agents.

• Only 16% of participants have had two Work Star interviews and 3% three interviews.

Recommendation

The Work Star needs to be used more frequently, with a regular and repeated cycle of assessment

taking place, and actions developed from this assessment. It is suggested that each participant

undertakes an assessment every 120 days (4 months) and that this data is analysed by Wavehill and

then sent as an information package to all delivery agents. The data will aggregate the progression

across all participants, and then for each of the ten COASTAL partners. This will then allow each

COASTAL partner to compare their data to the whole of the COASTAL project.

Recommendation 4

Issue

The core concept of COASTAL being about employability and employment of participants needs

emphasis.

Evidence

• Interviews with stakeholders identified a drift towards provision of care settings for participants.

Recommendation

Some ‘mission drift’ is noted amongst COASTAL stakeholders. This may be partly as employment

outcomes are, so far, very few amongst the COASTAL projects. However, the WEFO / ESF contract is

explicit in its requirement of a focus on employability and employment. It is also clear from

participant interviews, and speaking to stakeholders, that some COASTAL participants are not ready

for employment, and probably never will be. These participants require a social/day-care model, not

an employability employment model.




Recommendation 5

Issue

Social firms may be required to create employment opportunities for COASTAL participants inside an

Intermediate Labour Market model.

Evidence

• Low employment opportunities being found from private and public sector employers.

• Successful models such as FRAME and PACK-IT.

Recommendation

The partners may wish to visit social firms from within Wales, but also wider afield to examine how

social care, employability and employment and social firms can work together to provide service

models with employment outcomes. If the partners wish, Wavehill can suggest several options for a

COASTAL delegate to visit.

Recommendation 6

Issue

The participation rate of males is greater than for females and the revised (2011) targets anticipate a

37% Female to 63% Male ratio.

Evidence

• Males are 2.7 times more likely to participate in COASTAL than Females.

Recommendation

The issue of inclusion equality was discussed in the 2010 report and still requires some attention.

Why are males 2.7 times more likely to participate in COASTAL than females? The cause may be

systemic, in that for some reason the COASTAL project is more likely to appeal to males than

females, or it could be sociological, as parents and carers may feel that searching for employment is

a more male activity than female and so less females are presented to COASTAL for inclusion. The

COASTAL team should be aware that there is a gender based equalities impact assessment in place

for the Convergence ESF area.




1. Introduction and Context

This is the 2011 and second report for the on-going evaluation of the COASTAL project which will be

undertaken over a five year period alongside the delivery of the project. The evaluation is being

undertaken by the social research and evaluation specialists Wavehill1. This first section will provide

the context for the remainder of the report by introducing the COASTAL project as well as the

evaluation.

1.1. The COASTAL project

COASTAL is a £52million Regional Strategic Project covering the six Local Authority (LA) areas of

Bridgend, Neath Port Talbot, Swansea, Carmarthenshire, Pembrokeshire and Ceredigion, aimed at

the promotion of vocational guidance, employment, skills training and lifelong learning

opportunities for individuals who are currently economically inactive as a result of illness, disability,

(mental illness, learning disability, physical disability, sensory impairment) substance misuse

problems and/or the serious social disadvantage associated with the transition from long-term care

into adulthood. Bridgend and Ceredigion Local Authority partner areas were not active in service

delivery to June 2011; however, delivery of services has now started in both these areas.

In addition to the Local Authority teams, there are four ‘cross-regional’ partners within COASTAL,

taking on a dual role of both COASTAL partner and service delivery:

• West Glamorgan Council on Alcohol and Drug Abuse (WGCADA)

• Swansea Drugs Project

• ‘PRISM’ – The Mid & West Wales Alcohol & Drug Advisory Service

• Community Chaplaincy – HM’s Prison Swansea

COASTAL also offers support, advice and guidance to employers, in order to raise their awareness of

illness and disability issues and to assist them in engaging with and, hopefully employing, COASTAL

participants.

The headline targets of the COASTAL project are described in the table below. Note that the current

targets and the proposed revised targets have been included.

1 www.wavehill.com




Table 1: COASTAL project outputs and results; including original (old) targets and revised and

proposed (new) targets 2011.

Output Category Target (old) Target (new)

Total participants (all economically inactive) 9,020 8,500

Female participants 4,100 3,160

NEET participants 740 675

BME participants 225 180

Older participants 2,542 2,380

Participants with a work-limiting health condition

or disability 8,610 8,113

Employers assisted or financially supported 20 10

Result Category Target (old) Target (new)

Participants gaining qualifications 5,412 6,500

Participants entering employment - Economically

inactive and unemployed 2,870 1,000

Participants entering further learning -

Economically inactive and unemployed 1,763 2,750

Participants gaining other positive outcomes -

Economically inactive and unemployed 8,570 8,100

Source: COASTAL project proposed revised business plan 2011

Further information about the COASTAL project is available from the project’s website:

www.coastalproject.co.uk




1.2. The evaluation

1.2.1. Aims and objectives

Over its lifetime, the evaluation will seek to answer four questions:

• How have participants benefited from the support provided?

• How effectively has the project been delivered?

• How has the project changed the way in which support is provided in each area?

• What has been the cost-benefit and impact of the project?

This is the second of a number of reports that will be produced as the evaluation progresses. It

focuses on the perception differences between what stakeholders perceive their involvement in

COASTAL to be and the economic and employment focussed outcomes anticipated by the WEFO

funding under ESF Theme 1- Priority 2.

1.2.2. Fieldwork undertaken for this report

The report itself will be split into two parts. Firstly, this section includes interviews with 20 strategic

level staff and related stakeholders, generally heads or directors of social care services for local

authorities, with responsibility for their element of the COASTAL project.

The agreed method for the 2011 fieldwork is to create 40 case studies of COASTAL participants,

randomly selected to give an unbiased picture of the type of clients, what benefit they derive from

participation in COASTAL and the change in their personal propensity to find employment. These

will also be supported with around 20 interviews with delivery agent staff from 10 COASTAL delivery

agents. The timing of the 40 case studies and supporting interviews with delivery agent staff will be

the basis for the next evaluation report which will be produced in early 2012.

1.3. Structure of the report

The remainder of this report is structured as follows:

• Section 2 - discusses the outputs, results and impacts of the COASTAL project to date;

• Section 3 - considers the strategic stakeholders perceptions of the COASTAL project to date;

• Section 4 - draws together the key findings of the reports and presents a number of

recommendations.




2. Outputs, results, impacts and

outcomes

This section will explore the outputs, results and impacts of the COASTAL project to date (June

2011). Due to the affects of a weak economy and potentially less participants due to Work

Programme, COASTAL has made an application to reduce the number of outputs and results

expected from the COASTAL project. Reduction has been applied for and the board is awaiting

formal confirmation from WEFO. At the time of writing formal approval has not yet been received

by COASTAL, although informal approval has, and so these output targets are included for reference

only in this report.

Key findings

• The rate at which employment and employability results are being realised needs to increase

significantly if the targets set for the COASTAL project are going to be achieved.

• The COASTAL Project Board needs to consider options for increasing the focus on achieving

employment outcomes.

• To meet the new revised targets of 8,500 participants by December 2013 COASTAL needs to

work with 630 new participants per quarter. With relatively few participants moving out of the

COASTAL model into employment or learning and training this participation target is doubtful.

2.1. Participation Outputs and Results

The following table sets out the targets that have been set for the COASTAL project in terms of

outputs and results. It also notes the numbers achieved to date (as of 30th June 2011) and what

percentage of the target that represents. Definitions for the outputs and results can be found in

Appendix 2.




Table 2: COASTAL project outputs and results; targets achieved to date including both old and new

(2011) targets

Output Category to June

2011

Target

(old)

Target

(new)

% of target

(old)

achieved

% of target

(new) achieved

Total participants (all

economically inactive) 2021 9,020 8,500 22% 24%

Female participants 698 4,100 3,160 17% 22%

NEET participants 1977 740 675 267% 293%

BME participants 33 225 180 15% 18%

Older participants 150 2,542 2,380 6% 6%

Participants with work-limiting

health condition or disability 1121 8,610 8,113 13% 14%

Employers assisted or financially

supported 0 20 10 0% 0%

Result Category to June

2011

Target

(old)

Target

(new)

% of target

(old) achieved

% of target

(new) achieved

Participants gaining

qualifications 114 5,412 6,500 2% 2%

Participants entering

employment - Economically

inactive and unemployed 31 2,870 1,000 1% 3%

Participants entering further

learning - Economically inactive

and unemployed 70 1,763 2,750 4% 3%

Participants gaining other

positive outcomes -

Economically inactive and

unemployed

6,724 8,570 8,100 23% 19%

Source: COASTAL project management data up to the end of June 2011

2.1.1. Change in Participant involvement and outcome targets.

The original participation and outcome targets proposed and agreed between COASTAL and WEFO

at the start of the COASTAL project have been realised to be too high, partly due to the impact that

Work Programme is having by reducing the number of potential participants for COASTAL to engage

onto its project. In terms of overall participants the total has been reduced from 9,020 to 8,500.

The most significant reduction in targets is for participants entering employment. This has a

proposed reduction of 65%, from 2,870 to 1,000. There are several factors operating that have

reduced the potential to deliver jobs to COASTAL participants, and these are explored later in this

report, but in summary they are a weak labour market, a weak economy, and delivery agents not

focussing on economic outcomes for participants.

It is also worth noting that the ‘participants gaining qualifications’ target has increased as lower level

skills that move participants closer to employment can now be counted and ‘participants entering

further learning’.




Table 3: COASTAL project original participation and output targets compared with 2011 revised

targets.

Outputs Original

Target

2011

Revision % change

Total participants (all economically inactive) 9,020 8,500 -6%

Female participants 4,100 3,160 -23%

NEET participants 740 675 -9%

BME participants 225 180 -20%

Older participants 2,542 2,380 -6%

Participants with work-limiting health condition or

disability 8,610 8,113 -6%

Employers assisted or financially supported 20 10 -50%

Results Target 2011

Revision % change

Participants gaining qualifications 5,412 6,500 +20%

Participants entering employment - Economically inactive

and unemployed 2,870 1,000 -65%

Participants entering further learning - Economically

inactive and unemployed 1,763 2,750 +56%

Participants gaining other positive outcomes -

Economically inactive and unemployed 8,570 8,100 -5%

Employers adopting or improving equality and diversity

strategies and monitoring systems 20 10 -50%


2.1.2. Analysis of Participation Outputs and Results

The original estimates of potential outputs and results made within a more sympathetic economic

climate and labour market, where finding employment opportunities for people with long term

unemployment histories, alcohol or drug issues, mental health and learning disabilities was easier.

The original participation outputs and results targets have proven too high for COASTAL to achieve.

This is demonstrated by the recent reduction in results targets from, for example 2,870 employment

outcomes to 1,000, and the probable difficulty noted in this report in achieving even these reduced

targets.

This is not unusual. Tendering is a competitive process, and in some cases a beauty competition that

requires the tenderer to become the most attractive funding option and high participation outputs

and results make projects appear attractive and good value.

Finally the various partners in the project were all anticipated to have delivered some results by this

stage of the project cycle, and notably Ceredigion and Bridgend have not made any contribution to

date as they have not started delivery.

The COASTAL Business Plan (page 3.) noted that in Swansea between 2001 and 2007, projects

working with the same COASTAL cohort had experienced a ~ 10% employment rate and ~ 62%

qualification rate.




In the 2010 evaluation report the following table was used to illustrate the short fall in participant

numbers. The graph suggested that the total number of participants at the end of the programme

would be around 3,520 instead of the 9,020 within the WEFO funding agreement.

Graph 1: The target and actual / forecast number of participants supported by the COASTAL project

during its lifetime – 2010


New targets have now been suggested and a similar straight line extrapolation has been calculated

for this report (Graph 2). The graph again predicts a shortfall; of approximately 3,000 participants.

This assumes that COASTAL works at the current rate of 321 new participants per quarter. To reach

the revised target of 8,500 participants 630 new participants will need to be included in COASTAL for

each quarter between September 2011 and December 2013.

With few participants moving out of COASTAL and into other more independent settings, or out of

COASTAL and into more dependent day care service settings, the concern is that it is not likely that

COASTAL will meet the outcome target for participants.

These revised rates demonstrate the importance of the contribution as a result of the roll-out of

services from both Bridgend and Ceredigion, which are not yet included in these calculations as they

were not active (i.e. providing support) at the time of this analysis.

-

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Nu

mb

er

of

pa

rtic

ipa

nts

Quarter number

Target number of

participants

Actual / forecast

number of

participants




Graph 2: Graph showing revised targets and the cumulative progress to date as two separate lines.

The solid line shows the anticipated targets towards Dec 2013 and the dashed line progress at the

current rate.


2.1.3. Analysis by Local Authority and cross-regional delivery agents

Graph 3 illustrates how the number of participants is distributed amongst the Local Authority

partner and cross regional delivery agents that make up COASTAL. The graph also shows that none

of the participants have yet been registered by Bridgend County Council or Ceredigion County

Council as COASTAL was not active to June 2011, however, delivery of services has now started in

both these areas.

Graph 3: A breakdown of COASTAL participants per Local Authority and cross-regional delivery

agents

N= 2,021 (June 2011)

-

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Cumulative target required to reach 2011 revised target (8,500)

Cumulative performance extrapolation to Dec 2013 (8,500)

8% 8% 8%10% 10%

21%

16%18%

0% 0%0%

5%

10%

15%

20%

25%




The main contribution in terms of participation has come from the Swansea, Neath Port Talbot and

Pembrokeshire Local Authority areas. However, the COASTAL project is not about participation, it is

about the achievement of greater independence through obtaining qualifications, employment and

other outcomes such as volunteering. Day Care settings have a high participation rate, but do not

focus on achieving the employability and employment results that the COASTAL delivery agents have

contracted to.

The chart (Graph 4) shows two indicators for each project partner. The blue bar represents the wide

definition in outcome achievement across the 10 projects. The red bar is the proportion of

participants that have gained the narrower employability outcome results.

Once the category which records ‘other’ outcomes is removed, only one partner (Pembrokeshire)

has recorded more than 20% of results for its participants. To put this into context, the revised

targets (2011) expect 76% of all participants to gain a qualification, 12% to enter employment, and

32% to enter further learning.

Graph 4: Histogram showing the proportion of positive results across the 10 COASTAL partner

regions and cross area delivery agents. The blue region showing ‘all’ results and the red region

showing ‘employability and employment’ results. The graph demonstrates the need to create more

employability (red) results.

N= 2,021 (June2011)

*Note a participant may undertake employment activities, training, and work towards a

qualification, which will register as multiple results, hence some delivery agents (Swansea Chaplincy)

show more than 100% of results as a proportion of their participants.

The COASTAL model plans to continue its activities post ESF funding, and will possibly then be paid

on an outcomes based model, similar to other outcomes based social support models, notably the

DWP Work Programme. If this is the case then it will require adequate ‘positive outcomes’ for the

model to sustain its staff and costs.

113%

94% 96%

37%

92% 88% 87%

99%

0% 0%

13%

3% 2%

9%13%

8% 8%

23%

0% 0%

0%

20%

40%

60%

80%

100%

120%All Results

All Results except

'other' outcomes




2.1.4. A review of separate WEFO result categories

The following four graphs review the WEFO results categories separately, but disaggregated by cross

area delivery agent and project partner.

Graph 5 shows the qualifications gained by participants across the project partners and cross area

delivery agents. It is clear that Pembrokeshire County Council, via its FRAME and Norman Industries

delivery agents, have created an environment for qualifications; 63% of COASTAL project

participants gaining qualifications come from the Pembrokeshire area. It is however noted that

there has been a technical issue with the level of qualification being set at NVQ Level 2 and that this

has now been reduced to NVQ Level 1 to accommodate the learning levels of some COASTAL

participants. Furthermore, confirmation of certification can take up to a year in some extreme

cases, and seven months for OCN accreditation, so many qualification results may not yet be

recorded.

Graph 5: Distribution of COASTAL participants gaining qualifications per project partner / area

delivery agent.

(N= 114) - In total 6% (114/2021) of participants have recorded qualifications.

The category for further learning (Graph 6) shows that some further learning is taking place across

the project partners and cross area delivery agents, with 3% (70/2021) participants recording further

learning results. Thirty-seven percent of COASTAL project participants undertaking further learning

are however from the Swansea area.

0% 0% 0%5%

8% 7% 6%

63%

0% 0%0%

10%

20%

30%

40%

50%

60%

70%




Graph 6: Distribution of COASTAL participants undertaking further learning per project partner / area

delivery agent.

(N= 70) - 3% (70/2021) of participants recorded further learning results.

The category for participants into employment shows that, for the COASTAL project as a whole,

under 2% (36/2021) of participants have recorded employment results. The Swansea Prison

Chaplaincy has recorded the most employment outcomes; 49% of COASTAL participants that have

recorded employment outcomes are from that element of the project.

Graph 7: Distribution of COASTAL participants exiting into employment per project partner / area

delivery agent.

(N= 36) - 2% (36/2021) of participants have recorded employment results.

8% 8%

0%

15%

1%

37%

23%

6%

0% 0%0%

5%

10%

15%

20%

25%

30%

35%

40%

49%

0%

11%8%

0%3%

11%16%

0% 0%0%

10%

20%

30%

40%

50%

60%




By far the highest category recorded is for ‘other’ outcomes. These are specified by WEFO2 as;

… the number of participants gaining intermediary outcomes as a result of participation in an

ESF-funded project. Intermediary outcomes, such as completing courses, entering voluntary

work, or attending a job interview, are those which are achieved as part of the journey to

achieving final outcomes, such as entering paid employment or gaining qualifications.

It is important to note that the ‘other’ outcomes category is recorded as ‘part of the journey …’ and

not the journey itself. This again places an emphasis on using a tool such as Work Star to evidence

and measure the journey, but also that the journey is towards more tangible employment and

employability results.

In total 78% (1702/2021) of participants have recorded other outcomes and it is important to the

success of COASTAL to identify how these move participants closer to employment and

employability results. Wavehill will give some focus to this issue when it visits the individual projects

in the next stage of this evaluation.

Graph 8: Proportion of participants with other positive outcomes per project partner / area delivery

agent.

(N= 1,572) - In total 78% (1572/2021) of participants have recorded other outcomes.

The analysis of the results categories above demonstrate a wide range of outcomes to date and

considerable variance between the Local Authority areas. For example, Pembrokeshire is finding

some employment outcomes, attributable in part to delivery agents FRAME and Norman Industries.

Other COASTAL partners are finding employment outcomes more difficult to achieve. Accordingly, it

may be that COASTAL, as a collective, or the individual delivery agents need to extend and

emphasise the role of employer’s liaison officer, responsible for identifying businesses willing to

work with COASTAL participants. The role could be to work between the COASTAL teams and the

local employers to negotiate employment places and employment experience.

2 ESF Indicator Definitions, 2009

10% 10% 10%

3% 3%

22%

16% 18%

0% 0%0%

5%

10%

15%

20%

25%




This system worked well within the Cer Amdani! project run by MENCAP across South Wales. Using

employer engagement created an internal employment rate of around 26%, approximately 400%

better than the background employment rate of 6.4% in England. If this rate was replicated into the

COASTAL project then employment outcomes of around 500 would have been achieved to date.

To summarise, the employment and employability outcomes are currently far lower than WEFO has

contracted with COASTAL. Although progression in the participants is being recorded as ‘Other

Outcomes’ it is not clear how far these activities move the participants towards the employability

and employment results.

It is possible that the majority of the COASTAL participants are far from the employment market and

in most cases may never find work or gain qualifications to increase employability and so may not be

the best cohort to participate in a programme that places an emphasis on employability and

employment. A more comprehensive analysis will be made in the next report, now that the

evaluators are able to link participant data to Work Star data and other outcomes.

A greater emphasis on identifying employment opportunities by officers dedicated to this role may

benefit COASTAL participants by finding more full-time and part-time employment opportunities.

2.1.5. Characteristics of participants

An analysis of the characteristics of the participants (please refer to Appendix 2) shows that:

• 65% (1,323/2,021) are male, 35% (698/2,021) female;

• 23% (471/2,021) are in the age group 15-24 years, and 69% (471/2,021) 25 to 54 years;

• 45% (900/2,021) are disabled;

• 1.6% (33/2,021) are from a BME group; and

• 36% (727/2,021) had no qualifications at the time they entered the project.

Perhaps the most notable of the above statistics is that 65% of participants were, as of the 30th June

2011, male and 35% female, which does not reflect either demographics in Wales or the target

participation rate of 45% of participants being female (please refer to Table 1).

The Swansea Prison Chaplaincy – is only open to men as HMP Swansea is a prison for men and 8% of

the participants to date have come from the Chaplaincy project. However even adjusting for this by

subtracting the Chaplaincy project gives a Male to Female distribution of 62%:38%.

The issue of inclusion equality was discussed in the 2010 report and still requires some attention.

Why are males 2.7 times more likely to participate in COASTAL than females? The cause may be

systemic, in that for some reason the COASTAL project is more likely to appeal to males than

females, or it could be sociological, as parents and carers may feel that searching for employment is

a more male activity than female and so less females are presented to COASTAL for inclusion. The

COASTAL team should be aware that there is a gender based equalities impact assessment in place

for the Convergence ESF area3.

3 http://wales.gov.uk/docs/det/consultation/110221genderequalityconvergence.pdf




Analysis by Swansea University4 of the 2001 Labour Force Survey shows that males (21.25%) are

more likely to be considered disabled then females (18.42%) in the work force. It is also worth

noting that 1.6% of current participants are from BME groups, against a target of 2.5% that has been

set for COASTAL (2.5% - see Table 1). This participation rate will also need to be reviewed using the

race equality impact assessment guide5.

2.2. Impacts and outcomes

2.2.1. WEFO monitoring requirements

WEFO has set specific questions that need to be addressed as part of the assessment of the impacts

of projects that are funded under Priority 2 of the ESF Convergence programme (please refer to

section 1.1.1). The questions that are applicable to COASTAL are:

• How many (net) participants have entered employment as a result of ESF assistance?

• How many of these participants are still in employment 12 months after receiving ESF

assistance?

Figure 2; Copy of WEFO Guidance on how to measure the employment impact of the COASTAL

project

Participants in employment at 12 months

Definition:

The net number of participants in employment 12 months following participation in an ESF-funded

project.

Net: achieved through Structural Fund activity and adjusted for deadweight, displacement, leakage

and multiplier effects.

Evidence: Evaluation

Evaluation questions:

∗ How many (net) participants have entered employment as a result of ESF assistance?

∗ How many of these participants are still in employment 12 months after receiving ESF

assistance?

∗ How many participants who had or were at risk of developing a work limiting health condition

have remained in employment as a result of ESF assistance?

∗ How many of these participants are still in employment 12 months after receiving ESF

assistance?

4 Disability, Gender and the Labour Market in Wales (Sloane, 2004)

5

http://wales.gov.uk/docs/wefo/publications/developingguidance/raceanddisability/090112reiaconvergenceen

.pdf




Currently 36 COASTAL participants have entered employment. The contributions by project are laid

out in table 4 below. It is clear that the individual delivery agents now need to make substantial

progress in moving participants on through the project and out into employment outcomes.

Table 4; Total participants entering employment as a result of the COASTAL ESF assistance

Entering Employment

Swansea Prison Chaplaincy 18 48.6%

PRISM 0 0.0%

WGCADA 4 10.8%

Swansea Drugs Project 3 8.1%

Carmarthenshire CC 1 2.7%

C&C Swansea 1 2.7%

Neath Port Talbot CBC 4 10.8%

Pembrokeshire CC 5 16.2%

Ceredigion CC 0 0.0%

Bridgend CC 0 0.0%

Source: Wavehill 2011

The COASTAL project partners and delivery agents need to make significant progress towards

developing employability and employment if the revised result of 1,000 participants entering

employment is to be met. The target may look large, however, shared across the delivery agents it is

more achievable. Each quarter 107 employment places must be found to meet the WEFO results

target. This equates to around 3.5 jobs per month per project. Whether this is achievable or not

needs further discussion with the project staff and a review of approaches to sourcing employment

opportunities from the management team.

An employment target of 1,000 from 8,500 participants equates to an 11.75% employment rate

within COASTAL. To give this some context the Bevan Foundation (20096) wrote “In 2009, the

proportion of disabled people who were working was very much lower in Wales than in Great Britain

– just 38.9% were employed in Wales compared with 47.9%.” Although the COASTAL cohort is not

solely made up of disabled people, approximately 50% are disabled.

Another report that can inform the rate of employment from COASTAL is a recently published study

on Employment Support Allowance cohort (DWP, 2011)7. This report covers ESA claimants in

England, not Wales, and the cohort is only similar, rather than exactly the same as the COASTAL

cohort, but the context is useful. The key point is that around 15% of ESA claimants who were not

working prior to ESA registration were in work six months later.

6 http://www.ey2010.bevanfoundation.org/8.html 7 Routes onto Employment and Support Allowance (2011); Paul Sissons, Helen Barnes and Helen

Stevens; DWP.




Fig 3; Insert from DWP cohort study of ESA claimants 2011

Overall, people who were not working immediately prior to claiming ESA were much less likely than

people who had been in employment to have returned to work. By the time of the baseline survey,

only six per cent of those coming from non-work backgrounds prior to their claim reported being in

work and working, compared to 30 per cent of people who had been in employment before claiming

ESA.

… By the follow-up survey, 15% of claimants from non-work backgrounds were in employment and

working compared with 35% of claimants who were in work prior to their ESA claim.

… Overall exactly a quarter (25%) of the whole ESA sample were in work and working at the follow-

up survey.

Source: Routes onto Employment and Support Allowance (2011); Paul Sissons, Helen Barnes and

Helen Stevens; DWP.

A recent study by Wavehill (2011) into the Merthyr based Family Support Service for long term

unemployed people found an employment rate of 18% (14/77) from within the survey cohort,

although this was not a statistically significant survey.

Fig 4; Reference: Research Review of the impact of Welfare to Work Policy in Merthyr Tydfil, Blaenau

Gwent and Rhondda Cynon Taff; Worklessness Sub-Group Merthyr Tydfil; October 2011

Interviewed 77 out of 200 vulnerable and workless individuals working with the Family Support

Service in Merthyr April – May 2011.

14 had achieved employment (6 full-time, 7 part-time, 1 on work placement) with 2 not having

worked at all before.

10 of the 14 say that their experience of the Family Support Service has made a lot of difference in

them getting a job.

Most findings focussed on softer indicators as a prelude to achieving employment. (See Table 5 next

page).




Table 5; Impact of Support Received on Employability

Employability Worse

(%)

Better

(%)

Much better

(%)

All Better

(%) Base

Your confidence to look for a job 0 33 23 56 48

Your confidence to apply for jobs 2 26 26 51 47

Your prospects for getting a job 4 32 22 54 50

Your prospects for improving your

qualifications 0 33 32 65 66

Your understanding of your strengths and

weaknesses 1 42 21 63 76

Source: Wavehill 2011

Finally, an excellent source of comparative data comes from the 2009 ESF Leavers Survey8. A key

finding was that 15% of economically inactive participants found employment prior to their ESF

intervention, and 30% after ESF intervention; although 67% of jobs were classified as part-time.

Seventy-five per cent of the ESF cohort gained a qualification (to NVQ Level 2 or above). This

suggests a correlation between achieving qualifications and gaining employment outcomes for

previously economically inactive ESF participants.

The probabilities of ESF participants being employed after participation in ESF Priority 2 and Priority

3 projects were;

Table 6; A comparison of findings from the ESF Leavers Survey 2009 and the COASTAL cohort

ESF Employment Model COASTAL context

Males are less likely to find work than females

(0.89)

COASTAL has more males (65%) than females

People with work limiting illness are 65% less

likely to work (B=0.35)

COASTAL works with 45% disabled (B=0.45)

∗ possibly more as disability is self reported on

enrolment

People gaining an NVQ Level 2 via ESF are 55%

more likely to find employment than those with

no ESF sponsored qualifications.

COASTAL has gained 1% (27 qualifications) at

NVQ 2 or higher.

People employed prior to ESF participation are 7

times more likely to work post ESF.

COASTAL participants are 93% unemployed prior

to starting and 74% with unemployment greater

than 1 year.

Source: ESF Leavers Survey 2009 and COSTAL cohort

In all categories in Table 6 above, the COASTAL participants are less likely to work than the ESF

survey and so it is much less likely that COASTAL participants will find employment outcomes

compared to the ESF Priority 2 and 3 survey group.

8 The 2009 European Social Fund Leavers Survey, WISERD, (2010)




To summarise, the labour market has deteriorated since 2009 (MENCAP) but has probably not

changed much since July-Sept 2011 (DWP) and April-May 2011 (Merthyr) so it could be concluded

that an employment rate of around 12% is achievable. The comparison with the ESF leavers’ survey

shows that the COASTAL cohort is less likely to find employment outcomes. However, the greater

employment rate from the MENCAP Cer Amdani! project suggests that a greater emphasis to engage

local employers may be beneficial.

2.2.2. Impact on employers

As noted in Table 1, one of the targets of COASTAL is to provide assistance to 10 (revised target)

employers in the region in order to raise awareness of illness and disability issues and to assist them

in engaging with and, hopefully employing, project participants. The main tool is to develop

procedures and plans for employers, so that they can understand the issues and legislation around

employing adults with learning disabilities and other health and mental issues.

Some progress has now been made on this element, primarily with public sector employers rather

than private businesses. For this report we will not undertake any further analysis of this element as

although some plans are being developed, none are complete yet and so it is probably too soon to

measure if these policies and procedures increase employers’ propensity to employ COASTAL

participants.

The COASTAL web site (www.coastalproject.co.uk) has several downloadable tools available to help

organisations develop better informed equality and diversity schemes and procedures.

2.2.3. Baseline economic data

Baseline data has been updated in Appendix 1. The main issues are that net business creation is

currently negative; more businesses are registering closure, than new ones being started. The labour

market remains weak, and the existing claimant count method of measuring unemployment may be

distorted due to the introduction of Employment Support Allowance.

2.2.4. Soft outcomes and distance travelled by participants - The Work Star

Model

Soft outcomes in the context of COASTAL are the changes in behaviour, skills, communication,

attitude, personal stability and so on that demonstrates that a participant has made progress whilst

participating in COASTAL. The overall positive changes are often called the ‘distance travelled’ by an

individual. The changes are often quite subjective and can fluctuate from day to day, especially in

more chaotic, less stable individuals.

As previously discussed, it is important to measure the progress made by participants towards

greater employability, in order to demonstrate that the range of COASTAL delivery agents benefit

the participants. These ‘soft’ outcome measures are also of interest to WEFO and can be used to

demonstrate the effectiveness of the COASTAL project. As hard results are very low across COASTAL

this places a greater emphasis on demonstrating success by using soft outcomes.

A tool that is often used to identify the amount of change across these subjective dimensions is a

star diagram. Each arm of the star identifies a dimension for measurement, and each arm has a

scale to score that dimension, the Work Star uses a 10 point scale from 1 to 10.




In the case of COASTAL, soft outcomes are being used to monitor how participants move towards a

situation where they are ready to move into employment and the Work Star is being used to collect

the necessary information from participants. The Work Star sets out five stages within a ‘journey

into work’ that an individual will take if and when they progress:

a) Not thinking about work: substantial barriers to work or advancement and you can't see a

way round them (Score: 1-2)

b) Thinking about work: substantial barriers but you are working out how to address them

(Score: 3-4)

c) Making progress: some barriers overcome but others are still there (Score: 5-6)

d) Work-ready with support: barriers mostly overcome or can be worked around; you need

support to maintain progress (Score: 7-8)

e) Self-reliance: in work, work-ready, or engaged in work-related training. By then you don't

need support from the programme (Score: 9-10)

As part of their monitoring procedures, the COASTAL delivery agents are tracking participants as

they move through these stages.

At the time the analysis for this report was undertaken, the Work Star data represents the opinions

of 1,375 participants, however, only 324 have been assessed twice, and 56 more than twice. This

greatly limits the accuracy and use of the data in this analysis.

To demonstrate the overall ‘soft impact’ of the COASTAL delivery agents, the mean change has been

calculated from Work Star assessment one to two, and then two to three, and an overall change.

The change to assessment three is not statistically reliable as so few participants (56) have actually

had a third assessment but does provide some indication of the distance travelled by the

participants in question.

Table 7 shows the baseline or starting point of the Work Star data for each of the categories

monitored:

• Job-specific skills: the extent to which participants have the skills and experience necessary to

get a job.

• Aspiration and motivation: whether participants know what sort of work that want to do, are

motivated to do it and believe that it is possible for them.

• Job-search skills: the skills participants need to find and get a job or training – researching

opportunities, confidence on the phone using a computer, feeling able to present their strengths

in a CV or interview, etc.

• Stability: improvements to the day to day pattern of participants lives; it covers issues relating to

drugs and alcohol, dept problems, insecure housing, etc. – issues that may affect participants

ability to get a job or further education/training.

• Basic skills: literacy, IT, numeracy, use of the telephone, etc.

• Social skills for work: how participants relate to others and meet expectations in a work or

learning situation. It includes, getting on with people, self-confidence, turning up on time,

behaving appropriately, etc.

• Challenges: the practical issues that participants may see as barriers to participating in work or

training; childcare responsibilities, health issues, disability, age, loss of benefits, etc.




Table 7; Work Star data, baseline 2011

N Mean

Std.

Deviation Low σσσσ High σσσσ

Basic skills 1246 6.0 2.7 3.3 8.7

Stability 1363 5.8 2.5 3.3 8.3

Social skills for work 1375 5.7 2.4 3.3 8.1

Job specific skills 1364 5.0 2.2 2.8 7.3

Aspiration and motivation 1375 5.0 2.2 2.8 7.2

Challenges 1363 4.8 2.3 2.5 7.2

Job search skills 1364 4.5 2.4 2.1 6.9

Source: Wavehill / Work Star 2011

The range of opinion for each of the Work Star categories is quite high, and so the standard

deviation of each mean value is wide. Taking ‘basic skills’ as an example, the average (mean)

response is a score of 6.0, but the standard deviation range of σ=2.7 means that approximately 68%

of all views or scores fall between 3.3 and 8.7 – which ranges between the stages (b) Thinking about

work and (d) Work ready with support in the Work Star ‘journey to work’ (see above). The mean is

lowest for the categories ‘challenges’ and ‘job search skills’ suggesting that those are the areas were

participants will need to make greatest progress before they are work-ready.

Graph 9 shows the percentage change in participants’ views. The COASTAL project is having an effect

on the participants. The green line shows a positive change in all areas except for ‘basic skills’. The

blue line shows that most of the change takes place between assessment one and two, and less so

between two and three, and there is some reversal of personal assessment between assessments

two and three.

Overall ‘aspiration and motivation’ (+22%) is the category that shows the greatest change, with

‘stability’ (+14%), and ‘job specific skills’ (+11%) next. The categories showing the least progress are

‘basic skills’ (-1%) which appears to make no difference across the cohort, ‘challenges’ (+3%) and

‘social skills for work’ (+6%). It appears that more work needs to be done in the ‘basic skills’ area

although the ‘baseline mean’ is highest for this category suggesting that other categories may need

to be prioritised. It must also be noted that some delivery agents have not entered any data into the

Work Star, and some very little data, so these findings are therefore not wholly representative.




Graph 9: ‘Spider’ chart showing the categories in the Work Star tool with the greatest rate of change

using paired mean scores

Source: Wavehill/Work Star 2011

Table 8; Table showing the mean change between Work Star assessments

% change 1 to 2 % change 2 to 3 % overall change

Aspiration and motivation 15% 6% 22%

Basic skills -2% 2% -1%

Challenges 9% -5% 3%

Job search skills 10% -1% 8%

Job specific skills 14% -2% 11%

Social skills for work 7% 0% 6%

Stability 15% 0% 14%

Source: Wavehill/Work Star 2011

The Work Star needs to be used more frequently, with a regular and repeated cycle of assessment

taking place, and actions developed from this assessment. It is suggested that each participant

undertakes an assessment every 120 days (4 months) and that this data is analysed by Wavehill and

then sent as an information package to all projects. The data will aggregate the progression across

all participants, and then for each of the ten partners provide a tailored analysis by project for each.

This will allow each partner to compare their data to the whole of COASTAL.

This is not intended to be used as a league table of results, but to give some clarity and quantitative

view about what is happening within the projects to the participants. Currently there is a ‘black-box’

approach to the COASTAL projects, as each project is undertaking its own activities in its own way in

order to suit its particular participants and their needs. The Work Star analysis will hopefully allow

some comparison of approaches, identify good practise and promote the capacity to share these

methods.

-10%

-5%

0%

5%

10%

15%

20%

25%

Aspiration and

Motivation

Basic Skills

Challenges

Job Search Skills Job Specific Skills

Social Skills for

Work

Stability

% change 1 to 2

% change 2 to 3




Also the Work Star data needs to be integrated with the monitoring data so that a more accurate

analysis of the benefits of participation with delivery agents can be recorded. The analysis will then

support greater detail and clarity. As an example, the graph below reviews the Work Star progress

made disaggregated by gender. This shows that males benefit more from Job Specific Skills and

Social Skills for Work, but females in the other categories. Again, this analysis must be treated with

caution as the data is not comprehensive enough to make any statistically certain statements. Also

the scoring mechanism by different Work Star users is not yet uniform, so one respondent may score

a 4 and another a 6 but both for a similar interpretation of the category, Basic Skills for example.

Finally an enhanced training programme is recommended for delivery agents, so they understand

how to use Work Star correctly.

Graph 10: Graph showing Work Star categories disaggregated by gender.

Source: Wavehill / Work Star 2011

To summarise, the Work Star data is currently not robust enough to be useful for the evaluation of

soft outcomes within COASTAL. The frequency of assessments must be made using a regular time

period and every four months is suggested. Linking the data to other demographic information such

as previous work history, age group, disability category, and so on will, as in the ESF Leavers Survey,

provide far more useful data and powerful analysis.

2.2.5. Benefits identified by participants

The 2011 method proposed using 40 case studies to show the range of participants, their

experiences on COASTAL and benefits and outcomes from participation. The method of selecting

appropriate case studies was discussed by the evaluators and the partnership board and a non-

biasing randomised selection method was proposed. Essentially this method prevents project

managers presenting for case study their most positive participants, which in turn may provide a

positive bias on participant outcomes and opinions.

The selection of participants is negotiated, so a participant is empowered to decide whether they

wish to be involved in a case study, or not. This process of selection and consent has taken

considerably longer than originally anticipated and so the participant’s views as case studies will not

be included in this report, and will be published as an separate report early in 2012.

57%

20%

31%

53%47%

37%

20%

49%

20%

29%

43%

58%

44%

16%

0%

10%

20%

30%

40%

50%

60%

70%

Aspiration

and

Motivation

Basic Skills Challenges Job Search

Skills

Job Specific

Skills

Social Skills

for Work

Stability

Female

Male




3. Views of Strategic Stakeholders;

2011 Key findings

• A funding risk exists to the COASTAL partnership due to the low employment and employability

outcomes and results.

• Work Programme issues and uncertainty used a lot of the COASTAL team time and resources in

2011.

• What additional support can be offered to COASTAL clients as they leave COASTAL and move

into employment?

• Work Star data could be centralised and analysed more frequently by Wavehill.

• Some ‘mission shift’ away from employment towards externalisation of services has been

observed in stakeholder interviews.

• COASTAL could develop social firms similar to FRAME in order to create employment

opportunities, and possibly sustainability, but outside of ESF funding.

• The employment market for COASTAL participants is weak and this is probably affecting

employment outcomes.

• What is the correct balance between COASTAL as an Employment project and COASTAL as a

Social Care project?

• Greater collaboration, especially between project delivery teams, should be encouraged.

This section discusses the views of the strategic stakeholders of the COASTAL project. The interviews

were undertaken during July and August 2011. The range of stakeholders includes senior

management (strategic level stakeholders) and senior staff who are providing support to participants

(referred to in the following discussion as ‘delivery agents’).

The discussions were led by a topic guide, the main direction of the topics being;

• How effective is the project being delivered to date now it has entered the post start-up stage

• How have participants benefitted?

• How has the project changed the way in which support is provided in your delivery area?

• Who now provides the delivery, has this changed in structure, from Local Authorities to Social

Enterprises / Third Sector for employing and training people, for example?

The key findings of the interviews are discussed below. These views will be supplemented with

approximately 20-30 interviews with staff from across the delivery agents. The interviews will take

place parallel to the beneficiary interviews where possible and so will also be included in the next

report due in early 2012. The interviews will include a focus on the important issue of how much

work with participants is within the context of employability and employment and how much is

developing stability and skills, and other factors.




3.1. Work Programme

The new Work Programme introduced by the DWP during 2011 has used up much time and

resources from April to July 2011 on establishing how COASTAL would be affected by the changes it

brings. Without reporting the nuances of the various interpretations of how COASTAL may have

been affected by Work Programme, the final understanding is that some COASTAL clients currently

on Incapacity Benefit (IB) and transferring onto Employment & Support Allowance (ESA) may not be

eligible for ESF support. Furthermore the Work Choice programme has also made some participants

ineligible for inclusion on the COASTAL project.

The estimated reduction in the number of COASTAL participants varies between partners and

delivery agents, with several estimates at around the -10% level, but with two partners noting that

they have around 40% - 50% of participants on IB/ESA and so the effect may be greater.

The targets for the COASTAL programme have been reduced from 9,200 to 8,500 to reflect the

reduction in participant qualification for COASTAL. The overall effect on COASTAL as an employment

focused project may not be too large as the greater issue is not currently with the supply of

participants into the COASTAL project, but transition within the delivery agents’ projects and results

into employability and employment.

3.2. Progress ‘Work Star’ data is not being captured or used by all

projects

As previously discussed, the Work Star progression tool was chosen by COASTAL from several

progression tools as the best tool available to track soft outcomes. Yet, it is not being fully used

throughout the COSTAL project. Other potential tools have been suggested by stakeholders and one

similar progression tool is used within the Swansea Chaplaincy.

The stakeholder interviews demonstrate that Work Star in concept is understood, in that it collects

the data to evidence the changes across participant groups, however there is a wide range of

reactions to it. Six stakeholders supported the Work Star, four were indifferent to it, three were

negative about its use and two stated they would not use it at all. One stakeholder mentioned that

Work Star was inappropriate for its participants because it had the word ‘work’ in its title. However,

when the Work Star was presented to the project teams at the November 2011 conference, the

reaction to it was more positive, and attitudes may have shifted as more training and support in its

use and analysis has now been offered.

WEFO will recognise soft outcomes as contributing to the overall understanding of a project.

Currently the COASTAL project has very low employment and employability outcomes and so it is

doubly important to demonstrate the progress made by COASTAL participants, and their increasing

employability. The Work Star should therefore be updated on a regular cycle (120 days or 4 months

has been suggested by Wavehill) in order to demonstrate progression.

The issue of data management and collection in general and Work Star specifically was a common

theme throughout the stakeholder interviews. There were several suggestions that workshops or

collective discussions should be undertaken to negotiate the central collection and analysis of data.

It may also be beneficial to discuss the centralisation and analysis of data. This will allow the

evaluators, central COASTAL team, stakeholders and delivery agents to discuss a unified solution.




The analysis of the Work Star and other data should be for a clear purpose, so that outcomes and

impact can be measured effectively. The key question of which groups benefit the most from

COASTAL and which the least is difficult to inform without a quantitative data source that cuts across

all stages of a participant’s progress through COASTAL. Understanding which individuals are most

likely to get employment or near employment outcomes and who are not is also important if the

COASTAL project is to be successful within its ESF contract.

3.3. An increased emphasis on the externalisation of social care

services and independence within participants

The original emphasis of the COASTAL project was to move service delivery models away from long

term social care settings into creating a different paradigm of learning, skills and employment for

COASTAL participants. The business plan emphasises the need to develop employability skills and

employment outcomes are currently targeted at 12% (1,000/8,500). During the interviews with

stakeholders a small shift was noted in the responses to the question about the core role of

COASTAL; there was less focus on employment and employability and more with developing

independence in participants and externalising services.

The externalisation of services is the movement of statutory care services from within statutory and

local authorities to external service providers, often charities and social firms. The authority then

remains as a contract holder, monitoring the quality and delivery of the service. This is a very current

debate amongst stakeholders as externalisation is seen by some as a method of introducing more

competition between service providers and as a means to reduce the financial costs of providing

social care from within Local Authorities and Health Services.

The COASTAL project does encourage project partners to consider the externalisation of services.

COASTAL allows for service delivery by competitive tendering of service provision, in theory at least

opening up the possibility of service provision from any EU member state. However, this is seen as a

secondary outcome of the COASTAL model and not its primary function. The primary function is to

work with participants, who may traditionally have been cared for within social/day-care settings, to

undertake employability skills development, learning and to find employment.

The externalisation of services are, it seems, being used by some partners to justify the participation

in the COASTAL partnership, rather than the core concept based around employment outcomes.

Inclusion in the COASTAL partnership does benefit the partners financially as their social-care service

budget is multiplied by the ESF contribution, but ESF Priority 2 requires a focus on economic

outcomes above social care outcomes and ‘Making the Connections’ collaborative delivery

structures.

Some ‘mission drift’ is noted amongst COASTAL stakeholders. This may be partly as employment

outcomes are so far very few amongst the COASTAL projects. However, the WEFO / ESF contract is

explicit in its requirement of a focus on employability and employment. It is also clear from

participant interviews, and speaking to stakeholders, that some COASTAL participants are not ready

for employment, and probably never will be. These participants require a social/day-care model not

an employability employment model.

Finally, stakeholders were more likely to emphasise the participation targets than their employment

outcome targets. This possibly explains the position of many COASTAL partners, as they are primarily

concerned with the provision of a service and setting for their clients, rather than delivering

employment and employability outcomes.




3.4. New Service Models – externalised Social Firms

The Pembrokeshire participants benefit from the externalised services of FRAME and Norman

Industries. With such a weak labour market and so few COASTAL participants reaching employment

outcomes, the partners need to identify methods of creating employment outcomes. A shared

vision of creating employment opportunities through social firms may be a positive outcome for

COASTAL at a strategic level.

Although ESF cannot be used to create social firms, the partnership can be used to shape the

strategy and encourage discussions with potential social firm suppliers. The interviews with

stakeholders support this, five interviewees stated that the development of social firms to deliver

externalised services, provide apprenticeships and ultimately employment, is an opportunity for the

partners.

The partners may wish to visit social firms from within Wales, but also wider afield to examine how

social care, employability and employment and social firms can work together to provide service

models with employment outcomes. If the partners wish, Wavehill can suggest several options for a

COASTAL delegate to visit.

3.5. The balance between social care services and employment

services

COASTAL is delivered differently from area to area and approaches to delivery between each Local

Authority partner and delivery agent can vary. However, there is a dichotomy emerging within the

COASTAL project as a whole. The funding and the business plan are based within the economic

context of participants moving from COASTAL into employment. This core reason behind the whole

ESF fund across Europe ‘… was set up to improve employment opportunities in the European Union

and so help raise standards of living. It aims to help people fulfil their potential by giving them better

skills and better job prospects.’ (DWP, 2011).

Priority 2: Increasing employment and tackling economic inactivity to raise levels of

employment and economic activity, and secure higher participation in the labour market.“

WEFO

The COASTAL participants are in many cases a very long way from the labour market and, as

demonstrated by the baseline data collected using the Work Star discussed in the previous section,

many have a very low probability of finding mainstream employment opportunities in the near

future. This places them in the category of not being able to work or not thinking about work. This

was illustrated using the graphic below in the 2010 report.

Figure 5: An illustration of the continuum of participants COASTAL will work with from 2010

evaluation report.

Not thinking Self-reliance

about work In-work

Work ready

Participants at this point Participants at this point

need in-depth support require less support




This was also further illustrated by the progression model (below), which was included as figure 6 in

the 2010 report. The point was made that much participant activity is based within overcoming

barriers and less within the categories of employability.

Figure 6; From the 2010 evaluation report illustrating categories of participant assistance and

development with COASTAL

Feedback from stakeholders in 2010 and 2011 suggests that the majority of those participants that

the COASTAL project is currently working with are overcoming barriers to employment. This places a

question about the propensity of the COASTAL participants to increase economic activity and

increase employment under the Priority 2 theme from which COASTAL is match funded.

The evidence for this comes from the proportion of COASTAL participants that are reaching ESF

outcome and result targets of entering learning, gaining qualifications, finding employment and

gaining skills that increase employability, measured as ‘other outcomes’. The proportions of

participants reaching these results are very low, less than 3% progressing into qualifications, further

learning or employment as a proportion of all participants. This suggests that the large majority of

the COASTAL cohorts still requires in-depth support and are probably attending some COASTAL

delivery agents primarily for social care, rather than for employability.

However the origination of many of the participants (from social/day-care settings) and their

distance from the labour market suggests that the majority of COASTAL participants will not

progress into employment outcomes and may not record employability results such as qualifications

or further training.

“In the first two years the aim was to enrol all known client groups, people who go to Day

Centres”

Source; Stakeholder interview 2011

The revised outcome targets (2011) propose around 1,000 employment outcomes from 8,500

participants. This is a 12% employment rate across COASTAL still leaving nearly 90% of participants

without employment and held within the various COASTAL delivery agents.




This leads to a fundamental question about what the role of COASTAL is;

1. Is COASTAL a social/day-care project where around 10% of participants gain employment and

the rest undertake personal and skills development; or;

2. Is COASTAL a project that gets participants to gain in confidence, enhance skills and training and

then ultimately employment?

The stakeholder interviews identify COASTAL more as a social care model. This issue of the balance

between providing stabilisation and care for participants and progression towards employability and

employment results was discussed in the 2010 evaluation report. The comments below support both

sides of this discussion.

Figure 7; Extracts from stakeholder interviews 2011

1. The funding allows us to develop the current services in the way they work and having more

resources via match funding to allow us to do more with the clients. We provide social care but

in a different way now, developing day services, then progression into COASTAL activities,

training, volunteering, etc. It allows progression and this is the main concept for us. We

measure the benefits as more independence and less dependency.

2. It provides opportunities for those that did not have opportunities before. Some people have

moved on with their lives considerably and although it is still small numbers that have achieved it

is still way beyond what statutory organisations would have achieved, some have better

emotional well-being and others have jobs as an outcome.

3. I think you do have to accept that if this service is an alternative to Day Services then some

participants may have been in Day Care for years and years and will be used to just achieving an

outcome … I think some participants will always need support and should we be saying it has to

be a work environment, can't it just be that they end up in a nice and safe environment?

The ESF contract and the core management team hold the opinion that COASTAL is about moving

participants into employability and employment results, whereas the stakeholders interviewed

appeared to consider the project is more about personal skills enhancement which lead to increased

employability, testing models for the externalisation of services, and in some cases social care

provision.




Figure 8; Extract from the COASTAL Business Plan (Business Plan Aims section) illustrating the

employment focus of the project aims

Aim 2; To develop a strategic direction to offer opportunities for learning and work experience to

move disadvantaged citizens from our target group, [i.e. individuals who are currently economically

inactive as a result of illness, disability, (Mental Illness, Learning Disability, Physical Disability,

Sensory Impairment) substance misuse problems and/or the serious social disadvantage associated

with the transition from long-term care into adulthood], from being economically inactive to active in

the labour market.

Aim 5; To support and enable access to mainstream employment and learning opportunities with

support where necessary.

Aim 6; To gain the support of employers to provide opportunities, by offering support, information

and guidance in all aspects of engagement with our participants.

Aim 7; To promote permanent employment, paid at the same rate of pay and same conditions as any

other employee.

Aim 9; To assist our target participants to obtain and retain work by developing employment skills

required by local employers.

Figure 9; Extract from the 2010 evaluation report in relation to the key role of COASTAL

Their concern was that this focus on moving participants into employment (rather than overcoming

the barriers to employment) made the project irrelevant to many of the participants that they

worked with. Further, there was a concern that participating in the project could even have a

negative impact on some individuals by jeopardising the progress that they had previously made.

“There’s no hope of some of the people that we work with getting a job” – stakeholder at a project

delivery level

“You can’t push the people that we work with that hard” – stakeholder at a project delivery level

The correct balance between COASTAL as an employment project and COASTAL as a Social Care

project needs to be agreed on between the partners as there are some significant differences in

opinion and potentially a funding risk.




3.6. Weak employment demand

Interviews with stakeholders clearly identified the weak labour market as both a weakness and a

threat to COASTAL. The economy in the UK and in West Wales is weak and deteriorating. The recent

monthly announcement of the state of the economy and labour market in South West Wales

(Statistics Wales, 20119) stated that;

• South West Wales had the lowest employment rate – 63.7% vs. 66.3% for Wales

• The second highest unemployment rate – 8.5% vs. 8.5% for Wales

• But positively the second highest average weekly earnings – wage index of 87.5 vs. 86.2 for

Wales

Finding employment opportunities for COASTAL participants is undoubtedly difficult, especially as

some sectors such as retail, where traditionally COASTAL clients have found work are particularly

bad.

The evaluation should now place some focus on;

1. How delivery agent teams are finding work opportunities for their clients.

2. Where delivery agent teams are finding other near employment opportunities, such as work

experience and volunteering.

3. How delivery agent teams promote employability learning, skills development/training and

qualifications.

3.7. Competition for ESF participants

Where several ESF funded projects have been commissioned to provide similar services to a similar

client group this has created inter-project (delivery agent) competition for ESF referrals. ESF rules

only allow one individual to have ESF funded activities recorded against their name, and so it is

important that ESF projects do not duplicate services.

The stakeholders do not recognise this as an issue across the COASTAL projects and have identified

some referrals coming from Genesis, Work Ways and Want 2 Work. The stakeholders do not notice

any duplication or competition for participants. However, referrals have fallen from an estimated 10

a month at the start of delivery, down to only 1 or 2 a month now, and this may be a result of not

being allowed to share outcomes with ESF/WEFO.

9 Regional Economic & Labour Market Profile – South West Wales – September 2011




3.8. Cross (Local Authority) border services

The stakeholder interviews identified the need to share good practise and resources across Local

Authority borders. One criticism is that service still stops at the Local Authority boundaries and

some clients are tied to these geographies and cannot select the service that is best for them, or are

not offered the range of services across the COASTAL partnership to choose from. Clearly issues of

transport and travel will restrict many participants to their nearest service but offering access to

services more widely demonstrates the effectiveness of collaborative partnerships. This is reiterated

in the Framework for Sustainable Social Services in Wales policy document.

Also, the highly influential Beecham Report and the subsequent policy paper ‘Making the

Connections; Delivering Services Across Boundaries’ promotes shared and collaborative delivery, as

do the Local Service Boards for social care services. The very fact that COASTAL exists proves that

such collaborations can be formed. The balance to this comment is that four delivery agents are

delivering services not based within Local Authority borders and as services increasingly externalise

it is anticipated that more will do the same.

3.9. Procurement of services and suppliers is complex

In the 2010 report the issue of the procurement requirements for contracts being complex were

covered in some detail. However, the same issues were again mentioned by four stakeholders

during interviews as a weakness of the COASTAL project.

3.10. Collaboration and sharing of good practise

The stakeholder interviews identified that although collaboration between partners is taking place at

the strategic level through various COASTAL meetings. The working practices and day to day work

between delivery agents are not widely compared or shared.

Collaborative projects become most effective when good practice is transferred across the

partnership, affecting change to make other partners more effective in terms of service delivery and

ultimately to the benefit of the participants. A Development Officer Group has been established

although it is not clear yet what they have so far developed or collaborated on. The evaluation visits

to delivery agents are due to commence shortly and will identify areas where collaboration and

sharing can take place so that delivery staff can share their experiences and good practise. A new

mandate to demonstrate collaboration and sharing between delivery agents on a bimonthly

frequency has been agreed.

Collaboration does not need to be limited to COASTAL partners, but could include Work Programme

teams and Job Centre plus staff too, so that a better understanding is created between these service

providers, all who are essentially trying to achieve the same outcomes across similar client groups

within the same geographical areas.

If the Development Officer Group wishes, Wavehill can suggest some ideas to form into actions to

increase the exchange of ideas and collaboration. Collaborative groups and partnerships follow the

policy direction from the Welsh Government under the Making the Connections policy.




3.11. A critical response to this report

A criticism of this report has been recognised (and valued) that “although only 10% of Participants

are expected to achieve employment outcomes, what about the service to the other 90% of

participants.” We acknowledge this but would stress that it is not possible to capture all views and

opinions in every report when using a rolling evaluation programme because different waves

capture different opinions. It is therefore worth noting that reports produced by the evaluation in

2012 will consider the following questions:

• What “other positive outcomes” have been achieved and does this show the progression WEFO

would want to see?

• Is this [above] reflected in the Work Star – is the service provision effective?

• Is training being delivered and achieved?

• Is training effective in moving participants towards employability?

• Are participants leaving the project?

• Where do participants go when they leave the project – is it further learning?




Appendix 1: Updated baseline data

Please note – the baseline data has been updated using a data series from Statistics Wales, and so

will be comparable in the future, and in fact accessible at any time on line from the Statistics Wales

web site.

Graph 11: Claimant Count – the claimant count peaked in August 2009. Claimants may have fallen

due to a systemic change in workless benefits rather than an improvement in labour market demand.

Source: Statistics Wales; October 2011

Graph 12: Disabled Employment Rate 2007-2009 – The employment rate amongst disabled people in

September 2011 is around 9% for Wales but significantly higher in Swansea, Bridgend and Neath PT

Source: Statistics Wales; October 2011

Wales Ceredigion Pembs Carms SwanseaNeath Port

TalbotBridgend

August 2007 2.1 1.3 1.3 1.9 2.2 2.3 2.2

August 2008 2.4 1.2 1.5 2 2.4 2.5 2.3

August 2009 4.2 1.8 3.2 3.3 4 4.4 4.7

August 2010 3.7 1.7 3.2 3 3.5 3.6 3.9

August 2011 4 1.9 3.1 3.2 3.8 4 4.2

0

1

2

3

4

5

Pe

r ce

nt

un

em

plo

ye

d c

laim

an

ts Claimant count based unemployment rate.

Wales

West

Wales &

Valleys

Ceredigion Pembs Carms SwanseaNeath Port

TalbotBridgend

Economic inactivity level 45.7 43 43.9 49.6 43 49.2 36 43.8

Economic activity rate 41.7 39.3 40.9 48.6 39.6 43.7 32 37.7

Employment rate 8.8 8.7 0 0 8.1 11.2 11.1 14

Unemployment rate 54.3 57 56.1 50.4 57 50.8 64 56.2

0

10

20

30

40

50

60

70

Pe

r ce

nt

lab

ou

r m

ark

et

acti

vit

y r

ate

s Labour market data for disabled persons - September 2011.




Graph 13: Net Birth of Enterprises 2007-2009 – net enterprise births and deaths is a strong but

lagging indicator of economic activity. The graph clearly shows that the Welsh economy is losing

businesses in 2009, was mixed in 2008, and creating businesses in 2007.

Source: Source: Statistics Wales; October 2011

45

75

90

115

50

75

-40

55

-30

-90

40

0

-60

-120

-135

-150

-90

-40

-200

-150

-100

-50

0

50

100

150

Net business births and deaths by Local Authourity area 2007-2009

2007 net

2008 net

2009 net




Appendix 2: Analysis of COASTAL

project management data The following analysis is based on the project management data for COASTAL as at the end of the

quarter ending the 30th June 2011.

Output and result definitions

Output / results Definition

Participants The number of individuals participating in an ESF funded project.

Participation should be linked to specific outcomes and require a

meaningful level of engagement, for example a training course. Therefore,

this excludes individuals attending conferences or individuals who simply

receive information.

Employers assisted or

financially supported

The number of employers that receive assistance or financial support,

through this Programme.

Assistance: advice, guidance and information which can be delivered

through the following media: face-to-face, telephone, web-based

dialogue, conference, seminar, workshop, or networks (OffPAT, 2005,

p.8). For example, provision of advice and guidance on managing health

conditions for employers.

Financial support: Receiving a grant or loan. For example, financial

support for workforce development by employers.


qualifications

The number of participants gaining a full, accredited qualification as a

result of participation in an ESF-funded project.


employment

The number of participants entering employment as a result of

participation in an ESF-funded project.

Employment includes self-employment and can be full-time or part-time.

However, employment must involve a minimum of 16 hours work a week

and must be paid employment. Projects must only report against this

indicator if the participant enters employment within six months of

completing provision. This indicator only applies to those participants who

were not in employment upon commencement of their participation in an

ESF-funded project.




Output / results Definition


further learning

The number of participants entering further learning as a result of

participation in an ESF-funded project.

Further learning can include formal academic or vocational education and

less formal skills development training (including soft outcomes). For

example, a young person classed as NEET (16-18 year olds not in

education, employment or training) entering vocational training after

receiving intensive careers advice funded by this Programme; or, an

individual moving into formal training following outreach work funded by

this Programme.

Projects must only report against this indicator if the participant enters

further learning within six months of completing provision. This indicator

only applies to those participants who were not in education or training

upon commencement of their participation in an ESF-funded project.


other positive

outcomes

The number of participants gaining intermediary outcomes as a result of

participation in an ESF-funded project. Intermediary outcomes, such as

completing courses, entering voluntary work, or attending a job interview,

are those which are achieved as part of the journey to achieving final

outcomes, such as entering paid employment or gaining qualifications.

Employers adopting or

improving equality and

diversity strategies and

monitoring systems

The number of employers adopting a strategy, which outlines the key

priorities for action by the employer and its staff to promote equality and

diversity and challenge discrimination (GLA, 2005), and monitoring

progress against these priorities. The equality strategies and monitoring

systems must have been adopted or improved as a result of Structural

Fund assistance or financial support.




Analysis of Coastal participants

Graph 14: Distribution of COASTAL participants by the L.A. in which they are resident

N=2,194

(please note total percentage doesn’t equate to 100 due to rounding)

Graph 14a: Distribution of COASTAL participants per project

N=2194

64

336

9

418498

860

5 4

8% 8% 8%10% 10%

21%

16%18%

0% 0%0%

5%

10%

15%

20%

25%




Graph 14b: Distribution of Coastal participants entering employment

N=2,194

Graph 15: Gender of participants

N=2,194

0.80%

0%

0.20%

0.10%

0.50% 0.50%

0.20%

0.30%

0% 0%

65%

35%

Male

Female




Table 9: Gender of participants by project

Project Male Female

Swansea Prison Chaplaincy 178 0

PRISM 124 61

WGCADA 132 45

Swansea Drugs Project 136 78

Carmarthenshire CC 135 85

C&C Swansea 281 187

Neath Port Talbot CBC 202 150

Pembrokeshire CC 248 152

Ceredigion CC 0 0

Bridgend CC 0 0

TOTAL 1436 758

N=2,194

Graph 16: Existing qualifications of participants

N=2,194 (please note total percentage doesn’t equate to 100 due to rounding)

36%

33%

17%

7%

4% 3%

0%

5%

10%

15%

20%

25%

30%

35%

40%

None Below NQF 2 At NQF 2 At NQF 3 At NQF 4-6 At NQF 7-8




Graph 17: Age group of participants

N=2,194 (please note total percentage doesn’t equate to 100 due to rounding)

Graph 18: Disability status of participants

N=2,192

24%

68%

7.5%

0%0%

10%

20%

30%

40%

50%

60%

70%

80%

15-24 25-54 55-64 65+

47%

53%

Yes

No




Graph 19: Ethnicity of participants: BME – yes or no

N=2,194

Graph 20: Employment status of participants

N=2,194(please note total percentage doesn’t equate to 100 due to rounding)

2%

98%

Yes

No

0.80% 0.10%

18%23%

57%

1%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%




wavehill ymchwil gwerthuso arolygon

research evaluation surveys

Tel / Ffôn: 01545 571 711

Email / Ebost: [email protected]

Website / Y We: www.wavehill.com

Wavehill Ltd, 8 Water Street / 8 Heol y Dŵr, Aberaeron, Ceredigion, SA46 0DG

Ongoing evaluation of the COASTAL project - Report 2

Documents

Transcript of Ongoing evaluation of the COASTAL project - Report 2