Presentation to Joint Meeting of DG REGIO Evaluation Network and ESF Evaluation

Partnership8 July 2011, Gdansk

Key Fifth Cohesion Report themes (pp256-7)

1. Move on from preoccupation with financial ‘absorption’

2. Fewer, simpler policy priorities (‘concentration’)3. Focus on results and impacts (i.e. performance). 4. More rigorous evaluation (e.g. CIE, CBA for quant;

case studies for qual; beneficiary surveys for both quant and qual)

5. Triangulation6. From productivity to well-being (happiness) and

sustainabilityFor the presentation I will focus on 4, 5 and 6

Making the right choice: The example of beneficiary surveys

o CED priority (P6 – Targeted Action for Key Deprived Areas) in 1994-99 Yorkshire & Humber Objective 2 programme

o ERDF Measure 6.21: ‘support for community based economic projects’ – 111 (of the 178) CED projects

o Main types of projects funded:- Micro-finance and credit- Social enterprises (‘community businesses’)- Labour market access, training and ILM- Education- Targeted environmental improvements- Transport access to jobs- Pure capacity building

What we didTwo-stage sampling method:

- managers of all the projects contacted for lists of beneficiary enterprises and other organizations- individual enterprises and organizations randomly sampled from the lists (simple random sampling)

Useable lists from 33 of the 111 projects, but all the biggest projects chased down

Random sampling of 242 organizations and enterprises (110 First Sector enterprises serving mainly non-local markets; 44 First Sector serving mainly local market; 15 community enterprises; 64 mixed social/market organisations; 9 purely social organisations)

Did we do anything right?Direct counterfactual questions

Did we do anything right?Lots of nice new stuff on ‘why it works’

What did we do wrong (in Fifth Cohesion Report terms)?Rigour• Should have used stratified sampling method• Only one bias check (sub-areas)• No proper confidence intervals Triangulation • Could have used open questions and a third stage of

SSIs as a qual-on-quant triangulation• Could have analysed secondary data sets for quant-on-

quant triangulationWell-being and sustainabilityBeneficiaries = organisations. Wrong. Should have done residents’ survey Should have added some sustainability questions

Making the right choice: beneficiary surveys guidanceInherent characteristic Comments

Best for quant information Can add open questions and SSIs

Big population to sample from OK, but let’s not exaggerate this

Good quality sample frame OK, but much easier if beneficiaries +non-beneficiaries surveyed

Clear effects and preferably known beforehand what they might be

OK, but nowhere near as rigid as many would have us believe

Direct counterfactual questions Triangulation with other quant

Can ask ‘why it works’ questions Better if effects clear and understood beforehand

Rigour Must improve: technical, time and money resources?

Triangulation ‘In-house’ qual/quant and wider quant/quant

Well-being and sustainability Excellent tool. Residents’ surveys?

Focus on results/performance Excellent tool; cost-effective too

Case studies: what’s being done well? Example: ERDF 2000-06 Work Package 4: Structural

Change and Globalisation Thematic: effect of, and policy response to globalisation Holistic: quantitative and qualitative Quantitative: Three main elements:

- By core team statistical analysis- By case study teams- Pilot beneficiary surveys; secondary data

analysis Qualitative:

- In-depth analysis of documents - Semi-structured interviews with regional experts, policy makers and beneficiaries (176 in total)

Case studies: what’s being done well? Multiple full case studies (12) Transparent and logical selection criteria: ‘representative’

approach rather than extreme/deviant, maximum variation, critical or ‘pardigmatic’ (Flyvbjerg, 2006)

In-depth knowledge of context, including historical context (flexible – back to early 1980s for many)

Narrative: ‘telling the story’ Mini-case studies Common template but with flexibility for regional teams Central core team Generalization: accumulation of evidence; power of the

‘deviant’ case

Case studies: what next? Rigour• Fundamentally a qualitative method, but needs

more quant• Beneficiary surveys• Proper CIE and secondary econometric data analysis?• More rigorous qualitative methods Triangulation• Already happening, but• Regional team SSI/document analysis and core team

quant• Beneficiary survey as quant-on-qual by regional teamWell-being and sustainability• Potentially excellent method

Making the right choice: case studies guidanceInherent characteristic Comments

Best for qual information Some quant by core and regional teams already; beneficiary survey

Particularly useful for thematic and cross-cutting evaluation?

OK, but is unusually flexible especially where min-case studies too

Best for complex, longer-term and where context important

Best where main effects not clear or understood beforehand

OK, but beneficiary survey would widen it to clear effects too

Poor for measuring counterfactual

Good for ‘why it works’ OK, but beneficiary survey would help

Rigour Must improve: technical, time and money resources?

Triangulation ‘In-house’ qual/quant; more qual/qual possible

Well-being and sustainability Excellent tool. Not cost-effective as quant

Focus on results/performance Excellent tool; residents’ survey?

From productivity to well-being: A warning from history: British enterprise zones in the 1980s

An urgent need to prioritize amongst the many well-being and sustainability measures.

‘Concentration’ will narrow the field for each programme, but…….. still and awful lot, especially for well-being

A ‘must’ for all future evaluations: distribution of income and wealth

A warning from history: Erickson and Syms, Regional Studies, 1986. Salford-Trafford Park EZ, Manchester.

Tax reductions (especially property tax – ‘Business Rates’ ‘Place prosperity’ versus ‘people prosperity’: who wins, who

loses. Distribution: combined property tax (rates) plus real rents: 36% of value of the property tax exemption went to industrial tenants and 64% to initial property owners, many ex-regional. We may not like the results we get.

From productivity to well-being and sustainability: Challenges for other methods?

Heavily affected?

- cost-benefit analysis- SWOT- Evaluability assessment, logic models, concept mapping- Delphi methods, focus groups, experts panels- Observational methods- Stakeholder consultation and participative techniques- Meta-analysis

Less heavily affected?- Input-output- Macro modelsCGE