1

Impact Assessment Development and overview

Paul Foley, Francesco Mureddu and Alberto Savodelli

pdfoley@tech4i2.com

JRC-IPTS MIREIAExperts' Workshop on Measuring the Impact of

eInclusion Intermediaries in EuropeGranada, 29th May 2013

The views expressed by the authors are not necessarily those of the EC

Thanks to all the intermediary personnel, survey participants and advisers that have taken part in the

project.

211.45

Context, Conceptualisation, & approach

EU27 Unemployment rate 12.1% March 2013, 11% March 2012March unemployment – ES 27%, IE 14% IT 12%, PL 11%,

Youth unemployment (under 25) EU27 in 2012 22.8% 2012 youth unemp – ES 53.2%, IE 30.4% IT 35.3%, PL 26.5%,

DAE Actions – 57; digital literacy, 59; digital skills for jobs, 60; women and ICT, 63; accessibility

Dublin #da13skills #da13smart

311.45

Context

4

Context

5

Cap Gemini (2012) Benchlearning Report:-• eInlcusion intermediaries have difficulties in defining targets

and planning monitoring or evaluation activities. • Where evaluation was undertaken activities were generally

regarded as immature. • Projects and policies ‘speak different languages’.Digital Britain report (2009) - Lack of appropriate alignment of

the initiatives to ensure that they are efficiently delivered and complement each other.

The MIREIA Impact Assessment Tool should help to overcome this lack of understanding.

Conceptualisation

6

Government• Achieve policy goals• As cost efficiently as possible• ‘What works?’• Share better practices

Participants• Enter a course that will improve life chances• Share experiences to enhance impact for others

Intermediaries• Achieve organisation goals• As cost efficiently as possible• As effectively as possible

MIREIA Impact Assessment Framework

Conceptualisation

7

Methodology underpinned by theoretical considerations• Institutional theory - Role of intermediaries• New Institutionalism - Cultural explanations for behaviour• Theories of change - Interventions and outcomes• Organisational learning - Feedback into the planning process

Wide variety of theoretical perspectives that can be utilised toexplain the multi-dimensional construct of eInclusion and

the role of intermediaries.

Conceptualisation

8

Misuraca G., Torrecillas, C., (2012), 'Methodological approach for developing the MIREIA eInclusion Intermediaries

Approach

9

Approach

10

Participants1.Online questionnaire for

current cohort

2.Online questionnaire for

previous cohort

3.Online questionnaire for

counter-factual group

Intermediaries1.General information about the intermediary

2.Intermediary inclusion activities/initiatives

3.Relative intensity of initiatives

4.Evaluation criteria previously used

5.Relative importance of evaluation criteria

6.Importance of evaluation criteria

7.Performance assessment model

8.Instruments feedback questionnaire

Eleven instruments or tools underpin the approach

Counterfactual considerations

1112.05

Evaluating the impact of policies is a complex task because one would like to know what would have been the value for a given output/outcome variable in the absence of the incentive. But evaluators cannot know what would have been the behaviour of a treated subject in the absence of treatment. Then the main issue is to identify the appropriate control group in order to estimate the counterfactual

Obstacles in estimating the counterfactual•The 'selection bias’ consists in the fact that target population differs from counterfactual population due to pre-intervention features. A solution is the introduction of identification hypothesis stating that pre-intervention variables are sufficient to 'reconstruct' the control group of non-beneficiaries (counterfactual)

•The presence of spontaneous dynamics, due to the fact that target population differs from control population for the trend of the result variable. A solution is the introduction of an identification hypothesis to take in consideration the spontaneous dynamics of the result variable trend

Propensity Score Matching (Instruments 9)For each initiative that has been treated, it is possible to find at least one non-treated

initiative that is “close” enough to the treated counterpart, meaning that it exhibits a value for the propensity score very similar to the one observed for the treated initiative. The propensity score is defined as the conditional probability of receiving the treatment. After having computed the propensity scores for all the firms in the dataset, it is possible to use this value to match firms in the treated group with at least one firm in the control group. This approach is useful for ex-post evaluation of impacts as in the case of Guadalinfo and INNECIA initiatives (Spanish case) that were concluded before MIREAIA project activities have started.

Randomised controlled trials (Instruments 9)When the treatment is made available to firms/organizations on the basis of a random

process, we do not expect structural differences between those who are treated (and receive funds or support) and those who are not, so that we can use the non-supported organizations as a control group for comparison with the former group. This approach is useful for impact evaluation of planned activities as in the case of Pane_Internet_Lavoro initiative (Italian case) that has been both planned and evaluated during MIREAIA project life-cycle.

Of

12

Counterfactual impact evaluation and instruments 7a and 7b

Propensity Score Matching Pathway

The data gathering campaign is still open

59 valid set of responses

280 valid set of responses

Counterfactual Impact Evaluation

for MIREIA IAF

Counterfactual Impact Evaluation

for MIREIA IAF

Assignment to the treatment based on project/entrepreneur

characteristics

yes

no

Propensity Score Matching Procedure

14

By using the propensity score matching procedure we are able to define a target (treated) and a control (non-treated) group so as to assess the impact of the treatment on the relative performance of the subjects avoiding selection and spontaneous dynamics biases

Example: Guadalinfo-INNECIA (Spanish case)

Impact variables• Number of employees, number of customers, stakeholders' commitment• Survival of the initiative, success of the initiative, financial stability• ICT knowledge of the responsible, entrepreneurial knowledge of the responsible

Matching variables• Age, level of education, gender• Former involvement in social innovation activities, support from other initiatives• Work experience, previous employment sector, type of product

Propensity Score Matching Results

Headline results from the Spanish Guadalinfo case study

15

Variables

Matching MethodologiesKernel

matchingRadius

matchingStratification

matching

Number of employees + + +

Number of customers + + +

Survival of the initiative + + +

Success of the initiative + + +

Financial stability + + +

Stakeholders' commitment + + +

ICT knowledge of the responsible + + +Entrepreneurial knowledge of the responsible + + +

Randomised Control Trial pathway

The data gathering campaign is still open

99 valid set of responses

21 valid set of responses

61 valid set of responses

Impacts evaluation (outside MIREIA

initiative)

Impacts evaluation (outside MIREIA

initiative)

Outcomes evaluation (outside MIREIA initiative)

Outcomes evaluation (outside MIREIA initiative)

Open call; priority list based on subscription order

yes

no

Randomised Control Trial ProcedureBy using Randomised Control Trial procedure we are able to randomly define a target (treated) and a control (non-treated) group so as to assess the impact of the treatment on the relative performance based on two sample of the same target population

Example: Pane_Internet_Lavoro (Italian Case)Impact variables• Behavioral changes in using internet for job searching (e.g. h/week of internet usage for job searching; monthly number of job searching on line advertisement replied; monthly number of job searching portals consulted; monthly numbers of job offer received by e-mail) • Behavioral changes in job-searching methods (e.g. direct contacts; relatives/friends contacts; Internet social network contacts; company on-line forms; job intermediaries on line forms)Control variables• Age, level of education, gender• Labor conditions in the last three years: sector; average yearly gross salary; average number of working months• presence of computer and interned at home• degree of knowledge in using computer and internet

17

Randomised Control Trial ResultsHeadline results from the Pane_Internet_Lavoro Italian case study

18

Ab

solu

te n

um

ber

s

Key Behavioral changes

Proxi of expected impact

T-sud0,05

T-sud0,01

T-sud<0,001

T-sud<0,01

EUROPEAN ADDEDD VALUE

SOCIAL OUTCOMES

Attribution issues and/or time lag

OUTCOMES ADDITIONALLY

Adequacy of funding strategy to the

eInclusion needs

Efficiency in allocation and deployment of

funding

Adequacy of Risk

management approach

Private and public funding

GLOBAL IMPACT: SMART & SUSTSINABLE

GROWTH

IMPUT FOR eINCLUSION

STRATEGY CHANGES

Motivation of the Instruments developed for the study

ParticipantImpact

EvaluationTool

Performance Assessment

Tool

Evaluation of eInclusion Intermediaries’ actions

impacts per single intermediary or groups

INPUTS

PROCESSES

OUTPUTS

BEHAVIOURAL ADDITIONALLY (Process-based)

OUTCOMES ADDITIONALLY

(Resources-based)

OUTCOMES ADDITIONALLY

(Resources-based)

Aggregate eInclusion measures of

Intermediaries’ actions impacts per single

intermediary or groups

Impact Measurement

Tool

Intermediaries in various organizational forms: single organization; network; value chain

12.10 to 12.30

MIREIA methodology tested and testing instruments

ParticipantImpact

Evaluation Tool

ParticipantImpact

Evaluation Tool

Context Analysis framework

Context Analysis framework

IntermediaryPerformence

Assessment Tool

IntermediaryPerformence

Assessment Tool

Impact Measurement

Tool

Impact Measurement

Tool

Data gathering

tools

Instruments 1;2;3

Instruments 9a, 9b, 9c

Instruments 4;5;6

Instrument7

12.10 to 12.30

Impact measurement problem

21

APPROACH

1- Analytic Hierarchy Model(Saaty, 1987; Mansar, 2006; Kahraman, Demirel, & Demirel, 2007; Saaty, 2008; Parra-Lopez, Groot, Carmona-Torres & Rossing, 2009; Saaty, & Vargas, 2012)

2 – Literature review(some example of reference used: ICT4ALL, 2008; Codagnone, 2009; Bentivegna S., Guerrieri P., 2010; Garrido, et All. 2012; Tinholt et All, 2012)

3 – Stakeholders engagement•Workshops (IRL; ITA; POL; SPA)•on-line and off-line interviews•Check-lists (instruments 4, 5, 6)

Instruments developed for the study

Impact measurement and instruments 4, 5

Instruments 4 and 5 – impact measurement criteria identification to measure an e-inclusion for employability initiative

EfficiencyEffectivenessSocial Value generatedSustainabilityAlignment (strategic; organizational; technological)Compliancy

Degree of usage (Instr. 4; Likert scale)Degree of importance (Instr. 5; Pairwise comparison)

Workshops (IRL, ITA; POL; SPA)

Interviews (on-line; off-line)

Weighting systems for the Impact measurement tool

From literature review –first set of criteria identified:

Pairwise comparison method

1. Each participant to the consultation express is pairwise opinion on criteria

2. The eigenvector of the matrix define the weighing system of each stakeholder

3. The eigenvectors’mean define the weighting system amongst participants to the consultation

ADVANTAGES:•More accurate participants responses•Transparent negotiation technique•Easier to reach consensus amongst stakeholders

Wide literature on the matter, e.g. Saaty,1997; Munda, 2004; Gasparotos et al., 2008; Nordstrom et al., 2012

Weighting systems

Measurement criteria weighting system (Italian Stakeholders feedback)

Measurement criteria weighting system (Spanish

Stakeholders feedback)

Impact measurement and instruments 6Instruments 6 – impact measurement indicators identification to measure an e-inclusion for employability initiative

Example: assessment variable for the “efficiency” criterion

3 dimensions analysed with a Likert scale:

• Degree of importance• Degree of measurability• Degree of usage

Workshops (IRL, ITA; POL; SPA)

Interviews (on-line; off-line)

Most important Impact measurement indicators on

an e-inclusion for employability action

Performance assessment and instrument 8

27

The PA model is based upon five critical success factors driving Intermediary organizations in their services provisioning (Lettieri, Borga, Masella & Savoldelli, 2004; Savoldelli, Borga & Lettieri, 2004; Codagnone, 2009; Limassol report, 2009; Osimo, De Luca & Codagnone, 2010).

Performance assessment and instrument 8

28

1 dimension analysed with a Likert scale:

• Degree of importance

Workshops (IRL, ITA; POL; SPA)

Interviews (on-line; off-line)

Most important Performance assessment

criteria to measure Intermediary’s

organization functioning

Participant instruments

2912.30

Current Cohort CounterfactualPrevious cohort

Background/Before: Gender, age, education, previous activity, ICT skills, ICT useActivity/during: Employability, basic digital skills, advanced digital skills, numeracy/literacy, start-up, community, inclusion

Activity/after: Employment/activity, training basic & advanced digital skills, numeracy/literacy, start-up, community, inclusion

Now, after/future perceptions: Employment activity & prospects, ICT use, Internet saving, income/benefits, inclusion, three years time

Participant instruments

30

Instrument appraisal

Intermediary views on the relevance of the instruments

3112.35 – 12.45

Intermediary views on the usefulness of instruments

32

Intermediary ranking of the importance of assessment criteria

33

Initial intermediary results

Intermediary ranking of the usefulness of assessment criteria

34

Participant results: Pane IT

35

12.45

Headline results from the Italian Pane_Internet_Lavoro case study

2: data gathering campaign still ongoing(*): on line interviews done three weeks after the treatment(**): sttaistical significance(***): statistical significance at p=0,05(****): statistical significance at p=0,01

Participant results: Pane IT

36

12.45

(*): on line interviews done three weeks after the treatment(**): sttaistical significance(***): statistical significance at p=0,1

Headline results from the Spanish Guadalinfo case study

37

12.50

Participant results: Guadalinfo, ES

Headline results from the Irish FIT case study

38

12.55

Participant results: FIT, IRE

393 June 2013

403 June 2013

413 June 2013

423 June 2013

43

Counterfactal group:-• 50% of those in paid employment received further training• Average paid employment income EUR 21,700 • 83% of those in education/training had undertaken courses

to improve their employability • 43% of those unemployed had undertaken further training

The previous cohort:-• 25% of those in paid employment have undertaken further

training • Average paid employment income EUR 23,500• 10% of those unemployed had undertaken further training

44

Indicative draft financial impact (not for quotation):-• FIT treatment removed 50% of participants (913) from unemployment -

welfare payments avg. EUR 13,000/year. FIT induced Treasury savings of EUR 11.9 million for one year.

• FIT additional unemployment impact (vs. counterfactual) was 13% of participants (237); (a) saving EUR 3.1 million

• FIT induced 25% (456) of participants into employment (self & paid) at an average income of EUR 23,500. Total income EUR 10.7m

• FIT additional employment impact (vs. counterfactual) was 10% ofparticipants (183). Total income EUR 4.3m for one year

• With a tax contribution of 10% - (b) Treasury income is EUR 0.43m• Net additional impact of FIT is EUR 3.53 million for one year (a + b).• FIT costs EUR 2.43 million per annum to run. Net benefit EUR 1.1m

.

Analysis and results

Consideration:

Workshop participants

4513.00 to 13.30

Conclusions

4613.58

• The MIREIA Methodology works, but could be simpler.• Policymakers obtain an insight to the performance and

outputs from intermediaries and the extent to which these are aligned with policy objectives.

• Intermediaries obtain an insight to performance and outputs achieved, subsequent outcomes and financial impact.

• Participants can obtain a better understanding of potential benefits – employment prospects, income.

• XQuestionnaire

47

Impact Assessment Development and overview

Paul Foley, Francesco Mureddu and Alberto Savodelli

Pdfoley@tech4i2.com@pdfoley

JRC-IPTS MIREIAExperts' Workshop on Measuring the Impact of

eInclusion Intermediaries in EuropeGranada, 29th May 2013

The views expressed by the authors are not necessarily those of the EC