Evidence, Innovation & Impact Evaluation:

who is the third wheel?

Some considerations emerging from the analysis

of preliminary findings of the IESI Research

Gianluca Misuraca

Senior Scientist, Information Society Unit

Institute for Prospective Technological Studies

European Commission, Joint Research Centre

Brussels 20-21 November 2014

Workshop on Evidence for Social Investment

The views expressed by the author are not necessarily those of the

European Commission

Joint Research Centre

Information Society Unit

Researching the impact of ICT-

enabled innovation on the EU

economy and society

Institute for Prospective Technological Studies

Understanding the links between technology,

economy and society

Serving society

Stimulating Innovation

Supporting legislation

Bridging science &

policy…

Or at least trying to…

'Evidence-based policy'

vs

'Policy-based evidence'

Po

licy M

akin

g

ICT in Social Policy

ICT-enabled

Assessing impact

Social Innovation

The evidence challenge

SUPPORT

SIP

Inventory MAPPING Initiatives

Outc

om

es

Case studies

Social Services

Integrated care

The master scholar should examine the present,

in light of the past and for the purpose of the future…

(John Maynard Keynes)

INDEPENDENT

LIVING

Social Return on

invest

ment

The role of science… …in support of policy

JRC-IPTS research on ICTs in support of social policies reforms

ICTs impact and support to:

ICTs for learning and education

ICTs for governance

ICTs for active and healthy

ageing focusing on

independent living and elderly

care at home

Inclusion of immigrants and

ethnic minorities (IEM);

Youth Inclusion;

The role of eInclusion

Intermediary actors

Limitations and challenges

Conceptual 'fuzzyness' and 'moving targets'…

Unknown universes and unclear 'unit of analysis'…

Unavailability / poor quality of data…

Lack of scientific rigorous evaluation…

Limited assessment capacities of 'actors' involved…

Lack of robust methodological tools for data gathering / impact evaluation

'Evidence building' for policy support / rather than just filling knowledge

gaps… with different time expectations and approaches…

7

Source: Codagnone, Misuraca, Savoldelli 2013

The fundamental problem

of impact evaluation

How to overcome the 'causality' limitation?

focus on

'employability'

Map eInclusion intermediary actors

across EU27 to better understand their

characteristics and policy potential;

http://is.jrc.ec.europa.eu/pages/EAP/eInclusion/MIREIA.html

11/ 14/ 13 1:48 PMJoint Research Centre - JRC - European Commission

Page 1 of 2http:/ / ec.europa.eu/ dgs/ jrc/

First EU e-Inclusion map measures the potential for improved digital literacy

An EU-27 survey of intermediary organisations operating on the education,social and employment sectors and providing IT training has produced afirst ever assessment of the e-Inclusion intermediary sector. It accounts fora total of 250,000 organisations, or one e-Inclusion actor per every 2,000inhabitants. One in two employs 10 staff or less and operates on a budgetsmaller than €100,000. Half of the e-Inclusion actors go further and offeremployment–related training. And for two out of three, local governmentfunding is the main financial resource.

14/11/13

Assessing the technological possibilities and economic value of electricity storage

Spurred by a renewed interest in power storage, the JRC provided severalrecommendations in a recently published report on how to improveassessing the economic value of storing electricity. Drafted in cooperationwith the R&D department of Electricité de France (EDF), it presents anoverview of the current research on the economic drivers or barriers (suchas high costs, restrictions to offer certain services) for electricity storage.13/11/13 Expand

First post-typhoon damage assessment for Tacloban City (Philippines)

More than 700 residential buildings were completely destroyed and morethan 1200 were damaged by the Haiyan typhoon in the Tacloban city alone(Philippines). This is the result of the first rapid damage assessment carriedout soon after the event by the Copernicus Emergency ManagementService (EMS), coordinated by the European Commission's Joint ResearchCentre (JRC). The damage assessment maps are expected to supportresponse activities and, in the longer term, reconstruction efforts.

11/11/13 Expand

JRC software to assess impact of cyber-threats against physical infrastructures

The JRC has developed innovative software to assess the cyber-security ofconnected critical infrastructures (CIs), such as railway systems, energynetworks or power plants. The AMICI software, which stands forAssessment platform for Multiple Interdependent Critical Infrastructures,provides a novel experimental approach as it takes into account both thevirtual and the physical aspects of modern interconnected CIs.07/11/13 Expand

New method to study mountainous terrains with satellite imagery

JRC scientists obtained remarkable results in correcting surface shapes ofsatellite imagery to study mountainous terrains, with a combination oftopographic correction algorithms and statistical methods. The JRCpresented its new method in a recently published article in the IEEE Journalof Selected Topics in Applied Earth Observations and Remote Sensing.05/11/13 Expand

Eight metals used in low-carbon energy technologies under risk of shortages

Search here Search

The Publications Repositoryprovides access to JRC researchpublications.

Business, citizens andenvironment to benefit fromfree access to EU satellitedataCommission publishes Factsand Figures on OrganicFarming

EC Report highlights thepotential of PersonalisedMedicine

€40 million for new EUresearch on resourceefficiency

14/11/13Scientific support for publichealth: existing actions, newchallenges and Europeanadded value29/11/13Round Table discussion onScientific Support to EnergyEfficient Buildings04/12/134th International ColloquiumScientific and FundamentalAspects of the GalileoProgramme03/03/142nd International Symposiumon fishery-dependentinformation

JOINT RESEARCH CENTREThe European Commission's in-house science service

European Commission JRC

FAQ Index Contact Search Services JRC websites Privacy statement Legal notice

The framework has been tested using counterfactual impact evaluation methodologies

in four case studies at national and regional level in Ireland, Italy, Poland and Spain

Design and test a methodological framework to enhance capacity of eInclusion

intermediary actors and engage them to collect data and to measure their impacts

Building evidence and assessing impacts of ICT-

enabled social innovation in support of the SIP

Overall objective: To contribute expanding the knowledge of how ICT-enabled social

innovation can support the implementation of social policy reforms and providing evidence

of initiatives that are relevant for the implementation of the SIP

Specific objectives:

To provide a deeper understanding of how EU Member States can make better use of

ICT-enabled social innovation to implement the actions suggested by the SIP

To contribute building a knowledge bank on social policies as foreseen in the SIP, by

providing the results of a structured analysis of ICT-enabled social innovation initiatives

To develop a methodological framework of analysis of the impacts generated - by ICT-

enabled social innovation initiatives `promoting/embodying´ social investment

Focus on social services

providers

• Systems productivity

• Access and take up of services

• Quality and cost-effectiveness

of services

• Policies/services meeting the

needs of final beneficiaries

Focus on Beneficiaries • Promote active inclusion and support

inclusive labour markets

• Support social inclusion, education

and training, employment and civic

participation, especially of youth

• Promote access and use of early

childhood education and care

• Increase the capacity of elderly to

manage self-care and independent

living

ICT-enabled social

innovation

Social Investment goals Spending more effectively and efficiently to ensure adequate and sustainable social protection;

Investing in people's skills and capacities to improve opportunities to participate in society and the labour market;

Ensuring that social protection systems respond to people's needs at critical moments during their lives

Socio-economic contextual factors /

Welfare system and governance model characteristics

Needs of specific target groups addressed

Conceptual Framework

Source: Misuraca, G. Colombo, C., Radescu, R., Bacigalupo, M.,(forthcoming), European Commission, JRC-IPTS (IESI D1.2.1)

What is ICT-enabled social innovation?

Our working definition

'A new configuration or combination of social practices which solves the unmet needs of

individuals throughout their lives and emerges from the innovative use of Information and

Communication Technologies (ICTs) to establish new collaborations and relationships among

stakeholders and foster open processes of co-creation or reallocation of public value.'

Policy relevant experiences and initiatives, involving ICT-enabled innovations, in designing

and implementing services, systems or social policies more efficiently and effectively,

addressing final beneficiaries, intermediary actors or public administrations.

Research scope

Where can it be found?

Our unit of analysis

2014 2015 2016

Experts and Stakeholders' Consultation (i.e. peer-reviews, advisory board, workshops, events)

IESI FRAME (Methodological framework of analysis of impacts)

Year 1

Mapping & case study analysis Year 2 Year 3

WS

2014

WS

2016

WS

2015

Mapping & case study analysis

Mapping & case study analysis

Conceptualisation Refinement & Consolidation

Validation & Reccomandations

Research design

Beyond public sector's provision of social services…'…

in search of maximising 'public value'…

Public Sector driven initiatives Private Sector driven initiatives

Third Sector driven initiatives Multi-sector partnerships

Social Assistance Employability & Employment

Education & Training

Child care and Social care

Social Inclusion AHA and LTC

Social Housing

14

Third Sector

Private Sector

Multi-sector

Public Sector

ICT-ENABLED

INNOVATION

POTENTIAL Incremental Sustained Disruptive Radical

TYPES OF SERVICE INTEGRATION

Intra-governmental

Inter-governmental

Inter-sectoral

Isolated

Pervasive

WEAK/FUNCTIONALIST STRONG /TRASFORMATIVE

Mapping initiatives against their ICT-enabled innovation potential and type of integration

15

The IESI Knowledge Map

2014 – A static view…

Third Sector driven initiatives

Private Sector driven initiatives Multi-sector partnerships

Public Sector driven initiatives

ICT-ENABLED

INNOVATION

POTENTIAL Incremental Sustained Disruptive Radical

TYPES OF SERVICE

INTEGRATION

Intra-gov

Inter-gov

Inter-sect

Isolated

Pervasive

Alison

Aurora

Apps for Good

Boot

iTEC

NotSchool

FreaqOut!

MdE

NetAcademy

EmYouth

Youth Reach

HSH@NetwOrk

DIMENSAAI

MIXOPOLIS Surf to the job

W2ID

Jobcenter Plus

JeunESS

Samasource

Slivers of Time

IBSETBO

Task Squad

YEAH

MoMo

KIMPALE

At Risk

Shadow World Flabs

Timely information for citizens

Fosternets

Be my eyes

Cybermentors

The Joint Strategy

DSDW

ServiCon

CBSS

Social Counters

EMSS

MSISS

BOM

Choices & Voices

MyVocab

BAC

MLeben

The Site

BTS

CHAIN

Savvy BrainAge

Taiwan Telecare

TDP

WDS

Kaiser Permanente Advanced Telecare

West Lothian

ACTION

PaPeRo WAD w SMS

TELBIL

ISISEMD

Te Whiringa Ora Telegerontology

eCare Bologna

PRISMA

Commonwell

Nexes

PACE

Torbay Care Truts

GiovaniSì

Single Stop USA

Some preliminary

considerations…

Mapped examples of ICT-enabled social innovations show:

High degree of service integration

The trend is moving towards inter-sectoral / multi-partnership initiatives with various levels of integration (e.g. funding, organisational, administrative…)

a functionalist conception of social innovation

100% are ‘Need driven/outcome oriented' and 28% presents ‘Open process of co-creation/collaborative innovation’

more than a transformationalist conception of social innovation

48% support a ‘Fundamental change in the relationships between stakeholders’ and 34% generate 'Public value allocation and/or re-allocation’,

However

59% of the initiatives use ICTs for transformative innovation (43% disruptive and 16% radical) whereas 11% use ICTs for an organisational/sustained innovation

With many caveats…

…due to the non-representative sample of initiatives mapped so far

Development of a methodological framework to assess the impacts generated - from

micro to macro level - by ICT-enabled social innovation initiatives promoting social

investment (IESI-FRAME)

to provide a framework of analysis of the initiatives collected through the Mapping

activities and as a guide to conduct the in-depth analysis of case studies

structured approach to analyse such initiatives and provide insights for their

replicability and transferability;

to serve as a framework for conducting analysis of social return on investment of

initiatives, which have as key component ICT-enabled social innovation

recommendations on how the European Commission and Member States

could analyse (ex-ante, in-itinere and ex-post) the impact of ICT-enabled social

innovations initiatives

Assessing impacts of

social investment

IESI Way forward

ICTs can support social innovation and social policy reforms

facilitating user-centred services and user involvement in the innovation process,

service design and provision;

creating new roles for stakeholders or new power-relations, enabling network effects

and increasing the outreach of the innovation process.

But, impact evaluation of ICT-enabled social innovation need to be embedded

into mainstream social policy reform instruments, e.g.:

ESF - European Social Fund with regard to ICTs for digital skills and inclusion

EaSI – Employment and Social Innovation programme

…including ex-ante and ex-post experimental approaches to impact evaluation

Following examples from the SISPE – Social Innovation and Social Policy

Experimentation approaches

Final Considerations

Way forward:

next steps in IESI

Expand the mapping by collecting additional initiatives to broaden the

coverage of the research

Explore how different welfare system and social service provision

models influence the emergence of ICT-enabled social innovation in

different social investment areas

Investigate what are the enabling factors that influence the emergence,

deployment and diffusion of ICT-enabled social innovations in different

contexts, what the drivers and the barriers to their mainstreaming

Develop a methodological framework to assess the social and economic

impact of ICT-enabled social innovation initiatives

Anyone can contribute… Provide your inputs !!!

•gianluca.misuraca@ec.europa.eu