Dimetic Pecs 07 / Evaluation of STI policy L. BACH 1 Evaluation of STI policy performance and...

Dimetic Pecs 07 / Evaluation of STI policy L. BACH 1

QuickTime™ et undécompresseur Graphismes

sont requis pour visionner cette image.

Evaluation of STI policy performance

and impacts

Laurent [email protected]

BETA, university Strasbourg

Pecs Session / Week 2 - July 2007

QuickTime™ et undécompresseur TIFF (LZW)



QuickTime™ et undécompresseur Graphismes


Evaluation of STI policy performance and

impacts

Pecs Session / Week 2 - July 2007



•A few definitions•The key question of additionality•(Socio-economic) Effects and impacts•Some examples•New challenges for an "ideal" evaluation scheme ?


EVALUATION:

“An evaluation is an assessment, as systematic and objective as possible, of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision making process of both recipients and donors.”(OECD 1987)

Analysis and assessment of goals, instruments and impacts (Meyer -Krahmer / Georghiou 1992)

1. A few definitions


EVALUATION OF S&T POLICIES:

Projects / programmesMechanisms / institutions(collaborative scheme, funding mechanism, IPR regulation...)

( Evaluation of local / wider S&T systems - Nat. or Reg. SI )

Policy

Individual actors (researchers...)

Teams, labs

Organisations (PROs, HEIs,

Tech Center, Science-industry

intermediaries...)

CompaniesNot-For-Profit orga.


CONCLUSIONDESIGN OF THE INSTITUTIONAL ARRANGEMENT- tools, modalities, rules, … -

IMPLEMENTATIONANALYSIS OF THE ENVIRONMENTALCONDITIONS

DEFINITION OF GOALSREALIZATIONtime

Evaluation at different steps of the process of public actions




DEFINITION OF GOALSREALIZATIONObjectivesInstitutional arrangementEffectstime


E1 - E4 : Relevance (content) and "quality" of conception (decision process)

E5 - E7 : Implementation and resultsE8 - E9 : Efficiency and legitimization

ObjectivesInstitutional arrangementEffectsEnvironmental

economic,

social,

political

etcconditions

E1E2E3E4E5E6E7E8E9

THE DIFFERENT LEVELS OF EVALUATION



Relevance (content) and "quality" of conception (decision process)

E1 : Relevance of objectivesE2 : Coherence of objectivesE3 : Relevance and coherence of the "institutional arrangements"E4 : Coherence between objectives and institutional arrangements


economic,

social,

political

etcconditions

E1E2E3E4E5E6E7E8E9



Implementation and results

E5 : Programme management (cost-timing-quality…) = monitoringE6 : Effects / outputs / impacts = effectivenessE7 : Match between effects and objectives = efficacy (1)


economic,

social,

political

etcconditions

E1E2E3E4E5E6E7E8E9



Efficiency and legitimization

E8 : Match between objectives - institutional arrangements and effects-outputs-impacts : do the same / do better another way =

assessing adequate funding, management, contractual behaviour in order for objectives to be achieved in a cost-effective manner = efficiency


economic,

social,

political

etcconditions

E1E2E3E4E5E6E7E8E9



Efficiency and legitimization

E9 : Ex-post relevance of objectives, given the results of the

other evaluations = assessing whether initial objectives are still valid in the light of evolving RTD, societal and environmental conditions =

efficacy (2)


economic,

social,

political

etcconditions

E1E2E3E4E5E6E7E8E9




DEFINITION OF GOALSREALIZATIONProgramme processEvaluationprocessEx anteEx postRE-DESIGNE1................................E4E5............E6.................E7........>E8.............E9


ADDITIONALITY and related concepts

What difference does State intervention (i.e. policy) make ?<=> What will happen (has happened) over and above what will happen (would have happened) anyway ?

Does this difference justify State intervention ?

= comparing situation «with policy» vs situtation «without policy»

• efficacy (1) : with policy vs objectives• efficiency : with policy vs with an alternative policy aiming at similar objectives• opportunity cost : with policy vs other use of public funds

2. The key question of additionality


resultstimewith policywithout policyADDITIONALITYbeforeafter

Different concept of additionality

Without policy : alternative scenario : null hypothesis ---> before/after comparison counterfactual scenario ---> “fictitious case”

control grouplevel of definition (policy, gvt,

actors,…)Time scale


Input additionalitywhether the public action adds to, or substitutes for the

agents inputs (usually financial) crowding out effects, displacement etc

Output additionalitywhether the same/different results would have been

obtained without policy action

• Neo-classical / standard approach (input-output approach)

Market failures

State : «out» of system Normative approach

Policy Rationales, Evaluation Rationales and Additionality

(Evaluator is independant)

Counterfactual -> substitution/crowding out - additive effects


Behavioural additionality (incl. Network additionality ?)the differences to the agent’s behaviour following the policy

actionthe persistence of differences to the agent’s behaviour

beyond actions carried out under policythe differences in network structures and actors' position in

structuresCognitive capacity additionality

whether the policy action changes the different dimensions of the cognitive capacity of the agent.……organizational add., structural add. ….

• Knowledge-based approach

• Innovation system approach

• Evolutionary approach

System failures

Knowledge/learning failures

Selection failures

State : part of the system Normative approaches ?

Policy Rationales, Evaluation Rationales and Additionality

(Evaluator is in the system)

Counterfactual -> complementary, multiplicative, catalytic effects etc


Additionality in FP5 assessment (Polt-Streicher, 2005)

What would you have done if the project had not received FP5 fundings ?(n=939 participants)

•57% would not have undertaken the project in the absence of EU funding

•36% of the remainder would have gone ahead :On a smaller scale 29%Less partners 23%Less ambitious objectives 15%Lower expectations of profit 14%More national partners 12%A longer time-scale 11%Less funds 7%

10% would have replace EU funds with their own funds20% would have replace funds with other external funds

13% would have done at least one thing "better" without EU funds


• Competitiveness and exploitation/market-related effects (products, prototypes, patents, processes, interfirm technology transfer…)•Individual and organisational learning effects (such as finding of new partners, networking, building of research infrastructure) and behavioural changes (such as increased collaboration skills)•Influencing norms and standards• Structural effects (firm size, regional distribution, trusts, cartels…)• Externalities : generic knowledge, RD flows, system and network externalities, impact on the environment, health, education …• Cohesion, diversity (in case of international programmes, especially European programmes)• Contribution to skills and research manpower

(Meyer-Krahmer / Georghiou 1992)

Economic evaluation vs economic effects, Direct vs indirect effects, short term vs long term effects, occasional vs steady effects, etc

3. (Socioeconomic) effects and impacts :




From ToolBox Report 2002

Codified - tacit ? Shared - distributed among actors ?Public - "club" - private …

One reference model : outputs - (outcome) - impact


Outputs vs•changes in routines•behavioural changes•structural & organisational changes internally to actors - between actors

Impacts

XXX

Inputs Outputs

System t=0 =Team, firm, project,consortia...

XXX

System t=1 =Team, firm, project,consortia...


Key points :PROPAGATION of effects

- short term -> long term- participants -> non participants- micro -> meso -> macro

SEPARABILITY among factors generating effects- project fallacy (timing and scope, cumulativeness)- complementary assets for innovation (incl. strategy)- between partners if collaborative projects

Outputs ++............. (outcomes).............> Impactsindicators indicators

Models, theorical backgroundactorsrelation between actorstime horizonpositive vs negative effects <=> reference


Dominant models 1 :

NO MODEL !! List of indicators "à la Prévert"


FONDAMENTAL RESEARCH

APPLIEDRESEARCH

DEVELOP-PMENT

MARKET INTRODUC.ProductionSales

USE by NON INNOVATORClient, imitator

Scientific publications

Scientific and Technical publications,

patents Technical

Publications Patents,

prototypes, pilots Products,

processes, services

+Sales, income… Effects on clients,

competitors, suppliers, partners, society as whole…

methods, models, data base,reports, lab notebooks, …

Dominant models 2 : linear model / input-output based

=> Econometric models (firm, sector levels)output = f (various inputs possibly incl. those with public origin, structural variables, dummy for participation to programme)


Firm 2product & processes

Market competition

(F2 consumer product)

•F2 profit•F2 customer benefit•other firms profit (F2 competitors, suppliers…)

other firms profit (F2 suppliers competing with F1)

Market competition(F2 interm.

product)

F1 profit

F1 profit

Firm 1 R&D

New knowledge

New/better products

lower costs

Market competition

(F1 consumer product)

F1 customer benefit

other firmes profit (F1 competitors, suppliers…)

Firm 3 knowledge

New/better products

lower costs

Market competition

•F3 profit•F3 customer benefit•other firms profit (F3 competitors, suppliers…)

Firm 4 R&D decision

effects

market spillover

knowlegde spillover

strategic spillover

Neo-classical microeconomic framework for the evaluation of R&D project :Private / public rate of return and spillovers (externalities)

Private Rateof Return

Spillovergap

Spillovergap

SocialRateof Return

Adapted from Jaffe

Dominant models 3 : microeconomic market based approachessales, price, consumers, suppliers, competitors, market interactions (market structures, externalities…)

=> private / public rate of return


Other approaches, more scattered :

•Network based approaches : Social networks : increase of number of

links, centrality, connectivity… => growing emphasis

Techno-economic networks CSI

•Evolutionnist approaches•Knowledge-based approaches

BETA approach (to some extent)


•EU FP - see Annex below for institutional settings and some references•EUREKA : http://www.eureka.be CSE (Continuous and Systematic Evaluation) + new evaluation study 2006 •USA : ATP programme http://www.atp.nist.gov

implementation of 1993 GPRA (Government Performance and Results Act) and 2001 PART (Program Assessment Rating Tool) => effectiveness, efficacy (type 1= goal attainment)•Various national experiences in :

Conf. OECD Paris 1997 ConferenceTokyo 2002 Conference Kuhlmann/Shapiro 2003 book, ASIF 2003 Toolkit 2002 Microeconometric apporaches (see Klette and al.)evaluation of basic research (2 conf in Vienna : http://www.fteval.at/conference06/)2005 Conf. of European Research Evaluation Network in Manchester

http://www.mbs.ac.uk/Research/engineering-policy/dti-evaluation-conference-2005.htm

4. Some examples


GRILICHES 1957 : hybrid cornMANSFIELD 1975-77 : industrial innovation (17)

1991 : fundamental research

1975-77PRIVATE SOCIAL

COST

•R&D + launch•RD adjustment for uncommercialized projects

•PLUS : R&D and launch (public funds)•if competitors : failure : R&D costs success : modify T

INCOME

•income linked to innovation•MINUS income related to displaced activities

•PLUS consumer surplus•MINUS income of competitors•PLUS income of imitators

• in average : PRIVATE Internal RoR = 25% SOCIAL (PUBLIC) Internal RoR = 56%•Private IRoR : from less than O% to 214%•Public IRoR : from less than 0% to 307%

•30% of innovation projects would not have been carried out if ex-ante evaluation<=> Justification of State intervention (output additionality)

INCOME - COSTS = Net Income (from t=0 to T)

EVALUATION OF THE PROFITABILITY OF INNOVATION


Case study of R&D efficiency in an ATP Joint Venture - A. Link 1997Case : Printed Wiring Board Research Joint Venture

91-96Cost $ 12.866 million ATP

$ 13.693 million industry$ 5.2 million US DoE--------------$ 31.759 million

AT&T, Texas Instruments, Hamilton Standard, Allied Signal, Sandia, Hughes Electronics, IBM

COUNTER-FACTUAL HYPOTHESIS - project basedprojects that would have been started in the absence of the ATP award (generally with at least one year delay) :91 : 7 / 2996 : 31/62

why ? : cost of research / related risk=> Evaluation only on those projects “comparing situation with ATP (actual) with what would have happened without (counter-factual)” input and output additionality Effects :•direct impact :

• Scale, scope and coordination efficiencies (estimated workyears saved by carrying out the research as a joint venture)•Testing materials and machine time savings•Other research cost savings•Cycle-time efficiencies : shortened time to put into practice new procedures and processes•Productivity increase in production

• indirect impact :•Technology transfer to firms outside the joint venture•International Comparison issues


BETA approach

RD programmes (RD content - project characteristics : limited in goals, time) especially collaborative ones

Ex. FP programmes - RD collaborative projects (public co-funding)

European Space Agence programme (public procurement)

Material Ireland (private/public funded technical centers)French national programme for SMEs (refundable grants)

Sampling (participants / projects)

Micro evaluation and sum up of results at sample level

Data collection based on direct interviews


•Effects : for participants onlyDirect effects : fit with objectivesIndirect effects : beyond objectives - various learning

effects affecting other activities than those related to evaluated activities

scientific and technologicalnetworking, relationalorganisational changes (internal)methods, modes of managementreputationcritical mass

•Identification of effects / learning processes (intra & inter partners)

•Quantification : products/services sales, cost reductions, new research

projects, revenues from licenses etc"value" of critical mass"coefficient of influence" : very seldom 100%, minimal

estimates


EUREKA 2006 : the "Iceberg" Model

Survey + Case studies - partly based on BETA approach

Sales of innovative productReduced process costLicence income--------------------Firm strategy, organisation and method learningUse of technology in other parts of the businessNew contracts/netwoks & prestigeEmployment, competences and training-------------------Spillovers to non participantsUse and social benefits


Trends in S&T policy making

Multi-levels multi-agents decisionVariety of goalsVariety of Inst. Arrang.Coupling with other policies

International collaboration / integration (EU)

Flexibility / adaptive - learning policy makers

Privatization / contractualizationDevelopment of competition-based programmes

Trends / challenge for evaluators

Complexity/SeparabilityMultiple stakeholders

–Coherence of objectives–Weight of different dimension of the evaluation–Handling/using the results

Evaluation of policy mix

Mix of « evaluation cultures »

Evolving schemesEvaluation of shifts

Legitimacy, project fallacy

4. New challenges for an "ideal" evaluation scheme ?


Globalization

Solving problemorientation

Interdisciplinarity

CooperationS-I linkages

Increasing knowledge content

IPR regulations

Devpt of ICT

Trends in research activityTrends / challenge for evaluators

International dimensionBenchmarking

Bias toward short termmarket-type output

Limits of peer reviews

Separability/ComplexityNetwork evaluation

Evaluation of knowledge/competences/capabilities / capacity

Limits of bibliometrics


Complexity : no single method ; multi-interdisciplinarity /quantitative and qualitative; WITH alignment between (implicit or explicit) theoretical basis about which effects are to be evaluated

Guaranteing the scientific value of evaluation methods (Robustness, Repetability, Appropriability, Transparence, Independance of the evaluators, Confidentiality, Sampling…)

Balance between systematic / standardized /simple approaches and exploratory studies

=> systematic : Evaluation vs monitoring, time fit between R&D activity and evaluation, Availability of data, Evaluation “fatigue“, Cost of evaluation

Towards a « good » evaluation system ?


Ensuring that evaluation takes place on a programmed and properly resourced basis

Providing « easy to understand » results while avoiding meaningless list of indicators / scoreboards

Providing a mechanism for feedback of the results into policy making (learning policy maker)

Interactions between academics, practitioneers, policy makers and research actors for better understanding of scope, relevance and needs => evaluation as a « social process »

Towards a « good » evaluation system ?


Evaluation as a « social process » (from L. Georghiou )

• Motives / interest relates to the actors :Those « being » evaluated

justification/legitimationlearning at operational levelgain new supports for public sources

Those who are the audience of the evaluation

accountabilityresources allocationlearning at policy level (pro-active evaluation)

Those performing the evaluation

academic interestconsultant business

Make purpose and context-dependancy clear before choice of approach


Evaluation as a « social process » (from L. Georghiou )

• Co-evolution of policy rationales/tools and evaluation rationale/tools

– 1970s modification of peer review to extend criteria– 1980s interest began in collaborative R&D programmes– 1990s rise of:

• performance indicators • emphasis on knowledge transfer indicators• commercialisation of research

– 2000s interest in;• evaluation of system capabilities eg national or regional

systems• aggregate or interactive effects of policies (“policy mix”)• effect of “soft” policy tools such as foresight• strategic and persistent effects on firms of public support

(“behavioural additionality”)


ANNEX : Institutional set-up for the evaluation :

the case of the EU Framework Programme

E1-E4 : EU Staff + Scientific/ Industrial/ « External » Advisory Committees + EURAB - European Research Advisory Board (design and implementation)

Selection (ex ante) of proposals : Panel of independant experts /call

E5-E9 : Monitoring and Evaluation System ( from 1995 on)FP 3 : validity, efficiency, effects + pro-active

dimensionFP 4 : focus and appropriateness, achievements

and effectiveness, management and efficiency

Further information on 5th EU FP evaluation system :http://www.cordis.lu/fp5/monitoring/home.html


2 levels of evaluation :Specific programmesFramework programme

2 types of evaluation :MonitoringFive-year assessment

Creation of an Evaluation Unit

+ various evaluations from various High Level Groups


Monitoring--> day to day : EC--> annual (continuous annual reports) : panel of

independant experts (June 2005 on 2003 activities)

methods :experts panels, indicators, qualitative evidence,

interviews,etc target audience :programme managers and committees, CREST, FP

management

cost-effective implementation E5progress in relation with objectives E6/E7objectives, resources still appropriate E8/E9

http://europa.eu.int/comm/research/reports/index_en.html


5-year assessment before new FP proposal : panel of independant experts

methods :experts panels, indicators, qualitative evidence, interviews, surveys, report from monitoring, … target audience :programme managers and committees, CREST, FP management,European Parliament, Council, Economic and Social Committe

relevance E9management and effectiveness E6-E7efficiency E8+ recommandations E1-E4


Example of 5-year assessment :dec. 2004 « Ormala Report » on 1999-2003

http://europa.eu.int/comm/research/reports/2004/fya_en.html

need for (among others) : -focus on science / radical innovation-precise and simple definition of European Added Value-higher involvment of industry, high tech SME-simplification of administrative procedures-more support on HR/mobility


Networks of Excellence• progressive and lasting integration of research capacities existing• advance knowledge by pooling a critical mass of competence and skills. =virtual centre of excellence / long term objectives.

evaluation• added value : the scope of evaluation will extend beyond activity directly funded by the Commission • multiple pathways by which research affects the economy (focus on production of people trained)• additionality will lie both in the immediate EAV generated and in the persistence of the integration beyond the funding period• long term perspective, incompatible with the precise specification of outputs and effects

European Research Area and FP 6


Integrated Projects•aim to reinforce European competitiveness or to contribute to the solution of important societal problems through the mobilisation of a critical mass of research and technological development resources and skills existing•clearly defined objectives in terms of scientific and technological knowledge

evaluation• socio-economic dimension• critical scale / meso-level evaluation



Integration of research policies• extend beyond the evaluation of the Framework Programme. • mutual understanding of :

what has been achievedthe balance of benefitswhat constitutes quality and excellence in each country.

• take into account widely varying institutional settings for the same work

BenchmarkingSee for instance : http://trendchart.cordis.lu/index.cfmMisuse of data without contextualisation

Mobility and Cooperation with Third CountriesTowards assessment of capability and potential.



Examples of strategic evaluations :

-clarification goals/roles of different instruments (old and new ones)-need of continuity of tools along different FPs-the question of the size of NoE and IP-administrative burden => two-step evaluation-special attention to SME and « new comers »

-clarification, diversity and coherence of tools (mission oriented strategic research JETI / bottom-up ERC / old tools / FP6 tools)-5-10% of budget for new initiaives - flexibility / openess-new specific instruments for SME (industrial PhD, SBIR-like, info centres)-stop budget cuts / evaluation for stricter selection (=focus vs disseminate)-better multi-level coordination (reg., nat., levels)

• Evaluation of the effectiveness of the new instruments (Mid-

term FP 6)Morimon High Level Group Report July 2004 E8

http://www.cordis.lu/fp6/instruments_review/

• EURAB report on FP 6 assessment (+focus on instruments and FP7)April 2005 E8-E9 (http://europa.eu.int/comm/research/eurab/index_en.html)



European Research Advisory Board (EURAB) recommandations for evaluation

•Excellence as a criterion•Ensuring a quality Evaluators database•Distinguish dissemnitation plans for basic/applied project proposalsSuggested Quantitative Indicators: MID-TERM Evaluation.

Number of: IPs and NoEs with scores above threshold

Researchers/countries/disciplines involved Companies/SMEs involved Joint publications (abstracts/proceedings) Ph.D. Students in training in the consortium Shared facilities & resources Staff exchanges per total no. of researchers Patent submissions

Percentage of: Proposals above threshold that were fundedUse of shared facilities by other participants Budget used in project administration Budget retained in the project account Private investment so far



Suggested Qualitative Indicators: MID-TERM Evaluation. • Real participation and role of key players of high reputation• Are joint research plans being made for a future period ?• Joint standards and methodology (progress, intent etc) • Extent to which network has been structured o Regular meetings o Consortium

agreements • Submissions to journals with top 20% impact factor ratings in relevant field • Base for new legislation • Image/ Advertising/ participation in technical workshops • Views of participants regarding Management etc • Activities aimed at developing public interest in science • Gender disaggregated data




Suggested Quantitative Indicators: FINAL Evaluation Number of: Companies/SMEs still actively involved

Joint publications & proceedings Newly trained Ph.D. Students Shared facilities Staff exchanges per total no. of researchers Patents and Patent submissions

Percentage of: Use of facilities by other participants

Budget used in project administration Private investment so far Increasing market share/ new/ income New Business Success in leveraging additional funding




Suggested Qualitative Indicators: FINAL Evaluation

• Involvement of key players of high reputation • Joint research & development plans • Joint plans for continuation of collaboration & activities to develop same • Degree of clustering with other networks • Joint standards and methodology • Publications in journals with top 20% impact factor ratings in relevant field • Bases for new legislation • Image development/ Advertising/ participation in technical workshops • Teaching, Training and Framework Development • Activities aimed at developing public interest in science • Gender disaggregated data



Ex-ante E1+ E2-E4 (broad principles)

Impact assessment and ex-ante evaluation (April 2005) Staff working paperSee in "basic documents" at http://www.kowi.de/en/fp/fp7/fp7.htm

Context - policy options - institutional arrangements options - additionality / proportionality

• Aggregate economic impact (econometric model for growth - jobs - competitiveness)• Aggregate social impact• Environmental impact• Contribution to other EU goals / policy : R&D intensity - Research employment (Barcelona) + Sustainable development - etc


• Double the EU research budget (€ 72.7 billion over 2006-13)Knowledge-Education-Innovation triangle - Lisbon/Barcelona strategy• Foster the fundamental research (European Research Council), notably by competition between teams (programme "Ideas")• Human potential / reinforce attractivity of European research ("People")• Reinforce European research infrastructure ("Capacities")• "Cooperation"

• Develop poles of excellences• Externalize part of the management• Simplify institutional arrangements• Reinforce coordination of national policies

Towards FP 7 ...main features (see F. Meyer-Krahmer)


Towards FP 7 ... Requirements for evaluation

More needs for evaluation

Evaluating critical mass, Euopean Added Value,

infrastructure, coordination, externalization

Evaluating fundamental research...

+Draw lessons from FP5/6 evaluation system ->

re-organization


From N. Reeve

Planning Programming Evaluation Directorate General Research , European Commission, Nov. 2005

FP6 FP7

Research objectives Robust hierarchy of outcome objectives with appropriate indicators

Data collection (contract process, reporting)

Systematic and simplified collection of data

Annual Monitoring by independent experts

- Monitoring of implementation by senior Commission management -Indicators to track progress

Mid-term evaluation (Science panels)

Five Year Assessment by high-level independent experts

Ex post assessment of an FP, 2 years after its completion by high-level independent experts

Impact surveys at FP level Strengthened programme of coordinated strategic-level evaluations

Evaluation studies at operational level

Evaluation studies at operational level (portfolio, programme)

National impact studies Coordinated national impact studies

Ad-hoc research-related activities

FP research on evaluation tools and approaches

Dimetic Pecs 07 / Evaluation of STI policy L. BACH 1 Evaluation of STI policy performance and...

Documents

Transcript of Dimetic Pecs 07 / Evaluation of STI policy L. BACH 1 Evaluation of STI policy performance and...