high impact

The Evaluation Partnership

Framework Contract No: BUDG-02-01 L2

Contract No: BUDG/01/EVAL/2003/08 (EVA6-CT-2004-000001)

Analysis of ‘high impact’ research activities

under Community Research Framework Programmes

FINAL REPORT

The European Commission

Research Directorate-General (DG RTD)

Submitted by:

The European Evaluation Consortium (TEEC)

The Evaluation Partnership Limited (UK)

Economisti Associati (Italy) Particip GmbH (Germany)

navreme knowledge development (Austria)

Authorised Representative The Evaluation Partnership Limited (TEP)

6 Cole Park Road, Twickenham, Middlesex TW1 1HW, United Kingdom

February 2005

Analysis of ‘high impact’ research activities under Community Research Framework Programmes Final Report


ii

Framework contract for evaluation and evaluation-related services

Analysis of ‘high impact’ research activities under

Community Research Framework Programmes

TABLE OF CONTENTS

EXECUTIVE SUMMARY............................................................................................ 1

I. INTRODUCTION ............................................................................................... 1

II. THE STUDY ...................................................................................................... 1

The work programme....................................................................................................................1 Scope and definition of high impact ..............................................................................................1 Limitations of the data...................................................................................................................2

III. KEY FINDINGS ................................................................................................. 2

Immediate Impacts (1st Group) .....................................................................................................3 Enabling Impacts (2nd Group) .......................................................................................................3 Broad Societal Impacts (3rd Group) ..............................................................................................4

IV. MAIN CONCLUSIONS ...................................................................................... 4

V. MAIN RECOMMENDATIONS ........................................................................... 6

ACKNOWLEDGEMENTS........................................................................................ 10

A: DESCRIPTIVE SECTION.................................................................................... 11

1 INTRODUCTION ............................................................................................. 11

1.1 BACKGROUND ......................................................................................................... 11 1.2 OBJECTIVES OF THE STUDY ................................................................................. 11 1.3 WORK PROGRAMME............................................................................................... 11 1.4 STRUCTURE AND CONTENT OF THE REPORT .................................................... 11

2 METHODOLOGY AND APPROACH .............................................................. 13

2.1 THE APPROACH ...................................................................................................... 13 2.2 DEFINING HIGH IMPACT ......................................................................................... 13 2.3 CASE STUDY SAMPLING AND SELECTION ........................................................... 14

2.3.1 Case study sampling...................................................................................................14 2.3.2 Case study selection ...................................................................................................15

2.4 CAPTURING THE EVIDENCE .................................................................................. 17 2.5 ANALYSIS AND IMPACT ASSESSMENT ................................................................. 19 2.6 DATA LIMITATIONS.................................................................................................. 20

3 NATURE OF THE SELECTED RESEARCH ACTIVITIES .............................. 21

B: EVALUATIVE FINDINGS.................................................................................... 27

4 IMPACT ASSESSMENT - ANALYSIS AND FINDINGS.................................. 27



iii

4.1 CHALLENGES IN MEASURING IMPACT ................................................................. 27 4.1.1 Adjustments in the definition of high impact................................................................27

4.2 IMPACT ASSESSMENT BY AREA OF IMPACT ....................................................... 29 4.3 1ST GROUP IMPACTS (IMMEDIATE) ........................................................................ 30

4.3.1 International Cooperation and Human Resources ......................................................30 4.4 2ND GROUP IMPACTS (ENABLING) ......................................................................... 33

4.4.1 Policy and standards ...................................................................................................33 4.4.2 Scientific (knowledge) .................................................................................................36 4.4.3 Technical.....................................................................................................................39

4.5 3RD GROUP IMPACTS (BROADER SOCIETAL) ....................................................... 41 4.5.1 Social...........................................................................................................................41 4.5.2 Economic ....................................................................................................................43 4.5.3 Quality of Life ..............................................................................................................46

4.6 CROSS-CUTTING IMPACTS .................................................................................... 49 4.6.1 European Added Value ...............................................................................................49

4.7 OVERALL ASSESSMENT ......................................................................................... 52 4.7.1 Implications for the classification of potential impact ..................................................56

5 CRITICAL FACTORS...................................................................................... 58

5.1 CRITICAL FACTORS ACCORDING TO THE PROJECT CYCLE ............................. 58 5.1.1 At the proposal stage ..................................................................................................58 5.1.2 At the implementation stage........................................................................................59 5.1.3 At the exploitation stage..............................................................................................62

5.2 OVERALL CRITICAL FACTORS ............................................................................... 63 5.2.1 Recognition .................................................................................................................64 5.2.2 Project theme, timing and level of innovation..............................................................64 5.2.3 Research Context and Attribution ...............................................................................65 5.2.4 The role of the EC (as a facilitator) .............................................................................67 5.2.5 Flexibility......................................................................................................................67 5.2.6 Unexpected factors .....................................................................................................68 5.2.7 Factors which may reduce impact...............................................................................68

C: CONCLUSIONS AND RECOMMENDATIONS ................................................... 71

6 PROJECT SUMMARIES ................................................................................. 71

6.1 MAFCONS................................................................................................................. 71 6.2 ONTOWEB................................................................................................................ 72 6.3 NESSIE ..................................................................................................................... 74 6.4 TTA............................................................................................................................ 75 6.5 REOXCOAT .............................................................................................................. 76 6.6 BO-BA ....................................................................................................................... 78 6.7 SCENES.................................................................................................................... 79 6.8 GALA......................................................................................................................... 80 6.9 GECS ........................................................................................................................ 81 6.10 CARE-W ................................................................................................................. 82 6.11 WETO 2030 ............................................................................................................ 84 6.12 CIEL ........................................................................................................................ 85 6.13 MAIZE ..................................................................................................................... 87 6.14 CHIN ....................................................................................................................... 88 6.15 FUNCARS............................................................................................................... 89 6.16 ENHSIN................................................................................................................... 90 6.17 FOREN.................................................................................................................... 91 6.18 GMO ....................................................................................................................... 92

7 CONCLUSIONS .............................................................................................. 94

7.1 IDENTIFYING AND MEASURING HIGH IMPACT ..................................................... 94



iv

7.2 HIGH IMPACT IN THE CASE STUDY SAMPLE........................................................ 96 7.3 CONCLUSIONS ON AGGREGATING FINDINGS AT FP LEVEL.............................. 99 7.4 CRITICAL FACTORS .............................................................................................. 100

8 RECOMMENDATIONS ................................................................................. 103

8.1 STRATEGIC CONSIDERATIONS FOR DELIVERING AND ASSESSING IMPACT 103 8.2 CONTINOUS IMPACT ASSESSMENT - ‘MANAGING FOR IMPACT’ ..................... 104

8.2.1 Steps to developing the results chain approach .......................................................105 8.2.2 Linking the proposed framework to FP6 ...................................................................107 8.2.3 Linking critical factors to assessment points.............................................................108 8.2.4 Proposed Impact Assessment Tool ..........................................................................109

8.3 OTHER MECHANISMS FOR MULTIPLYING IMPACTS.......................................... 112

APPENDICES (Separate document)

Appendix 1: Terms of Reference Appendix 2: Original Project Methodology Appendix 3: Critical Assessment Appendix 4: Project Selection Methodology Appendix 5: Stakeholder List Appendix 6: Impact Assessment Interview Tool and Guidelines Appendix 7: Project Bibliography Appendix 8: Project Evaluation Grids-Individual Case Studies (Containing full and detailed impact assessments and causality analysis) Appendix 9: Proposed Impact Assessment Framework Appendix 10: Structure, Content, and Development of the 4th, 5th and 6th Framework Programmes

TABLES

Table 1: Details of the projects selected for assessment........................................ 24 Table 2: Overall rating of international cooperation impact ...................................... 30 Table 3: Evidence of international cooperation impact ............................................ 33 Table 4: Overall rating of policy impact .................................................................... 34 Table 5: Evidence of Policy Impact .......................................................................... 35 Table 6: Overall rating of scientific impact................................................................ 36 Table 7: Evidence of Scientific Impact ..................................................................... 38 Table 8: Overall rating of technical impact ............................................................... 39 Table 9: Evidence of Technical Impact .................................................................... 41 Table 10: Overall rating of social impact .................................................................. 42 Table 11: Evidence of Social Impact........................................................................ 43 Table 12: Overall rating of economic impact ............................................................ 44



v

Table 13: Evidence of Economic Impact.................................................................. 46 Table 14: Overall rating of quality of life impact ....................................................... 47 Table 15: Evidence of quality of life impact.............................................................. 49 Table 16: Project end dates and highest impacts .................................................... 55

FIGURES

Figure 1: Criteria for Rating Impact .......................................................................... 18 Figure 2: Groups of Impact ...................................................................................... 29 Figure 3: Balance between actual and potential impacts in the sample................... 52 Figure 4: Average impact score for 3rd group impacts.............................................. 55 Figure 5: Average impact score for 2nd group impacts ............................................. 56 Figure 6: Average impact score for 1st group impacts.............................................. 56 Figure 7: Top Critical Factors for High Impact:......................................................... 63 Figure 8: Impact assessment framework with 4 appraisal points ........................... 107



vi

LIST OF ACRONYMS

DG Directorate General of the European Commission EAV European Added Value EC European Commission EESD Energy, Environment and Sustainable Development ERA European Research Area EU European Union FP Framework Programme GDP Gross Domestic Product GM Genetic Modification HIRA High Impact Research Activities IHP Improving Human Potential INCO International Cooperation IST Information Society Technologies JRC Joint Research Centre MS Member State NIS Newly Independent States OECD Organisation for Economic Cooperation and Development PHIRA Potential High Impact Research Activities QoL Quality of Life R&D Research and Development RTD Research and Technological Development SME Small and Medium Sized Enterprise TEEC The European Evaluation Consortium TOR Terms of Reference



vii

Submitted to: The European Commission The Directorate-General for Research (DG RES) Submitted by: Mrs. Mariell JUHLIN, Team Leader, Contract Manager, TEP Ms. Alice ROWLEY, Consultant, TEP Professor Keith A. HARRAP, Senior Associate Consultant Dr. Bernd BAUMGARTL, Executive, navreme Dr. Jochen FRIED, Programme Director, navreme Mr. Mihaylo MILOVANOVITCH, Research Assistant, navreme Quality controlled by: Mr. John P. WATSON, Managing Director, TEP



1

Executive Summary

I. INTRODUCTION

Historically, it has been difficult to measure progress of Community Research Framework Programme (FP) supported research activities against the EU’s higher level policy objectives, i.e. the Lisbon-agenda, the creation of ERA, etc., because of the perceived ‘attribution gap’ between research outputs and higher level outcomes. There is however a commonly held belief that ‘one thing automatically leads to another’, which perhaps has reinforced the use of quantitative targets and performance indicators as evidence of attainment of these higher level impacts, even if this is not necessarily the case. This study differs from previous studies in that it focuses on the attainment of higher level objectives at broader societal level and attempts to measure progress made towards these using a new and innovative methodology. The study was commissioned as one of several inputs into the Five-year Assessment of Framework Programmes conducted in 2004, and it covers the period 1999-2003. The aim of the study was to generate an understanding of high impact research activities and how they occur using a bottom-up, case study approach and by examining projects that had not previously been assessed. It further aimed to contribute to the setting-up of an overall evaluative framework for assessing Community supported research by developing a generic case study methodology which could then be used to assess and provide comparisons of high impact across future Framework Programme areas.

II. THE STUDY

The work programme The study was divided into two main stages: an initial desk study phase in which previous impact assessments of Community supported research activities were critically reviewed in order to inform the case-study methodology, and a case study selection, analysis and assessment phase in which 18 cases studies (4 within FP4 and 12 within FP5) were identified and assessed. The work was carried out by a team from The European Evaluation Consortium (and led by The Evaluation Partnership) on behalf of DG Research, during 10 months from March to December 2004. A total of 58 face to face and telephone interviews with project partners and external stakeholders were conducted.

Scope and definition of high impact The study aimed to go beyond the identification of mere ‘success stories’ by identifying and measuring outcomes, i.e. ‘high impacts’ on European science and society. This necessitated agreement on a definition of ‘high impact’, which was reflective of the objectives of previous and current Framework Programmes and against which performance could be measured. In short-listing high impact projects for study, it was agreed that projects that were seen to have the following attributes should be included:



2

• Making a difference, particularly on European level. • Fulfilling a need in society. • Improving policy-making, standards, job creation, quality of life, competitiveness

and/ or career development. • Changing the scientific agenda/ the Research, Technology and Development

(RTD) environment for the benefit of society. • Changing people's perceptions beyond the world of science. Based on these criteria a qualitative, evaluative framework was constructed in which qualitative ratings of impacts could be measured and compared across sectors and FP sub-areas, backed by quantitative and qualitative evidence collected through interviews with implementers and other stakeholders. The ratings criteria for impact reflected the list of attributes outlined above, with high impact being defined as any impact that went beyond successful implementation or outcomes to make a difference beyond for example one country, or additionally, where there was high sectoral impact/ impact on industry/ a regionalised population or a regional economy. Very high impact would mean making a difference on a European-wide level or beyond. To achieve the most objective result possible, and to avoid any bias through self-assessment, project partner ratings were triangulated with the perceptions of other stakeholders.

Limitations of the data It is important to stress, that these projects were selected on the basis of perceptions regarding their high impact and as such, do not comprise a representative sample of the entire research activity across FP4 and FP5. Nevertheless, by taking into account a vast array of additional criteria during the selection process, including representativeness across FP sub-programme areas, SME involvement, geographic spread, size and timing, the sample was seen to reflect the widest spread of projects possible while still emphasising the key question in the first place: high impact.

III. KEY FINDINGS

In addition to the qualitative ratings, based on the evidence collected from the 18 case studies, impacts were categorised into three groups according to type of impact and time frame in which they occurred to facilitate cross-project assessment. The time frame was a particularly important factor considering the diverse end-dates among sample projects (from 1998 to ongoing). The three impact groups included immediate impacts (those likely to occur within the project cycle), enabling impacts (essential building blocks for higher level societal impacts likely to occur within 3 years following project completion), and higher level - broader societal - impacts (likely to occur between 3 to 10 years after project completion). The difficulties experienced by many interviewees in providing positive evidence for actual impact, while convincingly reporting about potential impacts, prompted the evaluators to assess both actual (to date) and potential impacts.



3

EXAMPLES Evidence of high scientific impact: • Innovative approaches to

combining genetic and agronomic strategies (MAIZE).

• NESSIE stimulated further cryptology research.

• The parameters and usability of the booster battery developed by BO-BA.

EXAMPLES Evidence of very high technical impact: • ONTOWEB developed a

sectoral technical standard with widespread uses.

• NESSIE achieved consensus on global ISO Standards for algorithms.

Immediate Impacts (1st Group)

International cooperation International cooperation was the main actual impact for five of the 18 projects and had a very high impact in 13 of the case studies. International cooperation is considered to be the main actual impact of the Commission supported research activities, greatly facilitated by the general FP requirement for institutions and researchers in European Union (EU) Member States (MS) to collaborate across the EU. Comparing findings with the relevant FP Programme Areas, it is evident that this general FP objective is being met by the sample of projects.

Enabling Impacts (2nd Group)

Policy and standards It is apparent that the group of projects under study has, and will have, a generalised high impact on policy and standards in the EU and beyond: 8 projects have recorded high or very high impacts to date and a further 7 have the potential to do so in the future. Policy was identified as the main area of impact for 3 projects (GECS, GMO and GALA).

Scientific knowledge Scientific impacts are broadly defined as changes to scientific knowledge and processes, including the generation of ‘new’ or ‘increased’ knowledge. Actual impacts have been recorded across all projects, although these vary in nature, scope and intensity. Six projects have recorded very high actual impacts and a further 4 are expected to generate very high impacts in the future. This is therefore the second highest area of impact after that of

international cooperation. Of the 3 projects which are funded under Framework Programme areas with an emphasis on science, GMO scored ‘high’ for actual impact, and FUNCARS and CIEL ‘very high’.

Technical Technical impacts relate to innovation and the development of industrial techniques, subjects, and the application of scientific knowledge. The evaluation revealed that because of the enabling nature of technical impacts they are important stepping stones to broader impact. Overall, 16 of the case study projects achieved actual impacts in technical areas, with 5 project scoring ‘very high’ and a further 6 ‘high’ impacts. In adherence with the FP objectives, technical impacts were found to be particularly strong for projects funded under sub-programmes BRITE/EURAM and Euratom, but also for those under GROWTH and IST.



4

EXAMPLES Evidence of high social impact: • 333 European jobs

resulted from REOXCOAT.

• Widespread European media coverage resulting from the GMO project raised public awareness of the issues of genetic modification.

EXAMPLES Evidence of high economic impact: • Support to the growth and

development of SMEs by REOXCOAT and TTA.

• WETO 2030 has been used by the Government of France to predict energy costs/ resources.

Broad Societal Impacts (3rd Group)

Social Social impacts were defined as improvements in the socio-economic knowledge base, job creation and increased public awareness. Whilst 7 projects recorded actual impacts, only four of these scored ‘mid’ and above. In accordance with the definition of higher level impacts, there were no ‘very high’ actual impacts in this category. Although IST and GROWTH projects were expected to perform better in this area, the JRC and other programme area projects performed equally well.

Economic Economic impacts assessed here, included increased competitiveness, the

development of new markets, creation of SMEs, and cost-savings, which are expected to materialise in the long-term. Notwithstanding, 9 projects have, already to date, reported actual economic impacts, with 8 in the ‘mid’ to ‘high’ range. An additional 9 projects have the potential for very high impacts in the longer term, and all 18 projects expecting some impact in this area, this could be the area of greatest overall impact in years to come. Relating impact back to the FP Programme Areas aiming to make a difference in this area, actual impact has been seen for the

majority of projects within Energy, Environment and Sustainable Development (EESD), IHP and Euratom, although with varying scores.

Quality of Life Identified Quality of Life impacts included improvements to the environment, and to the health and safety of citizens. As with other areas of higher level impacts, only a minority, six projects, have so far scored actual impacts in this area, and these are distributed between the ‘high’, ‘mid’ and ‘low’ categories. Four projects have the potential for even higher future impact and another 8 separate projects have potential for future impact. In linking impacts back to FP Programme Areas, it is clear that this is one of the areas in which impacts need a longer time period to materialise (e.g. 3-10 years).

IV. MAIN CONCLUSIONS

• There is a need for a ‘holistic’ approach in trying to understand high impact across sectors and types of projects. Impact is not just about quantitative indicators (i.e. turn-over generated), but about ‘making a difference’, e.g. changing people’s perceptions or creating alternative solutions to fulfilling societal needs.



5

• Since it is almost impossible to neatly separate the qualitative and the quantitative dimensions of ‘impact’, particularly in the short- to medium term, to measure impact solely in quantitative terms would be a reductionist approach. Projects need to be considered, not in isolation, but also in their leveraging capacities (both for motivating further activity, and in stimulating further funding). A qualitative approach backed by quantitative and qualitative evidence could therefore be used to measure and compare progress against attainment of higher level objectives and to identify far reaching impacts across Framework Programme areas.

• The time frame within which to expect impacts to occur is crucial, particularly

since socio-economic and quality of life impacts take considerable time to materialise after project finish (3 to 10 years). Hence, next to High Impact Research Activities (HIRA), it might be useful to consider a second and equally legitimate type of project - Potential High Impact Research Activities (PHIRA) – that seek to develop a potential for high impact.

• Impacts were attained in all three groups of impact (immediate, enabling and

broad societal). High socio-economic and quality of life impacts consisted of increased public knowledge, job creation, economic savings (and minimisation of economic losses), and environmental protection. Impacts that enabled ‘high’ impacts included the development of technologies, frameworks and products, tools for policy-making, generation of new knowledge and the refinement and application of existing knowledge. Underlying the international dimension of EU funded research was the immediate stimulation of international cooperation.

• In general the sample projects were highly innovative and EC funding

stimulated both international collaboration in their implementation and the pace and quality of the research. The momentum of change instigated is likely to be sustained over a longer time frame (there are numerous follow-on projects and uses being made of the products of research).

• European Added Value is a significant cross-cutting impact of these

research activities, e.g. eleven out of 18 projects would never have taken off without European funding. Only a small minority of projects would have occurred in any form without EC funding and these would have been on a far smaller scale.

• By generating added-value at the European level, it is clear that projects under

FP4 and FP5 have already contributed to the fulfilment of a number of the Framework Programmes’ over-arching goals and are contributing to the fulfilment of objectives under FP6. For example, the majority of case studies have been relevant to socio-economic concerns in Europe – a key feature of FP5. Nevertheless, since the sample projects were selected due to existing perceptions of high impact, they cannot be seen to be representative of the entire projects across FP4 and FP5. For this reason findings have not been multiplied to represent analysis of research activities at the aggregate FP level.



6

• Projects funded under FPs may have very different targeted outcomes and potential impacts. There is a simple divide between those research projects that seek to develop ‘new’ knowledge (infrastructural) and those that are seeking to apply it (super-structural). Within the strategy of Framework Programmes, both are of relevance.

• There are several ‘key critical factors’ that are relevant to the occurrence or

potential for high impact of the majority of projects across the project cycle. These include the people implementing projects, the functioning of partnerships, recognition attained by the project, the degree of focus on impact by implementers and the quality of initial needs assessment and flexibility. The greatest identified factor (15 out of the 18 projects rated this highly or very highly) was human resources, or the ‘people’ factor, which is seen as providing key foundations for the delivery of high actual and potential impacts.

• While the critical factors identified are conducive to impact, on their own, they

are not a sufficient condition for impact. Only if and when impact is understood and defined as an integral part of the goals and objectives of a given project right from the start, do these factors become relevant.

V. MAIN RECOMMENDATIONS

1. Qualitative measures of impact should be used alongside more quantitative tools and indicators since quantifiable and conclusive proof of impact of RTD projects is rare, particularly among the 3rd Group of broader societal impacts.

2. Since awareness of potential high impact on societal level at the outset of a

project can be a factor for it to materialise, applicants should be invited to indicate whether the successful outcome of a project will result in super-structural (use of present scientific knowledge) or infrastructural (fundamental scientific knowledge) impact and justify their choice at the project conception - ex-ante stage. Applicants for super-structural research should further be encouraged to conduct a proper needs assessment, involving users, at the proposal stage. This would ensure that partners consider their projects as triggering a chain of events towards impact, as well as identifying key risks, opportunities and assumptions. It would also help to determine the appropriate approach for subsequent programme management at the EC.

3. Assessment tools appropriate to the measurement of super-structural impacts

should not be used to measure the impact of infrastructural research activities and vice versa, since the scope and time-lag for achieving impact is likely to be different.

4. Competitive research funding seems to be a better vehicle for

infrastructural projects, while collaborative research seems more likely to generate applied projects with high impact potential. Ideally, the Framework programmes should support both, although collaborative research funding should still be granted on a competitive basis.



7

5. The EC should consider impact assessment as a continuous activity

towards ‘managing for impact’. Using a causal results chain rather than a linear model, would allow a causal analysis to be established at the ex-ante stage and then updated at various points in the project cycle. By involving the partners themselves in continuous monitoring and self-assessment, commitment to addressing the specified need can be made intrinsic to the relations among the partners involved.

6. In order to provide the evidence to support comparative qualitative scoring,

the EC should consider the development of qualitative impact ratings and indicators within all FP Programme Areas. The use of qualitative scoring would be appropriate and would respond to the need for flexibility.

7. The EC could measure impact using the causal (results chain) case study

methodology at four key points in the project cycle. Rather than requesting additional documents from researchers, the proposed impact assessment framework, could be linked to the current deliverables (proposal, interim report and final reports).

8. The following Figure illustrates the proposed causal case study methodology

and assessment points throughout the project cycle and beyond. The first point would take place at the project conception phase, the second mid-term and third point at the end of project. The EC could identify High Impact Research Activities (HIRA) or Potential High Impact Activities (PHIRA) at stage 3 and follow up with ex-post assessments at stage 4.

Figure: Impact assessment framework with 4 assessment points 9. Key elements of the proposed impact assessment tool would be:

• Assessment of need (including the involvement of users); • Identification and rating of impacts (according to the groups of impact

identified during this assessment), and the causal sequence required; • Elaboration of the relation between the project impacts and the

objectives of the FP and sub-programme areas;

1. Project Conception

3. Project end

Immediate Impacts Within the project cycle

Intermediate Impacts Enabling: Project end to 3 years

Higher Impacts Broad, societal: 3 to 10 years

2. Mid term assessment

4. Ex-post assessment



8

• Elaboration of assumptions behind impact expectations; • Elaboration of possible risks over time; • Identification of possible future opportunities; • Identification of feasible indicators, for each group of impacts; • Assessment of relevant critical factors by project partners and external

reviewers; • Identification and reporting of indicators.

10. Project partners, external reviewers and EC managers should have

specific roles at every stage of the process, each contributing to the assessment from their perspective. This impact assessment framework is intended to stimulate a stronger awareness of impact as an indispensable prerequisite to the attainment of impact. For example,

EC project officials may use these results chains for reflective, reactive

monitoring during the project cycle. Project partners would be required to consider their project in relation to

potential impact, and the overall objectives of the Framework Programme; to devise indicators; and to report using the results chain and indicators.

External reviewers would assess some of the ‘critical factors’ at various

stages.

11. A clear analysis of the new manner in which FP6 is managed (or indeed as planned under FP7), should be a pre-condition for any change of tools, methodology or reporting requirements.

12. The EC should also consider adopting mechanisms for multiplying and

sustaining impact of FP projects.

• For example where ‘winners’ are identified at the end of the project cycle, there could be systematic follow-up with participating organisations, to gain their perspective on how impact could be maximised and sustained. In this context there may be a need for a different instrument e.g. development research funding.

• To avoid a time-lag (impairing impact), shortly following the end of the

project, a further competitive stage for follow-up FP funding following characteristics could therefore be considered to ensure that products remain relevant and that impact is maximised:

• Lower amount per year; • Longer period; • Competitive; • Training provision; • Support to networks.



9

13. Another option for consideration the revision of the dissemination strategy towards the end of the project, so that targeting, of potential users, and relevance to them is constantly appropriate.



10

Acknowledgements

The TEP team wishes to express its appreciation to all those who have helped to make this impact assessment possible. To desk officers and other parties at the Commission for taking the time to brief the evaluation team, for giving thought to the definition of high impact, and for providing us with relevant contact and background data; to project lead and project partner staff and managers who in interviews and meetings provided the team with the data which formed the basis for the overall assessment, and to the many external project stakeholders that took the time to provide the team with their feedback.



11

A: DESCRIPTIVE SECTION

1 Introduction

This section outlines the background, objectives, timetable and structure of the study.

1.1 BACKGROUND

This study was commissioned to form part of the Five-year Assessment of activities under the Community Research Framework Programmes to be conducted in autumn 2004 and covering the period 1999-2003. The study of “high impact” research activities is to complement other impact studies which have or are being completed in parallel.

1.2 OBJECTIVES OF THE STUDY

The two main objectives of this study were to: • Generate an understanding of high impact activities and how they occur by in-

depth analysis of a select number of projects (cases) not previously assessed under the 4th and 5th Framework Programmes, which are already seen to have, or to have had, a ‘high impact’ by the Commission units involved in coordinating the FP sub-programmes that funded them;

• Contribute to the setting-up of an overall framework of Community research

evaluation for monitoring of future Framework Programmes by developing a robust case study methodology which would allow comparable impact assessment across future Framework areas.

The original ToR for this assignment can be found in Appendix 1.

1.3 WORK PROGRAMME

The current study was conducted during 10 months, from March to December 2004, by a team from the European Evaluation Partnership (TEEP) on behalf of DG Research.

1.4 STRUCTURE AND CONTENT OF THE REPORT

This final report consists of the following Sections and Chapters: • SECTION A: DESCRIPTIVE SECTION, including:

o This introductory section (Chapter 1).



12

o An outline of the methodology and approach taken to the impact assessment, including the definition of high impact used for the purpose of the study; the targeted stakeholder groups; and the evidence collected from these to answer the evaluative questions set out in the ToR (Appendix 1). It also details the sampling methodology used to identify high impact research activities (projects) and selecting them for in-depth study, as well as an overview of the limitations of the data collected (Chapter 2).

o A brief descriptive overview of the 18 research activities (projects) selected for study, by type of project, FP and FP sub-programme, duration, end date, geographic location of partners, etc. (Chapter 3).

• SECTION B: EVALUATIVE FINDINGS, including:

o Findings and supporting evidence emerging from the 18 individual impact assessments, through horizontal and counterfactual analysis, including levels of impact per type and area of impact, and links to FP overall objectives (Chapter 4).

o Critical factors that either facilitated or impaired the attainment of high impact organised according to stages in the project cycle and overall (Chapter 5).

• SECTION C: CONCLUSIONS AND RECOMMENDATIONS, including:

o Individual project summaries and analysis presenting key impacts findings and drivers of impacts for each of the 18 projects (Chapter 6).

o Conclusions with regard to the assessment of high impact within the sample and at aggregate FP level; to critical factors driving impact and methodologies that facilitate identification and measurement of high impact (Chapter 7).

o Recommendations for multiplying and extending impact; strategic considerations for enabling future across-the-board assessment of high impact among Community supported research activities; and proposals for a case study methodology (Chapter 8).

The report is supplemented by a number of Appendices (see separate document), which go into greater detail on aspects of the methodology, findings and analysis. These include:

• Appendix 1: Terms of Reference. • Appendix 2: Original Project Methodology. • Appendix 3: Critical Assessment of Previous Impact Assessments. • Appendix 4: Project Selection Methodology. • Appendix 5: Stakeholder List. • Appendix 6: Impact Assessment Interview Tool and Guidelines. • Appendix 7: Project Bibliography. • Appendix 8: Project Evaluation Grids - Individual Case Studies. • Appendix 9: Proposed Future Impact Assessment Framework. • Appendix 10: An introduction to the European Commission’s 4th, 5th and 6th

Framework Programmes, which aims to contextualise the study findings.



13

2 Methodology and Approach

This section presents an overview of the methodology and approach used, to select and assess ‘high impact’ among the research activities which were put forward for the exercise by Commission services. It outlines the definition of ‘high impact’ as employed in the study, and the approach taken to capture the evidence from each of the 18 selected case studies.

2.1 THE APPROACH

The study was essentially divided into two main stages of which the first was an exercise to critically assess previous ‘high impact’ studies of Community sponsored research activities. The second was the actual selection and assessment of ‘high impact’ projects under the Community framework programmes. A full outline of the approach can be found in the original project methodology in Appendix 2. While the first part of the exercise used a top-down approach to identify how impact or high impact had been defined and measured in past studies, the second part followed a bottom-up approach aiming to capture evidence of high impact from a sample of individual case studies. Essentially, the methodology used for the impact assessment consisted of: • Definition of high impact; • Collection of the perceptions of implementers and other stakeholders; • Utilisation of a highly structured framework to achieve consistency in the

measurement of impacts across sectors and projects; • Identification of critical factors; • Deployment of both quantitative and qualitative indicators and evidence to

identify and measure high impact; • Triangulation of qualitative ratings provided by informants’ with those of other

stakeholders’ and secondary sources; • Development of a case study methodology for future measurement and

comparison of high impact research activities across Community sponsored research activities.

2.2 DEFINING HIGH IMPACT

During the first stage of the study, the critical assessment of previous impact assessments (see full outline of results in Appendix 3), it became evident that very few of these had made any direct reference to ‘high impact’, and that none had in fact defined what differentiates ‘high impact’ from ‘impact’. In addition, the emphasis of these studies was more on describing ‘success stories’ and on finding quantifiable evidence to support these, than to look for the occurrence of the sorts of longer-term impacts which would be more in line with the higher level objectives of the Community Research Programmes. The overall conclusion of this analysis was that



14

there was a need to develop a definition of ‘high impact’ which would link back to these objectives. Since the aim of the current study was to try and go beyond the strictly quantifiable outputs usually identified as ‘impacts’ in previous impact assessments, the following definitions of what is to be seen as ‘high impact’ projects were developed and agreed upon with the project Steering Group. This defined the criteria for short-listing projects which were seen to have the ‘highest impact’. The definition of a high impact project was:

1. A project that wasn't ‘just’ successful (in its execution or in having an immediate tangible output such as higher sales) but that was seen to have made a difference, particularly on European level;

2. A project that fulfilled a need, or which was considered to have great

potential in fulfilling a need, in society, such as for example improving health or creating sustainable development at European level and/ or beyond;

3. A project that improved policy-making, standards, job creation, quality of

life, competitiveness and/ or career development, particularly at European and international level;

4. A project that changed the scientific agenda/ the Research and

Technological Development (RTD) environment for the benefit of society;

5. A project that changed people's perceptions beyond the world of science (e.g. through education).

The definition of high impact evolved during the project, as the team started gathering the evidence. This is further described in the main body of the report in section 4.1, analysis and findings.

2.3 CASE STUDY SAMPLING AND SELECTION

From an early cluster analysis of various features of previous impact studies, factors to be taken into account in devising a new and improved sampling strategy and the tools for selecting and screening projects were identified. The selection of individual projects for in-depth study then followed a number of steps of which the main stages are described below.

2.3.1 Case study sampling From an initial list of over 300 projects provided to the team during the inception phase, the list was narrowed down to a short-list of 29 projects. This short-list was arrived at by asking scientific officers to nominate a maximum of 2-3 projects per sub-programme area based on the impact definitions outlined above. To further facilitate the short-listing in those cases where many projects fitted these impact definitions, officers were asked to also take into account the following four additional criteria:



15

1. A project which would never have taken-off without European funding/ backing (European Added Value)1;

2. A projects that works across borders, within Europe or beyond;

3. A project for which appropriate and relevant documentation is available;

4. A project for which the potential for gathering inputs from stakeholders

outside the project itself is attainable. Apart from the added value perspective, these were neither closely related to impact nor end results. However, by trying to assess the importance of EC funding and the degree of feasibility of a fast assessment, these additional criteria were seen to be relevant.

2.3.2 Case study selection Initially background information on each of the projects on the short-list was gathered through web research and contacts with Scientific Project Officers and Project Beneficiaries. In order to make the selection of the 18 case studies from the second short-list of 29 projects, a selection tool (see Appendix 4) against which to ‘score’ each project was developed. The descriptive and evaluative criteria included in this tool originated from definitions of potential high impact developed during the critical assessment phase, feedback from the Steering group, and from the team’s initial research during the inception phase of the study. They included: Overall Context • Is the Project positioned in an area of recognisable need? • Does it aim to make a difference? • Does it have a clear European or wider context? • Is the Commission selection rationale clearly stated? • Is the Project representative of FP research and relevant for the 5yr assessment

process? Project characteristics • Which FP and programme area does it relate to? • Did it require European funding to take-off? • Was it managed and implemented successfully? • Is there sufficient appropriate documentation available? • Does it demonstrate satisfactory communication and dissemination outputs? • Is it collaborative by nature? • Is the RTD environment clear and understandable? Sector characteristics

1 One definition of European Added Value in research could be that what is achieved at a European level is greater than the sums of impacts of projects at a national level. Since this can be rather difficult to measure, another ‘crude’ way of measuring EVA, which has been applied in this case, has been to analyse the counterfactual, i.e. whether the activity would have taken place at all without European backing.



16

• Is it located within clearly recognisable sector with known features? – defined as policy; legal and regulation; scientific advance; commercial/economic; quality of life; societal issues; and education/ training.

• Are relevant trends, influences and perceptions of the sector identifiable? • Is research trusted/ valued/ recognised in the sector? • Can routes of exploitation be readily identified? • Is sectoral information accessible in various ways? Potential features of impact • Can the focus for the impact be recognised? • Is the impact capable of analysis either quantitatively or qualitatively? • Is there the potential for recognising both tangible and intangible impacts? • Does attribution of impact to project outputs looks possible? • Can the timescale for impact be taken into account? • Can project and personal or partnership outcomes be differentiated? • Can constraints or limitations to impact effectiveness be identified? • Does opportunity exist to assess user friendliness/ practical usefulness? • Are there other unique features? These features and characteristics were intrinsic to selecting projects that might best demonstrate potential for high impact. Where features seemed to be apparent from the available information amassed during the desk research a ‘Yes’ (Y) was inserted in the appropriate box in the selection tool. For convenience, in the selection process the characteristics were grouped as indicated above into those relating to the overall context of the project’s positioning, those of the project itself, those of the sector relevant to the project and those that might be a feature of the impact and how it occurred. Using this selection methodology on the 29 short-listed projects for which project officers had provided inputs, and through information gathering from CORDIS and through personal knowledge of certain projects and the sectors involved, 18 projects which clearly demonstrated a significant number of key features to underpin their high impact potential were selected. The final selection of 18 cases was in principle based on the overall number of Yes’ the candidate projects obtained. However, a ‘reality check’ was then performed to see whether the methodology employed to identify projects with potential high impact had also resulted in a ‘representative’ selection across:

• FP programme and sub-programme areas2; • Types of instruments; • Project budget sizes; • Countries (i.e. geographically spread and distribution of partner institutions);

2 It should be noted that the benchmarking for impact potential was always the main underlying essence in the selection of the projects. Programme names cannot be viewed synonymously as areas. Some are very diverse, which means that some selected projects that might come under the same programme name were suggested by project officers in different areas of the programme. Also some programmes were much bigger than others and some had different forms of instruments or actions. Therefore, the team has to the best extent possible tried to capture this diversity if and where relevant.



17

• Large and small players (i.e. involvement of SMEs versus larger institutions); • Accession and new member states3.

In order to agree a representational position of projects with the Steering Group and the Evaluation Unit (DG Research) it was always recognised that some final adjustment could be necessary. However, since the Commission services in most cases provided adequate information to support the revised selection, it was not seen to jeopardise the basic rationale of the selection process, meaning that it by and large confirmed the appropriateness of the selection tool for the overall sample. In terms of the selection of projects from the various FPs, it should be noted that the majority of projects put forward by the Commission were from FP 5. This meant that the final list of case-studies included a majority of projects from FP5, a small number of projects from FP4 and no projects from FP3. Possible explanations to this could be that newer projects were less likely to have been assessed previously and that they would be more present in the institutional and personal ‘memories’ of the sector units and the evaluation contacts in these. Past studies have indicated that there can be difficulties in identifying individuals with involvement in projects implemented some years ago. Hence it may not be surprising that more recent projects (under FP4 and FP5) were the ones of choice although such projects may have had insufficient time to generate impacts.

2.4 CAPTURING THE EVIDENCE

The evidence, upon which the subsequent analysis and impact assessment is based, was primarily gathered through one-to-one and group interviews with the project lead partner during field visits. Where possible, other project partners and/ or external stakeholders were also in viewed in the field, or otherwise contacted via telephone during follow-up. A total of 40 face to face and 18 telephone interviews with project lead partners, other project partners and external stakeholders were conducted. A list of the stakeholders interviewed for this evaluation can be found in Appendix 5. A detailed interview tool and guidelines for collecting and collating the interview data were developed (see Appendix 6 for more details), to capture evidence of impacts at project level and beyond. The former was primarily used for structured interviews and discussions with the project coordinators and project partners. Where applicable, parts of it were also used to gather feedback from external stakeholders and end users. The face-to-face interviews generally lasted for between one and a half to two hours, and the telephone interviews for about an hour. To allow comparison and assessment of impacts between the diverse groups of projects involved, informants were asked to rate relevant impacts (i.e. social, technical, scientific, policy, quality of life, economic and other) according to the following qualitative criteria, based on the high impact definitions outlined previously.

3 In practice it seems clear that, where adjustments to the case study selection were deemed necessary, these were due in the main to the absence of relevant information from the Commission at the time that the selection process had to be undertaken rather than as a result of any defects in the rationale used for the process of selection using the selection tool.



18

The geographical criteria below were developed by the evaluation team in collaboration with the Commission and according to the current OECD definition of impact4.

Figure 1: Criteria for Rating Impact

LOW Any impact that went beyond successful implementation or

outcomes, at a local level (i.e. impact meaning making a difference beyond the immediate field of activity – answering a need in society, which could be: ‘Positive and negative, primary or secondary long term effects, directly, or indirectly, intended or unintended’).

MID As above, but impact is limited to one region/ country/ sector/ population.

HIGH Impact has extended to make a difference beyond one country, additionally there has been a high sectoral impact/ impact on industry, regionalised population or regional economy.

VERY HIGH As above, and impact has extended to a European-wide level or beyond.

Ratings on their own could not explain why impacts occur. Therefore, a number of ‘open’ questions were also asked as part of the discussion, to explain reasons why these impacts are seen to occur and to explain the factors (internal and external) that contributed to the impact. The task of ‘capturing the evidence’ was implemented by sharing the approach to interviewing and the reasoning behind it and in a similar manner emphasizing the justification for the approach as there was a need to achieve consistency across the case study analysis in order to maximise the gathering of generic features germane to high impact. Interviews centred on project partners and lead partners, who are highly involved in their area of research (so that even when there has been little formal tracking of impact, they are often aware of impacts through informal monitoring). Other stakeholders’ perceptions were sought as validation of these findings. Advantages of this type of assessment include that it is not time-consuming or overly costly. However, it does face the constraints of a lack of randomisation and baseline data. Evidence of impact was requested during all interviews, and is presented in the Findings Section of this report (Section 4.2). All documents consulted during the course of the evaluation are presented in the bibliography (Appendix 7).

4 Crucially, high impact goes beyond successful implementation or outcomes. Impact means making a difference beyond the immediate field of activity – answering a need in society at a national, European or international level, it can be: ‘Positive and negative, primary or secondary long term effects… directly, or indirectly, intended or unintended’ (OECD/DAC 2002 /24).



19

2.5 ANALYSIS AND IMPACT ASSESSMENT

Another tool – an evaluation grid - was developed to allow for the synthesis and analysis of evidence gathered from all interviews, relating to any one project. This included a general descriptive section on the background to the project as well as sections which reflect the evaluative assessment and ratings in the interview tool. The information gathered in the grids formed the basis for the subsequent overall impact assessment and comparisons of impacts across projects, sectors, and programme areas as outlined in the analysis and findings section (chapter 4 below). It is important to point out that the final assessment of impacts in these grids were based on the overall evidence gathered for each project. While taking into account stakeholder ratings, the final assessment was made by the evaluation team. Detailed, individual impact assessment grids for the 18 case studies, including the causal path of impact, can be found in Appendix 8. It should be noted that while they contain the bulk of findings from the 18 individual impact assessments, for the sake of brevity, only key details and key causal paths have been included in this Final Report in chapter 6. The detailed impact assessment grids in the Appendix all follow the same standard structure: • Project background

o Project title and key information (budget, programme area, etc.); o Lead Institution (and partners); o EC Rationale (for further study); o Background and Genesis (including nature and characteristics of the

sector, and trends); o Key players and stakeholders in the Sector; o Typology of the project; o Specific project objectives; o Summery of work undertaken; o Characteristics of the outcome.

• Section A: Impact Assessment o Impacts (scope, expectations, etc.); o Areas of impact (qualitative scores); o Counterfactual and sustainability.

• Section B: Evidence o Evidence of impact; o Indicators to measure impact.

• Section C: Factors and Drivers o Internal and external factors; o Reasons for choice.

• Section D: Sector specific and additional comments o Additional comments.



20

2.6 DATA LIMITATIONS

The main imperative of the tools was to collect a mix of quantifiable and qualitative data, which would provide comparative information across all projects. Quantitative data was collected where possible and utilised to provide evidence for the system of qualitative ratings presented above. The framework for assessment (ratings) allowed for analysis of impact in a comparative way across the sample of projects as well as between areas of impact. In addition to being challenging to gather, quantitative data alone would not allow for this sort of cross-cutting assessment, or to systematically link findings back to the objectives of the Framework Programmes. Quantitative data is generally only available for tangible and attributable impacts, not for some of the higher impacts (economic, social and quality of life) that are also subject of this assessment. It must be further underlined that the sample of case studies selected falls short from being statistically representative of either the FPs or sub-programmes they derive from. The candidate projects from which the sample projects were drawn were already biased, in that they had been put forward by the Commission services because they were seen to have a high impact. This means that any attempt to quantify the results of the study, which are largely qualitative, with the view to scale them up to FP or sub-programme level, would not only be impossible, but statistically unsound. Due to the fact that the evaluation team met only with the lead partner in the majority of project visits, while contacting other partners by telephone, the impact of the projects on the capacities of smaller partners (other than the lead partners) may have been underestimated. Lead partners may for example not have been fully aware of, or prioritised, the impacts on human resources that the projects may have generated for their smaller partners, which means that some impacts may have been missed or only partly reported. Other challenges and limitations are explored in more detail in section 4.1.



21

3 Nature of the selected research activities

Of the selected 18 projects, 16 fall directly within Framework Programmes (4 within FP4 and 12 within FP5) and the remaining two projects were funded under different arrangements:

• 1 project was undertaken directly with Euratom: o Title : ‘Composants internes et Limiteurs’ - CIEL. Main objective: To upgrade Tore Supra to ensure high power heating of internal components (both by convection and radiation).

• 1 project was implemented by the Joint Research Centre (JRC) Spain on

behalf of DG Agriculture: o Title : ‘Scenarios for co-existence of genetically modified, conventional

and organic crops in European agriculture’ - GMO. Main objective: To develop scenarios and best practice for the co-existence of genetically modified, conventional and organic crops in European agriculture.

As expected following from the selection criteria (which emphasised high impact), the sample of RTD projects selected for this assessment is primarily conditioned by perceived high impact on European policy, the European economy and/ or European society. As such it is not representative of the average research activities funded under Framework Programmes 4 and 5. Nevertheless, by taking into account a vast array of criteria in the selection process, including representativeness across FP sub-programme areas, SME involvement, geographic spread, size and timing, it was thought to reflect the widest spread possible while still emphasising the key question of high impact. Within Framework Programme 4, selected projects were drawn from 3 of the 12 specific programmes:

• 2 projects fell under the BRITE-EURAM specific programme – Industrial and Material Technologies:

o Title : ‘Enhanced economy of magnetron sputtered resistive oxide coatings by improved target and magnetron design in large area applications’ - REOXCOAT.

Main objective: To solve the present technical limitations of the process and to utilise fully the clear advantages of magnetron sputtering technologies. o Title : ‘Booster Battery for High Power Demand’ - BO-BA. Main objective: To develop high power booster batteries as a key component of hybrid vehicles.

• 1 project came from the Information Technologies (ESPRIT) programme: o Title : ‘Time triggered Architecture’ - TTA. Main objective: To implement time-triggered computer architecture for fault tolerant distributed real-time systems, and to effectively deploy it in safety-critical transportation systems.



22

• 1 project was in the horizontal programme area of International Cooperation (INCO):

o Title : ‘Improving equity and deficiency of China’s urban health services’ – CHIN.

Main objective: To analyse how changes in health finance have affected equity and efficiency of urban health services and to contribute to a new agenda for reform in China’s health sector.

Framework Programme 5 reorganised projects into 4 broad thematic areas. Within the group of case studies examined, there is at least one project from each of these areas:

• 1 Quality of Life project: o Title : ‘Managing fisheries to conserve groundfish and benthic

invertebrate species diversity’ - MAFCONS. Main objective: To develop a management protocol to predict the consequences of setting particular TACs (Total Allowable Catches), on benthos and fish species diversity.

• 2 Information Society Technologies (IST) projects:

o Title : ‘Ontology Based Information Exchange for Knowledge Management and Electronic Commerce’ – ONTOWEB.

Main objective: To involve researchers and potential stakeholders from both the ontology-relevant and application-relevant areas, to promote the harmonisation of efforts. o Title : ‘New European Scheme for Signatures, Integrity and Encryption’

- NESSIE. Main objective: To develop secure algorithms for data exchange.

• 2 GROWTH projects: o Title : ‘Modelling and Methodology for analysing the interrelationship

between external developments and European transport’ - SCENES; Main objective: To develop a Europe-wide transport forecasting model. o Title : ‘Galileo Overall Architecture Definition’ - GALA. Main objective: To formulate the bases for Galileo (users, design, technical, service definition).

• 3 Energy, Environment and Sustainable Development (EESD) projects: o Title : ‘Greenhouse Gas Emission Control Strategies’ - GECS; Main objective: To address the economic assessment of climate change policies. o Title : ‘Computer Aided Rehabilitation of Water Networks’ - CARE-W; Main objective: To develop a rational framework for water network rehabilitation decision-making. o Title : ‘Poles world energy / technology outlook to 2030’ - WETO 2030. Main objective: To develop a European predictive energy model.

In addition to the FP5 thematic areas, case studies were drawn from the FP5 horizontal programmes:



23

• 1 International role of Community Research (INCO2) project: o Title : ‘Maize for sustainable cropping systems on tropical acid soils –

from molecular biology to field cultivation’ - MAIZE. Main objective: To increase maize production and productivity using biotechnology.

• 3 Improving Human Research Potential (IHP) projects: o Title : ‘Foresight for Regional Development Network’ - FOREN; Main objective: To promote the effective integration of Foresight processes into regional development, policy and strategic planning. o Title : ‘Research Training network: Chemical Functionalisation of

Carbon Nanotubes’ – FUNCARS; Main objective: To increase the functionality and processibility of carbon nanotubes (CNT). o Title : ‘European Natural History Specimen Information Network’ -

ENHSIN. Main objective: To enable the development of a shared interoperable infrastructure of natural history specimen databases in European institutions.

The end dates of the case study projects are highly varied. The earliest ended in 1998 (TTA and REOXCOAT), 3 projects completed in 2004, and MAFCONS is still in mid-implementation and scheduled to complete only in 2006. The geographic scope of the projects was evenly divided between those with a European and global reach. Average durations were around 40 months, with noticeable variations. The shortest project duration was 12 months (GMO) and the longest 120 months (CIEL). As with the Framework Programmes themselves, the 18 selected projects cover a diverse range of sectors and have diverse objectives. However, some common features are identifiable within the project sectors, objectives and expected results. For example, several projects within the sample aim to develop tools or models for use in policy testing at the EU level (these include MAFCONS, SCENES, GECS, and WETO 2030). Another ‘cluster’ of projects can be identified as those aiming to develop standards at the European/ global level (NESSIE, TTA, ENHSIN, ONTOWEB and GMO). Finally, a third cross-cutting cluster could be defined as those undertaking basic pilot research for an imminent technology leap (ONTOWEB, TTA, ENHSIN and MAIZE). There are also some commonalities in types of organisations participating in consortia; universities in 12 of the case studies, large companies in 6 of the case studies, and small and medium sized enterprises (SMEs) in 6 of the assessed projects (TTA, ONTOWEB, REOXCOAT, BO-BA, SCENES and CARE-W). Only 2 of the projects were implemented by sole organisations (CIEL and GMO), and as indicated above, they do not fit with the regular categories of FP programmes. The table below presents some of the main features of each project.



24

Table 1: Details of the projects selected for assessment

PROJECT: FP FP Programme Area End Date Duration

(month) Project Cost

(Total) in EURO

Number of Partners

Scope Sector Lead Institution

MAFCONS 5 QoL Sustainable Agriculture, Fisheries and Forestry

30/06/2006 42 €3,172,459 6 North sea

Sustainable fisheries

University of Wales, Swansea. UK

ONTOWEB 5 IST Information Access

31/05/2004 36 €1,866,332 21 Europe Global?!

Ontology-based information exchange

Institute of Computer Science, DERI. University of Innsbruck. Austria

NESSIE 5 IST Identification and authentication

31/12/2002 36 €2,740,215 7 Global Secure handling of digital information

Katholieke Universiteit Leuven. Belguim.

TTA 4 ESPRIT Information Technologies

30/11/1998 36 €1,866,332 21 Global Automobile and transport sector

Daimler Benz AG, Berlin. Germany

REOXCOAT 4 BRITE/EURAM

Industrial and materials technology

14/06/1998 24 €960,000 6 Europe Magnetron sputtered processes for oxide coating

Surface Engineering Consulting OY (Savcor), Mikkeli. Finland.

BO-BA 4 BRITE/EURAM

Industrial and materials technology

30/06/1999 36 €3,400,656 5 Global Automobile industry

VARTA Batteried AG

SCENES 4 GRTH Competitive and sustainable growth

31/03/2001 28 € 2,750,696 18 Europe Transport modelling and sustainable mobility

ME&P, Cambridge. UK

GALA 5 GRTH Competitive and sustainable growth

10/06/2001 18 N/a 12 Global Communications through satellite navigation

Alcatel Space Industries, Toulouse. France



25

PROJECT: FP FP Programme Area End Date Duration (month)

Project Cost (Total) in

EURO

Number of Partners


GECS 5 EESD Energy, environment and sustainable development

31/07/2002 24 € 1,140,039 9 Global Environmental protection

Centre National de la recherché scientifique, University of Gernoble. France

CARE-W 5 EESD Energy, environment and sustainable development

31/01/2004 36 € 3,296,916 11 Europe Water management and supply networks

Foundation for Technical and Industrial Research at the Norwegian Institute of Technology. Norway

WETO 2030 5 EESD Energy, environment and sustainable development

30/06/2002 18 € 120,571 4 Global Environmental protection

Enderdata SA. France

CIEL N/A EURATOM

2002 120 € 9,660,000 1 Europe Nuclear fusion

Associsation Euratom,-CEA. France

MAIZE 5 INCO2 International role of community research

31/10/2003 36 €1,248,209 10 Global Research for development / food security

Institute of Plant Nutrition, University of Hanover. Germany

CHIN 4 INCO International cooperation

31/01/2002 40 € 430,000 40 Global Health Insurance

School of Public Health, Fudan University. PR China

FUNCARS 5 IHP Human research potential and the socio-economic knowledge base

31/03/2003 36 € 1,660,000 9 Europe Nano-technology

Institute for Organic Chemistry, University of Erlangen. Germany



26

PROJECT: FP FP Programme Area End Date Duration (month)

Project Cost (Total) in

EURO

Number of Partners


ENHSIN 5 IHP Human research potential and the socio-economic knowledge base

01/04/2003 39 € 200,000 7 Europe Biological science - information

Natural History Museum, London. UK

FOREN 5 IHP Human research potential and the socio-economic knowledge base

31/01/2002 24 € 728,467 26 Europe Regional Innovation strategies - planning

CM International, Paris. France

GMO N/A JRC Apr-04 7 € 200,000 1 Europe Environment, agriculture, policy

Instituto de Prospectiva Tecnológica (IPTS)- JRC. Spain



27

B: EVALUATIVE FINDINGS

4 Impact Assessment - Analysis and Findings

This section contains an aggregate analysis of the data obtained and analysed individually in the project analysis grids in Appendix 8. Before going into detail on the high impacts observed across projects, some of the challenges experienced in measuring impact and how this affected the methodology and the subsequent analysis and assessment will be outlined.

4.1 CHALLENGES IN MEASURING IMPACT

The interviews provided a great deal of evidence regarding the multi-facetted nature of what can be called ‘impact’ and ‘high impact’. While a large number of these were captured by the questionnaire and the grid, there are others, often of a more general character, that did not seem to fit within the framework. In order for these aspects not to ‘fall between the cracks’, some adjustments were made to the generic and specific definitions of impact as well as the time frame for different types of impacts. In addition to conceptual and measurement challenges, the delay in project short listing and the subsequent summer holiday period created difficulties for completing interviews on time (this particularly affected the assessment of GALA, WETO 2030 and GECS).

4.1.1 Adjustments in the definition of high impact During the early stages of the field work it became evident that interviewees sometimes found it difficult to provide evidence for actual impacts (let alone ‘measurable’ or ‘verifiable’ impact), but they could convincingly report about potential impact. There might be a variety of reasons why an impact is ‘only’ classified as being of a potential nature, some of which may have to do with the temporal nature of ‘impact’ (the impact of a particular piece of R&D can often be considerably delayed – Michelangelo’s early R&D on airplanes might serve as an example) whereas in other cases the lack of actual impact is entirely beyond the control of the researchers: e.g. legislation, like in the case of genetic engineering, or lack of corporate interest/ vision, or simply a technology which is not yet ‘ripe’. Another factor is that researchers, in particular, still very much reflect their achievement in terms of academic ‘excellence’ as their measure of success, and not in terms of ‘relevance’ - which is closer to impact. At first, the interview and assessment tools (the questionnaire and grids) did not capture the potentiality of impact. However, based on an early assessment of the evidence, the interview tools and the comparative assessment framework were fine-tuned to take into account the following four distinct but interconnected aspects of impact:

1. Actuality of impact: A differentiation made between actual and potential impact.



28

2. Intensity of impact: Rated against sector/ population/ industry criteria. 3. Scope of impact: Rated against geographical criteria, region/ country/

Europe or beyond. 4. Classification of impact: Categorised within 3 Groups of impact: immediate,

intermediate and higher impacts. The main difference from the original methodology was the emphasis on assessing impact within an accurate time frame for the particular type (group) of impact. This is elaborated in detail in the section below.

4.1.1.1 Time-lag and Groups of impact One essential challenge in measuring the impact of the 18 case studies was the wide range in project end dates. One project (MAFCONS) is not scheduled for completion until 2006, and the earliest end date is 1998 (TTA and REOXCOAT). Evaluations of research programs face the particular challenge of a significant time-lag - possibly between 3 and 7 years from the end of the project to eventual results. Therefore, at this stage, an impact assessment for projects which completed after 2001 could be a largely theoretical exercise. In order to overcome this obstacle, the evaluation made the primary distinction between actual and potential impact. This is clarified by the addition of a time horizon to the analysis. Certain impacts can occur immediately, in the table below these are classified as impacts within the project (on its participants and participant organisations). Intermediate impacts may generally occur within the timeframe from the end of the project, up to 3 years following it. These impacts are classified as enabling. A final set of impacts are those which we could classify as ‘high impact’ without hesitation – broad societal impacts, which may have a 10 year time frame. Another motivation for introducing this distinction is the complexity of attributing impact. In general, but particularly for quality of life and social impacts: the further away from the project, and the less tangible the impact, the more difficult it can be to make attribution. The fact that the causal path from original research to a long-term effect is often indirect further complicates assessment. The classification of impacts into three groups – based both on time horizons and on the distance of the impact from the project - will contextualise potential impacts as well as providing a causal framework for comparative analysis (see Figure below).



29

Figure 2: Groups of Impact

1st Group Impacts

2nd Group Impacts

3rd Group impacts

Immediate impacts – within the project

Intermediate impacts – enabling (<3y)

Higher impacts – Broad, societal (3-10y)

Example: Examples: Examples: International Cooperation

Technical Quality of Life

Human Resources

Policy Economic

Scientific Social Cross cutting / intrinsic impact

European Added Value

It should be noted that the differentiation of groups of impact alongside a timeline does not mean that different groups of impact occur according to this sequence each time, or that the causal link between groups is necessarily linear. This model is to portray a general pattern as well as to facilitate a framework for assessment.

4.2 IMPACT ASSESSMENT BY AREA OF IMPACT

Findings presented in this section assess impact across the 18 projects, under a number of key areas of impact. These are in turn, organised according to the three groups of impact outlined above, starting with the impacts occurring within the project cycle (1st Group) and ending with higher level impacts (3rd Group).5 Following these is a section on the cross-cutting impacts of European Added Value generated through Commission support, an overall findings section, and a section on the potential implications for the classification of impact. While the individual project assessment grids (Appendix 8), as well as the project summaries (Section 6 below) contain the causal analysis per project, this chapter highlights general issues, and is structured according to type of impact. Under each sub-header, a table presenting the ratings of impact per project, including actual and potential, is followed by analysis of general findings and an evidence box for the assessment.6 This analysis also looks at the links between the higher level objectives of the Framework and sub-area Programmes and the types of impacts achieved for projects in different areas. A more detailed analysis of the 4th, 5th and 6th Framework Programmes, including their objectives and evolution, can be found in Appendix 10.

5 It is worth noting that this model does not presuppose that these groups of impact necessarily follow in a linear manner – hence the ‘circled’ groups of impact. These groups are defined by the type of impact, rather than a sequential assumption. 6 Impacts have been counted only once during this system. Therefore, a project scoring ‘very high’ in actual impact received an ‘n/a’ score for potential impact – even where there is a high degree of possibility of further high impacts (these details are captured in the individual project grids). Potential impacts have been used to identify areas of impact for which there has not yet been an actual impact, or areas where an increased impact (i.e. from mid to high) is expected in the future.

Actual - - - - - - - - - - - - -Potential

3 2 1



30

4.3 1ST GROUP IMPACTS (IMMEDIATE)

1st Group impacts can be defined as immediate impacts, i.e. as those occurring within the project cycle. By definition these impacts are therefore largely expected to occur within the organisations participating in the research. The assessment of impacts within this group focused predominantly on international cooperation, although related impacts on capacities, skills and human resources were also captured.

4.3.1 International Cooperation and Human Resources An important objective, for all projects within FP4 and FP5 was to increase cross-border cooperation and collaboration both within the EU and the wider world. This was aimed at increasing the networking between different nationalities and teams. In addition to cooperation, a key related area of impact was the development of capacities among participants. By their nature, the consortia undertaking these research projects were composed of a variety of organisations (often a mix of organisations from the original 15 MS as well as the new 10 MS), with diverse sizes, histories and expertise. All the FP sub-programmes had an international cooperation objective as one of their stated goals. For INCO and INCO2 Programmes (the relevant projects are CHIN and MAIZE), international collaboration and cooperation formed part of their high level goals.

Table 2: Overall rating of international cooperation impact

IMPACTS Actual Potential VERY HIGH 13 FUNCARS

ONTOWEB NESSIE TTA REOXCOAT BO-BA SCENES GECS CARE-W MAIZE ENHSIN FOREN GMO

1 MAFCONS

HIGH 4 MAFCONS CHIN WETO 2030 GALA

-

MID - - LOW - -

NONE / N/a CIEL Within five of the 18 projects, international cooperation was perceived to have been one of the main actual impacts of the project. It was deemed to have been a very high impact for 13 of the case studies. International cooperation is considered to be one of the main actual impacts of the Commission supported research activities, greatly facilitated by the general FP requirement for institutions and researchers in EU MS to collaborate across borders, with the only exceptions of CIEL and GMO (it



31

should however be noted that CIEL, whilst having only one contractor, was a Euratom project – itself a pan-European programme of many years standing). In rating the impact on international cooperation for projects, consideration was made of ongoing links and networks between participating organisations. For one project (MAFCONS), which is to be completed in 2006, there is potential for very high impact. MAFCONS, along with the other projects in its cluster (i.e. policy testing at the EU level, see chapter 3), has stimulated extensive networking between European partners. By examining the projects across the board, the impacts can be observed in the following ways: • All but two projects (CIEL and GMO) were implemented by a consortium of

partners, 15 of which were based on partnerships between EU Member State organisations. In several instances, the ongoing collaboration between EU MS organisations, and between EU and accession country organisations was perceived as a high impact (e.g. FOREN, SCENES and CARE-W).

• Within the project consortia there were on average 2-4 larger and more

experienced partners, and several newer and less expert partners. Due to the methodology of the field work (project leaders were visited and other partners contacted by telephone), the impact of the projects on the capacities of partners (other than the lead partners) may have been underestimated. For example, in the SCENES project, as the modelling was fairly new to a few of the smaller partners, an inevitable result of the collaboration was to raise capacities and skills among these. However, such effects were not necessarily captured in enough detail. Another project (FOREN) involved both academic and cross-EU partners, and was focused largely on skills development. The demonstrable use made of the FOREN guide and Country Specific Tools are evidence of the high impact that the project had on the capacities of its participants, as well as those who use the guide.

• The consortia have stimulated networks which have persisted after the end of

the actual projects. o In the SCENES project, 18 partners across Europe collaborated (some

with a history of collaboration but many were other, new partners). This has continued beyond the project end.

o FUNCARS has led to 3 ongoing training networks. • Ongoing collaborations with new partners and with new sectors, leading to

innovative ways of working, was a general impact of these projects (for the majority but not all):

o Within TTA the type of cooperation within the project was highly innovative and went beyond the sectors within which individual actors usually operate.

o In GALA, multidisciplinary cooperation on a global level created a sense of mission between partners that had not been evident previously.



32

• For projects where international collaboration beyond the EU formed part of their project objectives (e.g. ONTOWEB, MAIZE and CHIN), international collaboration was considered a main area of impact.

o For CHIN for example, the collaboration with Chinese researchers and health authorities was seen as being a two-way learning process and the European researchers’ knowledge of Chinese systems has increased their general expertise in international healthcare areas. The international cooperation is also something of a foundation for other potential impacts (social, quality of life), due to the stimulation of additional research stemming from increased European understanding of Chinese academic and policy contexts.

o For MAIZE, the EU-sponsored project served as a catalyst for substantially increased South-South cooperation (in this case, between South and Central America on the one hand, and Sub-Saharan Africa on the other).

International collaboration is an immediate impact (1st Group) but overlaps with enabling impacts (2nd Group) since cross-fertilisation in ideas, techniques and methods of working were common in the descriptions of impacts generated from international collaboration. Comparing findings with the relevant FP Programme Areas, it is evident that this general objective is being met. Furthermore, the projects funded under the INCO Programmes (CHIN and MAIZE) achieved high and very high actual impact on international cooperation respectively. The linked impacts on human resources are generally considered to have resulted from innovative collaborations. Existing skill bases were critical factors for the achievements of several projects (WETO-2030, FOREN, CHIN, ENHSIN, MAIZE, CIEL, CARE-W, GECS, GALA, BO-BA, ONTOWEB and NESSIE). For example, although REOXCOAT had some impact in terms of skill development, it relied heavily on experts who had been working in the field for decades. Just a few projects had skills development as one of their key impacts: TTA provides an example of this. Since existing skills were generally perceived to be necessary requirements for the projects, any impacts on skills development were therefore perceived as low. It is worth noting, that in the majority of project visits, the evaluation team met the lead partner and only undertook follow-up telephone interviews with other partners. Therefore, many of the face-to-face interviewees may not have been fully aware of, or have prioritised, the impact on human resources and capacities. Due to the diversity of organisations participating, it is therefore likely that the smaller, less experiences, and often new MS based organisations have seen a higher level of capacity and skill development than the more established lead partners. The two projects with a scope beyond Europe (MAIZE and CHIN), raised capacities both of European partners, and more notably with overseas partners. The greatest impact of CHIN was the development of research and analytical skills among the Chinese participants. For MAIZE partners, human and institutional capacities were raised by also increasing the capacity to find local solutions.



33

International cooperation is also the impact for which providing evidence and making attribution is most straight forward:

Table 3: Evidence of international cooperation impact

EVIDENCE – INTERNATIONAL COOPERATION Evidence for very high impact on ongoing international collaborations is extensive and often includes follow-up projects generated by EU project funding:

o A coatings seminar in 2001 involved REOXCOAT project partners, 3 years following the end of the project.

o Several scientific and commercial follow-ups including partners of TTA. o Many ongoing collaborations between SCENES partners and beyond

in a variety of consortia, some of which are supported by DG Environment, including TREMOVE, EXPEDITE, TEN-STAC, SPECTRUM and the IASON/TIPMAC Cluster.

o Ongoing collaborations including with the IAEA (International Atomic Energy Agency) and MAIZE partners.

o Collaborations at all levels between CHIN partners, particularly between researchers and health authorities within Chinese cities.

o Several research follow-up projects from ONTOWEB, under leadership of project partners.

o A follow-up EU-funded research project with participation of BO-BA partners.

Several projects have also led to the creation of networks involving partners of the project:

o FOREN was highly influential in the establishment of The Blueprints Group.

o The Knowledge Web research network followed on from ONTOWEB. o The European Network of Excellence in Cryptographic Research

(ECRYPT) followed on from NESSIE. o Three research networks: EMMA, WONDERFULL and FORCARBON

resulted from FUNCARS. o ENHSIN strengthened the existing network of European Taxonomists

(CITAF), and partners currently collaborate in the Synthesis project. o JRC knowledge and experience in the GMO area has resulted in the

creation of a European network of GMO laboratories, coordinated by the JRC.

4.4 2ND GROUP IMPACTS (ENABLING)

The 2nd Group of impacts, although overlapping with the 1st and 3rd Groups, can be distinguished by its sectoral scope – within an industry, sector or theme. These impacts do not always lead to end results in themselves, but are often facilitative of the broader 3rd Group of higher level impacts.

4.4.1 Policy and standards This section looks both at impact on policy, as relevant to policy-makers, and on broader standards, as relevant to a wider spectrum of stakeholders (scientists,



34

industry, and consumers). For simplicity, both types are referred to as policy impacts. These policy impacts were assessed from a geographic definition, and included impact on MS policies, that of the EU, and beyond. Both FP 4 and 5 included overarching policy objectives such as providing inputs into governmental regulation and standards, bringing about a change in EU-wide standards or directives. INCO and INCO 2 for example (CHIN and MAIZE), have the long term objectives of contributing to the EU’s external policy.

Table 4: Overall rating of policy impact

IMPACTS Actual Potential VERY HIGH 5 NESSIE

GECS FOREN GMO GALA

3 MAFCONS ONTOWEB SCENES

HIGH 3 ONTOWEB SCENES WETO 2030

4 BO-BA CARE-W MAIZE ENHSIN

MID 5 TTA BO-BA MAIZE CHIN ENHSIN

-

LOW 1 CARE-W - NONE / N/a REOXCOAT / CIEL / FUNCARS

As these impacts do not lead to change directly, but are considered to be enabling of broader impacts, they are classified within the 2nd Group of impacts outlined in previous sections. As with international cooperation, policy impacts were (or are expected to be), generated from a majority of the projects. In fact, only 3 were thought to have had no impact on policy or standards. Policy was identified as the main area of impact for 3 projects (GECS, GMO and GALA). Several of the projects within this sample were instigated with an explicit aim of developing tools for EC policy-making and testing (SCENES, GECS, WETO 2030, MAFCONS). For these projects, other eventual impacts (they all rate very highly in actual or potential impact) rely to a very significant degree on the use of these tools/ models in policy-making and testing (predominantly at the EC level). Others (e.g. GMO), have deeply influenced policy-making, primarily at EC level, due to their technical quality, and secondarily in Member States due to a ‘snow-ball’ effect. Among the strongest risk factors for the achievement of potential impact is the lack of use of these tools once they are developed (This risk proved largely non-existent in the case of SCENES, whose trans-European transport network model was developed in response to a genuine need, the usefulness of which was demonstrated both within the project itself - in the form of actual policy recommendations drawn by SCENES from the simulation of various policy



35

measures within the model - and as a building block in a variety of further projects and studies of several DGs). Some projects have had a high impact on the development of standards at the European and international levels precisely because of the credibility and recognition that their EC funding and Europe-wide partnerships have provided (NESSIE, ONTOWEB, TTA and ENHSIN). In contrast, REOXCOAT, FUNCARS and CIEL have had no impact on policy areas, and this is in keeping with their objectives. Considering the challenge and often perceived time-lag between research and impact on policy/ standards, it is interesting to see that 14 of the projects have had actual policy impacts, and that 8 of these were rated as high or very high. However, the intensity of these impacts varies in part due to the broad definition of ‘policy’ used here. For example: • FOREN stimulated the EC’s follow-up project to produce ‘Country Specific

Guides to regional foresight’ (with some impact on setting standards for regional foresight).

• In contrast, CHIN had a very high impact on policy, as the recommendations of

the study were taken up within the Chinese cities. However, due to the geographic emphasis of the scoring criteria it was rated ‘mid’ in this assessment.

In relating this impact back to the Framework Programmes, it is evident that there will be a generalised high impact on policy and standards from this set of programmes (8 projects have recorded ‘high’ or ‘very high’ actual impacts, and further 7 have the potential to do so in the future). Whilst both MAIZE and CHIN did impact policy in the relevant non-EU countries, it has not been possible to assess the impact of INCO and INCO 2 projects on the EU’s external policy.

Table 5: Evidence of Policy Impact

EVIDENCE – POLICY The most tangible evidence of policy impact includes the following:

o The adoption of EC guidelines on Genetically Modified crops following recommendations of the GMO study.

o The take-up of the FOREN guide at the EC, including ongoing activities with new MS and the decision to tailor the guide as a follow-up project.

o Use of the SCENES model in EC policy testing projects (including EXPEDITE itself a building block for SUMMA; the IASON/TIPMAC cluster; SPECTRUM) and studies (TEN-STAC, TREMOVE and PRIMES).

o Acceptance of standard for Semantic Web by 3WC (developed by ONTOWEB).

o Both CHIN and MAIZE had higher impacts in the Non-EU countries (China and Mexico respectively), than they did within the EU. One Chinese city has implemented the recommendations for health



36

EVIDENCE – POLICY insurance schemes developed by CHIN.

o ENHSIN developed much of the methodology used in the ongoing 6th Framework Programme, developing an integrated infrastructure (SYNTHESYS).

o The EC SYNTHESIS project and its standards for the classification of data have in turn been adopted by GBIF (the OECD run Global Biological Information Facility).

Other anecdotal evidence of policy impact includes:

o The GECS computer model for predicting gaseous emissions impacts has been taken up at European and International Policy levels.

o Widespread use of the WETO 2030 model by high level European policy-makers.

o GALA helped to justify Galileo and had a high impact on policy decisions.

o NESSIE resulted in 12 recommendations for ISO (cryptology) standards which can be used across industries.

4.4.2 Scientific (knowledge) The scientific objectives of FP4 and 5 sought inventions and innovations with a concrete or material outcome. The JRC (GMO) was designed to group teams of scientists in a first instance, with a view to creating networks working on specific outcomes. Euratom (CIEL) was working on results-oriented tasks. In addition, the IHP horizontal programme (FUNCARS) sought to improve the human research potential and socio-economic knowledge base.

Table 6: Overall rating of scientific impact

IMPACTS Actual Potential VERY HIGH 6 MAFCONS

NESSIE BO-BA CIEL FUNCARS FOREN

4 CARE-W MAIZE EHNSIN GMO

HIGH 10 ONTOWEB TTA REOXCOAT SCENES GALA CARE-W WETO 2030 MAIZE ENHSIN GMO

-

MID 2 GECS CHIN

-

LOW - - NONE / N/a -



37

In this category, scientific impacts are defined as advances in scientific knowledge and processes. Again, this definition is fairly broad and actual impacts have been recorded across all projects, although these vary a great deal in nature, scope and intensity. Six projects have recorded very high actual impacts and a further 4 are expected to have very high impacts in the future. This is therefore the second highest area of impact after that of international cooperation. Scientific impacts are also perceived to be enabling of broader impacts, although the use of new knowledge and application of improved processes can lead directly to 3rd group impacts, particularly economic and quality of life. One conceptual approach for assessing scientific impacts is to divide them into fundamental and additive impacts, here referred to as super-structural (“the building”) or infrastructural (“the foundation”). Scientific impacts are largely related to increased knowledge. Super-structural (existing and additive) impacts depend on the use of infrastructural (new and fundamental), scientific knowledge directed and applied to a need or problem to provide a solution or alleviation. The impact is therefore generated as a result of application (and/ or modification) of existing scientific knowledge to a problem. While superstructural application of science, and the impacts it generates, are essentially needs or problem-related, the more fundamental infrastructural advances have the potential for more profound and far-reaching impacts. A few of the projects have developed ‘new’ or infrastructural knowledge. • For example, a by-product of the development of a model for managing fisheries

(MAFCONS) is a vast amount of data on the North Sea bed. Project partners believe that the use of this data will have an eventual scope of use much wider than the project itself. This represents a fundamental impact on scientific knowledge (and is likely to have many other eventual impacts, when this data is put to use by other projects). CIEL also developed new knowledge and techniques (relating to the development of fusion energy), which will have impacts beyond the project. At the highest intensity of impact in the scientific field is NESSIE, which is already seen as a benchmark among industry and the scientific community.

Other very high science impacts include those from the development of technologies or the exposure of the need for them: • CHIN developed simple and innovative methodologies for monitoring equity in

health care, as did FOREN for regional foresight, TTA also fits into this category of impact by providing a basic electronic function which allows coordination of dozens other processes in cars. NESSIE made visible the need for new techniques in algorithms, and the GMO project clearly spurred the revival of landscape modelling and the science of co-existence.

Other impacts were achieved through the refinement and advancement of existing (super-structural) knowledge:



38

• For example, CARE-W, SCENES, WETO 2030, GECS (and MAFCONS – although this is not yet complete) all applied and refined an existing methodology to modelling (although GECS to a lesser degree), and REOXCOAT had an impact on the understanding of production processes. A common factor among these modelling projects is their Europe-wide scope. Impacts within this category are therefore high or very high (with the exception of GECS).

• Europe-wide collaboration in research is a common objective of the vast majority

of the above projects, and impacts are related to innovative collaborations and methodologies. Impacts on the infrastructure of research are also therefore included in this category of impact (the interdisciplinary approach of FUNCARS provides a good example of this). ENHSIN too has seen a main impact resulting from the scientific infrastructure developed as a result of innovative collaborations. Similarly, scientific impacts of GALA were the outcomes of a multidisciplinary team, resulting in new working methods.

Of the 3 projects which were funded under Framework Programme areas with an emphasis on science, GMO scored high for actual impact, and FUNCARS and CIEL very high.

Table 7: Evidence of Scientific Impact

EVIDENCE – SCIENTIFIC For those projects scoring high and very high actual impact:

o The parameters and usability of the booster batteries developed by BO-BA had a very high impact, demonstrated by the resulting 6 patents on battery systems management.

o NESSIE, CARE-W, GMO, and REOXCOAT have all stimulated further scientific research in their and adjacent areas (i.e. cryptology research; water and waste water network rehabilitation; co-existence of Genetically Modified (GM) and non-GM crops, and the development of oxide resistive coatings).

o Recognition and credibility of the GMO findings are evidenced by its publication in the journal ‘Nature’.

o There is potential for the knowledge developed in CARE-W to be applied to other sectors (gas, electricity and even road networks).

o MAIZE developed innovative approaches to combining genetic and agronomic strategies.

o The scientific publications resulting from FUNCARS provide evidence of the impact of collaboration between industry and science on carbon nanotubes technology (CNT).

o The prototype European-wide specimen database is a result of the network established during ENHSIN.

o The guide to regional foresight is evidence of the development of these processes (FOREN).

o Funding for the follow-up study on co-existence under FP6 (SIGMEA) demonstrates the impact of the GMO project on the scientific agenda.

o The definition of thematic research areas by ONTOWEB was retained and followed by all subsequent European research activities in the field.



39

4.4.3 Technical The technical objectives under FP4 and 5 included the development of actual products or technology where end results were expected. For example, this applied to REOXCOAT and BO-BA, both funded under FP4’s BRITE/ EURAM (Industrial and material technology) sub-programme, which included an SME element. Technical objectives also particularly applied to the cases of JRC (GMO) and Euratom (CIEL), where specific groups of organisations were set up to respond to a specific technical need.

Table 8: Overall rating of technical impact

IMPACTS Actual Potential VERY HIGH 5 ONTOWEB

NESSIE TTA REOXCOAT BO-BA SCENES

4 MAFCONS CIEL FUNCARS GMO

HIGH 6 MAFCONS GALA CIEL FUNCARS GMO

2 CARE-W MAIZE

MID 4 CARE-W MAIZE ENHSIN FOREN

-

LOW 1 GECS - NONE / N/a WETO 2030 / CHIN The final category of impact within the 2nd Group of enabling impacts, are technical impacts. Technical impacts relate to the development of industrial techniques, subjects, or the application of scientific knowledge. Overall, 16 of the case study projects achieved actual impacts in technical areas, with 6 scoring very high and a further 4 with potentially very high impacts. This impact was not perceived to have been one of the main impacts for the set of projects, with the exceptions of NESSIE, TTA and REOXCOAT. This is linked to the enabling nature of technical impacts. As such, they can be seen as stepping stones to broader impact – with a strong emphasis on their enabling role. This means that there is a higher degree of risk to 3rd Group potential impacts, should the technical impacts (related to innovation, recognition and quality) not be achieved. • For example, the findings for the assessment of the MAFCONS project, which is

due to complete in 2006, include a degree of uncertainty as to whether the potential technical impact will be attained (this is particularly important because technical impacts are the foundation for 3rd Group impacts under MAFCONS). Projects which develop technical products, such as modelling tools, and where eventual use will very much depend on final technical quality, can therefore be difficult to assess prior to completion of the product (model).



40

The largest group of technical impacts are related to the development of products (techniques, modelling systems, etc.): • TTA, which scored very high on actual technical impacts, developed a device

which was clearly satisfying a need of industry, and which has been accepted as a standard mechanism. A very similar result was recorded for REOXCOAT.

• MAIZE also developed bio-technological screening techniques for genetically

modified organisms that are likely to have impacts within its sector. Other technical impacts relate to the development of credible frameworks and structures for further research. Projects scoring high to very high actual impacts in this area were generally developing new structures: • ONTOWEB developed a technical structure that will be the framework for further

developments in the field of ontology-based information exchange (the ‘future world-wide web’).

• FUNCARS also produced innovative techniques (in the field of functionalisation

of carbon nanotubes), although these are still to be applied, (FUNCARS therefore still has a very high potential impact).

• GMO developed tools for the assessments of the co-existence of organic and

genetically modified crops. • Large-scale space project Galileo will largely be based on the systems designed

during GALA. • NESSIE, SCENES and BO-BA also fit within this category of technological

impact. Other impacts in the development of frameworks and structures were building on and refining existing technical processes or applying them to different contexts and have generally scored highly: • Projects belonging within this category include CIEL, ENHSIN, GECS and

FOREN (the score is at mid for both GECS and FOREN, due to the less intense impact – they were not highly innovative technically, whilst SCENES scored high due to its new mix of components and scope).

In reflecting on the impacts of projects with very clear technical objectives, the evaluative findings underline that the relevant projects did meet their objective in this area, i.e. impact in the case of the Euratom would be ‘high actual’, plus with ‘very high potential’ (as in the case of the CIEL project). This was due to an increased collaboration of scientists who would further enhance cross-border efforts between European teams and global networks. REOXCOAT and BO-BA (BRITE/ EURAM) both scored very high for actual impact. Additionally, impacts were also as expected for the QoL, IST, Euratom and IHP programme areas.



41

Table 9: Evidence of Technical Impact

EVIDENCE – TECHNICAL For those projects scoring very high actual impact:

o REOXCOAT successfully developed a mass productions coating line for sputtered resistive oxide coatings. The functioning coatings line in the Mikkeli plant of SAVCOR, as well as the establishment of other facilities in 2 other European MS’, is clear evidence of the technical impact of REOXCOAT.

o ONTOWEB developed a sectoral technical standard which will have widespread uses.

o Similarly, NESSIE is seen to have already achieved consensus on global technical solutions (i.e. ISO Standards for algorithms).

o BO-BA developed a new system for battery management, which is likely to be applied in most battery sectors.

o The SCENES model innovated through adding a new mix of components and scope, applying the modelling to all transport sectors European-wide, and has been used in many contexts, within the EC and outside.

For those projects scoring high actual impact:

o GALA had a high impact on the systems to be used in Galileo. o The impact on the EC of the recommendations developed during

GMO. For those projects scoring mid actual impact:

o The breeding and quick screening techniques developed by MAIZE.

4.5 3RD GROUP IMPACTS (BROADER SOCIETAL)

The final group of impacts are situated beyond the project and even the relevant sector. In assessing these types of impact, attribution and time-lag between project and impact is generally a key challenge. Evidence of impacts has been gathered where an actual impact has already resulted and estimates of potential impacts were clearly visible.

4.5.1 Social High level social goals under FP5 were generally to advantage the average EU citizen through job creation, future wealth, social well-being, and manifest improvements of the socio-economic knowledge base. In this respect, the goals were rather forward-thinking since job creation and wealth would take time to develop. A particularly relevant sub-programme area is GROWTH with an objective to develop European cross-sectoral projects in these areas (SCENES and GALA), and the IST FP area (under which ONTOWEB and NESSIE are funded) which has an overall goal of developing a coherent information society.



42

Table 10: Overall rating of social impact

IMPACTS Actual Potential VERY HIGH - 3 ONTOWEB

REOXCOAT FOREN

HIGH 3 REOXCOAT GALA GMO

5 NESSIE BO-BA CHIN SCENES MAIZE

MID 1 CHIN 3 TTA CARE-W FUNCARS

LOW 3 TTA MAIZE FUNCARS

-

NONE / N/a MAFCONS / GECS / WETO 2030 / CIEL / ENHSIN Social impacts, defined as improvements in the socio-economic knowledge base, job creation, increased public awareness, etc., are by definition expected to occur between 3 and 10 years following the end of the project although this of course is not always applicable. Whilst the assessment of 7 projects recorded actual impacts, only four of these scored ‘mid’ and above. In accordance with the definition of 3rd group impacts occurring between 3 and 10 years following the project, there were no ‘very high’ actual impacts in this category. REOXCOAT finished in 1998 and scored highly - as expected as this project is one of the oldest. Three projects have a potential for very high impact in this area; 5 ‘high’; and 3 ‘mid’. Social impacts were seen as being the main area of impact for CHIN alone. No actual social impacts have been recorded and none expected for 5 projects (MAFCONS, GECS, WETO 2030, CIEL, and ENHSIN), as such, this is the area of impact with the lowest level of either actual or potential impact. This is in part due, however, to the narrowed definition of social impact at this level of analysis, which does not include potential social impacts from wealth generation. Impacts in the economic and quality of life spheres are likely to also impact in the social sphere, and these causal links are elaborated in the individual case study evaluation grids (see Appendix 8). Actual social impact was highest for REOXCOAT, GALA and GMO, and these included very high levels of job creation. REOXCOAT had an impact on regional development (Mikkeli) as well as high job creation in three European countries and a potential safety improvement in the use of mobile phones. GALA underpinned safety, security and employment in the systems designed for Galileo, and GMO was also rated high for social impact, due to increased public awareness of the issues studied. Potential impacts have been identified most frequently in the area of job creation. The potential creation of new markets, services and jobs were identified for NESSIE.



43

Other project include TTA (jobs in producer companies), BO-BA (battery maintenance specialists), CARE-W (water management specialists), FUNCARS (members of the Young Researchers Network), and FOREN (regional job creation). Potential job creation is always dependent on commercialisation, the application of results or other downstream effects. Another potential impact is related the socio-economic knowledge base through the facilitation of EU citizens’ empowerment by access to information (ONTOWEB). The non-EU geographic scope of both MAIZE and CHIN again reduced the score for potential impacts at European level, although their potential social impacts were very high at a regional/ non-EU level. Potential negative impacts were noted for both MAFCONS and CARE-W. In the first case, this would be a logical result from the application of the fishing model in legislation – reducing both overall catch and effort - and therefore potentially reducing jobs in the fishing industry. For CARE-W, potential increased efficiency in the management of water networks could result in job losses. Looking at impact in relation to FP Programme Areas, IST projects (NESSIE and ONTOWEB) both have the potential of high or very high impact. Under GROWTH, both SCENES and GALA (via the much larger GALILEO initiative) are also expected to perform in these areas. Whilst the assessment of SCENES did not record an actual impact, it is expected that high social impact will result from more sustainable transport (although this potential is highly reliant on the application of the model to policy testing).

Table 11: Evidence of Social Impact

EVIDENCE – SOCIAL Evidence for actual impacts in social areas is both quantitative and qualitative:

o REOXCOAT provided the most tangible evidence for social impact –333 jobs have resulted within the EU.

o The widespread European media coverage of GMO is a good proxy indicator of increased public awareness.

o Linked to the policy impact, GALA has a high social impact by embedding safety, security and employment as a basis for Galileo.

o Social impacts emanating from the focus on equity in health insurance policy at the city level in China is a direct result of the adoption of CHIN recommendations.

4.5.2 Economic FP objectives in the economic sphere include increased sales, profit, and competitiveness. The long-term expected result would be to change the economic landscape of Europe. FP Programme Areas that have objectives in the economic sphere include EESD (sustainable development through economic competitiveness in Europe - WETO 2030), IHP (to contribute to the Community's competitiveness - ENHSIN), Euratom (making current technologies more economical - CIEL) and GROWTH (promoting competitive and sustainable growth – GALA and SCENES).



44

Table 12: Overall rating of economic impact

IMPACTS Actual Potential VERY HIGH - - 9 ONTOWEB

NESSIE TTA REOXCOAT BO-BA CARE-W CIEL FOREN GMO

HIGH 5 TTA REOXCOAT BO-BA GALA GECS

2 MAIZE ENHSIN

MID 3 NESSIE WETO 2030 FOREN

3 MAFCONS SCENES FUNCARS

LOW 1 CIEL 1 CHIN NONE / N/a - Economic impacts, here including increased competitiveness, development of new markets, creation of SMEs, and cost-savings, are expected to be more evident in the long rather than the short-term. Notwithstanding, 9 projects have, already to date, reported economic impacts, with 8 in the ‘mid’ or ‘high’ range. With 9 projects having potential for very high impacts in the longer term, and all 18 projects expecting some impact in this area, it is possible that this will be the area of greatest overall impact in years to come. However, this could also be due to the greater tangibility (and prediction) of impacts in the economic area compared to social or quality of life. Economic impacts were identified as the main impact for just one project (BO-BA). Actual impacts in the economic sphere are mostly related to savings, at both MS and EU levels. NESSIE generated savings at the MS level and scored ‘mid’ by reducing the likelihood, and therefore cost, of the need to recall ID cards in Belgium. The ‘bus’ system (for coordinating electronic functions in vehicles for the entire automobile sector and beyond) developed by TTA is a high actual impact, and this relates again to savings resulting from the application of one standard to the whole transport sector. The application of the FOREN guide, and follow-up Country Specific Guides has led to savings for implementing authorities. Other actual impacts included sectoral ones. REOXCOAT had a high impact on the competitiveness of the European mobile phones industry, by developing more efficient mass production processes. • Similarly, BO-BA stimulated the batteries industry and demand for specialist

knowledge.



45

• GALA stimulated markets by maintaining the involvement of large companies, with subsequent impacts on their suppliers.

• GECS provided predictive assessments of gross and net costs of emission

abatement policies, directly supporting the Emissions Trading Directive. • WETO-2030 is likely to lead to savings by the energy sector by developing more

accurate predictions of energy costs and usage, and the model has already been used in a number of contexts such as economic implications of Kyoto, effects of future energy technologies, and rises in oil prices.

The potential impact of ONTOWEB was sectoral impact (e-commerce), which is likely due to the follow-up project ‘Knowledge Web’. Similarly, the increased security offered to banking and telecoms sectors, resulting from NESSIE, is likely to lead to high impact in these sectors. ENHSIN too has potential high economic impacts in certain sectors. Other potential impacts include those on savings, either to industry (from fraud prevention by NESSIE), or to government (including more efficient regional planning and innovation strategy from FOREN and more efficient and sustainable transport planning utilising SCENES). CARE-W, if implemented and widely used, could lead to a number of savings (for utilities in efficient development of networks and avoidance of penalties, and lower the cost of economic damage resulting from disruptions and failures). The development of tools by GMO to prevent cross-contamination of non-GMO with GM crops will minimise economic losses, particularly for organic farmers. Similarly, MAIZE may well impact the economic performance of small and medium producers of maize in developing countries. CIEL is likely to have widespread economic benefits in the future as it is crucial to the achievement of longer reaction times vital to achieving the steady state conditions required in future nuclear fusion power generation that will provide low cost and safety. Regional economies in CHINA are likely to have an economic boost from the implementation of the recommendations. Similarly, as more regions apply the guidance emerging from FOREN, it is probable that the resulting innovation strategies will boost the economy, potentially on a global scale. Relating impact back to the FP Programme Areas, actual impact has been seen for 2 of the EESD projects, (GECS – ‘high’ and WETO 2030 – ‘mid’), and impact is expected for the other EESD project (CARE-W – ‘very high’). For the IHP area, impact is expected for all relevant projects, the intensity and scope are also varied (FOREN – ‘very high’, ENHSIN – ‘high’, FUNCARS – ‘mid’). CIEL, part of Euratom, has recorded an actual low impact, although the potential for this over the longer term is very high.



46

Table 13: Evidence of Economic Impact

EVIDENCE – ECONOMIC For those projects scoring high:

o TTA developed an industry standard for automobile electronics, with end results being applied in the car industry. Its impact also included the higher turnover and creation of specialised SMEs. Moreover, the application of TTA technology in the railway and airplane industries already led to a high economic impact.

o REOXCOAT saw the growth of one SME (SURFCOAT) from 2 people to over 1,000 and the expansion into 2 additional European countries and to China, Brazil and the US. SURFCOAT turnover was €19.9M in 2002.

o Existing evidence for the economic impact of BO-BA can be seen in the changes to public transport management in some cities in Germany due to savings via the use of new battery systems – cities in Lower Saxony (Hanover) started piloting the use and management of booster battery powered buses.

o GECS has been used to assess the costs of emission abatement policies and support the implementation of Emissions Trading Directives.

o WETO 2030 has been used by the government of France in relation to predicting energy costs/ resources and there have been approaches from the World Energy Council.

For those projects scoring mid:

o The technology developed in encryption by NESSIE has already generated savings in the production of ID cares in Belgium.

o Regional authorities applying the FOREN guide have made savings in cost of their planning processes.

4.5.3 Quality of Life The Quality of Life (MAFCONS), IHP (FUNCARS, ENHSIN and FOREN) and Euratom (CIEL) FP Programmes Areas primarily targeted the improvement of the quality of life of citizens as a main objective; with human potential, the environment and health and safety as some of the components.



47

Table 14: Overall rating of quality of life impact

IMPACTS Actual Potential VERY HIGH - 8 MAFCONS

ONTOWEB NESSIE REOXCOAT SCENES CARE-W ENHSIN GMO

HIGH 2 GECS GMO

3 CIEL MAIZE FUNCARS

MID 2 NESSIE WETO 2030

2 CHIN FOREN

LOW 2 CIEL CHIN

-

NONE / N/a TTA / BO-BA / GALA Quality of life impacts identified include improvements in the environment, and on the health and safety of citizens. As with the other 3rd Group impacts, the majority of projects have not led to actual impact in this area to date. As expected, measuring these impacts is challenging at this stage (between 0 and 5 years since the end of the projects). Nevertheless, 6 projects have scored actual impact, again distributed between the ‘high’, ‘mid’ and ‘low’ categories. Four of these projects have the potential for higher impact in the future, and a further 8 projects have potential for future but not actual impact. No impacts in the quality of life area were identified for TTA, BO-BA or GALA. Actual impacts in the ‘high’ category include those from GMO, such as the adoption of the study findings for elaborating guidelines for GM Maize farmers in Catalonia, Spain, which are having an impact on the environment there. The high dissemination of the findings of GMO may well push MS governments to act preventatively, and this will lead to other very high environmental health and safety impacts. In the ‘mid’ category, NESSIE had immediate and wide ranging impacts on reliability and trust (through the definition of NESSIE approved algorithms). Widespread use of NESSIE algorithms will generate broad impacts in the areas of safety and across the public and private sectors. Potential impacts relating to the environment and thus to quality of life are potentially very high for MAFCONS (although these impacts face many obstacles, not least resistance by the fishing industry). The main impact of ENHSIN will be on the environment and health, on a global scale and relating to enhanced environmental protection and the discovery of new species for medicine. Other potential impacts on quality of life are indirect, for example the human potential impacts likely to be generated through the research infrastructure established by FUNCARS.



48

Similarly to MAFCONS, another project with very high prospective impact on quality of life, but with significant risk factors, is REOXCOAT (this relates to health impacts but depends on the impacts of the coatings on safety being proven). The SCENES project also fits closely to this category of endeavour. If utilised, the model will facilitate sustainable development and improve quality of life across the EU. High level quality of life impacts in non-European settings are expected to result from both MAIZE and CHIN: MAIZE by allowing for growth of maize on acid soils and CHIN, by introducing equity analysis and research methodologies to health insurance scheme development in China. Other areas of potential impact include those with effects on human potential: • For example, ONTOWEB will likely increase information and communication

among populations who to date, have had little access to the products of the information revolution. Increased safety in communications and other industries, combined with fraud avoidance, is likely to be beneficial to public as well as private sectors, and have widespread quality of life impacts.

• CARE-W also fits this category, as very high and wide ranging impacts are

potential results from improved water networks. • By increasing organisational capacity to facilitate innovation, the use of FOREN

guides is likely to have a mid impact on quality of life. • For GECS, there is a key environmental impact through assessment and

eventual controls on greenhouse gas emissions (for example through emissions trading systems) and so alleviation of man-made climate change.

• WETO 2030 forecasts energy resources and costs in various scenarios thereby

supporting energy conservation and usage. • CIEL contributes to the scientific foundations on which the safe cheap

production of energy in nuclear fusion generation depends. • FUNCARS has led to the creation of 3 research networks, and has also has had

an impact on the development of capacities in the carbon nanotubes research field. This in turn is likely to lead to a high impact on human potential and quality of life.

In linking impacts back to FP Programme Areas, it is quite clear that time (3-10 years) would generally be necessary for quality of life impacts. The only actual impact recorded under programme areas targeting quality of life was for CIEL and that was ‘low’. None were attained by projects coming under other relevant overall objectives (e.g. QoL – MAFCONS, or IHP - ENHSIN, FOREN and FUNCARS). Interestingly, where the quality of life objective was not necessarily a prime concern, it nonetheless occurred on an actual level, in a few cases (e.g. GECS and GMO projects).



49

Table 15: Evidence of quality of life impact

EVIDENCE – QUALITY OF LIFE For those projects scoring high:

o As a result of GMO, Catalan farmers have been provided with guidelines on how to separate GM from non-GM crops to reduce risk of cross-contamination.

o Greenhouse gas emissions are better controlled through in abatement strategies and specific actions such as use of Emissions Trading Directives (GECS).

For those projects scoring mid:

o The definition of NESSIE approved algorithms and their widespread recognition.

For those projects scoring low: o The implementation of recommendations, and the results from an

equity approach to health insurance at the city level in China (CHIN).

4.6 CROSS-CUTTING IMPACTS

In addition to the groups of impact analysed above, structural, cross-cutting impacts also emanated from the projects’ effects on research infrastructure and from their European scope.

4.6.1 European Added Value The small number of projects selected for assessment has not allowed the evaluation to characterise different clusters of activity, in a definitive way. However, there do appear to be several trends emanating from this cluster of projects: • One is the creation of a variety of EU level research infrastructures. In line

with the emerging status of European level research, within many sectors, these projects are initiating networks, collaborations and infrastructures for European research. A good example of this type of activity is ENHSIN and the piloting of European inter-operable databases. Just a few examples include the following: ENHSIN project partners enthused about the multiplying effect of collaboration as well as the establishment of shared infrastructure (a database). GALA and GECS resulted in the development of equivalent infrastructure, and both the ONTOWEB, FUNCARS, GALA, NESSIE projects developed a framework for ongoing collaborative effort between EU level partners. The infrastructural impact is particularly noteworthy in the establishment of frameworks and infrastructures for ongoing cooperation between the older and new EU MS.

• Another key trend at the EU level relates to increased capacities, and this is in

large part related to the characteristics of the project consortia. The general mix of old and new MS ensures that a high degree of skills building is a by-product of this research, on both sides. Some participants have learnt new skills, whilst others have gained experience of applying skills to differing contexts (increased human resource capacities resulted from the majority of the 18 case studies, most notably, CHIN, SCENES, BOBA, MAIZE, and FOREN).



50

• The projects with an SME element (BO-BA, ONTOWEB, TTA, REOXCOAT and

SCENES) all had positive impacts on the participating organisations. Within TTA the main impact on the SMEs was skills development and organisational specialisation, whilst REOXCOAT saw an SME develop into a multinational medium sized firm. Both these SME impacts were related to technological developments inherent in the project.

4.6.1.1 Counterfactual This section focuses on the importance of European Commission funding to the initiation of each of the 18 projects and hence their subsequent impacts. A vast majority of the projects (11 out of 18 were classified as unable to have taken off without European Commission funding, whilst approximately half (7 out of 18), recognised that funding had had a great influence on the projects’ impacts, although some of the impacts may have occurred anyway. The results indicate that none of the projects would have been able to take off without at least some EC funding. Several projects indicated that the entire impact of the initiative was due to support from the European Commission. These projects were often relevant mainly to the EU level, such as in the case of the transport modelling project SCENES, which was instigated by the EC and would not have taken off at all without its funding. Other examples indicate that European Commission funding opened up avenues for exploring information. For example in the case of NESSIE, academics were given the opportunity to conduct cryptology research for the first time, whereas previously, such research was primarily driven by Governments’ national security needs. Similarly, in the case of WETO, without support from the European Commission, the US Dept of Energy and the IEA would have remained the only sources of such information. Furthermore EC support to GALA ensured that Galileo was not dependent on the military driven inputs of the US, the project was entirely dependent on public sector (EC) funding at this stage. Other projects indicate that European Commission funding was an important element in complementing existing funding sources. For example, the MAFCONS project is partially funded by universities and indicated that there was no possibility for the project to have been funded from sources other than the European Commission. Similarly, in the case of TTA, the cost would have remained too high for individual firms, so European Commission funding was necessary in order to share the financial burden. Moreover, the much shorter timeframe of industrial research would not have allowed for embarking on crucial but long-term research goals. In leveraging additional funding, the EC support has been critical to a few projects (for example FOREN, REOXCOAT, GALA, FUNCARS and ENHSIN). The EC label attached to these projects has also indicated to external stakeholders a level of relevance and quality. REOXCOAT partners leveraged additional support from local government and attributed this entirely to the EC label. For CHIN the leveraging and multiplier factors were closely related to skills development among Chinese researchers and officials. A proportion of the projects indicate that some impact would have been evident, but that the additionality of European Commission funding allowed a faster pace of development over a shorter time-scale, with a greater scope of activities to be



51

envisaged. Such evidence can be seen in the cases of BO-BA and CHIN, where increased research possibilities were available and on a larger scale, due to support from the European Commission. The particular characteristics of the FP funded projects include the requirement for cross-EU collaboration. Some of those interviewed perceived that this requirement, and the difficulties in working with less experienced and sometimes less engaged partners, limited the qualitative outcomes of the projects. However, this must be balanced with other desired outcomes such as capacity development and the establishment of European research infrastructures, which this evaluation has found proof of.

4.6.1.2 Sustainability of impact beyond EC funding In terms of the likelihood of project impacts continuing without the support of the European Commission, nearly all (16 out of 18), indicated that there was a very high or high possibility that project effects would be sustained. It is important to note here, that these scores refer to the likelihood of actual impacts being sustained over time rather than potential impact being achieved. However, importantly, the results indicate that none of the projects would completely fail to demonstrate any sustained impact, although in some cases, there are indications that impacts would not be as far-reaching without at least some continued support from European Commission funding. Many of the projects, particularly those with clearly identifiable results, demonstrated that the likelihood of project effects being sustained beyond Commission funding was very high. For example, in the cases of BO-BA and ENHSIN, project momentum was recognised as an indicator of sustainability. In the cases of FOREN and FUNCARS, tangible research evidence, such as guides and publications, provided proof of project impact and furthermore, demonstrated potential for further project development and expansion. In the example of MAIZE, national sources (based in Mexico and Colombia) provided evidence of the high relevance of the project and proved that its results had been adopted and that work is ongoing. In some cases, it was perceived that the level of ongoing impact would be relatively low, without continued European Commission funding. For example, in the case of CHIN, only the personal relationships factor would be sustainable, but it would not be possible to demonstrate concrete tangible activities without further Commission support. For the MAFCONS project, it was indicated that continuous support from the Commission would be required in order for the model to be updated and upgraded on a regular basis. This is also applicable to the other projects developing models for policy testing (i.e. SCENES, GECS and WETO 20-30). Other results indicated that alternative opportunities would be sought out in the absence of European support. For example, in the case of ONTOWEB, it was emphasised that the OWL standard was a sustainable effect in itself and research activities would continue in the USA, if opportunities were not available in Europe. Many projects highlighted the fact that although the likelihood of sustainability would be reasonably high without European Commission funding, the effects would not be as extensive without its support. This was evident in the case of SCENES, which



52

indicated that longer term funding and measures to ensure continuity were required to amplify impacts.

4.7 OVERALL ASSESSMENT

The chart below portrays the balance between total actual and total potential impacts (which have been quantified from the ratings7). As can be seen, the majority of actual impact has occurred within the 1st and 2nd Groups of impact (international cooperation - 1st Group, technical, policy and scientific impacts - 2nd Group). Although some 3rd Group impacts have been realised (social, economic and quality of life), the potential for this group is far higher than the actual.

Figure 3: Balance between actual and potential impacts in the sample

7 In quantifying the ratings of impact, ‘Very high’ scored 4, ‘High’ scored 3, ‘Mid’ scored 2, ‘Low’ scored one and none scored 0.

0

20

40 60

80 100

120

140

First

Group Second Group

Third Group

Potential Impacts

0 20 40 60 80

100 120 140

First Group

Second Group

Third Group

Actual Impacts



53

Table 16 below clusters projects according to their end dates – the duration of time elapsed since the end of the project – grouping the projects into 3 clusters: • A – Projects that are ongoing or have completed within a year; • B – Projects that completed between 1 and 3 years ago; • C – Projects that completed more than 3 years ago. Whilst all projects achieved actual impact in the 1st and 2nd Groups of impact (immediate and enabling), the attainment of actual impact in the 3rd Group (broad societal) is more mixed. There is an obvious correlation in this table between projects classified in Cluster C (those that completed before 2001), and demonstration of impacts in the 3rd Group areas (economic, social and quality of life). In fact, all projects classified in Cluster C, defined by length of time since the project end, have achieved ‘high’ impact in the 3rd Group of impacts with the only exception of SCENES. These impacts were felt in all areas, although more were recorded in the economic sphere than in social or quality of life. Of the 9 projects classified in Cluster B (those completing between 2001 and 2004), 8 have recorded impact within the 3rd Group of areas of impact. The only one not to score in the 3rd group – ENHSIN - was a demonstrator project, and as such it was not expected to score direct and high impact within the 3rd group. ENHSIN has however, led to other projects which are likely to achieve high 3rd group impacts. This is a marked difference however from the 3rd Cluster, as only 1 project scored ‘high’, 4 scored ‘mid’, 3 ‘low’, and one has not had an impact in the third group to date. As expected, none of the Cluster A projects (those ending in 2004 or after) scored any actual impact on 3rd group areas. In accordance with the three groups of impact model, the ratings of impact demonstrate that actual impacts have been made mostly within the 1st (during project duration) and 2nd (< 3 years after project end) groups, and that the majority of potential impacts are towards the 3rd group impacts (3-10 years after project completion). These findings also demonstrate that in order to assess ‘high’ (broader societal) impact, examining both potential and actual, a period of between 1 and 3 years is necessary. However, in examining 1st and 2nd group impacts, and constructing a causal analysis to link these groups to eventual 3rd group impacts, it can be possible to assess potential impact at an earlier data. It is interesting to note that the ‘main impacts’ identified during interviews and presented in the table below are not ordered according to the time elapsed since the end of the project. For example, Cluster A projects (that are still ongoing or that have completed within a year), identify both 1st Group (international cooperation) and 3rd Group impacts (quality of life) as their main impacts. This range of ‘main impacts’ is also apparent for the group of projects that have completed before 2001 (international cooperation, technical and economic impacts are all identified). It seems therefore that ‘main impacts’ can be the identified within the project cycle, be they potential or actual.



54

Only 5 of the projects identified the main impact as being social, economic or quality of life, and these projects came from all 3 clusters. An explanation for the comparatively few 3rd group impacts being identified as the ‘main impact’ may be that project leaders and partners do not always position their research within a broad, societal context. It could also be due to the fact that impact in these areas is often highly contingent on external factors. In addition, project leaders and partners may be aware of some of the impacts, out of their own interest in its development, but are not generally monitoring systematically for impact. For example, the use of the NESSIE technology in the ID card system in Belgium was only known to the managers of the project because they were approached to provide support to it. This is the case for many of the projects assessed here, and particularly for those producing a product (e.g. FOREN, NESSIE) which enters the public domain.



55

Table 16: Project end dates and highest impacts

PROJECTS: FP Prog Area

End Date End Date A=2004 / B=2001-4 / C = 1998-2001

Actual group 1 impacts



Highest group 3 score

Highest Group 3 Score Area

Main Area of Impact

Actual / Potential

TTA IST 30/11/1998 C yes yes yes High Econ International cooperation / Technical

Actual / Actual

REOXCOAT BRITE/EURAM

14/06/1998 C yes yes yes High Soc / Econ Technical Actual

BO-BA BRITE/EURAM

30/06/1999 C yes yes yes High Econ Economic Actual

GALA GRTH 10/06/2001 C yes yes yes High Econ / Soc Policy Actual

GECS EESD 31/07/2002 B yes yes yes High QoL / Econ Policy Actual

GMO JRC Apr-04 C yes yes yes High Soc Policy ActualCIEL EURATOM 2002 B yes yes yes Low QoL / Econ Scientific Actual

MAIZE INCO2 31/10/2003 B yes yes yes Low Soc International Cooperation / Scientific

Actual / Actual

FUNCARS IHP 31/03/2003 B yes yes yes Low Soc Scientific ActualNESSIE IST 31/12/2002 B yes yes yes Mid QoL / Econ Technical Actual

WETO 2030 EESD 30/06/2002 B yes yes yes Mid QoL Scientific ActualCHIN INCO 31/01/2002 B yes yes yes Mid Soc Social /

International Cooperation

Actual / Actual

FOREN IHP 31/01/2002 B yes yes yes Mid Econ Scientific ActualMAFCONS QoL 30/06/2006 A yes yes no n/a n/a Quality of Life Potential

ONTOWEB IST 31/05/2004 A yes yes no n/a n/a International cooperation

Actual

SCENES GRTH 31/03/2001 C yes yes no n/a n/a International Cooperation

Actual

CARE-W EESD 31/01/2004 A yes yes no n/a n/a Quality of Life Potential

ENHSIN IHP 01/04/2003 B yes yes no n/a n/a Scientific / Quality of Life

Actual / Potential

The time factor is most important for 3rd group impacts, as can be seen from the following charts which cluster the projects into A, B and C according to end-date (A is 2004; B is 2001-2004; and C is 1998-2000). The charts summarise total actual impacts by quantifying the ratings of impact (1-Low, 2-Mid, 3-High, 4-Very High). As can be seen from these charts, the time factor is the most critical for the assessment of higher (3rd Group) impacts:

Figure 4: Average impact score for 3rd group impacts

0

1

2

3

4

5

A Projects B Projects C Projects

Total Average Group 3 Impacts by end date



56

Figure 5: Average impact score for 2nd group impacts

0

2

4

6

8

10

12



Figure 6: Average impact score for 1st group impacts

3.2

3.3

3.4

3.5

3.6

3.7

3.8



4.7.1 Implications for the classification of potential impact There seems to be a simple divide between those projects that seek to develop ‘new’ knowledge and those that are seeking to apply it (perhaps by refining existing scientific knowledge and then applying this to practical situations.) Within the strategy of Framework Programmes, both are of relevance. The more fundamental research activities examined as part of this assessment include MAFCONS, CIEL and NESSIE. In each of these projects, new information or techniques have been developed and these are likely to act in an infrastructural way, in relation to application and eventual broad societal impact. Infrastructural impact derives from changes in scientific thinking or approaches that plough new furrows in scientific knowledge rather than just applying the knowledge base that already exists to satisfying a need or a problem. It is infrastructural in that its impact is to add to or alter the very foundations – the scientific knowledge itself. Here the impact ‘makes a difference’ because of changes in scientific thinking, rules or knowledge that have been created through observing new phenomena, validating new hypotheses, elucidating new information and so forth in the more classical mode of scientific discovery or creation of new knowledge as a result of the project. The judges of the impact here are more likely, certainly initially, to be the scientific community – other stakeholders will make a judgement only as the new knowledge comes to be applied to specific problems at a super-structural level.



57

Super-structural impact depends on the use of infrastructural scientific knowledge (the foundations of the building in the diagrammatic analogy below) directed and applied to a need or problem to provide a solution or alleviation (the superstructure of the building). The impact is therefore at this super-structural level – that is as a result of application (and/or modification) of existing scientific knowledge to the problem. This application of the science (the building) and the impact created in consequence are essentially needs/problem- related. Users and other stakeholders are the key judges of the impact achieved as a result of the science undertaken pushing current scientific thinking into the area of need in order to ‘make a difference’. Whilst this divide may not be as rigid as portrayed here (MAFCONS is in fact super-structural and the infrastructural impacts are not its main objective, ONTOWEB might be a project where infrastructural and super-structural considerations are almost interchangeable), it may still be useful to distinguish, at the ex-ante stage between these types of project, in order to better understand and measure the impacts expected to result from the various types of projects.

SUPER-STRUCTURAL

(the building) • application of current scientific

knowledge to a need or problem generates the impact

• stakeholders involve users and developers who have the need

• impact is best judged by these ‘customer’ stakeholders

ground level

INFRASTRUCTURAL (the foundations)

• the current scientific knowledge or understanding • stakeholders here are primarily the scientists themselves –

and those deciding on the funding of scientific endeavour • impact results from changes in or additions to this scientific

knowledge or understanding – direct impact on needs can be far more long term but when it comes can be fundamental and multi-faceted

• impact is best judged by the scientific community itself



58

5 CRITICAL FACTORS

Evidence of critical factors which facilitate or impair impact have been gathered and triangulated with various stakeholders during the interviews. The information to support these come from the scoring exercise of impact drivers (as presented in the project grids), and factors that have been analysed from general discussions with stakeholders. While some of the critical factors seem to be relevant to any project at any point in time, there are others that clearly are more important at different stages of the project cycle. Therefore, where possible, an effort has been made to link the latter to the stage in the project cycle where it is seen to be most relevant, starting from the proposal stage and ending with the exploitation of results. It is important to note that in identifying these critical factors, both potential and actual impacts were considered.

5.1 CRITICAL FACTORS ACCORDING TO THE PROJECT CYCLE

The first section of this chapter focuses on outlining critical factors which are seen to be particularly important at specific stages of the project cycle.

5.1.1 At the proposal stage

5.1.1.1 Needs assessment Research activities which were designed to fill a clearly definable need among its end users, be they consumers or policy-makers, society or the economy, were more successful in attaining impact, including:

• Bottom-up, market-driven projects such as TTA, BO-BA, FUNCARS and GALA which achieved 3rd group economic impacts. For example, TTA was developed in real-time situations with the end user in mind; therefore, the protocols and systems that resulted as the final products were developed with the need clearly understood. ENHSIN used a questionnaire to form its needs assessment which was critical in achieving high impact of the project.

• Demand-driven projects such as CHIN and GMO, which achieved 3rd group

social impacts. CHIN was put into place on demand by the Chinese government and GMO on demand by DG Agriculture.

Other examples include:

• WETO 2030 which clearly addressed the Commission’s specific needs of planning future research and development projects;

• CIEL which very effectively addressed the need of developing a deep

understanding of constraints related to nuclear fusion generation, and

• FUNCARS which developed the scope and character of its research activities based solely on an initial needs assessment.



59

The needs assessment helps to raise awareness among project partners of the eventual impacts. This in turn helps project partners to maintain the focus on the needs throughout the life of the project. This was a critical factor in many cases, and particularly in the development of ENHSIN, and the coordination of GALA.

• By the same token, a poor initial needs assessment can reduce the actual impact such as in the case of CARE-W and MAFCONS (e.g. creating a product for industry and producing a model for the EC).

The highest impact is achieved if a project identified a theme just before it broke-through, when it was just an emerging issue. REOXCOAT is one out of several sample projects which illustrates this point. The project partners were very aware of the nascent nature of the mobile phone industry, particularly in Finland, and set-out with the needs of this industry very much in mind.

5.1.2 At the implementation stage

5.1.2.1 Human Resources Human resources were considered to be key impact drivers in 15 of the 18 cases and were seen as “high” in the remaining 3 cases. Emerging from the analysis, it is crucial that project leaders have the authority, credibility and excellence to achieve high impact. Strong leaders attract the best partners, and are able to keep partners involved. The drive to be part of projects lead by strong leaders has a stronger influence on impact than any project management tools. Moreover, strong leaders are credible to impose sanctions for poor performance of partners. Within the sample, there were several occasions where the project leaders deployed tactics (such as clear division of labour, and presentation duties) to engage ‘sleeping’ partners. Several other factors have been found to be determinants of the high significance of the ‘people’ factor:

• The level of interaction between experts; • The project management and coordination skills; • The quality of the expertise brought to the project; • The personal attributes of the coordinator; • The level of sharing of a common experience; • A commitment to deliver final outputs and outcomes.

Excellence and quality, leadership and vision of the people involved in the project have been found to be of critical importance for attaining impact, including:

• In the three projects which reported International Cooperation as one of their main impacts (MAIZE, ONTOWEB, and TTA). These were driven by strong leaders: gurus or authorities in their fields, whose credibility, vision and leadership helped to achieve the impact. GALA, for which policy was the main impact, was also driven by a determined coordinator with personal charisma.



60

• In CIEL and FOREN, effective project management was a vital success factor leading to impact.

• In BO-BA, inspiration and lead of highly-skilled scientist with strong managerial capabilities.

• Capacities of scientists and researchers, and their strong personal relations were some of the most critical impact drivers of FUNCARS.

• In NESSIE, the credibility of the research team and the positive interaction between scientists and industry is contributing to its actual impact on technical standards and its long-term impact on the economy and quality of life. Similarly in CHIN pre-existing personal relationships between senior health officials and universities were a very important factor. In GALA, CIEL, MAIZE and CARE-W, the credibility of the whole group of researchers, not just the leaders, is contributing to actual and potential impacts;

• In GECS and REOXCOAT, the teams were tightly knit which facilitated their management. The complimentary skills and experience of the people involved intensified the impacts.

• In ONTOWEB and ENHSIN, the interaction among the experts was the core element leading to the high impact of the project.

The fact that the credibility and excellence of the people involved in the projects are important impact drivers is perhaps not surprising in the current sample of projects. Firstly, given that the Framework Programmes promote some of the best of European research and only 18% of FP applications succeed, a high level of excellence of the people involved would be expected. In addition, a second level of ‘selecting the best’ is added to our sample, in that the 18 cases identified, were seen as the most likely to have achieved high impact by EC project coordinators. In this process, it is possible that the best partners with the best scientists were selected by the best coordinators. In summary, the ‘people’ factor is seen as an extremely important impact driver in all the cases studied, providing one of the key foundations in delivering high real and potential impacts. The overt training added value was not the most significant in many of the cases. Many of the project leaders interviewed during this evaluation began the project with an appropriate level of expertise. However, the impact on capacities within the smaller (often the new MS) partners should not be underestimated as ongoing sustainable networks and partnerships resulted.

5.1.2.2 Partnerships Partnerships seem to be more effective when there is a high-level of trust between individuals in the partner organisations. This is why partnerships based on pre-existing relationships are clearly seen to be working better. A good example of the positive collaborations between existing networks was demonstrated in the REOXCOAT project.

• Another example is the CHIN project – there a critical factor was the existing relationships between senior health officials (in the cities) and the researchers (within Chinese universities). Exceptionally, a high level of trust and well-functioning working relationships (with wide-spread sharing of information and learning) can be achieved between new project partners. This is



61

demonstrated by CARE-W where consortium partner staff got together on a regular basis for social and scientific events plus encouraged medium-term staff placements (exchanges) among participating organisations.

Projects that were instigated by one of the partners (usually the lead) generally worked better. In contrast, one example where the EC entirely coordinated a consortium and then passed the management to one individual (MAFCONS) demonstrates that this approach (and the creation of networks with little previous contact) can potentially impair the project itself.

• Another problematic project in terms of partnerships was FOREN, where

there were tensions between core partners as well as between the 22 participants.

Within some projects, specific strategies have been devised for managing the partnerships with active and engaged project leadership. Some examples include:

• Some ‘best practice’ for managing partnerships comes from CHIN, where the lead partners divided responsibilities between partners and ensured that each partner had a role in presenting the end city reports at high-level meetings. SCENES project leaders devised roles for differing partners – this approach to managing partnerships is perceived to only arise following experience of working within complex multi-country partnerships.

Regular meetings are of course crucial in establishing free flow of information between partners, which in turn can have an impact on the quality of a product or result, increasing the impact of it. A good working relationship between partners in CARE-W meant that project partners ended up doing more work than they were paid for. Regular meetings with all project staff facilitated the building of this relationship. The strong partnership did in turn facilitate the creation of an integrated technical solution, which if implemented will have potentially high economic impact. Similarly in GALA, regular meetings helped to create a sense of mission that was lacking at the outset – even to the extent of some scepticism on the part of some of the organisations involved. Several projects intensified the partnership through an ongoing exchange of and mobility by junior researchers or post-docs. Examples are:

• Research visits within the FUNCARS project (which had targeted training and mobility of researchers as a main objective).

• Scholarships for extended mobility of researchers at partner universities

(MAIZE) which created thematic and personal ties between partners. • Placement of Marie Curie fellows among partner organisations helped to

strengthen collaboration as well as improve the technical solutions in CARE-W.

These factors may be seen as necessary for and conducive to impact, but not a sufficient conditions for impact alone. Only if and when impact is understood and



62

defined as an integral part of the goals and objectives of a given project right from the start, do these factors become relevant.

5.1.3 At the exploitation stage

5.1.3.1 Dissemination Whilst the EC puts strong emphasis on dissemination (e.g. in their project reporting guidelines), it is only seen to be a critical impact driver for a handful of the sample projects. Several projects reported no specific dissemination plans or tools (beyond the standard presentations at conferences, scientific publications and a web-site). EU research is funded with taxpayers’ money, and should be visible – although it is not always a relevant factor for impact. Also, dissemination to the public or policy-makers was perceived by some of those interviewed to be irrelevant, too early or too specific, by some projects. However, dissemination is required for impact at the policy level, and research projects may need to be strongly encouraged to produce specific products for the use of policy-makers, and stakeholders. For two projects, FOREN and CHIN, dissemination of the end result (models) was a key enabling factor in their subsequent impact. In both these cases, it was thanks to the EC providing extra funding at the end of the project that they were able to realise the dissemination plan. For the ONTOWEB project, external visibility is low and this is an insufficiently addressed gap in public understanding. Whether or not dissemination it crucial to attainment of impact varies largely dependent on the type of project and the expected end result. Also exceptionally, projects such as GMO show that unexpected dissemination can be far more crucial to a projects impact than a conscious dissemination strategy. The GMO final report received one of the highest disseminations in media observed across the sample of projects thanks to unexpected “leakage” of the final report by EC officials to Greenpeace just before the official launch. This led to an unprecedented media coverage including a wide number of citations in journals and newspapers, which in turn impacted on the awareness of the public, perhaps more than would ever have been possible had it been planned. There is little evidence to suggest that limited dissemination impacted negatively on projects such as TTA in which dissemination was limited beyond the partner network, REOXCOAT in which dissemination beyond Nokia was limited; or on BO-BA in which exchange of information between selected interested parties was limited. Whether dissemination is deemed relevant largely depends on the type of project. For more fundamental research, dissemination of results in the scientific arena would be regarded as more relevant than dissemination to the general public. Finally, highly successfully and well-funded projects should also be aware of their responsibility for the less-privileged research community, those who are not able to achieve. Dissemination needs to begin early in a project, to allow outsiders to follow project progress and results – a web-page and a publication at the end of the project are not always sufficient in themselves.



63

5.1.3.2 Commercialisation and IPR In CARE-W, potential 3rd Group of impacts in the QoL area are entirely reliant on the scientists being able to commercialise the product and to make it relevant to the need of users and for users (water utilities and decision-makers in local municipalities) to see the rationale behind investing in the model versus existing solutions. Critical factors in this process will be to get the commercialisation vehicle for CARE-W to work, i.e. get partners to agree on copyrights and division of labour, and to implement a first demonstration project in a water utility or municipality to ensure wider application. The issue of IPR and copyright can, if not handled carefully reduce project impact. TTA and FOREN are examples of projects in which clashes about these issues have occurred, although they did not necessarily hamper the impact. Copyright and IPR issues have also been apparent in ENHSIN and SCENES. A significant proportion of the ENHSIN project addressed and defined standards for copyright, and the end impact of SCENES was possibly hampered by a lack of access to MS level data for the creation of the database. In projects where copyright-likely products are to be expected, a clear regulation should be agreed on beforehand, in order not to cause tensions during implementation. In addition, there were some perceptions from interviewees that the EC needs to take on a clear role of ensuring access to MS level data, this is particularly important for EU modelling projects.

5.2 OVERALL CRITICAL FACTORS

Overall, the critical factors have been analysed to develop the following chart, which displays the 6 top factors, according to the assessments:

Figure 7: Top Critical Factors for High Impact:

Top Critical factors - 1 to 6

0.000.501.001.502.002.503.003.504.004.50

People

Inno

vatio

n

Skill B

ases

Awaren

ess

Partn

ersh

ips

Need



64

As can be seen, the most critical factor in impact is ‘people’, and this relates to those undertaking the research.8 This is followed by innovation, existing skill bases, awareness of impact, the functioning of partnerships and the intensity of need. All these factors scored an average of 3, which means that they were rated as of ‘high’ importance to the impact. The 5 other significant factors following these, and scoring just below ‘high’ are the relevance of the project to the need, the recognition of the outcomes, dissemination, and the downstream effects. The following factors are seen to be critical to the attainment of impact across research activities and throughout the project cycle.

5.2.1 Recognition Recognition is similar to “knowledge and skills bases” - a necessary precondition for research. In impact terms however, recognition is a critical factor when we talk about project cohesion or leadership. Only widely recognised project leaders do have the authority that a) everybody wants to be in their project; b) will perform highly to continue to be in the project; and c) wants to be involved in the next project which is often prepared already. For example:

• In NESSIE the potential impact of becoming recognised by peers (as having had an algorithm recommended by the project), was the main basis for involvement of a large number of submitters (from academia and industry). In practical terms, it first led to a high number of submissions, and then drove the process throughout by motivating submitters to contribute voluntary time, which produced an end product of better technical quality than anticipated. Because of the ownership that many submitters took of this open but competitive process, NESSIE standards are more likely to be well understood and implemented across disciplines and sectors.

• The OWL standard developed by ONTOWEB for future development in the

ontology sector gained high recognition and credibility and is unlikely to be replaced.

• In GMO, the success of the project has translated into recognition of the JRC

among scientists and policy-makers, which in turn has helped them to get follow-up funding and greater exposure for new research. Similarly in SCENES and WETO 2030, recognition has come through the wide use of the research following the end of the project further enhancing the initial impact. In both ENHSIN and FOREN, recognition has lead directly to leveraging and multiplier effects.

5.2.2 Project theme, timing and level of innovation Similarly to the requirement for initial needs assessment, the level of innovation and the timing are critical factors in the achievement of high impact, and require ongoing monitoring. Impact is to a large degree contingent upon the initial choice of theme or subject for RTD. Broadly speaking, the more strategically selective and sensitive a project is regarding the theme and the timing of the subject of its (RTD)

8 Ratings have been quantified (Low-1, Mid-2, High-3 and Very High 4) and the average score per factor calculated.



65

work, and where project leaders are aware of and engaging with these issues, the more likely the project is to accomplish impact. In terms of thematic choices, conducive impact factors identified include:

• Focus on a theme that is about to make its breakthrough but is not yet perceived as having major economic implications. The GMO study is a prime example of this. At the same time, one must bare in mind that the study was so innovative that it may not have been funded had it gone through an open FP tender procedure since at the time of funding it was not among the FP priorities. The timing and opportunities for exploitation were also critical factors to the eventual social and economic impacts of REOXCOAT.

• Ability to ‘contextualise’ and to shape the theme by having a sound

understanding of the need and the (socio-economic) environment to which the scientific work is expected to respond.

• Departing from the trodden path and the mainstream, breaking fresh ground

(which, of course, also involves a higher risk). As in the examples of MAFCONS, NESSIE, TTA, BO-BA, MAIZE, SCENES, and GMO, the innovative nature of a model definitely inspires further contributions and increases impact, which is often endured by a range of parallel or follow-up projects. As outlined previously under recognition, projects with a highly innovative character can equally motivate stakeholders to stick to a project and be active in it.

5.2.3 Research Context and Attribution In comprehending the impact of RTD projects it is important to contextualise them within the research background. For some, (perhaps several) case studies, a linear series of projects means that they cannot be assessed in isolation, and attribution can be a challenge. For instance the eventual impact of ENHSIN is entirely dependent on both preceding and follow-on projects. These, potential impacts, are however, attributable to some degree to ENHSIN, because subsequent projects were developed, in part, due to the technical achievements and quality of the original. Technical, policy, knowledge and educations impacts can be attributed directly to EHNSIN. Other examples of a solid research foundation include:

• The impact of WETO2030 and GECS depended on an earlier progression of work (on the POLES model and related positions) by the same teams. However, GECS impacts are directly related to the use of the model it developed.

• CIEL built upon foundations within a sector with a long history of existing

work, however a linear attribution from CIEL to its impacts is perceived. • CARE-W is entirely dependent on commercialisation of the model for it to be

implemented by end users.

Other projects have significantly impacted on the research context:



66

• ONTOWEB was the trigger for a series of nationally and EU-funded research

programmes. Attribution is linear from the project outcomes (publications), and directly linked to the development of the OWL standard. This is also evident in the ongoing collaborations.

• FUNCARS had strong multiplier effects and resulted in a national research

call on nanotubes. Impact is directly applicable through the widespread dissemination of its 60 publications.

• TTA was followed by several application projects. • MAIZE continues with national funding in several partner countries and the

close interrelation between outcomes and impact of this project is almost entirely due to the design of the work.

As demonstrated above, innovation within the sector is a key critical factor, both in advancing the sector (multiplication and leverage), as well as to generate impact. Innovation among this sample has arisen both out of new approaches, techniques etc., as well as through new collaborations, resulting in European level outcomes:

• MAFCONS is currently applying untested theories (i.e. Huston’s Dynamic Equilibrium Model).

• SCENES applied existing technology at a highly innovative European level –

its impact is directly applicable to project outcomes. • CHIN used existing material and approaches, but applied these to a new

context. Attribution is linear both from the application of the recommendations in one city, and in the ongoing international collaborations.

• FOREN adapted existing methodologies, and introduced managerial tools

that were highly innovative. The project outcomes were a successful network and the guides themselves. Impact in terms of knowledge, enhanced capacity and impact on policy is therefore directly linked to the project outcomes.

For the set of projects, impacts were almost invariably related to the project in a linear or direct way, in addition to the examples above:

• The impact of NESSIE is a direct result of the robustness of its testing, the involvement of many different actors in the process and the openness of results.

• The impacts of REOXCOAT are attributable to the direct relationship to

project outcomes, through the perfection of the technical process. The major impacts were both tangible and measurable.

• The impacts of BOBA are due directly to project outcomes, i.e. continuing

collaborations, academic achievement, and skills improvement.



67

• GALA impacts are attributed directly through the continued development and

international importance of Galileo.

5.2.4 The role of the EC (as a facilitator) Within the set of projects under assessment, several specific examples of EC management practices driving impact were revealed. This is distinct from the use of policy oriented tools, which is discussed in Section 5.2.7. In FOREN for example, the initial push towards making the project focused on a useable end product came from the EC. Management by the EC was perceived to have been very ‘hands on’ and involved (for partners in REOXCOAT, MAIZE and WETO2030 this was also seen as a critical factor). Following the end of FOREN, the particular manager at the EC pushed for the follow-up project, producing the Country Specific Guides, and these are deemed by all to be very strong factors in achieving high impact. Many of the projects identified that European Commission funding was important in terms of being able to increase collaboration with partners and therefore foster expanded networks, such as in the case of CARE-W. Another example is MAIZE, where the INCO programme provided a forum for partners to co-operate across developed and developing countries. This concept of contributing to an expanded focus is also evident in GEC’s, and REOXCOAT, where it is clear that the scope of the project moved from a narrow perspective to a level of global interest, based on European Commission funding.

5.2.5 Flexibility Analysis across the case studies has revealed that flexibility has been an important factor, both in management style (tools) as well in responding to changing situations. The management style was considered by all interviewees an important factor to keep the project on course. Some coordinators took the opinion that successful projects are easy to coordinate and manage, but most thought that an effective and inclusive management style is required for project success. As part of the ratings exercise for critical factors, partners were asked to rate the importance of the use of logframes and other traditional management tools. Interestingly, across the board, these project management tools were deemed to not be of particular use, or relevance, to the success of RTD projects. At the same time, EU expectations of dividing a project into milestones and work packages seem to have permeated into the standard planning procedures of most scientists without major resistance, and are clearly instrumental in ensuring completion and success. However, flexibility can also be hampered by project management rules of the EC. Late information on success of a proposal and late payments can make it difficult for project partners to hire the best people for a project. Extensions of the duration seem to be a legitimate and meaningful answer to such delays. In addition, and for REOXCOAT in particular, eventual impact was very much linked to a flexible approach from management to the direction of the project. In fact, REOXCOAT did not meet all of its initial objectives, relating to specific industrial applications (application of the developed coatings process to other products). This



68

ability to alter the project, in light of opportunities (which would be marked as a negative result in traditional project management systems), is the most significant factor in the actual high impact achieved (1,000 jobs globally).

5.2.6 Unexpected factors A component of excellent and high impact research is also the appearance of unexpected results, or results that exceed expectations. Such over-fulfilment applies to results being of relevance in other scientific or economic areas, but also to new ideas and methods applied on a broader scale than foreseen in the project (e.g. MAIZE). As mentioned earlier, one example is GMO, where one of the main factors that let to its wide distribution and dissemination among the public, scientists and policy-makers was the fact that the final report was leaked to Greenpeace. This generated a media interest which the JRC could not have achieved even with the most ambitious dissemination strategy. As outlined above, REOXCOAT resulted in different outcomes than those expected, and this is very closely related to its ultimate high impact.

5.2.7 Factors which may reduce impact Examining factors which may reduce impact is important to contextualise the potential impacts examined above. Internal factors are presented first, followed by external issues, which are beyond the control of researchers and largely beyond the project cycle.

5.2.7.1 Internal factors The greatest perceived internal risk factor came from difficulties arising in the coordination of large numbers of partners, and there are examples across the project cycle. At initial stages this related to establishing partnerships (ONTOWEB for example faced administrative difficulties when trying to partner directly with US institutions); to their management in some cases, with communication identified as a challenge (MAFCONS, FOREN); and to IPR and copyright issues once the project were complete (TTA, FOREN, SCENES). In a situation where inactive partners do not participate, the fixed budget can be a hindrance, as it is not associated with performance. Although improved, the administrative and financial relationship between project partners and the Commission was seen as a factor which may have reduced impact for two projects (FUNCARS and SCENES). One common factor to projects dealing with large quantities of data came both from difficulties in different operation systems among potential users (CARE-W), and potential reductionism from the selection of some data over other (EHNSIN), and also a potential for misinformation on a larger scale, should poor data be provided and then distributed widely. In the assessment of several projects, limitations to the technical quality and achievement were highlighted as potentially reducing impact (FOREN and CHIN). This factor is of high significance for science oriented projects (CIEL and GMO), and is something that can be expected as a result of experimentation. Challenges included limited data (due to issues of access in SCENES) and a lack of field data (GMO).



69

Some interviewees reported a dilemma, and sometimes even tension, between academic excellence and broader relevance of a project. Whereas ‘excellence’ is seen as the standard currency determining the value/recognition of scientific work, relevance (which is closer to ‘impact’) often tends to be regarded as being more of an extrinsic factor. The dilemma is further compounded by the very stiff competition for research grants and the immense proliferation of research results which requires academics to focus their entire energy and resources on striving for excellence, often at the expense of taking equal account of relevance and impact. This dilemma is not simply a recurrence of the old dichotomy between basic and applied research. It reaches deeper and poses the question of the need for “Re-Thinking Science”9. Impact will perhaps only become a more prominent criterion of scientific achievement if the traditional (self-referential) mode of defining scientific quality via excellence is being challenged. This tension was played out in some instances where academic and other stakeholders had differing expectations and perspectives (e.g. FOREN). A needs assessment, not only involving stakeholders, but also project partners, may have overcome these issues of differing expectations between partners. Additionally, there was a perception that within research projects, a lack of awareness, or clear focus on long-term impacts, may undermine eventual results. For one project at least (MAIZE), too much emphasis on (too) short-term) results may have eroded higher impacts.

5.2.7.2 External factors External factors which may reduce eventual impacts relate largely to changing contexts and the use and relevance of project outcomes, which often are unpredictable factors at the project inception. The time-lag between the conception phase of RTD projects and their completion is a risk factor in itself. This is related to a number of factors. Projects aiming for impact within policy spheres may not be applicable or as useable as was predicted, due to policy change. One example is the POLES model developed by WETO 2030, which may not be entirely applicable to the policy outlook in 2030, (other projects for which this is relevant include MAFCONS, SCENES and MAIZE). In many instances, projects had developed frameworks or tools that are not considered to be ‘perfect’, but a starting point. For these projects (SCENES, FOREN, GMO, and CHIN), follow-up projects and smaller scale updates and applications were deemed critical to the achievement of eventual impact. The impacts of projects aiming to deliver products for use in the policy sphere are also highly dependent on use – normally by the EC. Stakeholders for SCENES, for example, perceived that the impact of the model would not be at its highest possible levels, due to the lack of extensive use (particularly at DG TREN).

9 To quote the title of a book by Helga Nowotny, Peter Scott and Michael Gibbons (London: Polity Press 2001), subtitled: “Knowledge and the Public in an Age of Uncertainty”.



70

MAFCONS is a special case because it is not due for completion until 2006. At a fairly early stage in its development, there are perceptions that the project is high risk. This is linked to the fact that for 3rd Group (i.e. long-term) impacts to be felt, it has to attain high technical and policy impacts.



71

C: CONCLUSIONS AND RECOMMENDATIONS

This final section sets out the conclusions derived from the study and recommendations for how the Commission could go about assessing, monitoring and planning Community supported research in the future in order to maximise attainment of higher level impacts. The first chapter focuses on the cases themselves, before making generalised conclusions about the overall samples and before dealing with future needs.

6 PROJECT SUMMARIES

The following section presents individual summaries for each of the 18 projects, looking at the type of impacts achieved and why. Full individual impact assessments, including detailed findings and causality analysis can be found in Appendix 8.

6.1 MAFCONS

Scientific knowledge of the impact of deep sea fishing is currently extremely limited and only descriptive. MAFCONS is a Quality of Life project, funded under FP5 and scheduled for completion in June 2006. It aims to quantify the impacts of fishing and to provide a modelling tool for managing fisheries (to conserve groundfish and benthic invertebrate species), through use in policy testing and in setting Total Allowable Catches (TAC). The linkage of ecological impact and effort statistics is a highly innovative approach to these issues. One push for the project came from the EC itself (as an Agenda 21 signatory) and its obligation to conserve the biological diversity of marine living resources and to restore biological diversity in situations where degradation can be demonstrated. MAFCONS is a collaborative effort being led by the University of Wales, and the consortium includes research institutes and universities from Norway, Scotland, the Netherlands, Germany and Belgium. The project is currently collecting and analysing data in its 2 year phase of fish trawl surveys. Prior to this, 2 international workshops were attended by all project partners, and the first phase involved the development of the theoretical framework. As this project is still 2 years from completion, it is a Potentially High Impact Research Activity (PHIRA), and it has been a challenge to assess its potential impact; however, some impacts have already been felt. Actual impacts to date have been on scientific knowledge, emanating from the data gathered from the bed of the North Sea; and on technical processes. The development of the model itself is innovative and is producing new techniques, and therefore it is raising the capacities of involved researchers. In addition, it is stimulating collaborations and all 6 consortium partners are academic organisations from 5 EU MS’s, therefore knowledge sharing between these partners is occurring. The main potential impact is on quality of life, with potential high environmental protection impacts across the EU, and perhaps even globally, although this requires extensive application of the tool by policy-making bodies – chiefly the EC. The



72

potential impact on policy and standards is also very high. In a best case scenario, the model will be used as a management tool by the EC, in regulation of fishing. In any case, the model will have an impact on the fishing sector, scientists and fishing researchers. Reduced fishing effort is likely to have high environmental impacts, by the avoidance of environmental damage. The scope of impact is potentially global, as no comparable modelling tools exist to date – it is likely that this could be adapted to many contexts. At this stage of MAFCONS, there is some concern among project partners that the project may not in reality achieve its full potential. The key drivers of actual and potential impacts include the coordination of the project – a complex task; and the innovative nature of the research. As discussed above, the model itself will be a critical factor for achievement. A key potential obstacle to the achievement of this impact includes the possibility that the model itself does not function to a high enough degree for use in policy testing. There is a possibility that the sampling used for the trawl fish surveys, will not capture an accurate representative sample of the diversity of the North Sea bed. In addition to this the project coordination itself has been a challenge. This partnership was not based on an already existing network, but many of the project partners were new. In addition, the coordination role was delegated from the EC. There is some perception that project partners are more involved in their own smaller part of the project (be it trawling, analysis or modelling), than in the overall project. MAFCONS has indeed provided the opportunity for extensive data collection that is of separate interest, apart from the model. The final potential obstacle is the political climate surrounding fishing, and the widespread belief that fishing effort statistics are frequently inaccurate and open to manipulation. For the eventual model to be effective in regulatory efforts, attitudes to fishing and reporting of effort, particularly within the industry would need to change. In summary, MAFCONS has a great potential to contribute to environmental protection and allow EC policy to adhere to the agreements of Agenda 21. It is co-financed by several universities and also by the EC. Without EC funding and instigation, the project would not be underway. An indicator of impact will be reduced fishing effort, although usage at the EC will be a key interim indicator of results. The momentum of the project will be continued in the longer term, regardless of its immediate results. MAFCONS partners are deeply committed to its goals, although the achievement of reduced fishing effort will require both enforcement and attitudinal change.

6.2 ONTOWEB

This Semantic Web thematic network, coordinated by DERI, Institute of Computer Science, University of Innsbruck (www.deri.org) has had high impact in raising awareness in the areas of:

• World Wide Web usability, • Machine-side content management and • the entire range of IT-based/facilitated communication (B2B, B2C, KM, IR).

http://www.deri.org/



73

“In the future, when you put a question to the (Semantic) Web, you will actually get not 100,000s, but 1 (one) answer. The system will be able to understand the question and give you the answer you need”. This quote from the ONTOWEB project depicts the simplicity of the driving idea behind the entire project - the visionary notion of a “next generation Web”, as formulated by the “Father of the WWW” Tim Berners-Lee. The goal of the ONTOWEB Network was to bring together researchers and potential stakeholders from both the ontology-relevant and application-relevant areas, and to promote/channel the harmonisation of efforts, thus strengthening the European influence on Semantic Web development world-wide. The simple idea to use “ontologies” - meta data that allows for machine processable classification and handling of “data about the data”- required the establishment of a broad research network. It has also contributed to the adoption of standards for a Web Ontology Language (OWL) by W3C (whose working groups act as standardization bodies in WWW). Moreover, a number of FP6 projects semantic web and web services, or information filtering, knowledge representation and decision support systems have spun off from this network. The critical impact factors of ONTOWEB was broad interaction among experts (more than 150 partners, see list in Annex). Innovation was further a genuine characteristic of the ONTOWEB project due to its timely focus on the future of the world-wide web. In addition, the role of dissemination was very high in terms of sectoral dissemination – mainly through the ONTOWEB public platform. A clear outline of contexts and schedules (6 project phases) for deliverables in accordance to the four FPs of the project, and strong and concise management, with clearly defined cooperation and penalty parameters provided the basis for a well functioning partnership. Indicators for impact are a series of publications, a structured set of project deliverables, usable as tools and founding stones for follow-up projects/research, implementation of aspects of the academic dimension of ONTOWEB research in governmental projects (Austria, Ireland) and the Basle II program. Awareness raising and consensus building were also impacts, made visible through the now global project outreach, as well as the business interest in project results. Recently (since February 2004), the standard developed by ONTOWEB was selected as the future standard for the Internet by W3C (www.w3c.org), the commission establishing rules in cyberspace. The practical importance of the ONTOWEB can be demonstrated by the following “specific use” examples:

1. Development of new type e-government, where the ONTOWEB technology will arrange the processes behind the gv.at domains of the Austrian government;

2. ONTOWEB technology will explain and also contain the process of preparation and application of SMEs for funding under the BASLE II agreement.

Real impacts are Europe-wide and global. They reach the whole community of IT development, the scientific community, and a selected number of fore-riders from



74

the business community (a few major IT firms, and a number of specialised SMEs). Potential impacts will follow once the methods are used by the public, which may need another 3-5 years research, development and application. ONTOWEB has exceeded expectations, and induced and established a self-sustainable research initiative, held together by a visionary idea of changing the World Wide Web. There is a broad perception of high impact within the ONTOWEB community. In conclusion, these impacts are dependent on leadership and personal excellence, guaranteed by the attraction of the best possible cooperation partners by personalizing the links to the top-quality experts and a timely theme. It has a strong potential to change employment patterns and statistics. Finally, the impact generated a very strong multiplier effect and was tangible, expected and positive.

6.3 NESSIE

NESSIE was an internationally competitive research exercise which aimed to gather large numbers of encryption submissions (various types of algorithms), from both academia and industry for thorough testing with the view to define a set of universal ISO standards. The work produced 17 recommendations for standards of which 12 were new and 5 existing. Overall, the most significant effect of the exercise was to increase confidence in existing standards, block ciphers in particular. The brand “NESSIE” has since the end of the project itself (in 2002) been used as a seal of approval in the world of cryptology.

The simplest and most striking aspect of NESSIE's achievement is that without Community funding the project would not have occurred, or at least not in the same form. The project has provided something of a breakthrough in the positioning of public sector research in the domain as a whole. During the last decade (i.e. 1992-98/99) cryptology research was basically driven by Governments’ national security needs and academics were not encouraged to conduct this type of research.

At the scientific level, NESSIE has produced major advances. Through deciphering all the stream cipher encryption systems submitted, it has exposed the fact that many algorithms are still highly unreliable and a great deal more research is needed in order to come up with better technical solution. It has initiated a race to produce new and more reliable solutions in the next 5-10 years. The mobile phone industry, which has a particular interest in developing reliable stream ciphers to allow safer transmission of data, for example over radio links, is actively engaged in a follow-on project. In general terms, NESSIE has become a benchmark, a point of reference in cryptology, which demonstrates the need for a Europe-wide policy to invest in cryptology research.

So far the project has generated some short-term actual economic impacts and there are several indications of longer-term effects to come. At the immediate level, an important by-product of NESSIE has been to prevent the use of insecure algorithms. In Belgium for example, the Government requested the University of Leuven (the NESSIE lead partner) to safety-test new ID-cards, which led to improvements which will help to prevent the recall of these ID-cards in the near



75

future. Over the longer term, the two main industries likely to benefit most from NESSIE are banking and telecoms; however there is a delay of 5-10 years before implementation of the technology is likely to be seen.

The very high technical and scientific impacts attained so far can be attributed to a number of factors, most importantly the high number of quality responses from the participants and the existing skills base of the people involved. The fact that it was a public exercise at international level led by a University rather than a Government, added an extra level of recognition and credibility to it which in turn contributed to the attainment of impacts. The contribution of the people involved in the project cannot be underestimated. While the project team were around 20-25 people including partners, between 80-100 people were actively engaged in supporting exercises (e.g. submitters engaged in improving their own submissions or cracking those of others). This has resulted in important developments at a structural level. NESSIE established connections between academics and industry experts around the globe and inspired the project leader to initiate a European network of excellence in cryptographic research (ECRYPT) involving some of the project submitters and partners.

6.4 TTA

Time-Triggered Architecture (TTA) was an ESPRIT project lead by Daimler-Benz AG, it involved 8 partners from academia and industry, and ended in 1998. The project aimed at the implementation of time-triggered computer architecture for fault-tolerant distributed real-time systems, and its effective deployment in safety-critical transportation systems. It was based on substantial previous research and lead to the first application by industry. Its key component was a communication controller executing the Time-Triggered Protocol (TTP). The chip's protocol firmware was developed by TTTech, a partner company. The need for time-triggered architecture was caused by a significant increase in the amount of electronics that have been introduced into the car, and this trend is expected to continue as car companies introduce further advances in safety, reliability and comfort. The introduction of advanced control systems combining multiple sensors, actuators and electronic control units are beginning to place demands on the communication technology that are not currently addressed by existing communication protocols. Within the next decade applications like brake-by-wire and steer-by-wire will be launched in production. These applications require high dependability and performance of the car's computer infrastructure. The long-term trend in the field is the move from mechanical devices, to electronic devices, e.g. in cars today 60-80 functions are executed electronically. Only a standardised “bus” system, as developed under “TTA”, allows for the coordination and combination on such functions. The development of such basic standard devices implies too high costs for individual firms, for a basic technology which by itself is not marketable, but provides a basis for standard (like plugging systems for electricity). TTA is therefore a typical case of meaningful financing of research which is not providing a comparative advantage to

http://www.daimlerbenz.com/

http://www.daimlerbenz.com/



76

a specific firm but to the whole sector. Moreover, this development has an impact beyond the automobile sector, and is extended to the railway and aerospace sectors. Geographically, this means a global impact. Crucial factors for success and impact were that partners knew each other personally from earlier cooperation, there was general enthusiasm "to be on board", and discipline in keeping dates, meetings, and the delivery of documents. The European lead in engineering of electronics was confirmed, as this chip is now used by the whole sector for the coordination of electronic steering devices, but due to the specialised subject during project implementation only little dissemination outside partner network, except for workshops and publication, took place. The impact of the product is highly visible through publications, specialised presentations in scientific and technical conferences (e.g. the yearly conference of German Industry with a keynote speech on the "bus-war"), through public discussion, workshops and roundtables at the SAE Detroit, via products like TTP, and finally via the turnover of SMEs. TTA also resulted in 2 (competing) standards of communication systems for advanced automotive control applications developed by 2 new SMEs: TTP and FlexRay. The first one stands for time-triggered protocol, and is developed by the network TTTech (www.tttech.com). The latter standard, was developed by the FlexRayT Consortium (http://www.flexray.com), which is an alliance of automotive and semiconductor manufacturers, working together to develop an advanced communication technology for high-speed control applications. FlexRay and TTP/C both use a TDMA bus access scheme, however, besides that, there are little common properties. Both standards are used and applied in industry, and not only in the automobile sector. “TTA” has triggered many follow-up projects and leveraged national funding for further development (e.g. DE, AT), it achieved significant human resource and expertise impacts, and also on SME creation. Two globally acting companies: DECOMSYS - Dependable Computer Systems, Hardware and Software Entwicklung GmbH (www.decomsys.com), and TTTech Computertechnik AG (www.tttech.com), both based in Vienna, are direct outcomes of the project. Several products derive from TTA, also in neighbouring transport sectors. As could be observed from this project, technology development needs to be subsidised by public funding at early stages, before the market can take over. Results are not always predictable, and funding thus needs to be distributed to a certain degree. The main result of TTA is the proof that it is possible to produce a chip which can steer a high number of different electronic devices – this was not evident before. The very chip convinced both researchers and industry of the functioning of a previously only academic research.

6.5 REOXCOAT

The impact of REOXCOAT, 8 years following the start of the project is the most demonstrable of all the projects examined. Actual impacts were high or very high in the following areas; technical, scientific, economic, social and international



77

cooperation. The most tangible impact was on the Finnish SME ‘Surfcoat’ itself, which has grown from a 2 person SME to a multi-national firm, employing over 1,000 people on 4 continents, and 3 EU plants (Finland, Hungary and Denmark). The project facilitated a better understanding of mass production coating processes and the interactions with other partners proved to be fruitful collaboration. Over the last 6 years it has become a global leader in condensed electromagnetic interference (EMI) shielding. Additional, potential impacts could be seen on both social and quality of life areas. Due to the inexorable growth of Surfcoat’s operations, impact (particularly social impact) is on a global scale. REOXCOAT certainly increased European competitiveness in telecommunications. The attainment of these impacts relies on the application of the technology to other products, and on the proof of a safety impact resulting from the application of the technology to mobile phones. This was the first application of economic reactive pulsed DC magnetron sputtering processes for depositing insulating oxides. The process was adapted to a large scale by using computer simulation. The project outcome was a functioning air-to-air coating line. REOXCOAT was initiated out of local need and an identified opportunity in the telecommunications industry. A small 6-month project at Surfcoat became the foundation for this BRITE/EURAM project funded under FP 4, it completed in 1998. It aimed to solve the technical limitations of the application of magnetron sputtering technology to large scale processes. The project connected SME’s with academic institutions and included 5 partners from 5 EU MS. The Lead Partner was Surfcoat, and all REOXCOAT partners have evidenced some degree of growth relating to increased understanding and skill in developing the coating processes. The project facilitated a better understanding, and the mass production of coating processes and stimulated a network which is still very much evident. In 2001 Surfcoat organised a coating seminar attended by all REOXCOAT partners, and a new position at the university in Mikkeli on coatings was established in 2004. The social impacts have largely been on the Mikkeli region of Finland, and SAVCOR (which Surfcoat is now a division of), has become a leading employer in an area of previous high unemployment. Potential quality of life impacts are very high but rely on the demonstration of increased safety aspects from the coating on mobile phone usage. The project led to European and global job creation, and has certainly increased European mobile phone competitiveness. Interestingly, the projects’ original technical objectives were not all met (the application of the coatings process to other materials, particularly marble), and the project leaders feel that they are only now starting to fully utilise its potential. The drive and determination of the project leaders must be seen as the key impact driver. Other drivers of these impacts were both internal to the project (flexibility in its implementation, and an understanding of the needs of the telecommunications industry); and external (timing and the particular need of Nokia were critical. According to project partners, flexibility in repositioning the project according to need was a key factor in impact. This meant not adhering to the pre-agreed workplan with the EC, and the flexibility and constructive approach of the project officer at the EC was appreciated by the team.



78

6.6 BO-BA

The Booster Batteries project was initiated and developed in close relationship with the European automotive industry. Its implementation proceeded in a context of strong market-orientation towards a predetermined set of envisaged deliverables, which were meant to be fit for a rapid and cost-effective industrial utilisation. The project research was built upon a commercial research infrastructure10. Project success was ultimately exposed to the fluctuations of the commercial sector and its supply and demand trends, and was not dependant on the outreach, depth and sustainability of project impact. The motivation for initiating BOBA was built upon predictions about the high demand for hybrid-powered vehicles in the short run and on the awareness about the competitiveness deficits of the automotive industry in this area and in the high-power battery research sector. The empirical rationale for the project was founded on extrapolation about the possible research outcome. Expectations for the program were very high and were based solely on market demand forecasts. While the excellent lead of the project, the precise estimation of the market potential of envisaged research outcome, and the constant pressure for cost-effective usability of outcomes ultimately led to the development of world-class, ready-for-production hybrid batteries, the market forecasts proved to be too optimistic and in need for a implementation timeframe adjustment. Due to the fact that production of ‘intelligent’ batteries is a multi-step process involving the development and synchronisation of products from several industrial sectors, BOBA nevertheless had palpable impact in few areas of direct project relevance. Initially this was the sector of battery management through IT, where BOBA registered with six pending patents. Through the standardisation of hybrid batteries development processes, BOBA had a strong scientific impact in the area of power cell research as well. The increased and sustained demand for skilled specialists in the framework of co-operations with educational institutions had actual impact on the profiling of educational fields of relevance for the high power cell research and development, and potential impact in the mid-run on the profiling of new type of jobs - new generation batteries maintenance experts. BOBA has without doubt helped raise awareness on the scope of usability of hybrid batteries and on their implementation in hybrid-powered vehicles in particular. The project will potentially have impact on management and market forecast policies in the automotive industry in Europe as its products are ready for implementation in the commercialised end-user automotive technologies.

10 The research activities of BOBA were based on the interplay of project partners within a specialized network of production units/suppliers from the industry sector. The network infrastructure was copied from the supply chains of common battery production processes, which remained relevant through the entire project duration.



79

6.7 SCENES

The SCENES project is a great example of high impact emanating from innovative European cooperation. It was a phase B project which built upon pervious projects (STREAMS and SCENARIOS). The main objective was to produce transport demand scenarios for the EU, and to develop detail forecasts of factors in this. The project was led by the UK ME&P in Cambridge and involved 17 other partners, including new Member State partners (Poland and Hungary).

The SCENES project has developed a transport model for the EU, which has been used both in policy analysis and in further research. In terms of policy analysis, it has contributed to the assessment of the impacts of pricing, on traffic volumes and modal split, as well as the assessment of the macroeconomic impacts of TEN infrastructure project packages for the revision of TEN guidelines. The model has been used to accessing the policies on pricing.

The project was instigated by the European Commission and would not have occurred without its funding. SCENES has been used extensively, both directly and indirectly, and the model is widely perceived to be relevant and valid. It can be said that the project has made a difference, by influencing thinking on a European level. In fact the impact has extended beyond EC institutions, as various MS and European institutions have adopted the model. One example is the use made of it by the UK Department for Transport.

The EU level scope of the project is highly innovative. The methodology included the creation of a European-wide transport forecasting model, and then the use of it to explore different specific scenarios. Perhaps the most important methodological outcome is the demonstration that a comprehensive transport model for Europe can be created. Within transport, and across the EU, it has potentially improved decision making processes, leading to substantial economic, societal and environmental impacts.

SCENES borrowed heavily from a previous project (STREAMS) and although the projects did not have the same team, the methodology used was the same. Some stakeholders perceive that improvements to the model could have been made if more time and resources had been available. There are some perceptions that the models’ impact will be increased, particularly if DG TREN extends its usage of it.

The consortium was formed by 18 partners from both Old and New Member States, who were organised into 3 thematic groups. Within the project, the engagement of smaller partners was actively facilitated, and the larger partners devised strategies for involving partners with less active performance in the project. This collaboration, of both more and less experienced partners, is likely to have raised capacities, both in modelling itself as well as in coordination. The partners continued collaborating well beyond the completion of the project.

The results of SCENES are being used in further research programmes, such as the IASON, EXPEDITE, SUMMA, TIPMAC and TREMOVE projects. These are also being used in the SPECTRUM project, which aims to develop a theoretically sound framework for defining combinations of economic instruments, regulatory and



80

physical measures in reaching the broad aims set by transport and other relevant policies.

6.8 GALA

GALA was a basis for Galileo, a satellite navigation system that has involved the European Commission and the European Space Agency (ESA) for several years. Galileo is meant to bring independence to Europe in terms of provision of networks – global as well as European. Such networks result in services related to navigation (global positioning) by providing location and time data to a whole range of user communities. Requirements for fast electronic communications are growing rapidly worldwide. Satellite navigation provides the basis for much of this communication with global ramifications in almost every sector of human activity. The potential market is therefore considerable and there are significant employment impacts as these develop. There is no all-embracing methodology. Global networking is dominated by the USA mainly originating for defence/military purposes. Europe wanted a global communication system for civilian purposes, and that was independent. GALA provided the architecture for Galileo both in terms of market orientation and technically for certain aspects such as systems design. It embraced the needs of many stakeholders such as the European Commission, European Space Agency, certain Member States, communications and space companies, and a whole variety of market sectors with transport, safety and security requirements. The objective of GALA was to formulate the bases of Galileo from various perspectives including users, design and technical aspects, service definition, legal issues, management and institutional aspects, and risks involved. All available information relevant to sectors of importance to Galileo was sought, surveyed and analysed by appropriate parties using a variety of techniques so that definition of the architecture for Galileo resulted that carried the support of the wide range of the stakeholder communities involved. There were over 50 sub-contracting companies involved in the project that were familiar and experienced in the relevant fields. One of the objectives was certainly in the area of policy/planning because at the outset Galileo was not fully decided. It had strong support from the Commission but the transport sector for example still needed to be persuaded. Initially the commitment of those involved varied somewhat. Some thought it was crucial to establish the need for, and implementation of, Galileo in the long term and believed in that, but others were less convinced. However of those that did not have high expectations initially many became converted to the merits of Galileo as a result of GALA involvement. Commission support for GALA made it not just European but global. Partnerships were complex and international but still initially dependent on the public sector. In this respect the support of a body like the Commission was vital. It was multi-disciplinary but created a sense of mission that had not been evident for Galileo across the board previously. A platform was created fully involving the end users and international cooperation became an integral element to the development of Galileo. By underpinning safety, security and employment for the individual as an eventual outcome from Galileo GALA was European in nature but its outcomes



81

resulted in a real international cooperative development of Galileo. Although GALA was a short-term project (18 months) it has had impact in the longer term to date in establishing Galileo. Before GALA, some had seen Galileo from a more technical and design perspective. In general that had perhaps been the ESA position. GALA positioning was much broader. It had some technical components but a much larger axis at the user level and with design of systems. The view was expressed that if GALA had not been funded then Galileo would have been stopped some years before. Although science-led through ESA involvement it had been essentially national for certain Member States. It was not market-led. That input was the real value of GALA. Although there were many sub-contractors, there were a limited number of people as partners and the programme initially was not fully structured or designed. There were many good robust meetings at frequent intervals. The market-sector dimension was particularly well-chosen and implemented; the technical area was complementary but not comprehensive in GALA. The project was indeed primarily market-pulled (though the second stage less so) and there had to be a trade-off between those interested primarily in the satellite navigation and those who might pay for a service or not. The overall outcome of GALA was that the key basis of Galileo was established and agreed by a wide community of those involved. This is still helping the Commission in its strategy and planning of Galileo, notably in terms of the high level definition required. ESA and the Commission became closer in their commitment to Galileo. GALA was innovative in that it brought together component areas (markets, technical, legal) that had previously been isolated. In summary GALA set the scene for Galileo, helped to justify it and created involvement across a range of involved sectors from provider to customer. It was crucial to policy decisions and still is.

6.9 GECS

The project provided a modelling tool that can be used to simulate the effects of Greenhouse Gas Emission Control Strategies. The original model was simply a development tool with no targeted applications. However around and after Kyoto, the model was extended to embrace gas emissions and other models were also made use of in preparation of the analyses of the economic assessments of climate change with options for gaseous emission reduction, cost implications of abatement strategies, and impacts on land use. The initial model was limited to a 10 year prospect but in GECS and other applications it was extended to 30 years. The main outcome was prediction of greenhouse gas emissions relevant to Kyoto mechanisms such as COPs (Conferences of the Parties) as established by international protocol. GECS outputs have been used, for example, in supporting discussions in the Council of Ministers relating to climate change and by Member State Ministries and by DG Environment in relation to European positions on emission control in the COPs (especially COP6 – The Hague) and in policy planning regarding Emission



82

Trading Directives. The project provided European decision-makers and negotiators with analytical and quantified information on the sectoral and economy-wide impacts of alternative schemes of emission entitlements, flexibility systems and policy instruments. It helped to define a European strategy in international negotiations and facilitated an approach to prediction of climate change impacts (especially in relation to emissions from a European origination) that underpins international negotiating positions. The nature of the impact was to extend the usefulness of computer models (one in particular) to a longer timeframe prediction and to embrace gaseous emissions impacts thereby providing a valuable tool for policy planners considering the implications of climate change on Man and his activities. This proved to be particularly relevant to the Kyoto mechanisms established by protocol for international action in this area and other policy requirements. For example DG Research had a definite need for a multi-gas emission model and DG Environment officials came at their own request to project management meetings in order to follow its development. So there was a very good match of needs with GECS even though this was not initially a strong feature in the conception of the project. The outcomes from the project have been taken up by core interests and had an influence on thinking not only at a policy level in Europe but beyond so that further efforts have been requested to push the approach further into timeframes up to 2050 through new projects. Presentation of the work has also been requested by the prestigious Stanford University Forum relating to environmental modelling. The project has its basis in the European positioning of climate change interpretation and would not have been funded as a project of global interest without European funding. But now there has also been a diversification of user in that both Member State governments and DG Environment interest has resulted in further work. The project coordinator is very keen on small well-balanced teams and the nature of the partnership maximised the impact achieved as the team was small yet complementary in experience and skills, amenable to close management and therefore highly effective.

6.10 CARE-W

Aging public water supply networks in Europe are increasingly vulnerable to structural failures, which affect both water quality and reliability and generate huge investment needs which few water utilities or municipalities are able to cope with under normal budgetary constraints. In addition, existing technologies and applications seldom generate enough information to allow decision-makers to make an easy, strategic decision about what pipes should be changed in what order to ensure that the most vulnerable areas of the systems are protected against disruptions. Rehabilitation decisions are therefore often reactive, e.g. based on recent failures or repeated failures, rather than proactive or forward-looking. The aim of CARE-W was to produce an integrated computer-based system, incorporating instruments already in use to estimate the current and future conditions of water networks as well as to develop new and better routines for



83

estimating long-term investment needs. By contributing to a more efficient reverse of deteriorating water networks and by improving reliability of water supply to vulnerable areas or users (e.g. hospitals, fire-brigades or other consumers relying on a consistent water supply), CARE-W would have the highest potential impact in the area of quality of life of European citizens. In addition, by enabling easy selection and ranking of rehabilitation projects, decision-makers would be able to make wiser investment decisions resulting in substantial cost-savings. According to estimates, the potential economic impact of a widely-used CARE-W could involve cost-savings of up to €1 billion a year in Europe alone. CARE-W originated from an EC supported COST action (i.e. C3 on wastewater) in which all 11 consortium partners participated (6 based in old and 1 in new Member States). One of the most significant actual impacts of the project to date has been the high-quality co-operation it has generated among partners, bringing together know-how and establishing strong working relationships. Ingredients that particularly contributed to the exchange of ideas and high level of cooperation included medium-term staff exchanges, facilitated by Marie Curie grants, and regular team-building events (twice yearly) involving all project staff. The resulting enthusiasm and excellent relationships among partners also meant that partners dedicated more resources to the project than the officially designated number, which in turned improved quality and testing. More broadly, via workshops open to consortium as well as external scientists, new and advanced mathematical models and performance indicators were developed and vetted. Seventy scientific articles and conferencing papers have so far been published on the project which has significantly raised its profile within the scientific community and beyond. It has also spurred scientists in other fields (gas, electricity, industrial productivity and even road networks) to capitalise on the methodology and knowledge developed by CARE-W. Without European funding, the project would have been far less ambitious and probably more locally focused, loosing the value added of an international approach. At the same time, because of the diversity of water networks in Europe at the present time, including the tools and structures used to manage these, far more resources are needed in order for CARE-W to suit the needs of local and national users. Following the end of the project earlier this year (January 2004), the project partners have yet to improve the usability of the product and build the necessary support infrastructure to enable commercialisation. A demonstration project somewhere in Europe would greatly improve the chances of selling the product to users, providing them with real life data on the gains of using CARE-W. Successful commercialisation and implementation of CARE-W more widely are crucial pre-requisites for attaining future higher level impacts (economic, social and QoL). An important lesson to be learned from CARE-W is the importance of an initial needs assessment involving potential users as well as continuous interaction with users throughout the project (where possible) in order to ensure that any project end-product is tailored to suit the needs of its future users. Having been essentially science-driven, it is possible that the current CARE-W solution is too “sophisticated” to suit the needs and capacities of potential users, at least in the short-term, and needs to be adjusted further to suit the needs of each individual client.



84

6.11 WETO 2030

This project was initiated as a response to a suggestion from DG Research to facilitate policy generation and future resource planning of R&D budget allocations in the energy sector. There was a requirement for a European predictive energy model rather than relying on USA or international models to assist in planning future research requirements. In summary the requirement that the project addressed was a projection of energy needs, usage and sources that showed where research effort should be made up to 2030. The project developed from earlier work funded from 1990 through the Framework Programme and elsewhere in which the POLES model was originated. The model is essentially a computer simulation model into which data from many international sources is inputted to provide a year by year energy supply and demand projection by region, through trading and therefore pricing, future technologies and other inputs. As the model has been developed more and more sectors have been added and further detail incorporated to capture the adjustments necessary. The model had its origins in IEPE in Grenoble but the contractor, Enerdata in nearby Gieres, is a spin-off company from this which specialises in accumulating/accessing huge databases from many sources. For WETO 2030 the POLES-based simulation was extended to cover a 30 year projection timeframe and take account of new potential variables such as climate change. It was conceived beyond the European level – on an international basis. The outcome of the project provided a European predictive analysis of the world energy scenario identifying areas for research effort and assisting future R&D planning by the European Commission. It demonstrated the need for even longer projection periods and a further project has now been started. There was also a multiplier effect in that those involved were invited to undertake other studies both in Member States and at a European level. International interest was also created. Because the originator of the research proposal was highly focussed on the dissemination of its eventual outputs these were widely publicised in the media by press conference, involvement of Commissioners, question and answer sessions and involvement of high level policy and planning officials. The Commission, the coordinator and partners saw this dissemination as a key feature of the project from the outset if the impact was to be maximised so the work was approached with this in mind. There have been numerous press citations and other examples of media coverage prompted by the press releases and conferences at a high level. One new product was a book setting out the study and its findings. The real value of the impact lies in the long-term nature of the predictions and the implementation of the work in Europe. WETO 2030 would definitely not have been undertaken in the absence of funding from the European Commission and the US Dept of Energy and IEA would probably have remained the principal sources for this type of information. In summary, the Commission had specific needs that the work addressed. Awareness of the outcomes was intrinsic to implementation and this ensured that



85

impact was maximal. The Commission had the need and initiated the work – the match was therefore very good from the outset

6.12 CIEL

Euratom and other individual nations internationally are keen to support safe nuclear fusion research to provide future energy needs. Nuclear fusion research has been supported by Euratom for a long time, with the eventual aim of constructing fusion power plants. Tokamak fusion devices use a circular torus structure to confine the ionised gas (plasma) for study at very high temperature. One such device, Tore Supra, using magnetic confinement is located at Cadarache in Provence. The CIEL project is a key upgrade of Tore Supra to achieve longer timescales than other tokamaks and so comes closer to the steady state conditions necessary in a generating plant. To achieve this longer duration – significantly longer than what is possible in the earlier JET Euratom project involved developing novel features for Tore Supra to achieve a significant step forward – was a valuable input for ITER, and the result of a highly significant international initiative. In addition certain industries are willing to be involved in prototype engineering with an eventual view to involvement in nuclear fusion generating capacity and other scientific spin-offs for the sophisticated engineering involved. This also occurred in CIEL and was crucial to its success. CIEL is characterised by highly specialised expensive and scientifically complex studies combining state of the art physics research with engineering often in highly innovative ways to achieve the ultimate aim of abundant energy production through nuclear fusion rather than fission with the significant theoretical benefits that can bring. The scientific and financial investment is large but the potential benefits are immense. Tore Supra demonstrated that electro-magnetic forces can be used to control the interfacing of the plasma with container walls. This means that theoretically the plasma can be maintained indefinitely. However, heating must be controlled increasingly as longer pulse times are achieved. This is the second vital step – to achieve cooling primarily through water in order to maintain longer durations for plasma/container interactions studies. The basis of the approach was to separate the very hot plasma from the cooling system using a skin. CIEL extended this approach dramatically through the use of innovative carbon fibre metal bonding to provide efficient heat conductance thereby achieving pulse times of around 6 minutes rather than the 2 min periods previously achieved. Heat transfer systems therefore have proved effective in maintaining fusion pulses for time intervals hitherto unachievable as a contribution to studying plasma interface conditions as a pre-condition to a steady state generation position. CIEL’s solutions were bold in concept and required advances in the physics involved and novel engineering approaches using carbon fibre metal bondings in a castellation to achieve effective heat transfer and maintain the fusion reaction for longer periods – the limit is not known. The approach was fundamental to further understanding of the requirements of fusion generation of energy under steady state conditions. Tore Supra could potentially give long pulses using superconducting



86

magnets but resulting heating effects had to be removed to improve steady state conditions. Without this scientists did not know the limits of the electromagnetic approach. The project was very bold conceptually in that the proposition that heat conductance could be achieved in this way broke conventional tradition. A significant step forward in fusion research resulted. In this respect it is worth noting that fusion energy generation is fraught with difficulties - indeed there have been many setbacks. CIEL represents a significant move forward, because carbon fibre use was the key to the longer pulse times achieved. Expectation of success outside the team does not seem to have been high and the approach was therefore seen to be one of high risk. In essence it seemed to be a step too far – though initially Brussels made the approach to CEA to instigate the project. The outcome has exceeded expectations and in some senses has changed the scenarios involved. CIEL has made a difference to the study of plasma-container wall interfaces in a way that allows study of conditions that come closer to those required for steady state generation in power plants. It is a significant advance also of relevance to ITER. CIEL provided a solution to a specific fundamental difficulty by providing heat removal and dealing with particle losses in fusion reactions, so that pulse time is much increased, maximising understanding of the situations that need to be achieved in steady state power generation. Achieving this impact required a mix of scientific and technological competences that are available at the contractor site in Cadarache. It also required industrial input through specific manufacturing requirements of key components (through a specialist company in Austria) involving carbon fibre-metal castellated bonding. This was not a linear technology transfer relationship but a fertile partnership with industrially-based R&D capability. Some of the engineering capability provided is very significant and potentially is capable of application elsewhere. This was essentially a Euratom project that would not be funded in this way from other sources. As such it is part of a carefully constructed European progression in nuclear fusion research supported in this way for a long time. In addition it is not just a matter of money – it was in essence and culturally a Euratom project (even though implemented at one site by one contractor) and would not be thought of otherwise - although perhaps there was a desire to make a contribution that was essentially from France even though within the Euratom ‘club’. Euratom is itself an early European Research Area with many of the scientists involved in European institutions being very mobile and known to, and working with, each other. Those involved are highly professional and competent people. This is essentially a scientific advance – the outcome of demonstrating that effective heat removal through the CIEL approach achieved longer pulse times for study has had a big impact on the science of fusion generation of energy. The effect of CIEL is now in place and understood so the fusion community of researchers is aware of the advance. Further developments may well need further Euratom nurture as has been the case for many years. However the emerging ITER initiative in fusion research is international (for example China has now joined in) and future developments may come from international funding, especially if ITER is located at Cadarache – a possibility that can only be enhanced by CIEL.



87

The socio-economic effects of CIEL if achieved would be far downstream but as with many advances at the forefront of science the implications for broad and deep potential impact are huge. CIEL might be a key step in the ultimate provision of safe abundant energy production for the benefit of mankind – but at this stage we just do not know about such downstream impacts There have been other impacts from this work. For example, invited scientific papers and reporting of the work to international committees; its high recognition as a significant achievement by peer groups; an effect on the physics involved at purely scientific levels opening new areas of experimentation and pushing forward ideas in basic physics; recognition of an innovative yet risky or bold approach whilst built on prior work and an understanding of its limitations. An awareness of this project leads one to the view that it is one of considerable substance with a large tangible body of evidence. The contribution itself is of genuine scientific significance and at this stage that is its real impact. However, its potential is enormous if it provides a genuine stepping-stone to fusion generation of energy.

6.13 MAIZE

The MAIZE project is an excellent illustration of a meaningful, impact-driven and mutually beneficial scientific cooperation, between research groups from within the EU and with third countries. It comprised a total of ten partners from three different continents (Europe, Latin America and Africa), who pooled their knowledge and expertise to address one of the most pressing and troubling problems in the world: hunger, malnutrition and starvation in developing countries caused by unfavourable environmental conditions (in this case, by aluminium-toxic acid soils in large parts of Sub-Saharan Africa and Latin America). By combining physiological, bio-molecular and agronomic methods and approaches, each of the partners was able to contribute an important element to the project’s main objective -- to identify innovative and effective strategies for increasing the productivity of small farmers through resource-friendly, sustainable and economic cultivation of maize. While it is too early yet for larger-scale economic exploitation of the research and development results of this project, there is a realistic expectation that combined genetic and agronomic methods offer great potential in terms of tackling the problem of acid soils for maize-based cropping systems. Apart from its strictly scientific outcomes (e.g., publications, new plant breeding and screening techniques), the project has furthered the understanding of how to adopt advanced biotechnological methods sensibly to the specific conditions in developing countries. By doing so, it has fostered the advancement of human and institutional capacity and paved the way for the adoption of local solutions from within the target regions. MAIZE has been a model project of North-South knowledge transfer, and South-South cooperation. The present and future (social, economic, policy) impacts of the



88

project are closely associated with its role as an effective platform of knowledge generation and dissemination. It is also an example how a strong commitment to impact of all project partners (promoting self-supporting maize production of small farmers in developing countries) can engender alternative scientific approaches. The European Commission played a proactive role in bringing together the project partners by suggesting to merge the proposals of several teams which had independently sought for funding, all dealing with maize cropping systems. The INCO programme provided a unique opportunity for synergetic R&D between partners from developed and developing countries for which at the time there was no equivalent neither on the national nor the international level.

6.14 CHIN

The CHIN project involved cooperation between several European academic institutions and Chinese academic institutions in developing understanding of the inequities of China’s urban health care services. The research project developed recommendations for health authorities based on a widespread survey of access to healthcare in two Chinese cities. The recommendations were taken up by the Nantong authorities. The most direct impacts of CHIN included those on the capacities of researchers. European researchers (from Sweden, Germany and the UK) have increased their knowledge of Chinese systems, and their understanding of health care in transition and middle/low income countries. More pronouncedly, Chinese researchers have developed skills in research aiming at evidence-based policy-making. Policy makers at the local city level have been encouraged to seek systematic ways of assessing need, and collaborations between the participating health authorities and Chinese researchers continues. CHIN ensured that Chinese researchers who were stilled in producing statistics, developed analytical and research skills. In this regard, the project had an impact on human resources in China. In addition, there has been cross-fertilisation of ideas and the impact has been multiplied via the involvement of Indian researchers in study tours and exchanges, which were organised as part of the project. Policy impacts are also evident. The impact of CHIN on policy formulation in the relevant cities, and at a regional Chinese level is evidenced in the take up of the recommendations. The strengthened research and policy link between Shanghai and Fudan University is ongoing, and the Nantong Municipal Government has contacted CHIN’s researchers (Chinese) to provide ongoing technical support. European project leaders have continued to work in these areas, applying their skills to Vietnamese contexts, and the study achieved high recognition within the sector. The policy impacts are also likely to be sustained, as the Chinese Ministry of Health has demonstrated its move to develop systematic methods of assessing needs and redeveloping policy. Additional potential impacts include that on European knowledge and expertise in health care research in transition countries (possibly



89

applicable to accession countries in the future), as well as the impact on European-Chinese relations and collaborations at a high level. EC funding was critical to the impact, and even if the project had occurred without it, the scale would have been vastly reduced. A key impact driver was the identification of need – by the Ministry of Health- ensuring that the project received high level attention by the relevant authorities. In addition, the pre-existing relationships between Chinese participants and stakeholders are perceived to have strengthened the implementation of the project, as well as the uptake of its recommendations. The partnerships were built on some previous collaborations, and in addition, each partner had a specific area of responsibility.

6.15 FUNCARS

FUNCARS was a Research Training Network on Chemical Functionalisation of Carbon Nanotubes, coordinated by the Institute for Organic Chemistry, University of Erlangen and involved 9 Partners from academia. The FUNCARS project was initiated at an early stage of carbon nanotubes (CNTs) research. The idea behind the project was to facilitate expert communication and to foster an interdisciplinary approach in dealing with CNTs through training of young researchers, through regular meetings and through channelling research outcome into individual and, most of all, joint publications. The scientific programme of the project included the synthesis and standardisation of carbon nanotubes, functionalisation and solubilisation, their characterisation and investigation of chemical, electronic, magnetic and optical properties to the search for future technical applications of these new nanomaterials. Selected European academic institutions, active in the field of CNT production and functionalisation, laid the fundaments of the future FUNCARS network. The key success of FUNCARS was novel interdisciplinary research (Chemistry, Physics, Material Science, Medicine, Solid State) in the field of functionalisation and processability of CNT. The training component was institutionalised in the young researchers network (YRN), i.e. through training-through-research by financing transnational appointments of young researchers, regular participations in project meetings, publications, switch of expertise to the industry via employment. There is a clearly identifiable trend towards involvement of industrial participation in the research of CNT, which is attributable to the publications and regular open meetings of FUNCARS. Current industrial stakeholders are BMW, Sony, Siemens, Infineon, Future Carbon. FUNCARS main outcomes were the creation of 3 follow-up CNT research networks, publications and presentations, which thus do not have a direct, immediate and measurable impact on the subject matter of research, but rather on the methodology and framework of research. In terms of direct impact, FUNCARS is perhaps more a PHIRA (Potential High Impact Research Activity) than a real HIRA - it is an excellent interdisciplinary scientific project in traditional basic research and possesses in certain aspects and in terms of CNTs as a novel research field, a high potential for further development.



90

FUNCARS has no impact on society or policy in Europe so far and it will as a genuinely research and training network for involved scientists most probably not have direct impact in any of these areas. In terms of indirect impact, through established patterns of research and networks for collaboration, FUNCARS can be marked as a HIRA. Its impact on the partnerships in the CNT research sector was crucial, and it triggered a new wave of follow-up interdisciplinary research of CNTs. The scientific publications (around 110) have maximised the impact of FUNCARS along with the regular meetings of project partners, which were also initiated after FUNCARS ended. Through screening the authorship and topics of the publications, it is easy to recognise that around 60 of them are based on interdisciplinary research. The scientific impact of the FUNCARS is therefore a tangible one, just as expected in the kick-off stage of project development.

6.16 ENHSIN

The Human Potential ENHSIN project has to be primarily considered in terms of its leveraging impact. The project aimed to facilitate access to specimen data by developing a demonstrator interoperable database for European institutions. There were two key pushes for the project, the first arose from the increased capacity of interoperable databases, and the requirement for speedy access to this sort of data. The other push was the 1992 Rio Summit where the repatriation of biologic information became a requirement. The project exceeded the expectations that its implementers had, and has been closely followed by BioCISE, a collection and specimen level database. Considerable leveraging has been demonstrating by a growth in funding from £1 Million (ENHSIN) to £2.3 million got BioCISE and £13 Million for Synthesis (another follow on project to ENHSIN). Similar leveraging has been achieved for international cooperation, EHNSIN had 7 European partners, and BioCISE is bringing 30 together. Furthermore the standards developed by ENHSIN have been used by the Global Biodiversity Information Forum (GBIF). In fact ENHSIN has to be seen as part of a tightly related chain of projects (including BioCISE, CITAF - Consortium of European Taxonomic Facilities - and Synthesis). Among the European community of taxonomists, there are perceptions that the project has helped to build a community between them, as well as stimulate interest in the field. The most evident actual and immediate impacts have been on the development of scientific knowledge and infrastructure. Other impacts include those on the policy sphere – this is evidenced by the sustained funding to the Synthesis project by the EC. In addition, the work undertaken on IRP issues, related to a widespread access to data was innovative and influential. Wider impacts beyond the academic and taxonomic sphere include the potential environmental and health (quality of life) impacts from the usage and repatriation of specimen data. The provision of information is a key facilitating factor for the



91

achievement of these potential impacts. Many of these longer term and higher level impacts will most likely be attributed more closely to subsequent projects. The partnerships strengthened by ENHSIN were based on an already existing network of European Taxonomists (CETAF); this combined with committed leadership was a big part of the success ENHSIN. The actual database was developed following a wide ranging survey of potential end-user needs, and this must be considered as the key foundation for eventual high impact.

6.17 FOREN

The Human Potential project FOREN had has surpassed every expectation and attained high impact in several different areas. In line with its objective of promoting effective integration of foresight processes into regional development and strategic planning, it has stimulated interest in, and awareness of, foresight activities across and beyond Europe. It has also stimulated additional foresight activity, and had a highly visible impact on policy-making circles. The subsequent development and application of regional foresight tools means that FOREN still may have high impact in the economic and social areas. The project brought experts and policy makers together in 6 simulation workshops from the technology foresights and the regional development policy community. The results of the project were published in a guide publication, and a series of follow-up projects were undertaken. A project that immediately followed involved the project leader in adapting the general FOREN guide into a series of country-specific guides. Both the original guide and these subsequent specific guides are very widely cited. The guides have been used extensively within the EU, in regional development projects. Another key follow on project is the EC’s Blueprints for Foresight Actions in the Regions expert group. The main impact of FOREN relates to the increased knowledge about regional development and enhanced capacities, both of participants in the network and users of the guides on a global scope. The project had also a high impact in the policy field both at the EC and Member States level. An explicit reference to the project was included in EC Com (2001) 549. There is also evidence that use of regional foresight will boost job creation; this is claimed by some regional authorities, including Manchester in the UK. The project also has significant potential high impacts in the economic, quality of life and social areas. The project has developed a framework for an appropriate use of measures and methods for regional foresight that will save expenses to governmental authorities. There is also evidence that the use of regional foresight may boost job creation. It has also had a particularly visible impact in Eastern Europe accession countries (which are more regionally structured - unlike other MS – and also in the Netherlands). Finally, the increased capability and organisational level capacity to facilitate innovation is likely to make some contribution to increasing quality of life.



92

The extent of the impact was therefore unexpected and is perceived to be ongoing. The only identified negative impact was some conflict between partners in the project. This is perceived to be related to ownership issues and is probably a product of the high impact. In general the perceptions of FOREN are that the guides are filling a gap that was missing before. Foresight was moving up the agenda for 4-5 years but at the regional level very little had been done. Perceptions among researchers are that the guide is very practical and not a method, ‘it’s more of a recipe that can be adapted than a toolbox’. Other perceptions are that high impact was closely related to the management style of the project officer at the EC, which was seen as entrepreneurial and using ‘accompanying measures’ as a window of opportunity for follow up activities.

6.18 GMO

GMO was a one-year study conducted by the Joint Research Centre and financed through JRC core FP funding. Commissioned by DG Agriculture, the study was initiated as a pan-JRC operation with external expertise channelled, mainly, via the European Science and Technology Observatory (ESTO). Using sophisticated landscape modelling, the study developed an innovative methodological framework to ascertain adventitious presence of GM crops in non-GM crops; identified farming practices that could reduce this presence, and estimated the cost of developing adequate monitoring systems to ensure targeted levels.

The main novelty of the study was that it showed how a pan-European and interdisciplinary study aiming at giving recommendations to policy could develop into a widely used methodology to avoid cross-contamination in agriculture and promoting high social impact in the public domain. The impact of the study since the official launch of the final report in 2002 has been significant in several areas.

The exceptional timing of the study and the scope – being the first real substantial study to look at potential impacts at both farming and policy level with respect to coexistence - significantly contributed to it being seen as a milestone within the policy, scientific, and farming communities.

The study has been extremely successful in placing the issue of GMOs firmly on the political agenda. At EC level, the GMO study findings formed the basis for the advance of EU-wide guidelines on the development of strategies and best practices to ensure the coexistence of genetically modified crops with conventional and organic farming. At MS level, e.g. in Spain and Denmark, it has stimulated the development of locally tailored guidelines and regulations in line with the study recommendations. A key factor contributing to the policy impact was the support the study received from top level officials in DG Agriculture, which in turn contributed to it playing such an important role in the development of the European Guidelines.

The study has placed GMOs high on the scientific agenda stimulating activity in the area of landscape modelling and helping to develop a “science of coexistence” across Europe. JRC knowledge and experience in the GMO area accumulated



93

through the study has further resulted in the creation of a European network of GMO laboratories coordinated by the JRC.

In the agricultural and seed sector, there are strong indications of potential long-term economic impact. In the event of cross-contamination, the tools developed by the study to minimize admixture in a co-existence environment could help to protect non-GMO farmers against economic losses, e.g. loss of income due to difficulties in selling mixed crops.

Through widespread dissemination of results, the study has already had significant social impact, for example raising public awareness on the issue of genetic modification, and educating the general public, farmer unions, and particular interest groups (e.g. biosocial groups) on the subject of coexistence. This was mainly obtained thanks to widespread European and International media coverage. The key contributor to this however, was external to the project: a “leakage” of the final report to Greenpeace before its official launch generated massive media coverage in Europe and beyond as it was placed on the Greenpeace website.

Although the project itself is over and has been for some time, its legacy is likely to live on well into the future through the many follow-up projects it has stimulated across the globe.



94

7 CONCLUSIONS

The following conclusions are gathered under four main headings, starting with the implications for study and measurement of high impact; evidence of high impact found in the sample of projects under study; the findings at aggregate Framework Programme level; and finally the critical factors that facilitate or hamper attainment of impact.

7.1 IDENTIFYING AND MEASURING HIGH IMPACT

• There is need for a ‘holistic’ approach in trying to understand high impact across sectors and types of projects. Although it is understandable that the EC in funding research is looking for quantifiable impact measurements, it is almost impossible to neatly separate the qualitative and the quantitative dimensions of ‘impact’, particularly in the short- to medium term. Trying to measure impact solely in quantitative terms would be a reductionist approach and deprive the discussion and analysis of some of its most valuable pieces. Additionally, to allow for comparative assessments across areas, qualitative and comparable frameworks are required. As this study conveys, impact and high impact in particular (as opposed to ‘outcome’/ ‘results’ or ‘successes’) are not all limited to:

o The number of new jobs created; o The contribution to the increased turn-over of companies; o The number of patents or licences, or similar benchmarks, at least not

if the aim is to look at impact across sectors. • High Impact, can also be strictly qualitative, and should therefore be measured

accordingly using qualitative impact indicators. This way, an impact such as ‘Making a difference” would be about:

o Changing perceptions concerning innovative, unusual, potentially path-breaking approaches to RTD (new paradigms);

o Creating new synergies across the different stakeholder domains; o Opening up the prospect to alternative solutions to societal needs.

• The study shows that it is possible to use a qualitative approach backed by

quantitative and qualitative evidence to measure progress against attainment of higher level objectives and to identify far reaching impacts (see 4.5 above).

• In addition to High Impact Research Activities (HIRA), it is important to

consider a further and equally legitimate type of project - Potential High Impact Research Activities (PHIRA) that seek to develop, or intrinsically embrace, a potential to achieve high impact.

• Qualitative impact indicators for HIRA as well as PHIRA are more

comparable when assessing project impacts horizontally, between different sectors or areas (for example between QoL or Human Potential projects, or between SME development or transport modelling projects), without running the risk of ending up measuring “successes”.



95

• There are intrinsic differences between the types of impacts different projects might generate. While more fundamental advances resulting in profound and far-reaching impact tend to be infrastructural (e.g. elucidation of the double helix and genetic code; the jet engine; the first working computer; radar), super-structural projects tend to be more applied (e.g. elaboration and refinement of existing knowledge for a specific purpose such as growing food crops in arid areas or developing new technological approaches to provision and monitoring of clean water).

• It is important to distinguish between applied (super-structural), and basic or

fundamental (infrastructural) research since the potential scale of impact for these can be very different. The latter is likely to take more time to show tangible results but when it does it may not only produce a new product but create whole new markets. Although few projects in this assessment could potentially “revolutionise” society, i.e. having an impact on many sectors, creating new markets and behaviours, (e.g. CIEL and possible downstream nuclear fusion energy generation), most of them have the potential for high impact on a particular field or area of society.

• Impact cannot be measured along the same timescale or timeframe for

different disciplines or scientific areas. Because of the fundamental difference in scope of infrastructural and super-structural research activities, a consequence is that the same time frame for assessing actual impact cannot be applied without running the risk of skewing results in favour of applied projects. In other words, more infrastructural research in certain fields such as clinical, nuclear or agronomic research, that involve lengthy cycles of experimentation and testing, cannot be expected to produce the same impact results within the same period as for example an RTD project in the software sector.

• In order to assess ‘high’ broader societal impact, examining both potential

and actual impacts, a period of between 1 and 3 years after project finish is necessary. However, in examining 1st and 2nd Group impacts, and constructing a causal analysis to link these groups to eventual 3rd Group impacts, it can be possible to assess potential impact at an earlier date.

• Collaborative research efforts like most of those funded under FP 4 and FP 5

seem more likely to encourage applied (super-structural) research while competitive research looks more likely to encourage fundamental (infrastructural) research.

• The fact that international cooperation is the main area of actual impact

across the sample and that economic impact is the main area of potential impact, (with limited actual impact) clearly demonstrates the need to take into account the time frame for evaluating different types of impact. By actively looking for evidence of impacts that correspond to the relevant timeframe of each project, a case study exercise like this could demonstrate the project’s actual progress towards achieving higher and longer-term impacts at societal or European level.



96

• Infrastructural research is more likely to be science-driven and the judges of

impact more likely, certainly initially, to be within the scientific community. Other stakeholders will make a judgement only as the new knowledge comes to be applied to specific problems at a super-structural level. Therefore impact of super-structural projects is much more dependent on a proper needs assessment at the outset and on involving end users in the project. Projects partners that demonstrated a clear awareness and understanding of the needs - in society and among users – that they were trying to address at the outset of their project also demonstrated higher levels of attainment of impact.

• It can be a challenge for scientists and researchers to see the link between the

work they are undertaking and the broader societal impacts which the FPs aspire to. This is in part due to the relative distance between the project, both in time, scale and its scope for influence, and visible social and quality of life impacts (it can also in part be due to a lasting divergence between broader EU-policy goals and the evolution of scientific advancement). However, those who are aware of potential end impacts during the project, identify this as being a critical factor itself.

• Evidence from this study seems to suggest that potential impacts could be a

better, or at least an additional, measurement for across the board comparisons of fundamentally different research activities. This is particularly relevant since some of the EC’s higher level objectives (e.g. European added value or relevance to EU policies) have remained more or less constant through generations of Framework Programmes. This means that by focussing attention on the attainment of these higher level objectives, visible progress towards potential higher level impacts could also more easily be made and captured.

• Projects which develop technical products, such as modelling tools, and where

eventual use will very much depend on final technical quality, can be difficult to assess prior to completion of the product.

7.2 HIGH IMPACT IN THE CASE STUDY SAMPLE

• European Added Value is a significant cross-cutting impact of these research activities. 11 out of 18 projects would never have taken off without European funding. Only a small minority of projects would have occurred in any form without EC funding and these would have been on a far smaller scale.

• There have been significant structural impacts across the EU in establishing

research infrastructures and this is in line with the emerging status of European research (particularly relevant to the European Research Area (ERA).

• At the EU level, research capacities are being increased, and skills have been

transferred between the older and new MS, as well as between more and less experienced researchers and organisations. The diversity of partnerships also leads to cross-fertilization between public sector research (universities) and research promoted by private companies including SMEs.



97

• Actual and potential impacts have been registered for all 18 projects within three main groups of impacts according to the time frame within which they occurred or are expected to occur:

1. Immediate impacts (1st Group), defined as those occurring within the

project cycle include international cooperation. This area of impact includes the creation of networks and innovative collaborations; capacity-building and skills development; exchange and cross-fertilisation of ideas and methods of working;

a. 13 of the 18 projects have already had a very high impact on

international cooperation, making a difference to collaboration at a European level. With 4 other projects also scoring “high” for this impact, it is a generalised high impact area in the overall sample;

2. Enabling impacts (2nd Group) defined as essential building blocks for higher

level societal impacts, are likely to occur in the medium term – generally within 3 years following project finish. These include policy, scientific, and technical impacts;

a. 5 projects have already had a very high impact on policy and standards, and a further 3 have the potential to make a difference to policy and standards at a European level;

b. 6 projects have had a very high impact on scientific knowledge, and a further 11 scored highly, making this the area with the highest impact among the 2nd Group of impacts. This area of impact includes both the creation of ‘new’ scientific knowledge (3 projects); the development of technologies (4 projects), and the refinement of existing knowledge (8 projects);

c. 6 projects attained very high technical impacts, a further 5 scored highly. These scores reflect the development of products (techniques, modelling), and the development of frameworks and structures.

3. Higher level impacts (3rd Group) defined as those broader societal impacts

that usually take longer to develop – between 3 to 10 years after project finish – include social, economic, and quality of life impacts.

a. 7 projects attained actual impact in the social area, although none scored ‘very high’, 3 did score ‘high’. High impacts within this area included job creation, public awareness, and safety and security measures.

b. 9 projects had actual economic impact, with 5 scoring ‘high’, and again none ‘very high’. Economic impacts included cost-savings, creation of SMEs, increased competitiveness, reductions in economic losses and the stimulation of markets. Among the higher level (Group 3) impacts, this is the area where impact has been the highest so far.

c. 7 projects have had impacts on quality of life, though only 2 scored ‘high’ and none ‘very high’. Quality of life impacts were linked to improvements in the environment and human potential.



98

• The measurement of broad societal impacts has been affected both by attribution and time-lag problems. The fact that only 6 out of 18 projects finished more than 3 years ago may explain why there were no ‘very high’ actual impacts obtained in these areas. As expected, none of the projects ending in 2004 or after scored any actual impact in the 3rd Group areas, but among those that completed before 2001 all but one achieved ‘high’ impact in this group of impacts. These were felt in all areas, although more were recorded in the economic sphere than in social or quality of life.

• Assessment of potential impacts is particularly relevant for the broader

societal impacts. 13 of the projects were seen to have ‘very high’ potential for these impacts although the actual attainment of these impacts is sometimes beyond the direct control of project partners.

• According to the ratings criteria used in the study, some projects lack the

potential for broad societal impacts. These include: o CHIN: Since many of its 3rd Group impacts occurred in China, the

project scored lower due to the geographical component of the assessment (i.e. national impacts scored lower than those on European level);

o WETO 2030: The possibility that it will be replaced by a new and improved model currently being researched has lowered its potential for impact;

o FUNCARS: Its focus on human research potential and European added-value is not directly linked to broad societal impacts in a linear way, this is why it has been rated ‘mid’ for potential economic and social impact;

o GALA: The societal impact of the project GALA will be subsumed by the Galileo initiative.

• The diversity of needs to be addressed by research projects can be illustrated

by the following examples: o In CIEL the need was science-based (infrastructural) to address a

problem in physics of trying to approach steady state fusion reactions - a requirement for any electricity generation this way;

o In GALA the need was to establish market viability in the design technology of Galileo. It was the launch pad (with other inputs) for Galileo, and Galileo would never have happened without it;

o In WETO 2030, an internal Commission/EU need was addressed for European energy prospectives (as distinct from American);

o In GECS the requirement was for prospective modelling studies to underpin the EU Kyoto position.

• Some projects had a high impact on the development of standards at the

European and international levels, exactly because their EC funding and Europe-wide partnerships provided credibility and facilitated recognition.

• There was a wide spread of types of organisations participating in the projects.

Universities participated in 12 of the case study project consortia, large



99

companies in 6 and SMEs in 6. Only 2 of the projects were implemented by sole organisations and these did not fit with the regular categories of FP programmes.

7.3 CONCLUSIONS ON AGGREGATING FINDINGS AT FP LEVEL

• Since sample projects were not picked at random for this exercise, they cannot be said to be representative of the general performance of FP supported research activities. On the contrary, because they were selected following the criteria of having been seen to have high impact by Commission officials, (the evaluation team also developed methodological approaches for selecting high impact projects and the selections were then modified and verified to satisfy Commission requirements), it is likely that they represent only a small percentage of overall FP projects. However, through their status as high impact projects, they provide an excellent opportunity to identify and measure high impacts among FP supported research activities as well as to explore what critical factors contributed to these high impacts.

• As expected, projects have made progress towards, and been remarkably

effective in terms of fulfilling, the overall objectives of Framework Programmes 4 and 5, particularly in the 1st and 2nd Groups of impact, comprising of international cooperation, policy and standards, scientific and technical impacts.

• The main focus of FP5 Programme Area objectives, broader societal impacts

(3rd Group) have been more difficult to measure and attribute to specific projects, particularly social and quality of life impacts. Economic impacts have been easier to measure in a quantified way. It is possible this will be the area of greatest perceived overall impact in years to come. However, this could also be due to the greater tangibility (and prediction) of impacts in the economic area in comparison to social or quality of life areas. Actual impacts in these areas vary in intensity and scope. For example, among projects targeting economic results GALA scored ‘high’ (GROWTH sub-programme), WETO 2030 scored ‘mid’ (EESD sub-programme) and CIEL scored ‘low’ (Euratom). For projects targeting social results, the only actual score - ‘high’ – was obtained by GALA.

• Projects are achieving results in the areas under which their main FP

objectives lie and beyond. In keeping with the generalised objective of FP5, projects in FP Programme Areas with an explicit goal to increase international cooperation (e.g. INCO and INCO2) have for example had high impact in this area like, for instance, MAIZE. Likewise, projects that fall under sub-programmes areas emphasising broader societal impacts are perceived to have a very high potential for obtaining these impacts. For example in the area of quality of life this includes IHP (FUNCARS, ENHSIN and FOREN) and QoL projects (MAFCONS). Very high potential social impacts have also been identified for the IST programme (ONTOWEB and NESSIE).

• The fact that one of the strongest elements of impact in the overall sample was

international cooperation suggests that the work in FP4 and FP5 towards



100

creating a ‘critical mass’ of excellence through networking between Member State capacities has materialised more broadly across the FPs. This in turn, is a solid foundation on which subsequent strategies, including the creation of a European Research Area (ERA) under FP6, can be built.

• It is clear that a key impact of all projects under review is added-value at the

European level. The majority are providing a European product (e.g. high level infrastructure, EU policy testing tools), or contributing to European competitiveness (technology, knowledge, development of SMEs), or European quality of life (environmental protection, sustainable transport, public awareness). This is proof that these projects have already contributed to the fulfilment of some of the over-arching goals of FP4 and FP5, and are contributing towards the fulfilment of objectives under FP6.

• Sample projects were highly innovative. EC funding has stimulated both

international collaboration in their implementation and the pace of the research. Numerous follow-on projects and uses being made of products resulting from the research show that the momentum of change instigated by these projects will be sustained over a longer period.

• The majority of case studies have been relevant to socio-economic concerns

in Europe – a key feature of FP5. However, because the projects were selected on the basis of perceived high impact, they may not comprise a representative sample across FP4 and FP5. Additionally it could take substantially more time for these impacts to become evident, as many of the projects are reliant on follow-on activities and the application of results.

7.4 CRITICAL FACTORS

• There are critical factors that both facilitate and impair impacts. • Overall enabling critical factors for impact include:

o Leadership and vision by a a vigorous and inspiring, ‘charismatic’ leader as well as excellence, quality and commitment of the people involved in the project, are essential impact drivers;

o Recognition at various levels helps to achieve impact. The high

reputation of key people involved motivates others to participate and excel; the prestige of being involved in a highly influential project motivates people to give their utmost, and the recognition among peers and beyond leads to higher downstream impacts;

o Accurate assessment of needs at the outset, particularly taking into

account the needs of end users in designing the project is crucial to generate impacts especially in the case of applied research;

o Flexible management approaches among project partners allow for

adjustment to changing circumstances. Small, tight teams avoid the



101

management problems associated with large multi-component partners, which often cause diversions away from the main aim. In the case of larger teams a transparent division of labour and an agreed-upon chain of responsibilities are vital to successful cooperation. Flexibility is particularly important since research impacts can be unexpected or unplanned;

o High functioning of partnerships. Projects based on pre-existing

relationships work better;

o Choice of research theme, timing and innovation - the more strategically selective and innovative a project is regarding the theme and the timing of the subject of its RTD work, the more likely it is to accomplish sectoral impact;

o Well planned and audience-targeted dissemination can significantly

increase the possibility of attaining impact, specifically in the case of projects that aimed to exploit or commercialise a model or a product;

o Research context - links to other projects, leveraging capacity and the

application of research are all related to potential impact. Projects within the sample that are furthering an existing research position, tend to achieve higher levels of innovation and impact.

o A hands-on approach by EC counterparts can help to “push” and

“drive”, to facilitate and maximise, impact. Involvement by the client as the work progresses can maintain required commitment to achieving the impact.

• Specific factors that have been seen to hamper impact include problems

associated with:

o Coordination of a large number of partners; o The administrative and financial relationship between project

partners and the Commission;

o Limitations in the technical quality particularly for science oriented projects;

o Tensions between academic excellence and broader societal

relevance of a project. Impact will perhaps only become a more prominent criterion of scientific achievement if the traditional (self-referential) mode of defining scientific quality via its intrinsic excellence is being challenged;

o Lack of awareness or clear focus on long-term impacts;

o Time-lag between project conception and completion, particularly for

RTD projects is a risk factor in itself;



102

o Lack of follow-on funding for projects to allow results updates, to

keep up the momentum or to progress development (development research).

It should be kept in mind that the sample of project assessed was selected due to perceptions of their high impact, and these identified factors will not necessarily be relevant to other projects.



103

8 RECOMMENDATIONS

This section provides recommendations both for the classification of projects (in relation to their potential impact), and for the continual consideration of impact (and potential impact), as a robust methodology for its assessment. It then provides suggestions for additional mechanisms to facilitate and multiply the impact of research projects.

8.1 STRATEGIC CONSIDERATIONS FOR DELIVERING AND ASSESSING IMPACT

• Foremost, it is recommended that qualitative measures of impact be used alongside more quantitative tools and indicators since quantifiable and conclusive proof of impact of RTD projects is rare, particularly among 3rd Group of broader societal impacts.

• Since awareness of potential high impact on societal level at the outset of a

project can be a factor for it to materialise, it would be critical to identify likely results, as well as to distinguish between infrastructural and super-structural types of research projects, at the project conception - ex-ante stage. This would help to ensure that partners are considering their projects as triggering a chain of events towards impact, as well as identifying key risks and assumptions. Applicants should therefore be asked to indicate whether the successful outcome of a project would result in super-structural and/or infrastructural impact and justify their choice at the start of the project. This would then affect the way in which the impact of the project research work should be monitored and assessed.

• It could be that some dissatisfaction with the impact of scientific research in the

past stemmed from the assessment of super-structural impacts using approaches more appropriate to the assessment of infrastructural impact (such as peer review procedures and similar), rather than seeking views of other stakeholders. Assessment tools appropriate to measure super-structural impacts should therefore not be used to measure the impact of infrastructural research activities and vice versa.

• For super-structural projects, which are likely to be the majority of FP supported

research activities, steps to fully understand the sector, or societal need to be addressed, ought to be taken from the outset. Applicants could be encouraged to conduct a proper needs assessment, involving users, at the proposal stage to examine how the project plans would need to evolve in order to achieve both immediate tangible results and longer term, higher level objectives, and to very clearly portray this in their proposal. A structured but flexible tool could be developed to facilitate this process. The link to the overall FP objectives should also be established at this ex-ante stage, in standard documents submitted in all proposals. To ensure relevance,



104

the proposals could then be evaluated not just by scientific peers but also by external stakeholders such as “user communities”11.

• It appears that competitive research funding is a better vehicle for

infrastructural projects, while collaborative research is more likely to generate applied projects with high impact potential. Ideally, the Framework programmes should support both, although collaborative research funding should still be granted on a competitive basis.

• In projects where copyright-likely products are to be expected, partners should

be encouraged to agree upon a clear position beforehand in order not to cause tensions during implementation.

8.2 CONTINOUS IMPACT ASSESSMENT - ‘MANAGING FOR IMPACT’

• A qualitative system of scoring (using well defined ratings), similar to that used during this assessment, would allow for comparison of impact between greatly diverse projects, and would also assist the monitoring and the structuring of impacts for individual projects.

• In monitoring and reporting results coming out of Community funded research

programmes, a causal results chain approach would be more appropriate than one that only seeks quantitative evidence of impact or follows a rigid hierarchy of objectives. The definition of the results chain methodology proposed here is different from the standard definition12. Rather than a strict sequence from activities to impact, it focuses on the ‘higher’ levels of outcomes and impacts, and has a strong emphasis on causal analysis, including assumptions, potential threats and opportunities.

• As opposed to a linear model, the results chain approach suggested here would

provide a platform for the continuous assessment and facilitation of impact or potential impact. In other words it would help to maintain the intellectual emphasis on the need and the impact to be fulfilled throughout the project cycle and beyond it (ex ante to ex post). Through continuous monitoring and self-assessment of impact throughout the project cycle by the partners themselves, commitment to addressing the specified need can be made intrinsic to the relations among the partners involved in the project. By asking project partners and stakeholders to evaluate the impact relevance of the project on a continuous basis, it is more likely that the project will adjust itself to maintain focus on the overall higher level objectives. In short, 'making a difference' under this framework would be about:

11 An example of current practise similar to this is the involvement of a Regional Panel during Step 2 of the evaluation process of INCO proposals. 12 ‘The (results chain is a) causal sequence for an intervention that stipulates the necessary sequence to achieve desired objectives beginning with inputs, moving through activities and outputs, and culminating in outcomes, impacts, and feedback’. OECD definition of results chain, DAC Glossary of Key Terms in Evaluation and Results Based Management, 2002.



105

o Understanding the domain into which the work fits and what it is trying

to do - so research on the need is required beforehand; o Maintaining commitment to the impact objective through the people

involved - people are more important than the processes; o Maintaining assessment/judgement of impact through the project cycle

to maintain matching of the “push of the science” to the “pull of the problem”;

o Use quantitative benchmarks such as numbers of jobs created, profits made, businesses set-up, etc., as supporting evidence. The impact is rarely linear in this way and a platform of involvements is a better concept for 'making a difference to society'.

8.2.1 Steps to developing the results chain approach In order for a different approach to be used in the monitoring and facilitation of impact for research activities a number of tools would have to be developed and implemented. However, in doing so it should be taken into account that these tools have the potential to alter the current bureaucratic relationship between project contractors and the Commission. • Rather than requesting additional documents from researchers, the proposed

impact assessment framework (based on a results chain approach), could be linked to current deliverables (proposal, interim report and final reports).

• The proposed Impact Assessment Framework tool (Appendix 9) could be

used as a template for selecting projects at the proposal stage (ex-ante impact assessment), and for assessment throughout the project cycle.

• Using the results chain approach, both project partners and EC project

officials would be able to consider impact from the beginning, link it to the higher level socio-economic strategy of the relevant FP or FP sub-programme, and manage for impact throughout the project cycle. By building a reflective monitoring process, progress towards results can provide continuous feedback to drive robust management decisions at both project and EC level. This would require a different approach to the proposal stage from both the EC and project contractors.

• The results chain tool requires the appropriate stakeholders to consider their

project as a causal sequence (assessments within this would be supported by quantitative and qualitative evidence, and the definition and measurement of indicators). In order to provide this evidence there is a need for the development of very specific, indeed project level, impact indicators within all FP Programme Areas.

• As demonstrated in Section 5.2.5, there is a clear need for flexibility and

reassessment of research projects throughout their life spans. Traditional project management tools (such as the logframe), whilst suited to ‘implementation’ projects, are not appreciated by coordinators of high impact research projects. These tools are not fully commensurate with, or designed for research (they might be partly or entirely incompatible with the requirements of



106

diverse projects). Also, the ability to alter a project in light of new opportunities could prove to be the most significant factor for attaining the eventual impact, although such flexibility would be marked as a negative result in traditional project management systems.

• Another finding from the assessment is that project coordinators do not use the

logframe because they either do not know how to apply it effectively/sufficiently, or they have other tools at their disposal which they use successfully to plan and implement a project. A causal analysis tool (in the form of a results chain) may therefore offer a standard way for project managers to plan and implement.

• The essential components of the causal, results chain approach would be

(where applicable):

o Assessment of need (including the involvement of users); o Identification and rating of impacts (according to the groups of impact

identified during this assessment), and the causal sequence required; o Elaboration of the relation between the project impacts and the

objectives of the FP and sub-programme areas; o Elaboration of assumptions behind impact expectations; o Elaboration of possible risks over time; o Identification of possible future opportunities; o Identification of feasible indicators, for each group of impacts; o Assessment of relevant critical factors by project partners and external

reviewers; o Identification and reporting of indicators.

• In terms of assessment points throughout the project cycle and beyond, it

would be advisable to keep to the current cycle of reporting during the project duration and to limit the number of post-project assessment points in order to increase compliance and to avoid “fatigue” among partners and external stakeholders.

• The first assessment point would take place at the project conception phase

(see figure below). An Impact Assessment framework (similar to the example in Appendix 9) could be submitted and completed as part of the initial proposal.

• Follow-up assessments, using the same framework at the mid-term and end of

project (Assessment points 2 and 3), would revise the original causal chain and measure indicators identified at the conception stage. Indicators, both quantitative and qualitative can be considered to be one means of identifying and monitoring impact, but not the only means. The revision of need, risk, opportunity and assumption analysis are simple means of ensuring that projects are both ‘on the right track’ and that their potential impacts are explicitly framed and continuously rethought and adapted.



107

Figure 8: Impact assessment framework with 4 appraisal points

• Continuous assessment would also ensure that the project remains relevant over the time-lag between project conception and completion. Rather than just re-assessing the need, this approach would link the reassessment back to impact, and therefore take account of changes in risks and opportunities. Over time, the ‘results chain’ framework would build more and more information on potential and actual impact. The length of time is less important than the measurement at ‘stages’ – thus this framework suits diverse RTD activities.

• At the end assessment point, there would be the opportunity for the EC to

identify HIRA and PHIRA projects and to consider possible follow on activities aiming at fostering/ multiplying/ sustaining impact. In this context there may be a need for a different instrument e.g. development research funding.

8.2.2 Linking the proposed framework to FP6 • Any introduction of new planning or management tools would need to

consider thoroughly their likeliness of being used by project coordinators and participants. The evidence collected does not indicate an aversion against the use of specific tools (e.g. logframes) due to their characteristics, but rather suggests some opposition of academics and scientists to the use of planning and management, let alone administrative and bureaucratic tools. They do currently, however, use a vast array of their own informal and formal monitoring tools, and therefore, flexibility and clear guidelines for the development of the causal chain would be critical to achieving buy-in.

• The specific features of FP6 (substantially broader project partnerships, higher

budgets, and more complex coordination and management needs), may almost inevitably lead to a higher degree of professionalisation of FP project management. This requires a particular expertise and the identification of it should not simply be serendipitous. A clear analysis of the new manner in

1. Project Conception

3. Project end

Immediate Impacts Within the project cycle

Intermediate Impacts Enabling: Project end to 3 years

Higher Impacts Broad, societal: 3 to 10 years

2. Mid term assessment

4. Ex-post assessment



108

which FP6 is managed (or indeed as planned under FP7) should be a pre-condition for any change of tools or methodology. Any shift of method or introduction of new tools causes high transaction costs. In the case of European research, the seemingly unimportant role traditional management tools play for project success may suggest that other steps of fine-tuning or improving programming may have a higher potential for positive yield.

• There are some precedents for this type of assessment. One example comes

from the mid-term review report for projects under FP5 within Directorate F – Health Research.13 This reporting tool asks expert reviewers to report extensively on the types of questions proposed here, including:

o Is the project well focused on meeting its objectives as set out in the

contract? o Are these objectives still relevant and still achievable within the time

and resources? o Is there evidence of meaningful cooperation between all the partners? o Does the project have clear and appropriate strategies for exploitation

and dissemination? o Do you consider that the project will contribute to the objectives of the

QoL programme? • The difference between what is proposed here and this current reporting tool

comes from the finding that an awareness of impact (among project partners) may be a crucial factor in the attainment of impact. Therefore, it is proposed that project partners themselves be involved in assessing the project from these perspectives, and that external reviewers are involved in other aspects of assessment.

8.2.3 Linking critical factors to assessment points • Building from the analysis of factors relating to impact (Section 5), the critical

factors identified have been mapped into the assessment framework, by identifying the relevant points for analysis. The EC could ensure the concurrent assessment and facilitation of impact by linking the case study impact assessment to the monitoring of factors that aid or impede impact. The parts of assessment to be carried out by project partners, or external reviewers are identified. As are the types of indicators (quantitative or qualitative), that are to be used for providing evidence of overall ratings of actual and potential impact.

13 ‘Template for drafting the mid-term review report for contracts awarded by Directorate F “health research” within the Community Research Programme “Quality of Life and Management of Living Resources (1998-2002). EC. Cordis. FP 5



109

POSSIBLE INDICATORS: Point 1 – Definition of indicators for all assessment points. Preferably a mix of both Quantitative and Qualitative indicators across all Groups of Impact

• Based on the above recommendations, the following assessment tool could be used to structure the evaluative framework for impact monitoring and evaluation at various points of the project cycle:

8.2.4 Proposed Impact Assessment Tool 1. First Assessment Point: To be submitted in proposal / contract documents:

1A. Causal chain elaboration (by project partners) - including:

Assessment of need

Identification of potential impacts (using Group structure)

Elaboration of the causal sequence required

Elaboration of the relation between the project impacts and

the objectives of the FP and sub-programme areas

Elaboration of assumptions behind impact expectations

Identification of possible risks over time

Identification of possible future opportunities

1B. Identification of indicators: Identification of feasible indicators, for each group of impacts: for 1st Immediate Group – International Cooperation and Human Resources; 2nd Enabling Group – Policy, Scientific and Technical; 3rd Broad Societal Group – Social, Economic and Quality of Life). 1C. Assessment of relevant critical factors by project partners and external reviewers;

To be analysed by Project Partners at the Proposal Stage

o Strategies for managing partnerships; o Legal issues (Copyright/ IPR).

Stakeholder (user) at the Project Selection Stage

o Needs assessment; o Expectations concerning the product(s) of the project; o Possible contribution to achieving high impact.

EC assessment (External Reviewers) at the Project Selection Stage o Research context, timing and assessment of innovation;



110

POSSIBLE INDICATORS: Point 2: Types of Indicators Measured:

– Knowledge/ skills (Training)

– Tools developed – Collaboration/

synergies

o Elaboration of the relation between the project and sub-programme areas;

o Validation of the causal chain and indicators; o Partnerships and key individuals.

2. Second Assessment Point: To be submitted in mid-term reports: 2A. Causal chain update (by project partners) - including:

Reassessment of need

Reassessment of potential impacts (using Group structure)







2B. Measurement of indicators: Reporting on indicators established at the First Assessment Point. Reassessment of indicators and proposal with justification for change/readjustment/specification (if required) to be included in the mid-term report. 2C. Assessment of relevant critical factors by project partners and external reviewers;

To be analysed by Project Partners in the mid-term report: o Functioning of partnerships; o Common understanding of impact orientation (promoting and

inhibiting factors); o Copyright and IPR issues.

EC assessment (External Reviewers): o Recognition/ credibility/ expertise and commitment; o Update on research context, timing and assessment of

innovation.



111

POSSIBLE INDICATORS: Point 3 Types of Impact Measured:

– Innovations / new knowledge

– Products / technology

POSSIBLE INDICATORS: Point 4 Types of Impact Measured:

– Social – Economic – Quality of Life

3. Third Assessment Point: To be submitted in final project report: 3A. Causal chain update (by project partners) - including:

Reassessment of need

Reassessment of potential impacts (using Group structure)







3B. Measurement of indicators:

End of project reporting on indicators. 3C. Assessment of relevant critical factors by project partners and external reviewers:

To be analysed by Project Partners in the Final Report:

o Dissemination (cross-fertilization, catalyst function); o Commercialisation and IPR issues.

EC assessment (External Reviewers): o Recognition / credibility; o Update on research context, timing and

assessment of innovation; o Identification of HIRA and PHIRA; o Consideration of follow-up.

4. Assessment 4: Ex-post analysis of HIRA (2– 4 years):

Validation of results chain with actual results. A template impact assessment framework is presented in Appendix 9.



112

8.3 OTHER MECHANISMS FOR MULTIPLYING IMPACTS

• In order to extend impact, there are a number of strategic decisions that could be taken at the EC level, in addition to the implementation of the proposed impact assessment framework. Considering the sample of projects selected (those with demonstrable high impact), a number of factors leading to high impact can be identified, which were not in accordance with standard EC procedures:

o The follow up project to FOREN was initiated by the EC and served to maximise impact by making the product specific to MS situations;

o The additional funding for the dissemination of CHIN products is closely linked to impact.

• EC needs to take on a clear role of ensuring access to MS level data or other

information resulting from the projects that would benefit from being in the public domain. This is particularly important for EU-wide modelling projects.

• For some types of project (including those resulting in products) the 3 year life

of a project is not always enough to achieve longer term impact. In addition, the current definition of dissemination plans at the proposal stage may be too rigid to allow for new contexts of opportunity or risk.

• Tackling this issue would rely to a large extent on the end of project assessment

of impact (EC assessment at stage 3 in 8.2.3 above). If constructed in a systematic way that allows for both quantitative and qualitative assessment of impact (using a generalised scoring system with indicators as evidence), the results chain approach is likely to identify a small proportion of “winners”.

• Where ‘winners’ are identified, there could be systematic follow-up with

participating organisations, to gain their perspective on how impact could be maximised and sustained.

• One option is to consider a further competitive stage for follow up FP funding

(development research), shortly following the end of the project, in order to avoid a time-lag impairing impact. With the following characteristics a further competitive stage would ensure that products remain relevant and maximise impact:

o Lower amount per year; o Longer period; o Competitive; o Training and ‘spin-off’ provision; o Support to networks.

• These HIRA or PHIRA are likely to be close to achieving substantial impact, i.e.

any additional support the EC would give would be going towards achieving a tangible result leading to a tangible impact.



113

• Another option for consideration would be to re-frame and re-direct the usual dissemination work package towards impact throughout the project cycle and towards the project end. Issues that could be considered at this stage include:

o Changed situations in terms of risks and opportunities that require

constant adaptation of dissemination goals; o Reassessment of potential end users (which may have changed during

the course of the research); o The value in taking a generalised product and making it more specific

and targeted towards potential users; o Opportunities for project partners to run training/ dissemination events

to maximise impact.

high impact

Documents

Transcript of high impact