Framework Programme 6 - European Commission...Framework needed more systematic planning, clearer...

Framework Programme 6

Meta-Evaluation

www.technopolis-group.com

Erik Arnold

November 2009

Framework Programme 6 i

Summary This meta-evaluation of FP6 aims to use evaluation evidence to understand Framework performance. Compared with earlier evaluations of the Framework Programme – the so-called Five Year Assessments – the recent evaluation of FP6 by the Rietschel group has benefited from a dramatically improved evidence base. This represents a big step forwards in the governance and accountability of the Framework.

There is a now quite long tradition of FP evaluation, which shows that the Framework has been good at producing knowledge and networks, strengthening European R&D capabilities across borders. FP6 represents a ‘step up’ in the policy ambitions of the Framework, intending to help build the European Research Area – stabling a European common market on knowledge and knowledge services and making European research and R&D more globally competitive by enabling the creation of critical mass and increased specialisation.

Evaluation evidence for FP6 shows that the Framework has continued its traditional, positive role. FP6 clearly involved high-quality research and members of the European scientific elite. It has also continued a long-running process of ‘academic drift’, with the presence of industry declining. The approach of designing ‘new instruments’ has been insufficient to trigger the structural changes needed for the ERA. The Commission has therefore rightly moved on to experiment further with large-scale mechanisms for engaging stakeholders across Europe in planning and implementing consensus-based scientific and technological visions. These new interventions – some of which have the potential to re-engage industry in the Framework to a greater extent – focus on generating consensus among the powerful players. There is good evaluation evidence that this is a powerful mechanism for reinforcing strength. A key potential weakness, however, is that they reinforce the Framework’s consensual role. There needs also to be space for new, disruptive, disequilibrating ideas and activities, otherwise the Framework will be a factor contributing to lock-in rather than innovation.

• Enhancing the impact of the Framework Programme will require better linkage between the Framework and other policies

• Developing the needed critical masses within ERA will require instruments that promote disequilibrium and restructuring to a greater extent than proved possible within FP6

• We lack an adequate arena in which the grand lines of the Framework strategy can be discussed. Should the ‘academic drift’ continue or is this an undesirable accident? Is the (changing) division of labour between the EU and the Member State level the right one?

• The Framework is committed to reducing gender inequalities but does not appear to have any mechanisms for doing so

• Despite the increased opening of the Framework to ‘Third Countries’, there is little in the way of a strategy for the role of the Framework in the globalisation of European knowledge. Strategy is urgently needed

• Continued innovation in evaluation methods and more complete coverage, especially of the thematic foci, are needed in order further to improve our understanding of the Framework and to improve its accountability

Framework Programme 6 ii

Table of Contents 1. Introduction 1

2. Recapitulation: What do we know from the previous meta-evaluation? 1

3. How FP6 differs from previous Framework Programmes 2

3.1 Policy context and objectives 2 3.2 Legal basis 4 3.3 Instruments and Interventions 4

4. The programme as a whole 7

4.1 Overall evaluations 7 4.2 Thematic Priorities 11 4.3 Member state perspectives 12

5. The Components and Instruments of the Programme 13

5.1 Networks of Excellence 13 5.2 Integrated Projects (IPs) 15 5.3 ERA-NETs 16 5.4 Technology Platforms 17 5.5 SMEs 18 5.6 The Joint Research Centre (JRC) 19 5.7 New Member States (NMS) 19 5.8 ‘Third’ Countries’ 20 5.9 Gender 22 5.10 Research Infrastructure 23

6. Networks and networking opportunities 23

6.1 Framework Programme Networks 23 6.2 COST 25

7. Conclusions 27

7.1 About FP6 27 7.2 About FP Evaluation 29

Framework Programme 6 1

1. Introduction

This is a meta-evaluation, in the sense that it synthesises evidence from other evaluations of parts of the Framework Programme in order to produce evaluative conclusions at a higher level – in this case at the level of the Framework Programme itself. It builds upon a meta-evaluation1 covering the period 1999-2004 that was among the background reports for the Five-Year Assessment of the Framework Programmes in 2004.

This paper was originally commissioned by the European Commission on behalf of the Rietschel panel, evaluating the European Union’s Sixth Framework Programme (FP6). Work had to be interrupted in October 2008 and an early version of this paper formed a major input into a draft of the main expert panel’s evaluation. This paper has therefore been finalised after the main panel report, as a result of which it is able to cover more of the FP6 evaluation studies than was possible for the main report, given its timing.

The amount of new evaluation-based evidence produced since the last Five Year Assessment is substantial. As a result, both the work of the Rietschel group and this paper have been able to benefit from an amount and quality of evidence about the Framework Programme that is without precedent and that represents a major increase in the transparency and accountability of the Framework Programme. More evaluations are in preparation at both EU and member state level, so the amount of evidence will continue to increase. Important gaps include a lack of new evidence about human resource and mobility aspects of the Framework and the still limited coverage of FP6’s seven major ‘thematic priorities’ that together accounted for about two thirds of the budget.

I gratefully acknowledge advice and inputs from members of the Commission services – especially Peter Fisch, Neville Reeve and Carl Dolan – as well as others in the field. However, this report reflects my own views and any mistakes are wholly my own. It should not be interpreted as representing the views of the European Commission, Technopolis or any other group or organisation.

2. Recapitulation: What do we know from the previous meta-evaluation?

The previous paper reviewed the record of mid-term and ex post evaluation of the Framework Programmes in the period 1999 to mid-2004 and distilled key messages from it. The analysis served as an input to the Five-Year Assessment 1999-2003 of the EU RTD Framework Programme (FP). The timing of the last Five Year Assessment meant that it had to consider parts of FP4, FP5 and the start of FP6. The new system, beginning with FP6, of focusing attention on a single Framework Programme has the advantage of freeing the evaluation from having to say something about three different FPs, each with its own pattern of emphasis and different policy goals.

The earlier meta-evaluation found that the Framework Programme provided a broad and permissive context for RTD programmes. Its high-level goals were to strengthen the research capabilities underpinning European industry and to improve citizens’

1 Erik Arnold, What the Evaluation record tell us about Framework Programme Performance, report to the European Commission, Brussels: European Commission, Directorate General for Research, 2005 http://ec.europa.eu/research/reports/2004/pdf/analysis_arnold_en.pdf


quality of life, while its low-level goals were largely to do R&D in its constituent Specific Programmes.

The available evaluation evidence suggested that the Framework broadly funded good quality work, in which universities and research institutes played a majority and increasing role. Framework participation was led by a ‘core’ of major beneficiaries, who sat at the heart of multiple European RTD networks. There was scope for greater industrial participation, which could be desirable in order to reach the Barcelona goal of spending 3% of Europe’s GDP on R&D. Administration was seen as more burdensome than that in national programmes, and – except where networking and scale are important – participants preferred to use national programmes. The FPs therefore focused in areas where they generated ‘European Added Value’. Projects were mostly ‘additional’ in the sense that they would not have been conducted without European funding. If attracting SME participation continued to be a policy objective, greater parts of the Framework should explicitly address their needs, which appeared poorly handled.

Framework projects primarily produced knowledge and networks, strengthening European-level human capital and RTD capabilities across borders. Their role was therefore quite distinct from nationally funded projects. There was evidence that the Framework Programmes had positive effects on the behaviour of the research community, competitivity, jobs, regulation and the environment.

Member states needed to improve national RTD strategies by taking better account of European-level programmes and policies.

The Five Year Assessment in 2000 concluded that the FP was not on its own sufficient to reach the Lisbon objectives. There was a need for a ‘real European RTD policy’, involving a more flexible FP and the incorporation of other instruments and actions. Since then, integration between FP design and wider EU policy for innovation and R&D appeared to have improved, though there was further to go in this direction.

The intervention logic that connected the high-level to the operational goals of the FP was poorly articulated, making an overall evaluation of the FP difficult. The Framework needed more systematic planning, clearer objectives and a stronger link to an evidence base. This would ease evaluation and, arguably, improve FP performance. Framework evaluation needed to be more clearly linked to understanding, diagnosis and wider policy for the European Research and Innovation System. This implied a more systemic approach to evaluation and to related studies needed to support policy development and planning. New and additional evaluation methods and further improvements in the organisation of evaluation were needed to link it better to an improved planning and policy framework for EU R&D.

3. How FP6 differs from previous Framework Programmes

This section discusses some of the history of the Framework Programme, explaining why and how FP6 is different from its predecessors, refers to the legal basis of FP6, provides soma basic participation statistics and identifies the role of FP6 in a process of developing policy interventions that stretches up to the present.

3.1 Policy context and objectives

The Framework Programmes date from the mid-1980s: the First (FP1) in 1984-7; the Second (FP2) in 1987-91. The Framework Programmes (FPs) had roots in earlier activities, for example the Multi-Annual Programme in the field of Data Processing (MAP, running from 1979-83 and subsequently incorporated into the ESPRIT programme, part of FP1.) The First Framework Programme was an amalgamation of


existing initiatives throughout the Commission in an attempt to develop a coherent research and development strategy.2 The initial focus was on Information Technology (IT) – actually as part of an OECD-wide push to increase IT research that followed the spectacular successes of Japanese industry in consumer electronics and telecommunications of the latter 1970s.

Over time, the Framework Programmes’ scope has tended to widen, so that they now cover a very broad range of themes and the repertoire of instruments has increased from the early focus on collaborative research.

Up to and including FP4, European Added Value in the form of networking, cohesion, scale benefits and so on was largely seen as sufficient justification for the FPs. In FP5, the focus shifted towards socio-economic benefits.

The launch of the European Research Area (ERA) via the Busquin communication3 in 2000 broadened the idea of European Added Value to include a role for the EU in ‘structuring’ the R&D supply side in Europe and coordinating national policies. This was very gently stated in the Communication, which focused on the need to network existing competences. A new Communication in 2002 was clearer about what ERA really meant, namely

• The creation of an 'internal market' for research – an area of free movement of knowledge, researchers and technology, which would contribute to an increasing co-operation, and would stimulate competition and a better allocation of resources

• A restructuring of the European research fabric; in particular by improved co-ordination of national research activities and policies

• The development of a European research policy which would not only address the funding of the research activities, but also all relevant aspects of other EU and national policies4

FP6 was designed at the time when the new ERA policy aimed to concentrate research resources and create a system whose most excellent parts could compete readily with those of the USA and Japan. This led to increased concern with research (compared with the earlier industry policy and impact focus), which should be excellent and in which Europe should build scale. FP6 therefore included new, larger instruments. The previous industrial strand continued but was less of a focus and – especially outside ICT – involved less effort. FP6 also marked the creation of Technology Platforms and ERA-NETs, in which the Commission encouraged groupings within the union to self-organise and try to develop cross-border groupings that would drive R&D and innovation policies for their sectors or technologies. By and large, these collect together existing strong interests and the thrust of the Technology Platforms is continued in FP7’s JTIs (Joint Technology Initiatives) and increased interest in Article 169 consortium arrangements.

The FP was launched after a period in which European R&D cooperation had blossomed on a multilateral basis, for example through CERN, EMBL, COST and ESF. In the period since then, the FP has become increasingly involved in funding aspects of these cooperations. Because the FP exists and is a simpler way to channel money than creating new multilateral organisations, there have been no significant new European R&D cooperations set up since the FP began in which the Commission is not central. There is one historical exception to the Commission’s monopoly of European action, namely Eureka, which was in effect Paris’ reply to what it saw as a shift of power towards Brussels. But that was in 1985 and even Eureka has now succumbed to the

2 Patries Boekholt, The European Community and Innovation Policy: Reorienting Towards Diffusion, Birmingham, 1994.

3 Towards a European Research Area, COM(2000) 6 final, Brussels 19.1.2000 4 COM(2002)565 of 16/10/2002, p. 4.


funding logic and taken the Commission’s money for the EUROSTARS programme. The FP6 evaluation argued that this concentration of power, and the risk of monopoly of thought that accompanies it, is problematic. The European Research Council (ERC) story serves as a useful symbol. Originally proposed as something that should belong to the scientific community and that could be based anywhere in Europe provided the Commission was not involved, it ended up as a budget line in FP7.

3.2 Legal basis

Article 163 of the version of the Treaty of Rome in force when FP6 was defined says, “The Community shall have the objective of strengthening the scientific and technological bases of Community industry and encouraging it to become more competitive at international level …5” It goes on to empower the Community to support both industry and knowledge infrastructure of research institutes and universities to this end. The Treaty empowers the Commission to define and operate the Framework Programme (Article 164). It also says, “The Community and the Member States shall coordinate their research and technological development activities so as to ensure that national policies and the Community policy are mutually consistent.” (Article 165) Articles 169 and 171 respectively enable the Community to support “research and development programmes undertaken by several Member States” and to “set up joint undertakings”.

There is specific enabling legislation for FP66 This explains that the programme will strive towards greater integration of research in Europe by means of

• Focused action in priority thematic research areas, using powerful financing instruments (integrated projects and networks of excellence) which bring together the research actors in appropriate configurations for the new challenges that these priority research areas represent, and with critical mass

• Systematic and coordinated planning and execution of research to support Community policies, and to explore new and emerging scientific and technological areas, taking account of needs expressed by the relevant actors throughout the European Union

• Promoting the networking and joint action of national and European frameworks for research and innovation, and the opening up of national programmes, in these priority areas, including where appropriate by the use of actions under Article 169 of the Treaty, as well as in other areas where such action would be of benefit to the performance of Europe’s research base

The legislation also stresses the need to involve ‘third countries’ in the FP, both in the thematic priorities and in “specific international cooperation activities with some groups of countries, as a support to Community external relations and development aid policies.” Participation by SMEs and candidate countries were also to be encouraged.

3.3 Instruments and Interventions

FP6 introduced two ‘new instruments’ within its seven thematic priorities

• Networks of Excellence, aiming to strengthen Community research by building critical mass

5 Official Journal of the European Communities, Consolidated Version of the Treaty Establishing the European Community, C325, 24.12.2002

6 Council decision of 30 September 2002, adopting a specific programme for research, technological development and demonstration: ‘Integrating and strengthening the European Research Area (2002-2006), (2002/834/EC), OJ 29.10.2002


• Integrated projects, which amount to very large versions of the FPs’ traditional collaborative research projects, where a significant part of the administration is devolved to the project itself, again with the intention of creating “critical mass”

The traditional collaborative research projects were renamed Specific Targeted Research Projects (STRPs). Other instruments were collective and cooperative research projects for SMEs, coordination actions and later the ERANETs. Technology Platforms were also encouraged but do not form part of the FP. Figure 1 shows participation data per instrument type.

Figure 1 Collaborative Research Projects in FP6

Contracts Participations EU Financial Contribution

Number % Number % Number %

Integrated Projects 703 14% 17,707 30% 6,657,330,257 48% Specific Targeted Projects 2,273 45% 21,398 36% 4,474,979,235 32% Networks of Excellence 171 3% 5,153 9% 1,262,017,551 9% Coordination Actions 486 10% 7,123 12% 609,104,922 4% Specific Support Actions 1,368 27% 8,224 14% 948,751,098 7% Totals 5,001 100% 59,605 100% 13,952,183,063 100% Source: DG-Research Directorate A, FP6 Final Review: Subscription, Implementation, Participation, Brussels: EC, June 2008

FP6 marked both a transition to larger instruments and to lower success rates. Some 18% of FP6 proposals succeeded, compared with 26% in FP5. This meant that 19% of FP6 applicants succeeded, compared with 24% in FP5.

Figure 2 FP5 and FP6 Basic Implementation Statistics

Source: DG-Research Directorate A, FP6 Final Review: Subscription, Implementation, Participation, Brussels: EC, June 2008

FP6 appears to have continued the long slow trend to declining industrial participation (Figure 3) – a major problem for a programme whose overriding objectives relate to economic performance at the EU level.


Figure 3 Shares of participations in successive FPs

Source: First 4 columns, FYA 1999-2004; last column FP6 Final Review

The sequence of new instruments and interventions introduced during and after FP6 shows clear tendencies towards larger interventions, delegating administration from the Commission to the research performers, promoting self-organisation by established interest groups, influencing member state research and innovation budgets and imposing forms of governance that involve actors at the level of member states but that often bypass the agents of the states themselves so that the member states are involved but disempowered in terms of agenda-setting. We can think of these as belonging to four generations.

Generation 1. Integrated Projects and Networks of Excellence were introduced in FP6 to generate disequilibrium or ‘creative destruction’ in the fabric of the R&D infrastructure. However, their success in doing this was limited overall, though there is evidence of individual successes. It might have been natural to replace them with something like ‘competence centres’ that would geographically focus R&D resources, but this step has not been taken.

Generation 2. The ERANET and ERANET+ schemes are becoming precursors of a form of joint programming that delegates agenda setting to member state agencies while the Commission retains some control of what is started and the number of organisations in the coalitions through funding competitions and its use of subsidy. The interest in the Nordic area in NoriaNets suggest that this scale of intervention is too big for some countries and that there is scope for similar initiatives at the level of Member States or networks of Member States, without necessarily involving the Commission. (In fact, many ERANETs appear to comprise an active core of participants and a larger periphery, who do not participate in calls for proposals, suggesting that in many cases a small network is the most relevant size.)

Generation 3. The European Technology Platforms launched in the latter part of FP6 allow actors – especially industry – to self-organise to define research strategies, which they naturally look to the Commission and to the member states to fund. Influencing the FP agenda is a specific intention. Some are evolving into Joint Technology Initiatives (JTIs) or even Article 169 arrangements. Here the stakeholders involved do the governance but the Commission and to a more variable degree the member states hold the purse strings.


Generation 4. In the last two years, a new style of intervention has emerged through the European Institute of Technology (EIT), the Recovery Plan and the SET-Plan. These effectively invite stakeholders to build coalitions that will co-fund R&D with FP7. They involve very large blocks of money (hundreds of millions and in some cases billions of Euros). Their governance comprises in some cases member state government representatives, in other cases other kinds of actors from the member state level. Unlike the ERANETs they do not appear to involve the national agencies.

The emerging Joint Programming Initiatives (JPIs) will be governed at the overall level by a High Level Group of representatives of national ministries, associated with CREST. They will work rather like JTIs but will be public-public research cooperations. Interestingly, the impact assessment of Joint Programming argues that the governance of individual JPIs should be done via a “strategic European process” in which experts from the national level advise on individual initiatives. This would extend the tendency to work with expert groups in FP governance – following a somewhat academic governance tradition in which those who govern do not represent or coordinate other interests but lend legitimacy and expertise.

The introduction of the European Research Council (ERC) in FP7 provided an extension of traditional academic self-governance into the FP and also an instance of NSF-style funding of Principal Investigators, as opposed to the consortia that have in the past been necessary in order to generate European Added Value. Beyond the ERC, there is little in the FP that encourages exploration of new possibilities in a way that is detached from established interest groups. While the process of FP design is not very transparent, it is nonetheless clear that the role of established lobby groups is large and that the Commission understandably can react cooperatively when offered coherent visions and road maps that show what the FP should do in future. The FP is a consensus-reinforcing mechanism, not something that sets new directions.

4. The programme as a whole

This section discusses first what is visible about the Framework Programme from Framework-wide evaluations. It then looks specifically at evidence about the individual Thematic Priorities before considering what new evidence has emerged from the Mamber State level.

4.1 Overall evaluations

An increasing amount of new evidence is emerging at the level of the overall Framework Programme.

4.1.1 Why participate?

The Innovation Impact study7 makes innovative use of the Community Innovation Survey (CIS) to complement survey and interview-based data on participation in FP5-6 to explore links between the FPs and industrial innovation. It found that compared with the average for their sector, industrial participants tended to be more R&D-intensive, better networked, more orientated to international markets and to patent more. The study found knowledge goals were more important than other kinds – but the questions used were inconsistent with those used in other surveys, which tend to find that networking as well as knowledge is important8. The FP6 participation study confirmed these goals as important to industry, although if also found that 60% of

7 Wolfgang Polt, Nick Vonortas and Robbert Fisher, Innovation Impact, Final report to the European Commission, Brussels: DG Research, 2008

8 See for example Atlantis, 2005


industrial organisations expect commercial returns within 2 years of the end of the project9. Small firms were more likely than other participants to regard commercialisation as a priority.

The growing share of FP participation by the knowledge infrastructure meant that the proportion of the effort that could lead directly to industrial innovation had declined.

Compared with internally funded projects, FP projects involve

• Lower commercial risk

• Longer term R&D horizons

• More interest in ‘peripheral’ technologies outside the core technologies of participants

• Focus on exploration (rather than exploitation) strategies

• Lower degree of flexibility

• Higher administrative burden

There was no meaningful difference between FP5 and FP6 projects in this respect, despite the use of new instruments – an interesting piece of evidence in support of the ‘project fallacy’ idea. That is, the notion that beneficiaries pursue their own ‘real projects’, often using multiple funding sources, and that the ‘official’ objectives agreed with funders or defined in funding programmes do not necessarily correspond with those of the research performers.

A study of the impact of FP funding on the behaviour of firms and research institutions10 confirms much of this picture. On the whole, FP6 provided opportunities for extended international and cross-sectoral networking, for projects of a greater scale (particularly financial scale), and for projects of a greater technical and scientific complexity – opportunities which would have been severely limited without the funds it made available. FP participation for firms is seen as a way to experiment in new areas to draw on expertise and infrastructure in the wider scientific community. Access to expertise, infrastructure and large-scale funding are also the dominant motivations within the knowledge infrastructure. The FP6 participation survey11 found that of the proposals which did not receive funding – and which characterised themselves as strategically important to the organisation - 54% were abandoned or modified. The most important reasons for abandoning a project were lack of funding and the loss of complementary expertise. The latter reason was more important for industry than other participants, indicating their greater dependence on the expertise of project partners12.

However, the behavioural study also noted a marked difference between large firms and the knowledge infrastructure and in their dependence on FP-funding and their responsiveness to call objectives and criteria. Universities and Public Research Institutes (and indeed small companies) were much more likely to tailor their proposals to FP calls and to cancel a project if it failed to receive funding. Larger firms, on the other hand, formulate their R&D strategies independently of FP calls and would partially or fully cover the cost of rejected projects from internal private resources. This indicates that, at the project or call level at least, FP has very little leveraging

9 Idea Consult, Participation Survey and Assessment of the Impact of the Actions Completed Under the 6th Framework Programme, 2009 p.32

10 Idea Consult, Does Europe Change R&D Behaviour? Assessing the Behavioural Additionality of the 6th Framework Programme

11 Idea Consult, Participation Survey and Assessment of the Impact of the Actions Completed Under the 6th Framework Programme, 2009



effect on the R&D strategies of individual firms. Universities and public research institutes also appear much more dependent on FP funding, with 72% of them having been involved in 2 or more FPs compared to the 50% of industrial organisations that were first-time participants13.

The relatively conservative, low-risk nature of FP research was confirmed by the 85% of funded projects that were characterised by participants as either 'incremental' or 'next generation' in terms of technology reach. Only 11% of projects were self-described as 'radical' i.e. making use for the first time of technology that is new to the sector. Of FP6 project proposals that did not receive funding, a much higher proportion were self-described as 'next generation' or 'radical'.

In general, companies in competitive high technology sectors were more likely than those in lower technology, oligopolistic or monopolistic markets to have strategies to commercialise results of FP research.

Despite relatively low success-rates and a reputation for burdensome administrative procedures, the best scientists are still sufficiently attracted by the benefits of FP collaboration to continue participating. A study of the research track-record of a sample of lead scientists participating in FP6 projects14 (as measured by the number of citations their publications receive) demonstrates that these researchers consistently outperform their counterparts publishing in the same field, regardless of factors such as discipline or country. A study15 of publishing by researchers at five Swedish universities found that FP participants were more successful than non-participants in terms of both citation rates and number of collaborations, already before participating in EU-financed projects. This suggests that one pre-requisite for being successful when applying for EU-funding is already to be an established researcher. Another conclusion is that the general trend towards an increased internationalisation of science has the effect that the differences between participants and non-participants have decreased over time. The traditional effort of the Framework Programme to increase networking in Europe seems to be hitting diminishing marginal returns.

4.1.2 Impacts

The Innovation Impact study tried and failed to find a linear connection between FP participation and innovation, which is what one would expect if there is not a simple cause-effect link between the two. It points out that many of the FP participations analysed were single links in longer chains of R&D projects – some involving the FPs, others not. Attributing innovations solely to the FPs would be problematic. Those going into a project in the expectation that they would innovate with the results were more likely to do so than those with no such initial ambitions. However, the majority of industrial participants reported at least one commercialisable result from their participation, although they reported a decline in exploitable results as compared to FP516. This appears to be driven by the more fundamental and longer-term focus of much of the FP6 work.


14 Technopolis, Bibliometric Profiling of Framework Programme Participants, 2009 15 Johan Fröberg and Staffan Karlsson, ‘Possible effects of Swedish participation in EU frame programmes

3-6 on bibliometric measures’, Appendix J in Erik Arnold, Tomas Åström, Patries Boekholt, Neil Brown, Barbara Good, Rurik Holmberg, Ingeborg Meijer and Geert van der Veen, Impacts of the Framework Programme in Sweden, Stockholm: VINNOVA, 2008



In view of the difficulties in measuring the impact of the Framework Programme on commercialisable innovations downstream, an FP-wide study17 attempted to measure the impact of programme funding on research behaviour and strategy. The study aimed to discover whether funding changed the scope, the scale and the speed of the project; whether it changed the nature of the research undertaken; and whether R&D processes have been formalised or there have been durable changes in the capability of organisations to manage research. The study found limited evidence for such behavioural changes, particularly amongst the established R&D performers who make up the bulk of FP participants. Strategic or behavioural changes were more evident in small or start-up companies or public research institutions in candidate countries for example. Overall, participants reported moderate effects on the time focus, the financial scale and the scope of the research projects. Such effects are lower than those found in national or regional funding programmes, which again reflects the fact that successful FP applicants typically have a strong track-record in attracting funding and that the assembled research consortia build on core long-standing partnerships.

However, the study did find that projects would have led to smaller range of potential applications and, to a lesser degree, marketable products if continued without FP funding.

Despite the high level socio-economic goals which are explicitly set for the FP from legislative to work-programme level, participants appear more concerned with intermediate goals such as knowledge generation and theoretical development. When asked about the contribution of their projects to such themes as employment, environment, health care, sustainable energy and quality of life, a large sample of participants estimated only a moderate impact as a result of their projects18. More remarkable is the number of participants who regarded such socio-economic categories as irrelevant, ranging from 25-40%. A similar result is obtained when participants are asked to estimate the impact of their research on policy-making, with high numbers of participants regarding such aims as irrelevant to their projects.

A programme-wide survey19 clearly demonstrates this focus on output rather than socio-economic outcomes, with the preponderance of FP6 outputs being more upstream (e.g. refereed publications, tools, methods and techniques) than downstream (e.g. new products, patents, intellectual property) – although this may be a natural consequence of taking stock of project outcomes at a relatively early stage of the programme cycle.

Taking part in FP projects is seen as a source of prestige and esteem by large numbers of participants, but the impact of mainstream collaborative FP projects on researcher mobility and on the immediate research environment and working conditions was reckoned to be limited20.

4.1.3 Administrative burden

The FPs’ administrative arrangements – and more generally, those of the Commission – are near-universal hate-objects. The last meta-evaluation suggested this makes the FP a ‘funder of last resort’ because where potential applicants can reach their goals by less administratively painful means they will do so. Correspondingly, where they lack

17 Idea Consult, Does Europe Change R&D Behaviour? Assessing the Behavioural Additionality of the 6th Framework Programme


19 AVEDAS AG, NetPact: Structuring Effects of Community Research – The Impact of the Framework Programme on RTD on Network Formation 2009, p.16

20 Idea Consult, Participation Survey and Assessment of the Impact of the Actions Completed Under the 6th Framework Programme, 2009. Note that this study did not include in its sample actions specifically designed to encourage mobility under the Marie Curie programme.


alternatives they will go to the FP, which was for example the major funder of Irish university research during the 1990s, when little national project funding was available. IST applicants from the New Member and Candidate states saw the administrative arrangements as a far less significant barrier to participation than those from the EU-1521. Whether this implies applicants from the New Member and Candidate states are already inured to complex administration is not clear. But it supports the widespread finding that the cost of writing proposals is a significant barrier to participation.

Nevertheless, there have been discernible improvements in programme administration compared to FP5. Participants have welcomed the improvement in terms support and information services, flexibility in project composition and call and evaluation procedures, although it should be borne in mind that the new instruments in FP6 have introduced new dimensions of complexity. There remains a good deal of dissatisfaction with programme implementation in terms of delayed payment and negotiation of contracts22.

The last IST 5YA23 assessed the administrative cost of the IST priority as 7.5% of the budget, placing it above the normal range of costs for national Research Councils (which tend to lie in the 3-5% range) and at the lower end of the range for Innovation Agencies (typically 6-12%).

4.2 Thematic Priorities

A growing proportion of the FP6 Priorities and sub-priorities have now been evaluated, providing more insight into the ‘core’ activities of the Framework, many of which persist between different FPs. More evaluations are in progress, and will over time round out the picture further.

The sub-priority ‘Global Change and Ecosystems’ aimed to tackle climate change and its ecological implications. The evaluation found that it produced a mixture of scientific and policy results. Publication performance and scientific quality are especially high in Climate Change, Water & soils, and Biodiversity & Ecosystems and work is frequently cited in IPCC reports, which represent the key international discussion forum about climate change. Other parts of the sub-priority are less visible in the scientific literature because they produce outputs that go more directly into policymaking. Their actual policy effects are variable – the work is generally relevant and of good quality, but where and whether it has policy impact is not under the control of the research teams. An element of chance is involved. The large instruments of FP6, especially the IPs, have been useful ways to sustain and support the work of existing networks. While in the industrially orientated parts of the Framework it is reasonable to expect that the positive experience of networking will make inexperienced companies more likely to collaborate without (or with less) subsidy in the future, this sub-priority brings together largely public sector researchers for whom research is not an input to a larger business but actually is what they do for a living. As a result, the networks are permanently dependent upon public funding – they serve a public purpose.

The part of the Aerospace priority that was funded by DG-Enterprise and that tackles services – specifically the Galileo satellite positioning system and the GMES Global Monitoring for Environment and Security earth observation system (renamed

21 Technopolis, Monitoring Strategy and Requirements for IST-RTD in the sixth Framework Programme, Part A Management Report. 08/2004, Version 1.0, p 16


23 Five-Year Assessment: 1999-2003, Research and Technology Development in Information Society Technologies, Final Panel Report 01/2005, p 18


Kopernikus in 2008) and satellite communications – has been evaluated24. These areas have in many respects more the character of engineering and innovation than of research. Projects were generally well executed. The link to end-users, however, was insufficiently strong and demand too poorly articulated to produce confidence that the eventual services to be provided would be commercially successful. In some cases, the justification for developing a service was driven by strategic considerations – the desire for Europe to have an independent system, for example – rather than markets, making it hard to assess benefit/cost aspects. .

In IST, a pilot impact study25 shows that the work of FP6 was more fundamentally orientated than that in FP5, but that the technologies involved were nonetheless aligned with European strengths and market needs. In FP6, the Leitmotif for the IST programme was the ‘Ambient Intelligence’ vision developed by ISTAG (the IST Advisory Group) – a group of 18 senior company people and 11 university and research institute people. The vision was therefore strongly shaped by an industrial and forward-looking research view of the challenges and opportunities to be tackled in order to address future markets. The IST programme made good use in places of the new instruments, especially IP, to build and reinforce research/industry communities around common visions or road maps. It produced considerable quantities of new knowledge and human capital as well as in many cases strengthening European technology and market positions. However, it had surprisingly little impact in Microsystems and Sensors, which is a area of traditional European strength – probably because the research community, the markets and the suppliers tend to be fragmented. Some of the IST Programme’s most important impacts were in pre-normalisation and standards creation. Its effects were therefore especially important where technologies and markets were maturing and it was possible to build and articulate a consensus. The Framework seems to have been less effective under conditions of research or market fragmentation and where technology road maps do not exist or are unclear. This finding mirrors one of the key conclusions of the Swedish longitudinal impact study of FP3-6.

4.3 Member state perspectives

Several national impact studies of FP6 are in press or in progress: Denmark, Ireland, the Netherlands, Norway, the UK and the Nordic Area as a whole. As a result, there is little that can be said about FP6 from the national level at the time of writing. The published studies that are evaluative (as opposed to desciptions of participation) are from Finland, Switzerland and Sweden.

The Finnish study26 points to a decline in the number of Finnish participations in FP6 compared with FP5 but also to an increase of their quality. Nationally, FP participation has been building up and this learning has translated into more realistic expectations about the benefits of taking part as well as to increased performance. I argues that “commercialisable output is not the core objective of the FPs but EU collaboration nonetheless contributes significantly to the creation of innovation.” The study investigates a handful of sectors in more detail, showing how Finnish organisations have become involved and making the point that there is a positive relationship between being active and organising networks and obtaining benefits. The report concludes that there is a need for clearer articulation of national strategy

24 James Leather, John Clark, Anca Dumitrescu and Effi Pitsaros, Ex post evaluation of the activities carried out by DG Enterprise and Industry under FP6 Innovation and Space Research Activities, Brussels: EPEC, 2008

25 Bea Mahieu and Erik Arnold, WING FP6 IST Pilot Impact Study, Brussels: EPEC, 2009 26 Soile Kuitunen, Katri Haila, Ilpo Kauppinen, Mikko Syrjänen, Juha Vanhanen, Paavo-Petri Ahonen, Ilkka

Tuomi, Pekka Kettunen and Teemu Paavola, Finns in the EU 6th Framework Programme: Evaluation of Participation and Networks, Tekes Programe Report 6/2008, Helsinki: TEKES, 2008


and how it takes account of the EU level and also for better ‘joining up’ of policies across different policy sectors, both at national and at EU level.

A Swiss study27 of participation in FP5 and FP6 found that participation generated both knowledge and jobs. The Framework added only limited value in terms of networking, because the Swiss industrial and academic systems were already very open. In the view of Swiss participants, the benefits of FP6 were larger than those of FP5, but the complexity of the new instruments and the wider administrative burdens associated with the Framework meant that the benefit/cost ratio had actually fallen between FP5 and FP6.

The one really new type of study28 is a longitudinal look at impacts in Sweden from FP3-6. It found that in the university context, the FPs have added quite a substantial amount of money to external research income. In so far as research (and education) are good things, then these are good things that should broadly lead to increased social and economic welfare. This funding is additional to national funding; the study did not find suggestions that national funding has been reduced to compensate. And Sweden’s excellent performance in bringing money home from the FPs means the bargain for Sweden has been a good one: she takes out more than she puts in and most of that additional money goes to the universities. However, the fact that the universities largely lack thematic strategies for their own operations and consistently lack strategies for handling the FPs is an important missed opportunity to use FP resources systematically to promote the development of critical masses and therefore to combat the fragmentation in the university system. This fragmentation puts it at risk, both in terms of the general need for critical mass and specialisation in an increasingly globalised university system (and, indeed, in support of the knowledge and manpower needs of key parts of Swedish industry) but also the specific need to specialise in the context of the focusing of resources that is intended within the future European Research Area.

On the industrial side, the study shows significant impacts on competitivity in vehicles and in electronics (especially telecommunications). These are essentially areas where European industry can articulate clear needs during the process of consultation on Framework Programme design and subsequently in the design of Work Packages. In sustainable energy and life health, where this is not so well done, the industrial impacts are more diffuse.

5. The Components and Instruments of the Programme

5.1 Networks of Excellence

The NoE instrument was innovated in response to the need to de-fragment the EU research community and build the strong centres of excellence needed to become globally more competitive that was identified in the ERA discussions. As Bonaccorsi et al point out29, the concept changed a number of times. At one point, NoEs were

27 Oliver Bieri, Andreas Balthasar, Ruth Fller-Länzlinger, Bernd Ebersberger, Jakob Edler and Sascha Ruhland, Evaluation of the Swiss Participation to the 5th and 6th Research Fraework Programme of the European Union as well as the Swiss Information Network Euresearch, Bern: State Secretariat for Rducation and Research, 2005

28 Erik Arnold, Tomas Åström, Patries Boekholt, Neil Brown, Barbara Good, Rurik Holmberg, Ingeborg Meijer and Geert van der Veen, Impacts of the Framework Programme in Sweden, Stockholm: VINNOVA, 2008

29 Andrea Bonaccorsi, Manfred Horvat, Austria, Toivo Maimets and Pierre Papon, Expert Group on the Future of Networks of Excellence Final Report, Brussels: Directorate-General for Research, European Commission, 2008


expected to be major funding instruments with tens of millions of Euros at their disposal and taking up the greater part of participants’ time.

The FP6 legislation offers a much weaker definition. It says, “The purpose of networks of excellence is to strengthen and develop Community scientific and technological excellence by means of the integration, at European level, of research capacities currently existing or emerging at both national and regional level. Each network will also aim at advancing knowledge in a particular area by assembling a critical mass of expertise. They will foster cooperation between capacities of excellence in universities, research centres, enterprises, including SMEs, and science and technology organisations. The activities concerned will be generally targeted towards long-term, multidisciplinary objectives, rather than predefined results in terms of products, processes or services.”30 By the following year, their focus on critical mass was clearer “integrating at European level the critical mass of resources and expertise needed to provide European leadership and to be a world force in that topic … an instrument designed primarily to overcome the fragmentation of European research … In all cases the number of participants … to be integrated should be compatible with a) the overall objective of a meaningful long-term integration of the research capacities of the participants and b) the manageability of the whole endeavour”31.

While NoEs seemed at this stage to be conceived as perhaps having 3-6 participants, the instrument was interpreted differently in different parts of the Commission and Commission publicity pointed increasingly to the need for NoE applications to have many participants. In the end, the mean number of participants was 30. The NoEs became very academic. Industry contributed only 7.6% of the participations (and received a mere 4% of the funding) in the NoEs. In practice, the instrument was greatly watered down from the idea of promoting critical mass, to the extent that some of the NoEs approach the pure focus on networking seen in COST and ESF programmes. Bonaccorsi et al reviewed the achievements of NoEs and concluded that a minority had implemented joint programmes of activities that provided a basis for more durable integration structures, and proposed that the Commission should fund their continuation for a period. Other NoEs achieved more diffuse and less sustainable results in training, scientific services, small outsourced research programmes and collaborative research among their members. In their overall judgement, the NoEs as implemented failed to address the problem they were designed to tackle. In some ways, this failure was due to a disappointing level of 'buy-in' or support at both institutional and national level, which meant that much of the collaboration and networking remained on a personal or project level32.

Despite these difficulties and misunderstandings, the level of support from the research community remains generally high. A minority of participants (38%) in a recent FP-wide survey33 believed that the size of the projects was too big (compared to 61% for the other new instrument – Integrated Projects), and their scale has increasingly been seen as a strength. This belies the notion that the networks are too big to function and indeed in some cases they have been seen as playing a catalytic role in building and sustaining new relationships in fragmented research areas – a central aim of the scheme. Furthermore, 86% of survey respondents agreed that they will continue to collaborate with other members on new activities after the network funding had been discontinued, demonstrating the value placed on the relationships that had been built.

30 Council decision of 30 September 2002, OJ 2002/834/EC 31 EC, FP6 Provisions for implementing Networks of Excellence, May 2003 32 European Policy Evaluation Consortium, (EPEC), Assessment of the Impact of the New Instruments

Introduced in FP6, 2009, p.6 33 Ibid, p.51, 57


Our interpretation is that the NoEs were initially conceived as contributing to the creation of critical masses under ERA. They were watered down by the Commission’s desire to sign up as many participants as possible to its agenda, with the result that they have tended to reinforce rather than decrease fragmentation. The viability of virtual centres of excellence as ways to decrease fragmentation is in any case not well tested, but the Commission is obliged by the subsidiarity principle to focus its efforts ant the international – and hence virtual – level. In the period since 2000, many Member States have begun investing in competence centres and other kinds of centres of excellence with the intention of reducing national fragmentation. We argue that there is scope for the European level to reinforce these efforts by investing in selected centres, thereby reinforcing their tendency to create disequilibrium in their respective fields and bring about a concentration of effort. This would imply an instrument focused on (but not limited to) physical and not just virtual centres of excellence.

5.2 Integrated Projects (IPs)

Integrated projects34 are intended to generate the knowledge required to implement the thematic priorities of FP6. They are intended to assemble a ‘critical mass’ of activity, sufficiently large to increase Europe’s competitiveness or to address major social needs. “Each integrated project should be aimed at obtaining specific results relevant either to increasing the impetus to Europe’s competitiveness or to addressing major societal needs. Its objectives may include more long-term or “risky” research.” The IPs were intended to be multidisciplinary programmes containing a number of modules, normally to cost in the tens of millions of Euro and to last 3-5 years. Part of each IP’s budget was to pay for consortium management. A little-discussed reason for initiating IPs appears to be reduce the EC’s own costs, given the limitation of management costs to 6% of the total FP6 budget by the Council Decision and DG-Research’s inability to persuade the Commission that it should increase staff numbers.

IPs tend to be led by people from industry or research institutes, who are equipped to take on the considerable managerial task of coordinating these large instruments. Interviews and surveys suggest that university people tend to be less comfortable with IPs than with STREPs; leading an IP is difficult for university people not used to managing large projects and a distraction from research. By and large university people seem to prefer STREPs both because they are small enough to be manageable form a university base and because their proliferation means there is a better chance of launching a STREP on a topic of interest to the researcher than of launching an IP.

A study35 of the original new instruments – IPs and NoEs – shows that participants do believe IPs help build critical mass both through scale and by involving a larger proportion of the relevant sector than is possible in STREPs. In most cases, however, they felt that the Call obliged them to use IPs and that this instrument in very few cases resulted in a reduction of the administrative burden – over 50% of respondents thought the burden was greater. But using IPs increased the size and diversity of the networks in which people operated and increased the degree to which problems tackled were international in character. As most surveys of FP participants find, the key outputs were networks and knowledge. Some 10% of respondents experienced major unexpected negative benefits, while almost 50% experienced major unexpected positive benefits form the new instruments – suggesting that the protests of the research community against the instruments in the early days must also be tempered by their subsequent experience.

34 FP6 Instruments Task Force, Provisions for Implementing Integrated Projects, Brussels: EC, 12 May 2003; available at http://cordis.europa.eu/documents/documentlibrary/66622311EN6.pdf

35 Isabelle Collins, Assessment of the Impact of the New Instruments Introduced in FP6, Brussels: European Policy Evaluation Consortium, 2009


One use of IPs is to reflect industrial consensus. This is clear from the Aerospace thematic priority, which is dominated by IPs and where the research agenda is to a high degree set by the European industry, with its ACARE platform as a focus. This has the effect of focusing FP effort on the interests of established EU industry. Similar evidence is emerging for the IST Programme

5.3 ERA-NETs

The ERA-NET instrument was innovated as one means to ‘structure’ research funding within the ERA by grouping together funders from different countries to make joint calls for research proposals in areas of common interest. These ‘variable geometry’ constructions were expected to work in four steps

1. Systematic exchange of information and good practices on existing programmes and activities

2. Identification and analysis of common strategic issues

3. Planning and development of joint activities between national and regional programmes

4. Implementation of joint trans-national activities, including joint calls and programmes

The Commission covered all the additional costs associated with the trans-national aspects of coordination. There were two types of ERA-NET: Specific Support Actions (SSAs) to establish new networks and Coordination Actions (CAs) to support the activities of the ERA-NETs themselves.

Overall, 26 SSAs and 71 CAs were selected for funding. These involved over 1000 representatives from 38 countries (25 EU Member States; 8 Candidate and Associated States; and 5 ‘Third Country’ States). A 2006 review36 found that many of the ERA-NETs were making good progress toward issuing joint calls and argued that they added value to the European R&D funding portfolio.

A subsequent impact study of which a summary was published in 200937 found that the ERA-NET scheme succeeded in reaching its objective of fostering cooperation among national funding authorities and had resulted in joint calls worth some €1.1 billion by the end of 2008 – with national funding amounting to over five times the EU contribution. It had major effects on attitudes to cross-border cooperation an the joint programming of funding activities but its overall impact was limited by national R&D policies and the limited role that national authorities were prepared to give to ERA-NETs.

Our own more narrow review of Nordic ERA-NET experience suggested that many agencies had rushed into ERA-NET participation without a strategy and found themselves over-stretched as a result. While many if not most EU agencies learnt how to co do cross-border calls with partners, this learning was inefficient and repeated in each ERA-NET. Only a minority of ERA-NET participants actively engaged in the eventual calls for proposals. “ERA-Nets brought important benefits. They enabled cooperative priority setting by sharing strategic intelligence to identify areas of common priority. They encouraged the synchronisation of national research programmes. Existing bilateral funding schemes were being extended to become multilateral. Small countries like Norway found that ERA-Nets enabled them to fill

36 Manfred Horvat, Ken Guy, Violeta Demonte Barreto, Jüri Engelbrecht and Ralf Wilken, ERA-NET Review 2006: The Report of the Expert Review Group, Brussels: EC December 2006

37 European Commission, FP6 ERA-NET Study, Summary of the Impact Assessment Study of the ERA-NET Scheme Under the Sixth Framework Programme,DG Research, Brussels: European Commission


gaps in the national research portfolio. They increased the exposure of national research performers to competition.”38 A continuing concern is that ERA-NETs increase the complexity of the funding system rather than simplifying it.

Both the TAFTIE network of innovation agencies and the Nordic research councils’ umbrella Nordforsk have since adopted the principle of ERA-NETS – but explicitly aim to foster such networks on a smaller scale (involving only partners who from the outset are likely to fund a joint call or other activity) and free from the EC’s timetable constraints and the bureaucratic burden involved.

5.4 Technology Platforms

The European Technology Platforms are a measure that originated with the Communication ‘Industrial Policy in an Enlarged Europe’39 rather than in the Framework Programme. They are considered here because they are a key policy complement to FP6. Their main policy objectives were to

• Support the development and deployment of those key technologies in Europe that are vital to address major economic and societal challenges

• Define a European vision and a strategic agenda for the development and deployment of these technologies

• Support the objective of increasing European private research investment by bringing research closer to industry and improving markets for innovative products40

As of December 2007, there were 34 ETPs. Their activities involve: developing Strategic Technology Plans for their own areas; identifying mechanisms to obtain the needed public and private investment; identifying relevant skill and education needs; and communicating the need for action at a European level. IDEA Consult’s evaluation shows that some 45% of ETP stakeholders are industrial while 40% are from the knowledge infrastructure. Generally, industrial members lead the creation of a strategic vision while the knowledge infrastructure is more involved in translating this into a Strategic Research Agenda (SRA). In many cases, the Commission has funded early coordination costs but generally it is industry that carried the greatest load for funding the secretariats and other common activities involved. The evaluation raised concerns about the ineffectiveness of SE participation and the comparative absence of end users and non-governmental organisations from the ETPs, which therefore tend to represent the more powerful industrial and research interests. Those stakeholders involved judge that the ETPs have been effective in coordinating ideas about R&D directions but (so far) only partly effective at raising new money. They have had little effect in triggering more joint R&D so far. The ETPs provide mechanisms that could coordinate research policy at regional, national and EU level, though the evidence for this actually happening is so far weak. There has so far been little or no impact on skills.

While it is still rather early to see effects of the rather slow policy coordination processes the ETPs are expected to trigger, there has apparently been an effect on the agenda of FP7 in some technologies, but not in others.

38 Erik Arnold, Annelie Eriksson, Sven Faugert and Tommy Jansson, Building Nordic Strength Through More Open Funding: The Next Step in NORIA, TemaNord 2006:576, Copenhagen: Nordic Council of Ministers, 2006

39 COM 2002, 714 final 40 IDEA Consult, Evaluation of the European Technology Platforms, BUDG06/PO/01/Lot 3, Brussels:

IDEA Consult, August 2008


5.5 SMEs

In the earlier meta-evaluation, we argued that – except in the case of the SME-specific instruments of the past – the designs of the FPs have not been especially friendly to SMEs, except in the special case of technology-based SMEs. We concluded that if the EU is to pursue a more traditional SME support policy, it would be better to use dedicated instruments rather than instruments that are essentially designed for other purposes. (This, however, raises the issue that outside certain high technology fields, small-firm support is probably best handled at national and regional levels.)

The evidence available about SME participation in the Framework is improving. DG Research commissioned an impact study that looked at SME-specific measures in FP4-6, specifically the CRAFT projects and their continuation in the form of Cooperative Projects in FP6; and the Collective Research projects piloted under FP5 and mainstreamed in FP6. Key findings included that

• All schemes have a high net effect • The Collective Research scheme has a high net effect • Economic and Technical Support Actions have a strong structuring effect, and few

alternatives are available • Further effort is needed with regard to business intelligence in order to increase

socioeconomic impact

The cooperative projects involve firms in solving innovation problems in partnership with other firms and members of the knowledge infrastructure while the collective projects tend more to involve collectives and the knowledge infrastructure solving problems in order to be able to transfer results to SMEs later on. In effect, therefore, FP6 has reinvented at the international level key parts of the repertoire of the traditional national Research Associations. In parallel with this reinvention has come a savage reduction in success rates to well below FP6’s already low average of 19%.

Figure 4 CRAFT, Cooperative and Collective projects in FP4-6

FP4 CRAFT FP5 CRAFT/Coop

FP6 Cooperative

FP6 Collective

Submitted projects 1,775 261 3,274 673

Funded projects 766 680 397 80

Success rate 43% 30% 12% 12%

Source: de Laat and Bitard, 2005; FP6 data from Horizontal Activities involving SMEs, Brussels: EC, 2006

De Laat and Bitard found that these schemes were highly additional and attracted technologically competent firms – more than 70% of which regard themselves as ‘technology intensive’. They speculated that the high technological capabilities of the successful firms might be a function of the low success rate. Just as tends to be found at national level in equivalent programmes, the project initiative is usually taken by the knowledge infrastructure rather than SMEs – with the counter-intuitive result that impacts are increased (because this makes the questions researched more generic). Collective projects depend even more on the knowledge infrastructure. Both projects have significant positive impacts on their beneficiaries, which could be increased with better business intelligence and improved information about cross-border partnership opportunities.

Rammer and Geyer41 studied participation by German SMEs in the FP and EUREKA. The study confirmed that participants in the thematic priorities and in EUREKA had

41 Alexandra Rammer and Anton Geyer, Erfahrungen deutscher KMU mit europäischen FuE-Kooperationen am Beispiel der EU-Forschungsrahmenprogramme und EUREKA, Studie im Auftrag des DLR, Wien: Technopolis, 2005


high technological capabilities while those in CRAFT has lower absorptive capacity. They saw clear added value in participationg in European programmes, which was connected with a need to internationalise their R&D as well as market relationships. However, they tended primarily to pursue knowledge goals, selecting EUREKA for close to market issues and the thematic priorities for longer-term questions. While the knowledge infrastructure or large firms usually initiated projects, the SMEs felt they played key roles, albeit not a project leaders. The administrative trouble of making a proposal to the FP was a major disincentive.

The key barriers to SME participation in the IST thematic priority were the nature of the consortia; financial issues; instrument related issues; and contract negotiations, as well as SMEs’ limited knowledge of the detail of the FPs42.

5.6 The Joint Research Centre (JRC)

The JRC has seven institutes and functions as the Commission’s ‘government laboratory. It was allocated €760m (5.8%) from the FP6 budget and was free to compete for more.

The thrust of the JRC’s support to EU policies lies in the provision of technical support on issues related to environmental protection, safety and security of the citizens and sustainable development. This includes risk assessment, testing, validation and refinement of methods, materials and technologies to support a whole gamut of policies ranging from safety of food products, chemicals, air quality, water quality, nuclear safety, to protection against fraud. Almost all this support will be carried out in close collaboration with laboratories and research centres in Member States and elsewhere.43

The JRC was heavily criticised in the run-up to the last Five Year Assessment for lack of customer orientation. An evaluation44 of the JRC’s directly commissioned programme of work under FP6 observed that the JRC was implementing all the recommendations made to it under the previous review and appeared to be producing work of good quality and policymaking relevance. The evaluation report is couched in positive terms, but points to the JRC’s inability sufficiently to reallocate resources from old to new activities, its reactivity and failure proactively to develop its own plans taking changing needs and internal resources into account, its need to implement more flexible human resource policies and reduce the negative effects of its internal ‘silo’ organisation. The evaluation panel, chaired by the UK’s former Chief Scientific Advisor proposed that the EC should have such an advisor within the Commission Services. Unfortunately, the evaluation says little about the JRC’s contribution to FP6.

5.7 New Member States (NMS)

The experience of the New Member States entering the EU during FP6 reflects the way networks behave and evolve. Success rates for proposals from EU15 member states were comparable (18%) to the success rates of proposals (16%) that included participants from new member states (10+2). The NMS’ shares of participations in the

42 Monitoring Strategy and Requirements for IST-RTD in the sixth Framework Programme. Part B Theme 1. The Use of the FP6 Instruments involving SMEs in the IST Thematic Priority. Revised report. Version 1.0. 0/2004, p19

43 Council Decision of 30 September 2002, adopting a specific programme of research, technological development and demonstration to be carried out by means of direct actions by the Joint Research centre (2002-2006), OK (2002/836/EC), 29.10.2002

44 David King et al, Ex-post Evaluation Joint Research Centre Direct Actions in the 6th Framework Programmes 2002-2006, Final Report, EC-JRC, September 2008


new large-scale instruments (IPs and NoEs)were especially low45. However, on average, NMS partners received only €112k per participation, compared with €250k for EU-15 partners. Even the highest average among the NMS at €112k is less than the lowest of the EU-15: Portugal, at €146k. This difference is partly driven by wage differentials (that should close over time) and partly by the NMS participants’ limited FP experience that limits the importance of their roles in many projects (reflected in their lower than ‘expected’ share of the lucrative coordinator positions). A significant body of analysis shows that FP networks operate much like any others: they evolve slowly so they are a little ‘closed’ and new members have to demonstrate their capabilities and build trust before being able to play major roles. The NMS therefore find themselves in the position of ‘new kids on the block’ and can be expected further to assimilate into the FPs over time, as others did before them. The key difference with the previous group of entrants (Austria, Sweden, Finland) is the lower incomes and research intensity of these NMS but the earlier examples of Ireland and Greece show that over time the FP can make a major contribution to developing R&D capacity and quality. Increasing FP participation requires increased effort at the national level. Some of the latest 12 entrants need similarly to build capacity and will do well to build links to Structural Funds in order to fund this. Greece provides a particularly strong example of a country that has used accession to build a powerful research community, and which therefore benefits substantially from the FPs.

5.8 ‘Third’ Countries’

The Framework Programme has contained an international development component since the outset, initially via the Science and Technology for Development (STDI) programmes from 1983 onwards, which focused on health and agriculture in less developed countries. This was supplemented fro 1984 by International Scientific Cooperation with countries with which the EC had signed economic cooperation agreements. This has been replaced by more explicit bilateral S&T Agreements over the past 10 or more years. These activities were merged in 1994, to form the International Cooperation Programme (INCO) in FP4.

Like previous FPs, FP6 continued to include an activity for cooperation with ‘third countries’ not associated to the FPs. However, in addition it opened the main thematic priorities to participation by these countries on a cost-shared basis and allowed them to participate in the Marie Curie human mobility programme. The FP6 budget allocates €320m to Inco itself and earmarks a further € 285m for Third Country participation in the thematic priorities. The legislation46 is rather woolly about the objectives of international cooperation in FP6 – probably because it tries to cover a wide rage of motivations and the need to pursue different tactics in different parts of the world.

The general objective of the international cooperation activities carried out under the Framework Programme is to help open up the European Research Area to the rest of the world. These activities represent the particular contribution of the Framework Programme to this opening-up process, which will require a joint effort by the Community and the Member States. Under this heading, the activities in question have the following particular objectives:

• To help European researchers, businesses and research organisations in the EU and the countries associated with the

45 Isabelle Collins, Assessment of the Impact of the New Instruments Introduced in FP6, Brussels: European Policy Evaluation Consortium, 2009

46 L 294 EN Official Journal of the European Communities 29.10.2002


Framework Programme to have access to knowledge and expertise existing elsewhere in the world

• To help ensure Europe’s strong and coherent participation in the research initiatives conducted at international level in order to push back the boundaries of knowledge or help to resolve the major global issues, for example as regards health and environment

• To lend support, in the scientific and technological field, to the implementation of the Community’s foreign policy and development aid policy

These aims can only be facilitated by the generally positive image of the FPs (and FP6 in particular) held by members of the research and science policy communities outside Europe47. The programme is associated with top-class research, good networking opportunities and advancement of research careers. One result of international activities being embedded in FPs over time is that the programme has a very good visibility amongst the funding and policy communities, although in the wider scientific community misunderstandings and ignorance concerning the opportunities available and the objectives of the programme persist. FPs compete with other established international programmes (e.g. those run by NIH, NSF and COST), but are considered better both in terms of career recognition and in terms of mobilising top-class researchers and institutes. The more generous funding provisions are also recognised.

An evaluation of the international cooperation programme INCO in 200548 focusing on FP5 but also discussing aspects of FP6, pointed out the fragmented and apparently unfocused nature of the programme – which has presumably been caused by its rather dispersed set of tasks. It pointed out that only some S&T agreements involved resource allocation; those without resources had little effect. According to the report, Inco has very few resources compared with the size of its global remit, lacks scientific coherence and is poorly differentiated fro the rest of the FP. Its main impacts are strengthening relations and building scientific partnerships between the EU and Third Countries. The evaluation suggests focusing Inco on ‘development science’ and ‘international cooperation science policy’ issues. The procedures should be simplified, Inco more clearly distinguished from other parts of the FP and its role better communicated. A coordination function is needed to make Third Countries’ access to the thematic priorities more systematic.

A panel evaluation49 of the EU-China S&T agreement in 2004 found that it was highly regarded on both sides . In practice, its meaning was to ‘open’ the FP up to Chinese participation but this needed to be complemented by strong mobility measures, wider communication and a specific instrument for cooperation with China. There was “confusion” about whether the Chinese Ministry of Science and Technology or the FP was responsible for paying for Chinese participation. The 2008 evaluation50 of the agreement found increased interest in co-funding on the Chinese side and argued for strengthening and coordinating the work of the EU and key Member delegations to China. A key issue was that mobility schemes from China to Europe were popular but that few Europeans used the schemes to visit China. There were over 300 Chinese

47 Idea Consult, Assessment of the International Standing of the 6th Framework Programme, 2009, p.6. 48 Evaluation Partnership, Impact Assessment Report on the Specific Programme of International R&D

Cooperation Fifth Framework Programme, Brussels: DG-Inco, 2005 49 John Watson, Keith Harrap, Xin Mingyi and Shi Guangchang, An impact assessment of the Science and

Technology agreement concluded between the European Community and the Government of the People’s Republic of China, Brussels DG-Inco, 2004

50 Manfred Horvath and Nannan Lundin, Review of the Science and Technology Cooperation between the European Community and the Government of the People’s Republic of China, October 2008


participations in FP6 but many were small scale, the FP was poorly understood and difficult to access from the Chinese side and the FP indeed provided no more than a framework: a strategy and a common platform with key Member States and a much larger commitment of resources were needed in order to make an impact, especially in the light of large US and Japanese efforts51. The USA, and indeed some EU Member States have been working with China since the late 1970s. The EU’s spend of about €5m per year in FP6 is paltry in comparison with these other efforts and the size of the scientific and industrial opportunity. In the absence of a strategy coordinated with the Member States, the FP will do little to strengthen the interface between China and the ERA.

An evaluation52 of the equivalent agreement with India also indicated that the main effect had been to open up the FP to Indian participation. Lack of activity by the steering committee and a lack of focus in the agreement itself meant that cooperation was largely generated bottom-up and was almost entirely limited to institutions in the Indian knowledge infrastructure. Only 35 Indian researchers had benefited from EU mobility schemes. The evaluation is rather quiet on the subject of impact – but clearly the fragmented approach and the lack of focused activity implies that impact is limited.

More generally, the ‘programme logic’ of international cooperation in the Framework Programme is poorly defined and needs articulation. For example, the different motives involved in industrial as opposed to academic research collaboration are not articulated and their implications for policy have not been analysed or translated into specific instruments .

5.9 Gender

The Amsterdam Treaty of 1997 laid the legal foundation for gender mainstreaming

• Article 2: The promotion of equality between men and women is a task of the European Community

• Article 3: In all its activities the European Community shall aim to eliminate inequalities and to promote equality between men and women

In 1999 the European Commission issued the Communication Women and Science: mobilising women to enrich European research53. This set a target of 40% female membership of all committees, groups and panel associated with the Commission’s research activities and integrated the gender dimension in research content.

A study of EURATOM and four FP6 priorities (together covering about one third of the Framework) Found that women’s representation in committees etc average from 10% (EURATOM) to 26% (NMP)54. The larger and more research-orientated the FP instrument studied, the lower was women’s participation. Only 11% of the coordinators in FP proposals were women and only 10% of the coordinators of FP6-funded projects were female. Some 20% of researchers in the priorities surveyed by the project were women.

51 Erik Arnold, Sylvia Schwaag-Serger, Neil Brown and Sophie Bussillet, Evaluation of Chinese Participation in the EU Framework Programme, Brighton: Technopolis, 2008

52 Vijay S Panday, An impact assessment of the Science and Technology agreement concluded between the European Community and the Government of the Republic of India, Brussels DG-Inco, 2006

53 COM (1999)76 final, 22 February 1999 54 Diego Herrera, Mercedes Oleaga, Mayra Amate and Ione Isasa, Monitoring progress towards gender

equality in the Sixth Framework Programme, Synthesis report Nanotechnologies and nanosciences, knowledge-based multifunctional materials, and new production processes and devices (NMP), Aeronautics and space, Sustainable energy systems, Sustainable surface transport, Euratom, Brussels: European Commission, 2008


In science and engineering in Europe as a whole, 33% of PhD graduates, 20% of researchers and 8% of professors are women. Given that FP project coordinators have to be relatively senior within their organisation, female participation in the FP seems to be broadly representative of that in the relevant research communities. Since there are no positively discriminatory measures in the FP, this is what we would expect. But it seems reasonable to question the logic of aiming for gender equality in the FP and then effectively doing nothing about it.

5.10 Research Infrastructure

Aspects of research infrastructure development have been present in the Framework since ESPRIT in FP1. However, FFP6 took up the issue in a more systematic manner, in response to the perception that common infrastructure is a useful way to contribute to the creation of ERA. The Commission has encouraged the European Strategy Forum on Research Infrastructures (ESFRI) to develop a road map for EU research infrastructure needs.

A study55 of the impacts of FP6 support actions on research infrastructure found these had allowed scientists to reach goals for improved service in the majority of cases, with national facilities often being opened up to international use. However, the bottom-up nature of the project proposal and selection process meant that projects were not necessarily aligned with the needs of entire EU research communities and failed to address the wider social and economic goals of the programme.

6. Networks and networking opportunities

This section describes networking aspects of the Framework Programme and discusses the success of COST – the European Cooperation in the Field of Scientific and Technological Research, which antedates the FP but which has long been funded by it to promote scientific and technological networking. .

6.1 Framework Programme Networks

Collaborative networks are at the core of the Framework Programme and are the main tools for achieving such aims as building a critical mass of research effort and, crucially for FP6, reducing fragmentation within the European Research Area. Numerous surveys have confirmed that this is valued by researchers and industry and is a primary motive for engaging with the Framework Programme. There is also evidence that such FP networks are delivering in terms of intermediate programme goals such as stimulating collaborative outputs, improving research quality and sustaining networks beyond the lifetime of the project. FP6 projects have led to increased co-publication activity between project partners56; these co-publications have a significantly higher impact (as measured by citation performance) on the scientific field than the world-average (up to twice as great)57; and over 80% of project partners intend collaborate with one or more partners after the project has ended (a higher proportion than FP5)58. Nonethless there are concerns that such networking effects are limited to the project-level rather than the structural level (i.e. inducing

55 Matrix, Rambøll, Manchester Institute of Innovation Research, Research Infrastructures in the Sixth Framework Programme: Evaluation of Pertinence and Impact, Synthesis Report, Brussels: European Commission, 2009

56 AVEDAS AG, NetPact: Structuring Effects of Community Research – The Impact of the Framework Programme on RTD on Network Formation 2009, p. 16

57 Ibid. 58 Idea Consult, Participation Survey and Assessment of the Impact of the Actions Completed Under the 6th

Framework Programme, 2009 p.39


more and better cooperation between universities, industry and business). Clearly, such networks and their formation deserve more investigation.

DG-Infso has led the use of social network analysis (SNA) as a way to explore the FPs. A priori, we would expect this technique to tell us new things about the FPs, which are highly networked activities. The first steps have shown for example that in FP5, the obligation to have international partners in IST projects led to more intense international networking than was the case in other programmes without this obligation59. Aspects of collaboration networks in FP IST projects are national, owing to pre-existing national relationships. Taken as a whole, network analysis of all the IST projects shows there are strong ‘hubs’ (for example, the Fraunhofer institutes) that build bridges among these smaller networks and connect to large numbers of participants. RAND found60 that the structure of IST networks changed at the start of FP6. Whereas in FP3-5, the average number of links (‘or degrees of separation’) between participants was 3.24-3.26, this number fell to 2.63 in the first Call for FP6. Of course, this is the arithmetical consequence of funding fewer, bigger projects with on average more partners in each. Whether the change in network topography changes inter-participant cooperation, learning and research effectiveness cannot be determined using SNA. Both interview evidence61 and subsequent studies62 have indicated that using bigger instruments does not necessarily increase the amount of collaboration among people who now appear to belong to the same network. Indeed an FP-wide bibliometric analysis found a higher partner co-publication share for smaller teams (maximum 5 members) as well as a stronger increase in partner co-publication share over time63.

A programme-wide network analysis of FP6 projects64 confirmed the findings of the ICT programme that FP networks are dominated by a small number of closely-knit large organisations, and that this structure becomes more embedded the more funding a thematic area receives. This structure tends toward the oligopolistic as one examines the relationships at instrument level within each thematic area65. In contrast to the ICT programme, industry plays less of a coordinating role and is only weakly embedded in networks in the rest of the Framework Programme, which is dominated by the major national research associations (e.g. CNRS, INSERM, Fraunhofer Gesellschaft, Max Planck institutes). Such dominance is even more pronounced in new Member States such as Hungary, Poland and Czech Republic.

Bocconi University explored66 the relationship between patenting and ICT business networks and FP-IST networks, finding a substantial degree of overlap, which they argued means the FP effort is well integrated into global innovation networks. While small-firm FP-IST participants tended to be more technologically active (measured in terms of patenting) than others, only 5.4% of the European SMEs holding highly-cited patents participated in the FP, tending to confirm the idea (coming from many

59 http://soleunet.ijs.si/website/html/euproject.html 60 RAND Europe, Evaluation of Networks of Collaboration Among Participants in IST Research, 2004 61 For example in Erik Arnold, Tomas Åström, Patries Boekholt, Neil Brown, Barbara Good, Rurik

Holmberg, Ingeborg Meijer and Geert van der Veen, Impacts of the Framework Programme in Sweden, Stockholm: VINNOVA, 2008; Erik Arnold, Sylvia Schwaag-Serger, Neil Brown and Sophie Bussillet, Evaluation of Chinese Participation in the EU Framework Programme, Brighton: Technopolis, 2008

62 AVEDAS AG, NetPact: Structuring Effects of Community Research – The Impact of the Framework Programme on RTD on Network Formation 2009

63 AVEDAS AG, NetPact: Structuring Effects of Community Research – The Impact of the Framework Programme on RTD on Network Formation 2009, p. 18

64 AVEDAS AG, NetPact: Structuring Effects of Community Research – The Impact of the Framework Programme on RTD on Network Formation 2009

65 AVEDAS AG, NetPact: Structuring Effects of Community Research – The Impact of the Framework Programme on RTD on Network Formation 2009, p11

66 Bocconi University, Evaluation of Networks of Collaboration Among Participants in IST Research, 2004


sources) that the FP is not a good vehicles for rapid innovation involving SMEs and a short time to market.

A subsequent Bocconi studys67 showed that while FP-IST networks are well linked to the knowledge infrastructure in Europe’s regions, they are poorly linked to the SMEs that constitute the main regional deployment actors for new ICT technologies. This supported the increased use of structural funds for innovation in ICT, and by extrapolation in other technologies.

6.2 COST

COST – The European Cooperation in the Field of Scientific and Technological Research – antedates the FP but has for some time been funded via an FP budget line.

The origins of COST are in the 1960s, when European countries began to formulate science and technology policies. After a long period of worrying about whether the Russians were coming, Europe turned its attention to Le Défi Américan68, and the fear that European industry would become colonised by US multinationals. In 1965, the fledgling European Community of six countries set up a sub-committee of its medium-term economic policy committee to deal with science and technology. This sub-committee – PREST, which became CREST with the enlargement of the Community in 1973 – proposed seven areas for scientific and technological cooperation in 1967. It took a further four years, interrupted by periods when both the Netherlands and France held the discussions hostage to wider disagreements within the Community, for the Six plus finally a further thirteen non-Community countries and the Community itself to sign a cluster of seven international agreements to cooperate in various areas of technology. The agreements defined by example what COST was to do, but there was never a statement of over-riding objectives. “In 1970-71 COST became the first instrument of European science policy, but completely lacked any strategy.”69 At the time, COST was seen as potentially fulfilling the task later taken on by the Community’s Framework Programme – extending to research funding, not just networking as in COST’s current form but the idea that COST could set European science policy disappeared as the European Commission gradually positioned itself to take on this role.

From the late 1980s, the composition of COST appears to have begun to change. It began increasingly to include Actions that were more scientific in character, notably in areas such as Chemistry. Not all disciplines were represented in this change – for example, the physics community famously tends to turn to ESF for support in networking. We have not been able to discover any decision that lay behind this ‘mission drift’.

Evaluations of the work of COST are consistently positive. The 1997 evaluation70 concludes that “COST has been a valuable instrument for fostering scientific and technological collaboration. It is quite distinctive in the panoply of European research.” The 2005 evaluation (based on peer review interviews and surveys) of the COST Domains in Physics, Fluid Dynamics, Chemistry and Materials71 from 1998 onwards said, “The key conclusions are very positive. The quality of the science is generally very high and the networks normally engage some of the best researchers in

67 University Bocconi, Networks of Innovation in Information Society Development and Deployment in Europe, 2005

68 Jean-Jacques Servan-Schreiber, Le Défi Américain, Paris: Denoel, 1967 69 Jean-Luc Roland, A Review of COST Cooperation, Brussels: Commission of the European Communities,

1988 70 PREST, CSIC/IESA, ESSOR, ISI and NUTEK, Cost Evaluation, (mimeo), 1997 71 Report of a Panel Chaired by Juan Rojo, Review of COST Domains in the Chemical and Physical

Sciences, Brussels: COST, 2004


their respective fields. The networking dimension of COST adds considerable value over and above existing relationships, and helps account for an important body of collaborative outputs.” A similar review of Agriculture, Biotechnology, Food Sciences, Forestry and Forest Products, Environment and Meteorology72 was equally positive.

In 2006, the panel reviewing Transport and Civil Engineering found the Actions useful and largely well performed but also took exception to assumptions in its brief that COST was only a scientific endeavour – very much speaking for the kind of areas on which COST originally focused73

The Transport and Urban Civil Engineering domains are very applied. COST Actions in these areas are not Science but provide important types of support to the research community. They need to be judged in these terms; both in this review and in the way COST assesses and manages them more broadly.

Accordingly, COST Actions in Transport and Urban Civil Engineering have little impact on basic research; nor should they be expected to do so. Since they are problem orientated, the Actions tend to be interdisciplinary. They are generally policy relevant and have some influence on the setting of norms and standards but their impacts would be greater given better dissemination of results. Some have been precursors of product, process and service developments.

COST has been subject to a series of more or less existential evaluations since the start of this decade. The Busch report74 urged the Commission to continue funding COST and said that outsourcing COST should involve greater decentralisation and professionalisation, including a more rigorous approach to assessing the quality of potential Actions. The Kneucker panel noted significant improvements in operating efficiency and responsiveness of COST75. The final review of COST after the end of FP6 found that significant improvements in processes and effectiveness had been made, and that it was important that these should be maintained.76.

The content of the COST Actions in this recent period remains largely use-orientated. However, the ‘distance to implementation’ has increased. For example, there is now little activity that is explicitly aimed at standardisation but a greater proportion of Actions that generate knowledge that could be input to later pre-normalisation or applications work. There is also a minority of Actions that appear to be strongly science-driven. Thus, the character of the work funded by COST appears to have been affected by ‘mission drift’ from applications-focused work of direct interest to government to use-orientated work with a longer time perspective.

These evaluations of COST domains show that COST only to a very small degree can substitute for FP activities. Rather, they tend to pursue research networking for its own sake or in order to build alliances that subsequently propose projects to the FP.

72 Report of a Panel Chaired by Morten Carlsson, Review of COST Domains in Agriculture, Biotechnology, Food Sciences, Forestry and Forest Products, Environment and Meteorology, Brussels: COST, 2005

73 Report of a Panel Chaired by Tony Ridley, Review of the COST Domains in Transport and Urban Civil Engineering, Brussels: COST, 2005

74 Neils Busch, David Coates, Reinhard Loosch and Luis Sanz Menendez, An Assessment of COST for the COST Ministerial Conference, Brussels: COST, 2003

75 Raoul Knecker, Isabelle Collins, Niels Busch, Helena Illnerová, Lucien Laubier and Georges Wanet, COST Mid-Term Review, Brussels: European Commission, 2005

76 Jeanne Monfret, Pim Fenger, Heikki Kotilainen, Toivo Maimets, Maria Luz Peñacoba and Patries Boekholt, Final Review of COST in the Sixth Framework Programme, Brussels: European Commission, 2007


RAND Europe has shown77 that FP6 participants are likely also to participate in COST and ESF networks but that FP6 networks are more inclusive than these. RAND’s comparison of FP6-IST networks with those in other European collaboration networks show that the FP tends to be more multi-disciplinary, to include the New Member States, to include patent holders and highly-cited organisations and to include SMEs than other European mechanisms, which tend to be either academic or industrial in nature and to focus on narrower questions.

7. Conclusions

This section draws conclusions first about FP6 and second about the process of evaluating the Framework Programme during FP6 and into the future.

7.1 About FP6

FP6 was the first Framework Programme with clear policy intent: namely, the creation of a European Research Area. But it also had many other purposes, including integrating the R&D communities of the New Member States into the wider Europe, in which it has performed well.

It is clear from the Framework-wide evidence that the FP does a solid job in its traditional ‘core’ – that two thirds or so of the activity that is pre-competitive, collaborative R&D and that tends to roll on from programme to programme. It increases industrial competitiveness, builds networks, creates ‘intermediate knowledge outputs’ that can be exploited in future R&D and, to a considerable extent it also leads to commercialisable outputs. The first longitudinal study spanning several Frameworks shows clearly that there are longer-term paths from ‘intermediate knowledge outputs’ that lead to new products and processes, and that at least in the Swedish case these have proved crucial to competitiveness.

FP6 clearly comprised high-quality research and involved members of the international research elite. The bibliometric evidence, the intensity of competition and the transparency of the assessment process clearly demonstrate that past insinuations by some members of the research community that the Framework was where the ‘B-team’ went to play are wrong.

The ‘new instruments’ of FP6 did not live up to expectations. The idea that the FP6 could ‘structure’ the European Research Area was over-ambitious. This will remain the case, at least as long as it is not accepted that there are losers as well as winners in the process of restructuring of capabilities and institutions implied in the more radical interpretations of the ERA concept. If Europe is to become a common market for knowledge and build the critical masses needed to improve its global research competitiveness, there has to be more specialisation. Not everyone can research into everything. Strategic choices have to be made in national research policy as well as at the European level and measures are needed – not further to network the already highly networked research community in Europe but to induce disequilibrium and therefore structural change.

It is also clear that the FP will not be effective in some areas (eg energy) unless there are moves to ‘join up’ research, innovation and other policies. Horizontal coordination is difficult in all state apparatus, and the Commission is no exception to this rule. The problem nonetheless has to be addressed.

77 Caroline Wagner, Jonathan Cave, Tom Tesch, Vera Alee, Robert Thomson, Loet Leydesdorff and Maarten Botterman, ERAnets: Evaluation of Networks of Collaboration Among Participants in IST Research and their Evolution to Collaborations in the European Research Area (ERA), Report to DG-Infso, Leiden: RAND Europe, 2005


A major policy success of FP6 was the ERANETs, which – together with the Open Method of Coordination (OMC) – have persuaded the member states of the value of joint programming. The 2007 Green Paper on the ERA reflects this. It is a ‘relaunch’ of the ERA idea but is bolder than the 2000 Communication because the idea of ERA has in the meantime become more credible, even if its realisation had by 2007 been minimal. FP6 also introduced the important new principle of ‘variable geometry’ in EU R&D policy – giving legitimacy to sub-sets of the member states working together by providing subsidy from the EU level (in the ERANET and Article 169 schemes).

The ERA goal has clearly shifted focus of the Framework further onto research, but the longer-term trend of declining industrial participation is worrying. The ‘mission drift’ of COST described earlier points in the same direction. A weakness of the complex and decentralised way the Framework Programme is assembled is that there is no easy way to debate or make a policy decision about this kind of balance. Good governance would require that these issues could be debated. Such debate needs also to consider the evolving division of labour between the EU and the national level, which is changing both as a result of the ERC and because of the new large-scale interventions that began with the Technology Platforms.

The newer FP arrangements like the Technology Platforms are moving it towards increasing use of self-organising communities to define agendas. There is evidence that this is likely to reinforce the existing strengths of the Framework Programme. While the design process of the Framework is opaque (and should be made less so), the Framework is fundamentally a consensus mechanism. Many of its most powerful successes are in areas where stakeholders (for example, in the ICT, automotive and aerospace industries) can offer the Commission explicit road maps and descriptions of the research trajectories they believe they should themselves pursue. This forms a virtuous circle of consensus building, planning and implementation that has significant and positive effects on research and competitiveness. The idea has been extended in FP7 via JTIs, Article 169 arrangements and so on.

However, the dark side of these arrangements is that they risk becoming clubs for the powerful incumbents, harking back to the strong European tradition of cartel building, with all its implications for declining competitiveness, lock-in and ignoring new and potentially disruptive ideas. There is an urgent need for debate about this because – despite its strengths – using a self-organising principle as the dominant way to define coming FPs involves important challenges

• Designing robust and accountable governance of systems where beneficiaries increasingly give FP money to each other

• Maintaining a coherent overall strategy

• Ensuring adequate access for small players (institutions, countries, stakeholder groups) with good ideas

• Transparency of programme design, allowing it to be debated and considered in rational terms

• Combating lock-ins

All experience of design in the mix of policies for research and innovation speaks of the need to mix bottom-up and top-down approaches. In the case of the FP, it would be reasonable to mix self-organised approaches with top-down organisation of responses to key challenges such as climate change and ageing and with a third, bottom-up but scientifically driven component. This third one may best be protected from the changes in the other two by ring-fencing its funding and moving it to an external agent, such as a strengthened ESF. For the corollary of emerging stakeholder monopolies in parts of the Framework is the monopoly of thought that we risk if a single organisation designs and manages a continent-wide R&D monoculture.

FP6 contained a small precursor to the European Research Council, in the form of the NEST programme. This disappeared during the Framework without explanation or


evaluation. But the trajectory from funding industrially relevant networked R&D to Principal Investigator led scientific research involves a big change. Given the continually increasing share of the Framework resources that has for some time gone to the academic community, it suggests we are moving over time towards the US model – where fundamental research is funded at the national level and where innovation is tackled by the member states. This would be fine if we believed that Europe did not have an industrial competitiveness problem. However, industrial competitiveness rightly remains the central concern of the Framework. It may be that the newer self-organising instruments prove to be a remedy for this academic drift – we have no evidence about that from FP6.

It is impossible to discuss the Framework Programme without touching on the administrative burdens it imposes upon participants. Indeed, such complaints are deeply embedded in FP evaluation rituals. Improvements will not come from further tinkering. A good deal of useful tinkering has already been done. Significantly reducing the burdens will require a more radical approach: namely, to move the basis for funding projects from trying to understand costs in detail and towards a grants basis. This requires radical overhaul of the financial regulations – a non-trivial task.

Gender issues are not much treated in FP evaluation. However, the evidence is that women are less well represented among FP participants than in the European research community as a whole. Clearly his is a waste of talent that should be rectified.

FP6 demonstrated that there are benefits for Europe in opening up the Framework to the wider world. In Europe, we have been over-excited for too long at our success in making a Union and we have failed to engage sufficiently with the developing global knowledge superpowers – not only China but also India, Brazil and new generations of industrialising countries. FP6 had no real strategy for tackling globalisation. While there have been timid moves since then, the EU still does not have a differentiated strategy to tackle its research and innovation relationships with the rest of the world, despite the glaringly obvious fact that a single strategy is not adequate to tackle the different needs and opportunities in, say, China, the USA and India.

7.2 About FP Evaluation

Evaluating the FP is extremely difficult – both because of its immense size and because, in reality, it is more of a framework than a programme. It has multiple purposes and does many different things. There is no single, over-arching design. This means that while its component parts need to be consistent with the high-level goals set out in the legislation, it is not possible to trace a unitary logic that can explains how and why, if we do all the component arts, this will add up to the overall objectives. As we said in our earlier meta-evaluation, the ‘middle’ of the logic is missing. Moe attention to this would improve the Framework Programme’s design – and, for good measure, make its evaluation more tractable.

A key methodological innovation in recent years has been the use of Social Network Analysis (SNA) in FP evaluation. Despite several studies now having been undertaken, however, this is telling us little new about the Framework. It confirms that there are major ‘nodal’ players who persist over long periods in the FP and has (somewhat tautologically) confirmed that the bigger instruments of FP6 involved larger networks of collaborators (which was, after all, the intention). But we still miss any activity that connects macro-level networking patterns with strategy, motivation and behaviour. Without such exploration, it is difficult to see how SNA can fulfil its promise.

Perhaps the most interesting methodological innovation in practice has been the Swedish interest in longitudinal study of the impacts of the framework on higher education and a range of industrial sectors. This has inspired other activity at member state level, but it will be useful to deepen our understanding through more longitudinal and sectoral study at the level of the overall programme and of the EU. Some indications about the comparative lack of impact of the FP in parts of the life sciences suggest potentially that its collaborative paradigm (essentially derived from


behaviour that is useful in many electrical and mechanical industries) is less suited to the life sciences and, especially, to the long time constants and high secrecy imposed by drug approvals regimes. This issue has major policy implications and should be explored.

It remains a little surprising that no significant modelling or econometric approaches have been attempted.

The growing complexity of the FPs has not adequately been reflected in all the evaluations. The Commission has done well to identify the need to evaluate the new instruments and has launched several studies in order to do so. But in many cases, especially where surveys are used, there is a tendency not to segment responses by instrument type and therefore potentially to obscure important differences among them and their respective participants.

The evidence base available to the FP6 evaluation is a big improvement on anything that has gone before. However, the size and complexity of the FP means that a very large number was studies was needed, and this produced constraints both on the supply and the demand side. The Commission struggled (not wholly successfully) to define and launch all the studies needed in a timely way. And the resulting spike put the R&D evaluation community under considerable capacity pressure. Both argue for a more continuous or rolling evaluation process, perhaps exploiting the huge continuities between Frameworks.

Technopolis Ltd 3 Pavilion Buildings Brighton BN1 1EE UK T +44 1273 204320 F +44 1273 747299 E [email protected] www.technopolis-group.com

Framework Programme 6 - European Commission...Framework needed more systematic planning, clearer...

Documents

Transcript of Framework Programme 6 - European Commission...Framework needed more systematic planning, clearer...