Bachtler - Evaluation of European Union Cohesion Policy

Introduction

Evaluation of European Union Cohesion Policy:Research Questions and Policy Challenges

JOHN BACHTLER� and COLIN WREN†�European Policies Research Centre, University of Strathclyde, 40 George Street, Glasgow G1 1QE, UK. Email:

[email protected]†Economics, Business School, University of Newcastle upon Tyne, Newcastle upon Tyne NE1 7RU, UK. Email:

[email protected]

INTRODUCTION

During the past 15 years, the Cohesion policy of theEuropean Union (EU) has become one of the mostintensively evaluated policies in Europe.1 Since thereform of the Structural Funds in 1988, successivephases of regulation have created an increasinglyrigorous system of appraisal, monitoring and evaluationcovering all EU-funded regional development inter-ventions. Evaluation of Structural and CohesionFunds programmes has to be conducted at definedpoints in the programming cycle: ex-ante to verifytargets; at the mid-point to establish the need for cor-rective action; and ex-post to assess outcomes. Thesenational and regional evaluations are complementedby meso- and meta-evaluation studies and by thematicevaluations by European Commission services, and byextensive Commission-sponsored research and debateon evaluation concepts, methods and practices.

The growing importance accorded to the evaluationof EU Cohesion policy is part of a broader international‘boom’ in policy and programme evaluation, reflectinga perceived need to legitimize and justify governmentintervention–albeit with major differences betweencountries (POLLITT, 1998; FURUBO et al., 2002). Inthe context of the EU, two factors help to explain thetrend. First, the Cohesion policy of the EU hasbecome, in budgetary terms, the most significant areaof Community policy. In the mid-1980s, the EuropeanRegional Development Fund accounted for only 7.5%of the Community budget (ECU2.3 million in 1985prices) (MICHIE and FITZGERALD, 1997) and wasoften dismissed as a ‘side payment’ (MARKS, 1992;POLLACK, 1995; ALLEN, 1996). In the 2007–13period, by contrast, the Structural and CohesionFunds will comprise 36% of planned Communityspending (E308 billion in 2004 prices) (EUROPEAN

COUNCIL, 2005). Second, the 1988 reform of theStructural Funds gave the European Commissionmuch greater influence on the distribution of regionaldevelopment funding, in particular with respect to thedesignation of eligible areas, the approval of MemberState development plans, the management and deliveryof programmes, and the control of expenditure. Thisinfluence has often been a source of tension betweenthe Commission and Member State governments,with protracted arguments over the spatial and thematicallocation of funding (BACHTLER and TUROK, 1997;BACHTLER and WISHLADE, 2004, 2005).

The combination of large amounts of expenditureand contested decisions on its usage have led to pressurefor more accountability in spending, which is reflectedin the creation of a steadily more extensive EU evalu-ation regime, as well as more stringent obligations forthe monitoring, financial management and auditing ofexpenditure. However, Cohesion policy evaluationhas had to serve the objectives of many different organi-zations, ranging from programme managers and part-ners through regional and national governmentauthorities to the various European institutions–theEuropean Council, the Commission, the European Par-liament and the European Court of Auditors–each withits own motives and interests in the results andimplementation of EU spending.

Further complexity arises from the nature of thepolicy being evaluated. Structural and Cohesion Fundsprogrammes are implemented under a common regulat-ory framework, but in widely differing national andregional circumstances with varied institutional arrange-ments for managing and delivering regional develop-ment policy. Programmes comprise a range ofinterventions–targeting physical infrastructure, econo-mic infrastructure, business development, humanresources, research, technological development and

Regional Studies, Vol. 40.2, pp. 143–153, April 2006

0034-3404 print/1360-0591 online/06/020143-11 # 2006 Regional Studies Association DOI: 10.1080/00343400600600454http://www.regional-studies-assoc.ac.uk

innovation, environmental improvement, tourism andcommunity development–through a mix of financialinstruments and many different types of beneficiary.Also, EU support has to be co-financed with nationalpublic or private funding that may originate in severaldifferent organizations or schemes.

With the greater importance attached to evaluation,the methodology for evaluating EU Cohesion policyitself has come under scrutiny and been subject toconsiderable debate. This is not surprising given themuch increased scale and role of Cohesion policy, butit is clear that there is no uniformity of approach.Current evaluation methodologies range from the‘bottom-up’, survey-based assessments of project andbeneficiary outcomes to the ‘top-down’, input–output models of aggregate programme impact, as wellas process studies of Structural Funds implementation.In addition, there are differences in the kinds of datacollected, the sophistication of application and thekinds of question addressed. On the one hand, this isto be expected, given the diversity of policy instrumentsand purpose, and the differing institutional arrange-ments for management and implementation. It couldeven be viewed as a strength of EU Cohesion policyevaluation. On the other hand, it makes the evaluationof the policy appear uncoordinated, and raises issuesabout the comparability and consistency of results.

In fact, the results reported for the effects of EUCohesion policy in the periodic reports of the Commis-sion claim substantial levels of job creation, investmentand other outputs attributable to the policy (EUROPEAN

COMMISSION, 1996, 2001, 2004). However, the poorquality of monitoring data on which many evaluationshave had to rely, the difficulty of isolating effectsattributable to EU funding, and the other metho-dological problems and limitations highlighted abovemean that some of the reported results have beentreated with scepticism, in particular the degree towhich national or regional convergence is attributableto EU Cohesion policy (EDERVEEN et al., 2002;TARSCHYS, 2003; SAPIR et al., 2004).2

Apart from disputes over the outcomes and impactsof Structural and Cohesion Funds programmes, therehas also been debate about the less-tangible effects ofEU Cohesion policy, referred to as Community‘added value’. Under this, it has been argued that theregulatory obligations, combined with the role of theCommission in promoting ‘good practice’ in evalu-ation, have encouraged a greater commitment to, anduse of, evaluation for the effective management of pro-grammes (EUROPEAN COMMISSION, 2002, 2004),although, again, the influence of the Structural Fundsin this area has been questioned (ECOTEC, 2003).More broadly, evidence has been presented on thegrowing emphasis of Cohesion policy evaluationbeing carried over into national regional policy-making, with a more widespread evaluation of domesticinterventions (BACHTLER and TAYLOR, 2003).

As the third programme period since 1988 draws to aclose, this is a timely opportunity to take stock of theevaluation of EU Cohesion policy, to consider whathas been learned, and to identify research questionsand policy issues for the future. This special issue ofRegional Studies aims to provide an overview of thecurrent state of EU Cohesion policy evaluation. Thecontributors are researchers from different academicdisciplines and policy-makers drawn from across theEU. To cover the variety of evaluative approaches, theinstruments and circumstances in which Cohesionpolicy is applied, as well as different perspectives, thespecial issue is divided into four parts, some of whichare cross-cutting in nature.

The first part (Leonardi; Mairate) provides overviewsfrom different perspectives on the rationale for EUCohesion policy and its effects, including ‘addedvalue’. The second part (Batterbury; Bradley; Martinand Tyler) provides a critical assessment of Cohesionpolicy evaluation, focusing on the current regulatoryframework. It examines contrasting macro-basedapproaches for determining the effect of policy, com-prising a model-based approach to ex-ante evaluate theStructural Funds and a shift-share, residual-basedmethod to evaluate interventions in Objective 1regions. This part offers a new estimate of the employ-ment effect of Cohesion policy. The third part (Florio;Basle; Blazek and Vozab; Eser and Nussmueller;Armstrong and Wells) examines the evaluation experi-ence in a variety of circumstances, including differentpolicy measures (Structural Funds, CommunityInitiatives and Cohesion Fund) and different kinds ofevaluation (ex-ante, mid-term and ex-post).3 It exploresother issues such as the use of cost–benefit analysis,the evaluation of community economic development(CED) initiatives and the preparation of programmingdocuments in a new Member State. Finally, the fourthpart has shorter contributions (Barca; Huber; Jakoby;Raines) that take a policy-maker perspective on evalu-ation culture in a variety of EU regions and countries.

As a starting point for the special issue, the presentpaper provides an introductory discussion of the mainissues, setting the context for the subsequent papersand highlighting some of the main issues being addressedby contributors to this issue. It begins by tracing theevolution of Cohesion policy evaluation from 1988 tothe present. It then discusses the concepts and methodsof evaluation, the credibility of the results obtained,and the organizational and cultural differences in evalu-ation practice across the EU. It concludes with somequestions on the way forward for evaluation.

EVALUATION OF EUROPEAN UNION

COHESION POLICY

The current approach to EU Cohesion policy evalu-ation originated in the reform of the Structural Funds

144 John Bachtler and Colin Wren

in 1988. Before the reform, the monitoring andevaluation of regional policy spending had beenaccorded a low priority and was subject to disagreementbetween the Commission and Member States over thenecessity for evaluation, and how it should it be orga-nized and conducted (BACHTLER and MICHIE, 1995).Evaluation techniques and capabilities varied greatlybetween Member States, reflecting different policyand public management traditions and practices(MEANS, 1993).

The 1988 Regulations introduced a structuredapproach to the monitoring and evaluation of theFunds. They required partnership-based MonitoringCommittees to be established for each CommunitySupport Framework (CSF) and programme, and struc-tural operations to be evaluated ex-ante and ex-post toestablish their effectiveness with respect to economicand social cohesion, as well as the impact of CSFs andindividual operations (EUROPEAN COUNCIL, 1988).In the subsequent programme period (1989–93), thebasic structures for monitoring and evaluation wereestablished, but compliance with the requirements forprior appraisal was patchy and often ineffective(EUROPEAN COMMISSION, 1990, 1993). ManyMember States were either unwilling or unable toprovide projections on the outcomes of Communityassistance, leading to the Commission undertaking itsown evaluations to establish the likely effects of policy(EUROPEAN COMMISSION, 1992). Criticisms of theinadequacies of monitoring and evaluation were madeby the Council, Parliament and Court of Auditors(BACHTLER and MICHIE, 1995).

In 1993, the Regulations were revised for the 1994–99 programme period. They strengthened the monitor-ing and ex-ante and ex-post evaluation obligations of theMember States and stipulated specific requirements forthe quantification of programme objectives, projectionsof impact and identifying the outcomes of previous EU-funded operations (EUROPEAN COUNCIL, 1993). Anamendment to the Regulations also introduced therequirement for an ‘interim evaluation’ to beundertaken after 3 years to provide a critical analysisof progress and to allow adjustments to be made.

The Commission used these regulatory changes toexert a powerful influence on evaluation practice inthe Community, in four main ways. First, the Commis-sion created specialist evaluation units within itsDirectorates-General to coordinate evaluation andensure that evaluation requirements were met by theMember States. Second, as part of the programmingprocess for the 1994–99 programmes, the Commissionobtained an independent ex-ante appraisal of each devel-opment plan submitted by the Member States in orderto strengthen its influence in the negotiation andapproval of plans (involving, inter alia, pressuringMember States to provide better and quantified indi-cators to permit monitoring). Third, the Commissionsignificantly increased its own evaluation activity.

Apart from ex-ante appraisals, it undertook ex-postevaluations of the 1989–93 period, as well as a seriesof thematic evaluations on subjects such as partnership,research technological development and innovation(RTDI), small- and medium-sized enterprises (SMEs),transport and financial engineering. The COURT OF

AUDITORS (1998) estimated that expenditure on evalu-ation by the Directorate-General for Regional Policy(DG XVI) increased from E9.96 million in 1989–93to E25.29 million in 1994–99. Lastly, the Commissioninvested heavily in evaluation research through theEvaluation Methods for Actions of Structural Nature(MEANS) programme, which undertook wide-ranging analyses of evaluation practices across the Com-munity, drafted guidance and evaluation manuals, andprovided support for evaluators and evaluation commis-sioners, such as through ‘exchange of experience’ foraand evaluation conferences (BACHTLER and MICHIE,1995; EUROPEAN COMMISSION, 1999).

A further reform of the Structural Funds in 1999introduced new Regulations for the 2000–06 period.With the aim of simplifying the implementation ofthe Funds, the Regulations decentralized severalaspects of programme management to the MemberStates, including responsibility for monitoring andevaluation. The requirement to evaluate programmesex-ante, at the mid-term point and ex-post was main-tained, and in some respects was strengthened (EURO-

PEAN COUNCIL, 1999). The content of the ex-anteevaluation was more clearly specified, with specialattention given to environmental, labour market andequality issues. In an effort to improve the quality ofmonitoring information, the Commission producedextensive guidance on ex-ante evaluation and monitor-ing indicators. Further, the results of the mid-termevaluation (MTE) were to be used for a mid-termreview of programmes and a ‘performance reserve’(constituting 4% of programme allocations) held backfor allocation to the ‘best-performing’ programmesmid-way during the programme period. The MTEwas to be updated a year later to check that the evalu-ation recommendations had been implemented and toprovide strategic input to the preparation of the2007–13 programmes. The Commission also contin-ued to invest in evaluation studies and research, withfurther conferences and an updated guide on evaluationdesign, implementation, methods and techniques(TAVISTOCK INSTITUTE with GHK and IRS, 2003).

The current situation is that the regulatory obli-gations for evaluating EU Cohesion policy comprisethree main stages of evaluation: ex-ante, mid-term andex-post:

. Ex-ante evaluation–or, perhaps more appropriately,appraisal–has to be carried out as part of the pro-gramme planning phase. It involves a structuredassessment of the social and economic situation inthe programme area, with particular attention being

Evaluation of EU Cohesion Policy 145

given to the environmental situation and equalopportunities, the expected impact of the proposedmeasures, and an analysis of the relevance of the pro-posed implementation and monitoring arrangements.In the 1999–2000 programming phase, this was gen-erally carried out by independent evaluators, either asa separate exercise or integrated within the pro-gramme development process. In previous pro-gramme periods, an independent ex-ante evaluationwas undertaken by the Commission, but in thecurrent period, this responsibility was left to theMember States.

. Mid-term evaluation, as the name suggests, is carriedout at the mid-point of the programme. In the caseof the 2000–06 period, the MTEs were mostlyundertaken during 2003. Conducted by independentevaluators, the main purpose is to establish whetherthe programme strategy is still relevant in the lightof economic and social trends, and to assess theinitial results of assistance and the extent to whichtargets have been attained. In some respects, this canbe regarded as the most valuable part of the evaluationsystem, since it enables the programme to be adjustedin the light of experience.

. Ex-post evaluation is the final stage in the evaluationprocess. Whereas the other two stages are the respon-sibility of the Member States, ex-post evaluation is theduty of the Commission, in collaboration withMember States. Carried out by independent evalua-tors, its remit is to examine the use of resources,and the effectiveness and efficiency of the assistanceand its impact, with a view to drawing conclusionsregarding policy and economic and social cohesion.These studies form the basis for the periodicCohesion Reports published by the EuropeanCommission.4 Although comprising the most com-prehensive and detailed assessment of the outcomeof EU Cohesion policy, ex-post evaluations areclearly limited by problems of data availability andreliability, and methodological difficulties in isolatingthe effects of EU support from other influences,especially in areas where Structural Funds play arelatively small role.

Reflecting on the evolution of Cohesion policyevaluation since 1988, it is possible to identify severalimportant trends and changes (BACHTLER, 2001;TAYLOR et al., 2001; TAVISTOCK INSTITUTE withGHK and IRS, 2003). First, evaluation has been fullyintegrated into the programme cycle from the start.Driven by regulatory changes and the influence of theCommission, programme management authoritieshave increased the range and quality of monitoring indi-cators, benchmarks and targets, as well as monitoringsystems and organizational arrangements to delivermore reliable data. More evaluation information isbecoming available and is being used for the planningof EU-funded interventions, through the

implementation phase of programmes, to the ex-postevaluation of outcomes.

Second, evaluation has evolved ‘from externallyimposed evaluation obligations to internally drivendemand for evaluation coming from programme man-agers and policy makers themselves’ (TAVISTOCK INSTI-

TUTE with GHK and IRS, 2003, p. 12). Instead ofbeing regarded solely as an administrative requirementfor accountability purposes, evaluation is becomingmore widely regarded as a management tool that cansupport the delivery of programmes and contribute toknowledge management.

Third, over consecutive programme periods, thegradual acceptance of the partnership principle and itsapplication to all stages of programme managementhave extended the range of partner organizationsinvolved in the design and implementation of evalu-ations. Evaluation studies have also served as a com-munication tool to engage local authorities, economicand social partners, voluntary and community groups,and environmental bodies in the design and manage-ment of programmes.

Notwithstanding these trends, there are stillimportant research questions and policy challenges con-cerning the future development of EU Cohesion policyevaluation. Some of these are measurement issuesrelated to the deficiencies of evaluation data thatcould be resolved through changes in monitoringpractices. There are also important debates about thephilosophical approaches, rationales and methods ofCohesion policy evaluation. Other issues are intract-able, deriving from the design of EU Cohesion policy,the different interests of European institutions andMember State governments, national evaluation cul-tures and resource constraints on evaluation. Theseissues are addressed below.

EVALUATION CONCEPTS AND

METHODS: POSITIVISM

VERSUS REALISM

At the heart of many of the debates and conflicts aboutthe evaluation process are the differing philosophicaltraditions underpinning evaluation. The Commission-sponsored guide distinguishes between three maintraditions (TAVISTOCK INSTITUTE with GHK andIRS, 2003): the classical, positivist approach based onthe premise that objective knowledge can be obtainedthrough observation; the more recent realist approachto evaluation that explores the mechanisms thataccount for changes in policies and programmesthrough social enquiry among practitioners; and con-structivism, which rejects objective knowledge andinvolves evaluation through orchestrated interactionwith stakeholders to understand the different views,values and interdependencies. These traditions, whichare discussed in several papers in this volume


(Batterbury; Eser and Nussmueller; Armstrong andWells), underlie the different purposes and methods ofCohesion policy evaluation.

The longest-standing purpose of evaluation ofCohesion policy is accountability, with a view tomaking judgements on the economy, efficiency andeffectiveness of Structural Funds. Adopting a positivistapproach, so-called ‘summative’ or ‘allocative’ evalu-ations have been conducted with a view to justifyingthe use of Cohesion policy expenditure (to theCouncil, Parliament and Court of Auditors), measuringwhat has been achieved in relation to targets, and deter-mining the appropriateness of the policy and specifictypes of intervention (in particular, for the Commis-sion). For the larger programmes (in the Cohesioncountries and major Objective programmes), macro-economic models have been used for ex-postevaluation–notably HERMIN (see Bradley in thisissue; ESRI, 2002), PARADISE (BLAAS, 1991),QUEST (ROGER, 1996) and dynamic input–outputanalysis (BEUTEL, 1997). They have been utilized forex-ante purposes in making projections of impact forthe EU-15 for the 2000–06 period (BEUTEL, 2002)or in the new Member States in 2007–13 (BRADLEY

and MORGENROTH, 2004) to establish impacts onGross Domestic Product (GDP) growth and employ-ment. Techniques involving the use of control groupsand other statistical methods have also been applied,but much more rarely (EUROPEAN COMMISSION,1999). In smaller programmes, a range of bottom-uptechniques has traced the micro-economic conse-quences of programme measures and activitiesthrough a combination of programme monitoringdata, surveys of beneficiaries and regional or subregio-nal statistics, principally to capture employment effects(EPRC and FRASER OF ALLANDER, 2000).

These methods involve a range of methodologicaland measurement difficulties. In the case of Objective1, Bradley concludes that macro-models are potentiallycapable of extracting the pure Structural Funds policyimpacts from the background of all the other domesticand external shocks that affect the economy at the sametime. ‘However, such results only provide imprecise,ball-park impact estimates, and the challenge remainsof reconciling bottom-up micro-analysis with top-down macro-analysis’ (p. 189). A key constraint is thequality of available data, which according to Martinand Tyler ‘varies considerably from being very poor to(at best) barely adequate’ (p. 203).

These problems increase in the evaluation of inter-ventions outside Objective 1. Based on experience ofex-post evaluation of Objective 2, Basle argues that itis near-impossible to obtain ‘a reliable and credibleassessment of impact, and a true legitimization for theEU interventions’ (p. 227), particularly because of themultiplicity of actors, the difficulty of isolating causeand effect, and the locality of actions and areas. Estab-lishing effects in such contexts would require a better

‘logical diagram of impact’ (i.e. a theory of cause andeffect), greater knowledge of the key factors determin-ing success and failure, and better regional quantitativeand qualitative data from improved management andmonitoring.

The emphasis on measuring impact and perform-ance, using such methods, is criticized by Batterburyas missing the main question of ‘why things work (ornot) in specific contexts’ (p. 187). Following a morerealist evaluation approach, this type of evaluation ques-tion has grown in importance since the mid-1990s.‘Formative’ evaluation techniques have been appliedto Structural Funds programmes to examine the effec-tiveness and relevance of implementation procedureswith a view to improving programme design and deliv-ery. The use of this approach, typified in MTEs and, to acertain extent, in ex-ante evaluation, has been driven bythe growing recognition of the usefulness of evaluationas a management tool and has been primarily of interestto programme managers and partners. The methodsused are mainly stakeholder interviews and case-studyresearch to assess programme management, partnership,project appraisal and selection, and monitoring arrange-ments. Over time, such methods have contributed toan important ‘learning effect’ both within and acrossprogrammes (see Mairate).

The ‘learning’ function implicit in formative tech-niques has not been without problems. Eser andNussmueller use the example of the MTE to illustratethe conflict between accountability and learning.When the MTE was included as part of the EU evalu-ation structure in the mid-1990s, insufficient consider-ation was given to the question of whether the mainfunction and purpose of the evaluation was to justifyexpenditure or to learn, and whether the role of theevaluator was to be a judge or a moderator. Suchtensions were heightened in the current programmeperiod (2000–06) with the decision to make theallocation of a performance reserve conditional on pro-gramme achievements at the mid-point of the period,to be determined on the basis of the MTE results.These problems are at least partly inherent to the‘hybrid governance’ structure of Structural Fundsimplementation which involves a mix of managementby results and ‘bureaucratic rule steering’ (LANG, 2001).

Further, some regard evaluation practice hitherto asbeing largely ‘managerialist’ with the absence of a‘democratic’ dimension that might open the process tothe wider community (TOULEMONDE, 2000). This isthe main theme of the paper by Batterbury, which con-siders the evaluation framework for Cohesion policy tobe limited to three core purposes–accountability,improved quality and performance, and improvedplanning–with other potentially useful functions neg-lected (i.e. capacity-building and learning). While thelearning function would open the possibility of carryingStructural Funds lessons into wider organizational andinstitutional systems, the Commission has not yet


engaged sufficiently with this broader approach. Other‘pluralistic’ or ‘participatory’ evaluation techniques todevelop a sense of ownership among programme par-ticipants or to empower specific groups of individualshave also been largely ignored.

Lastly, it is important to note that Cohesion policyevaluation is conducted primarily at the programme,regional or Member State levels. The paper by Floriodraws attention to the critical role of project-levelevaluation in the context of the Cohesion Fund. Thisfinances large infrastructure investments in transportand the environment, which potentially have longtime horizons, and for which cost–benefit analysis is aformal requirement. However, market prices will gen-erally be inadequate to examine the economic benefitsand costs of these projects, necessitating the calculationof special ‘shadow’ prices that reflect the real costs ofcapital and labour, and of time itself, the social discountrate. This paper raises intriguing questions about theappropriate prices, which are by no means trivial.

While project-level evaluation is an obligationfor the Cohesion Fund, Armstrong and Wells considerthat there is a need, within the Structural Funds also,to ‘bring to bear the full power of economic analysis’(p. 269) at the project level and within project designand monitoring processes. This is particularly relevantfor CED initiatives that have expanded within Struc-tural Funds programmes over the past decade, butwhich according to these authors have not been ade-quately evaluated. Armstrong and Wells see a third, con-structivist evaluation approach, within the mix ofphilosophical traditions, as having potential merit inorder to overcome the general hostility of CED organ-izations to orthodox economic evaluation.

CREDIBILITY OF EVALUATION RESULTS

As noted above, the origins of Cohesion policy evaluationlie in the results-oriented approach to Structural Fundsmanagement introduced by the 1988 reforms. Despiteshifts in the conception, methods and organization ofevaluation, the need to justify the results of policy andaccount to the Council, Parliament and Court of Audi-tors remain important objectives, especially through theex-post evaluations (BOUGAS, 2001).

As Leonardi points out, EU Cohesion policy hasprovided significant amounts of investment (5.5% ofnational GDP) in the Cohesion countries and has‘favoured the convergence of less-developed regionstowards the EU mean in terms of GDP per capitarates of annual economic growth, employment levelsand unemployment between 1988 and 1999, and there-after’ (p. 164). According to the evaluation researchreported in the Third Cohesion Report (EUROPEAN

COMMISSION, 2004), GDP in real terms at 1999 wasbetween 2.2 and 4.7% higher than it would otherwisehave been in the four EU-15 Cohesion countries:

Greece, Spain, Ireland and Portugal. The analysis ofemployment change in assisted regions conducted byMartin and Tyler also shows good evidence of a relativeturnaround in employment growth in Objective 1regions in the 1990s compared with the 1980s, i.e.the period when Structural Funds support wasincreased. They estimate employment creation in theorder of 1 million jobs at 2002.5

Nevertheless, qualification of such figures for impactis important. The paper by Bradley highlights theimportance of taking account of other factors; researchsuggests that the direct impacts of the Structural Fundsin isolation are modest, and that the real, long-termbenefits of EU Cohesion policy are associated withthe responsiveness of lagging economies to externalopportunities for trade and investment. In similarvein, research by EDERVEEN et al. (2002) found thatEuropean support as such did not improve nationalgrowth performance but that it ‘enhanced’ growth incountries with the ‘right’ institutions. With respect toother effects, BOLDRIN and CANOVA (2001, p. 242)found that ‘there is no statistical evidence that StructuralFunds have had a positive impact upon the growth ratesof either labour or total factor productivity in thepoorer regions’. Regarding the location of economicactivity, MIDELFART-KNARVIK and OVERMAN (2002)concluded that while the Structural Funds expenditurehad attracted R&D-intensive industries to the less-favoured regions, this effect had mainly acted counterto regional and national comparative advantage(except in the case of Ireland).

In the context of Objective 2 regions, Commission-sponsored research has attempted to calculate employ-ment effects associated with Structural Funds duringthe 1990s (ERNST & YOUNG, 1997; CSES, 2003).After making adjustments for ‘additionality’, displace-ment and indirect effects, Ernst & Young attributedsome 450 000–500 000 net additional jobs to theObjective 2 programmes in the 1989–93 period,while CSES estimated net job creation at 560 000 forthe 1994–99 period. However, such estimates arehedged with major reservations about the quality ofthe data, as noted above. In the UK, for example,research on the quality of monitoring data for the1994–99 period demonstrated significant deficienciesin data collection and interpretation practices and therealism of impact estimates (EPRC and FRASER

ASSOCIATES, 2002). Problems of disentangling Struc-tural Funds effects from other interventions in countrieswith small amounts of EU funding are also significant.Research in Sweden, for example, was unable to ident-ify any discernible effects of EU programmes onregional development indicators such as income perhead, population and employment (OXFORD

RESEARCH, EUROFUTURES and UMEA UNIVERSITY,2004). In addition, Basle points to the possibility of an‘announcement’ effect rather than an ‘implementation’effect, and casts some doubt on the latter without


suitable calibration of the results, i.e. cross-checking ofmicro-based results against relevant and objectivelymeasured, aggregate-level indicators.

The effects of the Structural Funds are not justidentifiable in terms of cohesion impacts. Mairatecites influences related to policy, implementationrules, learning and visibility, although varying accordingto the programme scale and type, administrative systemsand maturity. Both Mairate and Leonardi note theimportance of the multilevel governance dimension ofStructural Funds in enabling previously excludedactors (i.e. subnational authorities, socio-economicpartners, community groups) to participate in thepolicy process and for building institutional and admin-istrative capacity for economic development in theregions. Cohesion policy is also regarded as a politicalexpression of ‘solidarity’ between Europe’s wealthierand poorer regions.

DIFFERENCES IN EVALUATION

PRACTICE

Evaluation culture and capacity

There are major differences in the way that evaluationhas evolved and is used in Member States, and in parti-cular the way in which it is integrated within thepolitico-administrative culture of individual countries(TOULEMONDE, 2000). This is reflected in the differentapproaches to the evaluation of regional policy andresponses to the evaluation obligations relating toStructural and Cohesion Funds (BACHTLER, 2001).

Historically, evaluation experience in the EU-15 hasbeen presented in terms of a ‘north–south’ divide(WADLEY, 1986; TAVISTOCK INSTITUTE with GHKand IRS, 2003). In parts of Northern Europe–theNetherlands, the UK, Germany and the Nordiccountries–the evaluation of regional policy has a stron-ger tradition, with regular evaluations of nationalregional policy interventions (TOEPEL, 2000; TAYLOR

et al., 2001). By contrast, in Italy and Spain, policyevaluation has not been well established within thepublic administrative culture (OSUNA and MARQUEZ,2000; CASAVOLA and TAGLE, 2003; MAP, 2004). InFrance, the focus of research has been on the politicalprocess of producing policy rather than the results,and while the evaluation of regional policy has grownsince the mid-1990s, the ‘managerialist’ accountabilityapproach has been contested (FONTAINE andMONNIER, 2002; BARBIER, 2003).

These differences in evaluation culture are reflectedin varied evaluation capacity across the EU, such asthe availability of qualified and experienced evaluators,and the presence of appropriate knowledge and skillsamong evaluators and evaluation commissioners, andthe institutional frameworks to manage evaluation andpromote evaluation practices; a point that is made bya number of papers in this special issue. Such factors

have influenced the way in which Member Stateshave responded to the obligations to evaluate EU Cohe-sion policy. During the 1990s, countries such as Greece,Italy, Portugal and Spain restricted their evaluationactivity to complying with the accountability require-ments of the regulations, while programme manage-ment authorities in other countries (such as Ireland,Denmark and the UK) used evaluation more activelyfor improving the management of the Funds (BACH-

TLER, 2001).The development of evaluation in the new Member

States is a case study of these issues. The first experienceof Structural Funds evaluation has been the ex-anteevaluation of the 2004–06 programmes. In thecontext of the Czech Republic, as well as other newMember States, the paper by Blazek and Vozab con-cludes that the ex-ante evaluation, while effective insome respects, was constrained by a shortage of experi-ence and knowledge of practices and techniques, thelack of clarity in the roles and responsibilities of evalua-tors and programmers, and the absence of an insti-tutional structure to facilitate evaluation. The timingof the evaluation findings was also critical: unlessresults were generated relatively early in the program-ming process, they would not be taken into accountby policy-makers.

Influence of European Union evaluation practice

The last question is: What effect has EU evaluation hadon regional development policies and programmes?Previous research argues that the evaluation obligationsof EU Cohesion policy have acted as a ‘driver’ of policyand evaluation in the Member States (TAYLOR et al.,2001; TAVISTOCK INSTITUTE with GHK and IRS,2003). The final papers in this volume consider thisquestion from the perspective of policy-makers infour EU countries. In general, they suggest that theEU evaluation requirements and practice have influ-enced policy choices, enhanced the role of evaluationas part of the policy process and stimulated policylearning.

In Italy, Barca states that EU evaluation obligationshave been used as part of a conscious strategy toimprove policy verifiability and to modify policy objec-tives and instruments. It has been part of a more generalstrategy of capacity building.6 According to Jakoby, inthe North Rhine-Westphalia region of Germany,evaluation has been a catalyst for changing programmesand management systems at different stages of the policylife cycle. It has also initiated a learning process amongstakeholders in the region and encouraged an exchangeof experience at the international level.7

From an Austrian viewpoint, Huber identifies anadded value for national regional policy through theEU evaluation requirements, especially with regard toevaluation methods. However, he also sees drawbacksin the overly bureaucratic and prescriptive way the


obligations are interpreted and applied by the Commis-sion, and is of the opinion that ‘tacit knowledge’ isundervalued. Raines echoes some of these points,from the experience of EU evaluation in Scotland.While the evaluation of the Structural Funds hasplayed an important role in policy development andencouraged policy learning, as currently practised, itdoes not have sufficient flexibility to adapt to nationaland regional local circumstances, policy needs andevaluation practice.

Finally, and perhaps contrary to the above, Blazekand Vozab find that evaluation cannot influence theconceptual basis, rationale or objectives of regionaldevelopment policies, and it cannot initiate majorsystem changes. Likewise, SCHEFFEL (2001) notes theobjection of the German government authorities toevaluation that challenges the underlying policy objec-tives and expenditure choices. It illustrates the need forevaluation to operate within the prevailing policy andinstitutional paradigm, and its inability to question thetheoretical propositions and assumptions on which thedevelopment and implementation of policy are based(SANDERSON, 2000). This is a point that is reinforcedby Huber and draws attention to the potential incom-patibility of the different values, and wider socialsystems, within which politicians, administrators andevaluators operate, and the different way that theyperceive the logic of evaluation (HUMMELBRUNNER

et al., 2005).

WHICH WAY FORWARD?

Looking forward, in many respects the evaluation of EUCohesion policy is at a watershed. Over the past 18years, an extensive evaluation regime has been con-structed to account for spending on Structural andCohesion Funds, to assess the economy, effectivenessand efficiency of the policy, and to support the planning,management and delivery of programmes. This hasbeen supported by extensive investment in evaluationmethods, the promotion of ‘good practice’ and the pro-gressive creation of an ‘evaluation culture’ in mostMember States.

Notwithstanding this activity, the results of evalu-ation are not conclusive on the impact of Cohesionpolicy. With respect to the accountability objective ofevaluation, on the one hand there is clear evidence ofeconomic catching-up by poorer parts of the EU-15over a period when major EU support was provided;research suggests that, at the very least, Structural andCohesion Funds have helped to accelerate growthwhere the national policy environment was receptive.On the other hand, even in the Cohesion countries,where EU transfers account for a sizeable proportionof GDP and investment, impact estimates are con-sidered imprecise, and the effects attributable to EUfunding alone appear (according to some studies) to

be limited. As Bradley indicates, there are major meth-odological challenges to be addressed if this situation isto improve, e.g. through the quantification of pro-gramme objectives, better physical monitoring data,and the reconciliation of macro- and micro-economicmethods for impact assessment. These are challengesfor the Commission and for the research community.At present, independent (i.e. not officially commis-sioned) regional economic research on the effects ofStructural Funds is limited by comparison with researchon national policies or research in the political scienceliterature on issues such as governance and partnership.Leonardi, Bradley, and Martin and Tyler identify a seriesof gaps where independent work is urgently needed.

This is not just an academic question. The recentdebate on the EU financial perspective for 2007–13showed that spending on EU Cohesion policy isheavily contested by the ‘net contributors’, part of thecriticism being justified on the grounds of inadequatepolicy performance (EDERVEEN et al., 2002; HMTet al., 2003; TARSCHYS, 2003; SAPIR et al., 2004).The December 2005 European Council budget agree-ment includes a provision for fundamental review ofthe structure of the EU budget in 2008–09 in pre-paration for negotiations on the post-2013 budget.Major questions on spending and policy direction willalways be determined mainly by political factors, butthere is a need for better evaluation and analysis toinform decisions.

A second set of research and policy challenges is howto make better use of evaluation at programme andproject level. In this area, it is arguable that some ofthe evaluation effort is currently wasted. As Baslepoints out, considerable efforts are made to quantifyprogramme performance in regions where the Struc-tural Funds comprise a relatively small part of economicdevelopment expenditure, and where it is impossible todisaggregate the effects of different funding streams.This problem will be exacerbated in the 2007–13programme period where many Regional and Compe-titiveness programmes will have less money than atpresent and potentially spread over a larger geographicalarea. Further, as Eser and Nussmueller note, there is aninherent conflict between the accountability and ‘learn-ing’ function of evaluation.

A different approach would be to focus programme-level evaluation efforts more strongly on capacity-buildingand learning objectives (rather than accountability), asBatterbury recommends, with a view to assistingpolicy choices and the implementation of programmes.Decisions such as the approach to evaluation, the choiceof objectives and methods, and the administration andtiming of evaluation should be made within the pro-gramme partnership. This would allow for a morediverse range of methods and, as advocated by Arm-strong and Wells, should encompass more evaluationat the project level and in areas such as CED whichare not well served by standard evaluation methods.


The aim would be to gain more insight into the effec-tiveness of interventions and delivery mechanisms andto understand ‘success’ factors and preconditions. Therole of the European Commission, under such anapproach, would be to ensure that an evaluationsystem is in place and, as at present, to promoteexchange of experience.

Lastly, the institutionalization of an evaluationculture still varies greatly among Member States. AsBarca, Jakoby, Huber and Raines illustrate, EU evalu-ation practices have brought tangible improvements tothe design and delivery of EU programmes in differentparts of the EU, but there remains an asymmetry inevaluation capacity and culture (BOUGAS, 2001). Themajor institutional challenges are in the new MemberStates, as outlined by Blazek and Vozab (and, to acertain extent, in the EU-15 Cohesion countries),which need to invest in evaluation management units,and the evaluation skills and competences of future eva-luators. In other parts of the EU, the situation is differ-ent. As Structural Funds focus more on the Lisbon/Gothenburg agendas in the next programme period(BACHTLER et al., 2004), EU funding may be usedmuch more for specialized areas of support such asregional innovation networking, specialist advisorysupport, environmental sustainability, financial engineer-ing, innovative applications in the use of InformationTechnology, and specialist training. These interventionspresent particular implementation difficulties forprogramme managers and challenges for evaluators,requiring new thinking and more interchange in evalu-ation approaches and methods across policy boundaries.

Acknowledgements – The Editors thank all of the

referees of the papers, who gave freely of their time, and

whose efforts are greatly appreciated. They are also grateful

for the secretarial assistance of Lynn Ogilvie and Rhona

Walker of the European Policies Research Centre, University

of Strathclyde. In addition, thanks are extended to the Regional

Studies Editorial Board at the University of Newcastle for their

support and to the current and former Editorial Assistants at

Newcastle, Gail Welsh and Sarah Wray, who processed the

material efficiently and were able to offer excellent assistance.

The Editors would also like to thank Frederike Goss,

Tobias Gross and Carlos Mendez of the European Policies

Research Centre for their assistance in the preparation of this

introductory paper.

NOTES

1. Cohesion policy refers to the Structural Funds and

Cohesion Fund, which are the EU’s main instruments

for supporting economic and social development across

the EU at both regional and national levels. There are

four Structural Funds: the European Regional

Development Fund; the European Social Fund; the

European Agricultural Guidance and Guarantee Fund:

Guidance Section; and the Financial Instrument for

Fisheries Guidance; while the Cohesion Fund finances

major projects in environmental and transport infrastruc-

ture. The Structural Funds are currently available in pri-

ority ‘Objective’ areas (i.e. Objectives 1, 2 and 3) and in

support of Community Initiatives (i.e. EQUAL,

INTERREG III, URBAN II and LEADERþ). The

Cohesion Fund is available in less prosperous Member

States, which are currently Greece, Portugal, Spain and

the ten new Member States. For the post-2006 objectives

of Cohesion policy, see EUROPEAN COUNCIL (2005).

2. For a detailed discussion on the effectiveness and efficiency

of EU Cohesion policy as part of a debate on the report of

the Sapir Group, see the ‘mini-theme special issue’ in

Regional Studies 2005, 39(7).

3. To see how these policy measures relate to Cohesion

policy, see note 1. Cohesion policy is wide ranging, and,

inevitably, for the purpose of this special issue it was

necessary to be selective, so that some elements of

Cohesion policy are not well represented, including the

European Social Fund, agriculture, fisheries and rural

development measures.

4. The latest being the Third Cohesion Report (EUROPEAN

COMMISSION, 2004).

5. However, it points to a general problem of the compara-

bility of results when different evaluation methodologies

are used. Martin and Tyler measure the jobs in net

cumulative terms at a point in time, but which is not

directly comparable with the estimates in the Third

Cohesion Report, which measure the net jobs over time,

i.e. a programme period (EUROPEAN COMMISSION,

2004).

6. Indeed, for the current period, the Italian government

supplemented the EU performance reserve with a national

reserve to increase the pressure on programmes to

perform, and to raise the importance accorded to

evaluation.

7. This is through a benchmarking project with Scotland as

part of the mid-term evaluation.

REFERENCES

ALLEN D. (1996) Cohesion and structural adjustment, in WALLACE H. and WALLACE W. (Eds) Policy-Making in the European

Union, 3rd Edn, pp. 209–233. Oxford University Press, Oxford.

BACHTLER J. (2001) Quod erat demonstradum? The evaluation of regional policy, in ECKEY H.-F., HECHT D., JUNKERNHEINRICH M.,

KARL H., WERBECK N. and WINK R. (Eds) Ordnungspolitik als konstruktive Antwort auf wirtschaftspolitische Herausforderungen, pp.

3–20. Lucius & Lucius, Stuttgart.

BACHTLER J., GROSS T. and MCMASTER I. (2004) Delivering the Lisbon and Gothenburg Agendas. Background Paper for the

UK Presidency Conference ‘Regional and Rural Development Programmes (2007–2013): Delivering the Lisbon and

Gothenburg Agendas’, Newcastle-Gateshead, UK, 7–8 November 2005.


BACHTLER J. and MICHIE R. (1995) A new era in EU regional policy evaluation? The appraisal of the Structural Funds, Regional

Studies 29, 745–752.

BACHTLER J. and TAYLOR S. (2003) The Added Value of the Structural Funds: A Regional Perspective. IQ-Net Report, European

Policies Research Centre, University of Strathclyde, Glasgow.

BACHTLER J. and TUROK I. (1997) Conclusions: An agenda for reform, in BACHTLER J. and TUROK I. (Eds) The Coherence of EU

Regional Policy: Contrasting Perspectives on the Structural Funds, pp. 346–372. Jessica Kingsley, London.

BACHTLER J. and WISHLADE F. (2004) Searching for Consensus: The Debate on Reforming EU Cohesion Policy. European Policy

Research Papers No. 55. European Policies Research Centre, University of Strathclyde, Glasgow.

BACHTLER J. and WISHLADE F. (2005) From Building Blocks to Negotiating Boxes: The Reform of EU Cohesion Policy. European Policy

Research Papers No. 57. European Policies Research Centre, University of Strathclyde, Glasgow.

BARBIER J.-C. (2003) Devising and Using Evaluation Standards, The French Paradox. Document de travail No. 23. CNRS/Centre

d’etudes de l’emploi, Noisy Le-Grand.

BEUTEL J. (1997) Evaluation of the Macroeconomic Impact of the Structural Policies in the Cohesion Countries (Greece, Ireland, Portugal and

Spain) 1989–1999. Report to the Directorate-General for Regional Policies (Commission of the European Communities).

University of Constance, Constance.

BEUTEL J. (2002) The Economic Impact of Objective 1 Interventions for the Period 2000–2006. Final Report to the Directorate-General

for Regional Policies (Commission of the European Communities). University of Constance, Constance.

BLAAS E. W. (1991) Evaluation of Alternative Methodological Approaches for Assessing the Regional Macro-Economic Effects of the Commu-

nity’s Regional Action: The PARADISE-Model. Economic and Social Institute, Free University Amsterdam.

BOLDRIN M. and CANOVA F. (2001) Europe’s regions: income disparities and regional policies, Economic Policy 32, 207–253.

BOUGAS A. (2001) Progress and challenges in the evaluation of European structural policies, Informationen zur Raumentwicklung

6/7, 311–314.

BRADLEY J. and MORGENROTH E. (2004) A Study of the Macroeconomic Impact of the Reform of EU Cohesion Policy. Economic and

Social Research Institute, Dublin.

CASAVOLA P. and TAGLE L. (2003) Building capacity for evaluation: lessons from Italy. Paper presented to the 5th European

Conference on Evaluation of the Structural Funds, ‘Challenges for Evaluation in an Enlarged Europe’, Budapest, Hungary,

26–27 June 2003.

COURT OF AUDITORS (1998) Special Report No. 15/98 on the assessment of Structural Fund intervention for the 1989–1993

and 1994–1999 periods together with the Commission’s replies, 16.11.98, Official Journal of the European Communities, C 347/1.

CSES (2003) Ex Post Evaluation of 1994–1999 Objective 2 Programmes. Synthesis Report to the Directorate-General for Regional

Policies (Commission of the European Communities). Centre for Strategy and Evaluation Services, Sevenoaks.

ECOTEC (2003) Evaluation of the Added Value and Costs of the European Structural Funds in the UK. Final Report to the Department

of Trade and Industry and the Office of the Deputy Prime Minister. ECOTEC Research and Consulting Ltd, London.

EDERVEEN S., DE GROOT H. L. F. and NAHUIS R. (2002) Fertile Soil for the Structural Funds? A Panel Data Analysis of the Conditional

Effectiveness of European Cohesion Policy. Discussion Paper No. 10, August 2002. CPB Netherlands Bureau for Economic Policy

Analysis, The Hague.

EPRC and FRASER ASSOCIATES (2002) A Feasibility Study of an Evaluation of the Impact and Added Value of the EU Structural Funds in

the UK. Final Report to the Department of Trade and Industry. European Policies Research Centre, University of Strathclyde,

Glasgow, and Fraser Associates.

EPRC and FRASER OF ALLANDER (2000) Methodologies Used in the Evaluation of the Effectiveness of European Structural Funds: A

Comparative Assessment. Final Report to the Scottish Executive. European Policies Research Centre and the Fraser of Allander

Institute, University of Strathclyde, Glasgow.

ERNST and YOUNG (1997) Ex-Post Evaluation of the 1989–1993 Objective 2 Programmes. Evaluation and Documents Series No.

4. Directorate-General for Regional Policies, Commission of the European Communities, Brussels.

ESRI (2002) An Examination of the Ex-post Macro-economic Impacts of CSF 1994–1999 on Objective 1 Countries and Regions. Final

Report to the Directorate-General for Regional Policies (Commission of the European Communities). Economic and Social

Research Institute (ESRI), Dublin, with Gesellschaft fur Finanz- und Regionalanalysen (GEFRA).

EUROPEAN COMMISSION (1990) Annual Report on the Implementation of the Reform of the Structural Funds. Office for Official

Publications of the European Communities, Luxembourg.

EUROPEAN COMMISSION (1992) Community Structural Policies: Assessment and Outlook. COM(92) 84 final, 18.3.92. Commission

of the European Communities, Brussels.

EUROPEAN COMMISSION (1993) Third Annual Report on the Implementation of the Reform of the Structural Funds. Office for Official

Publications of the European Communities, Luxembourg.

EUROPEAN COMMISSION (1996) First Report on Economic and Social Cohesion. Office for Official Publications of the European

Communities, Luxembourg.

EUROPEAN COMMISSION (1999) The MEANS Collection: Evaluating Socio-Economic Programmes. Office for Official Publications of

the European Communities, Luxembourg.

EUROPEAN COMMISSION (2001) Unity, Solidarity, Diversity for Europe, its People and its Territory, Second Report on Economic and Social

Cohesion. Office for Official Publications of the European Communities, Luxembourg.

EUROPEAN COMMISSION (2002) Community Added Value: Definition and Evaluation Criteria. Working Paper. DG Regional Policy,

Commission of the European Communities, Brussels.

EUROPEAN COMMISSION (2004) A New Partnership for Cohesion: Convergence, Competitiveness, Cooperation, Third Report on Economic

and Social Cohesion. Office for Official Publications of the European Communities, Luxembourg.


EUROPEAN COUNCIL (1988) Council Regulation (EEC) No. 2052/88 of 24 June 1988 on the tasks of the Structural

Funds and their effectiveness and on coordination of their activities between themselves and with the operations of the

European Investment Bank and the other existing financial instruments, Official Journal of the European Communities

L185, 15.7.88.

EUROPEAN COUNCIL (1993) Council Regulation (EEC) No. 2081/93 of 20 July 1993 amending Regulation (EEC) No. 2052/88 of 24 June 1988 on the tasks of the Structural Funds and their effectiveness and on coordination of their activities between

themselves and with the operations of the European Investment Bank and the other existing financial instruments, Official

Journal of the European Communities L193, 21.7.93.

EUROPEAN COUNCIL (1999) Council Regulation (EC) No. 1260/1999 of 12 June 1999 laying down general provisions on the

Structural Funds, Official Journal of the European Communities L161, 26.6.99.

EUROPEAN COUNCIL (2005) Presidency Conclusions of the Brussels European Council, 15–16 December 2005. CONCL 3, 17

December 2005. Council of the European Union, Brussels.

FONTAINE C. and MONNIER E. (2002) Evaluation in France, in FURUBO J.-E., RIST R. C. and SANDAHL R. (Eds) International

Atlas of Evaluation, pp. 63–76. Transaction, New Brunswick, NJ.

FURUBO J.-E., RIST R. C. and SANDAHL R. (Eds) (2002) International Atlas of Evaluation. Transaction, New Brunswick, NJ.

HMT, DTI and ODPM (2003) A Modern Regional Policy for the United Kingdom, H. M. Treasury, the Department of Trade and

Industry and the Office of the Deputy Prime Minister, HMSO, London.

HUMMELBRUNNER R. with HUBERW. and ARBTER R. (2005) Process Monitoring of Impacts: Towards a New Approach to Monitor the

Implementation of Structural Funds Programmes. OAR-Regionalberatung and Austrian Federal Chancellery. Mimeo.

LANG J. (2001) Improving structural policy under conditions of hybrid governance: multi-actor constellations, integrated feedback

instruments and the use of evaluation results, Evaluation 7, 7–23.

MAP (2004) Final Report by the Commission for the Study and Creation of a State Agency for the Evaluation of the Quality of Public Services

and Policies. November. Ministerio de Administraciones Publicas, Madrid.

MARKS G. (1992) Structural policy in the European Community, in SBRAGIA A. (Ed.) Europolitics, Institutions and Policymaking in

the new European Community, pp. 191–224. Brookings Institution, Washington, DC.

MEANS (1993) Methods to Give Meaning to the Evaluation Obligation: The Conclusions of the MEANS Programme. MEANS/93/12/EN. CEPOS, Lyon.

MICHIE R. and FITZGERALD R. (1997) The evolution of the Structural Funds, in BACHTLER J. and TUROK I. (Eds) The Coherence

of EU Regional Policy: Contrasting Perspectives on the Structural Funds, pp. 14–28. Jessica Kingsley, London.

MIDELFART-KNARVIK K. H. and OVERMAN H. G. (2002) Delocation and European integration: is structural spending justified?,

Economic Policy 17(35), 323–359.

OSUNA J. L. and MARQUEZ C. (2000) Guıa para la evaluacion de polıticas publicas. Instituto de Desarollo Regional, Fundacion

Universitaria, Seville.

OXFORD RESEARCH, EUROFUTURES and UMEA UNIVERSITY (2004) Effektutvardering av de geografiske malprogrammen inom EG:s

strukturfonder. ITPS A2004:009, June.

POLLACK M. (1995) Regional actors in an intergovernmental play: the making and implementation of EC structural policy, in

RHODES C. and MAZEY S. (Eds) The State of the European Union, Vol. 3, pp. 361–390. Longman, Harlow.

POLLITT C. (1998) Evaluation in Europe: boom or bubble?, Evaluation 4, 214–224.

ROGERW. (1996) Macroeconomic evaluation of the effects of CSF with Quest II. Paper presented at the European Conference on

Evaluation Methods for Structural Funds Intervention, Berlin, Germany, 2–3 December 1996.

SANDERSON I. (2000) Evaluation in complex policy systems, Evaluation 6, 433–454.

SAPIR A., AGHION P., BERTOLA G., HELLWIG M., PISANI-FERRY J., ROSATI D., VINALS J. and WALLACE H. with BUIT M.,

NAVA M. and SMITH P. M. (2004) An Agenda for a Growing Europe: The Sapir Report. Oxford University Press, Oxford.

SCHEFFEL K. (2001) Evaluierung – Chance oder Last? Eine Betrachtung aus Bundessicht, Informationen zur Raumentwicklung 6/7,

315–316.

TARSCHYS D. (2003) Reinventing Cohesion: The Future of European Structural Policy. SIEPS Report No. 17. Swedish Institute for

Growth Studies, Stockholm.

TAVISTOCK INSTITUTE with GHK and IRS (2003) The Evaluation of Socio-Economic Development: The Guide. Tavistock Institute,

London.

TAYLOR S., BACHTLER J. and POLVERARI L. (2001) Structural Fund evaluation as a programme management tool: comparative

assessment and reflections on Germany, Informationen zur Raumentwicklung 6/7, 341–357.

TOEPEL K. (2000) Evaluation in der Regionalpolitik, Vierteljahrshefte zur Wirtschaftsforschung 69, 395–405.

TOULEMONDE J. (2000) Evaluation culture(s) in Europe: differences and convergence between national practices, Vierteljahrshefte

zur Wirtschaftsforschung 69(Heft 3/2000), 350–357.

WADLEY D. (1986) Restructuring the Regions: Analysis, Policy Model and Prognosis. Organisation for Economic Cooperation and

Development, Paris.


Bachtler - Evaluation of European Union Cohesion Policy

Documents

Transcript of Bachtler - Evaluation of European Union Cohesion Policy