3rd+Evaluation+Report

8/9/2019 3rd+Evaluation+Report

1/182

Evaluation of South East Europe

Programme 2007-2013

Final report

Client: VTI Hungarian public non-profit company on regional development & town planning

Rotterdam, Budapest, Sofia, Madrid, 25 November 2013


2/182


3/182

Evaluation of South East Europe

Programme 2007-2013

Final report

Client: VTI Hungarian Public Nonprofit Company on RegionalDevelopment and town Planning

Marie-Jose Zondag &Elvira Meurs [NL]

CsillaKarlik [HU]

IrenaVladimirova& Daniel Nigohosyan [BG]

JavierFernandez&ValentinaPatrini [ES]

Rotterdam, Budapest, Sofia, Madrid, 25 November 2013


4/182

2

II22696

About Ecorys

At Ecorys we aim to deliver real benefit to society through the work we do. We offer research,

consultancy and project management, specialising in economic, social and spatial development.

Focusing on complex market, policy and management issues we provide our clients in the public,

private and not-for-profit sectors worldwide with a unique perspective and high-value solutions.

Ecorys remarkable history spans more than 80 years. Our expertise covers economy and

competitiveness; regions, cities and real estate; energy and water; transport and mobility; social

policy, education, health and governance. We value our independence, integrity and partnerships.

Our staff comprises dedicated experts from academia and consultancy, who share best practices

both within our company and with our partners internationally.

Ecorys Netherlands has an active CSR policy and is ISO14001 certified (the international standard

for environmental management systems). Our sustainability goals translate into our company policy

and practical measures for people, planet and profit, such as using a 100% green electricity tariff,

purchasing carbon offsets for all our flights, incentivising staff to use public transport and printing onFSC or PEFC certified paper. Our actions have reduced our carbon footprint by an estimated 80%

since 2007.

ECORYS Nederland BV

Watermanweg 44

3067 GG Rotterdam

P.O. Box 4175

3006 AD Rotterdam

The Netherlands

T +31 (0)10 453 88 00

F +31 (0)10 453 07 68

E [email protected]

Registration no. 24316726

W www.ecorys.nl


5/182

Table of contents

3Evaluation of South East Europe Programme 2007-2013

Executive Summary 5

Conclusions 5

Recommendations 9

Abbreviations 11

1 Introduction 13

1.1 Context 13

1.2 The evaluation and this third and final evaluation report 14

2 Project generation and selection 15

2.1 Scope of the evaluation of the project generation and selection 15

2.2 Assessment of project generation and selection process and instruments 15

2.2.1

Project generation and selection 15

2.2.2 Assessment of supporting tools 21

2.3 Conclusions and recommendations on project generation and selection 22

2.3.1 Conclusions 22

2.3.2 Recommendations 24

3 Monitoring and implementation 25

3.1 Scope of the evaluation of monitoring and implementation 25

3.2 Performance of the programme 25

3.2.1 Summary of previous findings 25

3.2.2

Assessment of the indicator system 253.3 Geographical & Thematic scope of projects 30

3.3.1 Geographical scope 30

3.3.2 Thematic scope of the projects 35

3.4 Monitoring system 37

3.4.1 Summary of previous findings 37

3.4.2 Monitoring Wizard 38

3.5 Programme management system 38

3.6 Conclusions and recommendations on monitoring and implementation 40



4 Results of the programme 43

4.1 Scope of evaluation 43

4.2 Results of the programme 43

4.2.1 Summary of the findings of the 1st and 2nd evaluation reports 43

4.2.2 Analysis of results achieved per PA and AoI 43

4.2.3 PA 1. Facilitation of innovation and entrepreneurship 47

4.2.4 PA 2. Protection and improvement of the environment 52

4.2.5 PA 3. Improvement of the accessibility 57

4.2.6 PA 4. Development of transnational synergies for sustainable growth areas 63

4.2.7

Additional analysis with regard to achievement of results 67

4.3 Evaluation of the added value 70


6/182

4

Evaluation of South East Europe Programme 2007-2013

4.3.1 Transnational added value 70

4.3.2 Contribution to European Policies 74

4.4 Future perspectives 80

4.5

Conclusions and recommendations on results of the programme and futureperspectives 82



5 Programme communication 87

5.1 Scope of the evaluation for communication 87

5.2 Key results of the 1stand 2ndevaluation report on communication 87

5.3 Progress in programme communication in 2012 and 2013 88

5.3.1 Progress in transnational level communication 89

5.3.2 Progress in national level communication 93

5.4

Conclusions and recommendations on communication 95



Annex 1: Treatment of recommendations 97

Annex 2: Performance of programme indicators 105

Annex 3: Results of the programme 109

Annex 4: Indicators on communication 113

Annex 5: Questionnaire 117


7/182


Executive Summary

This executive summary contains the conclusions and recommendations of the third and final

evaluation report from the ongoing evaluation of the transnational programme for South East

Europe 2007-2013.The focus of the report is on the results of the programme and the threeevaluation reports. Furthermore the monitoring and implementation, the communication activities

and the added value of the programme have been evaluated.

Overall it can be concluded that the transnational programme for South East Europe 2007-2013 is

functioning well. More details are provided below.

Conclusions

Conclusions on the project generation and selection procedure

Q1. Do the project selection criteria effectively support the selection of the best quality transnational

projects?

1. A clear trend towards the fine tuning/narrowing of the spectrum of selection requirements to

more concrete assessment criteria can be evidenced;

2. A steady adoption of a flexible approach to formal checks en route to a more in-depth analysis

of project content, transnational partnership development and consistency has been detected;

3. Gradual reduction of the weight of the administrative burden by the introduction of more flexible

approaches to formal and eligibility checks;

Q2. Is the project selection procedure clear and transparent?

4. Increasing evidence of strategic awareness of project applicants and better project fitting and

integration into the policy transnational framework.5. Although the duration of the procedure has been reduced in the 4th call with respect to the 2 nd

one, this has still resulted in a long process.

6. Despite the partial reduction of the administrative burden (for instance by accepting scanned

documents and faxes), which is judged as a positive improvement, the (sometimes necessary)

administrative requirements may still represent an obstacle for potential quality projects to

apply for financial support;

7. The geographical and leadership imbalances of projects, although decreased during the

programme life cycle, can still be observed and still need to be reduced.

8. A trend towards offering more information and feedback on the non-selected project proposals

has been observed by the national bodies and JTS thus increasing the awareness and

knowledge of what worked wrong with non-successful applications. This, without any doubt,has contributed to the enhancement in the quality of future project proposals.

9. Also, the use of a two-step approach has supported the generation of quality projects in the

context of the Programme, especially with the third top down and targeted call.

10. A more strategic evaluation analysis has been observed, with a shift from an administrative

check towards a more thematic approach while safeguarding objectivity in the project selection

procedure through the 6-eye principle.


8/182

6


Conclusions on performance of the programme

11. The South East Europe Programme is performing well. The ERDF funding has been fully

committed to 122 projects after 4 calls for proposal. Originally it was foreseen to support 130

projects of average budget of 1.5 MEUR each. However the specific character of 3rd call

strategic projects resulted in much higher allocations per project.

12. During the first and the second evaluation report only a limited quantitative assessment was

possible. In the meantime improvements have been made in the monitoring system. The

introduction of the Monitoring Wizard tool ensures that project achievements can now be linked

to programme objectives. The results will be available in the coming Annual Implementation

Report.

13. The Wizard shows that the programme is on schedule with achieving its objectives. 55% of the

output and result indicators are on schedule in achieving their targets (21 indicators have

achieved 45-65% of the target) or are far above the expected figures, 16 indicators have

achieved more than 65% of their target). 45% of the indicators are slightly or largely behind

schedule: 18 indicators have achieved 35-45% of their target and 13 indicators have realised

less than 35% of their target.

Conclusions on the geographical & thematic scope of projects14. In comparison with the situation in the first and second evaluation reports the geographical

balance of projects over the programme area remains largely the same. All programme

participating countries are covered.

15. Of the IPA countries the participation of Croatia and Serbia are the strongest in the SEE

programme with respectively 59 and 80 project partners.

16. The country most cooperated with is Romania as they are in most projects (107 projects) and

with whom is being cooperated most (701 relationships), followed by Hungary, Italy (both in

102 projects and respectively 668 and 670 relationships), Bulgaria (in 92 projects and 600

relationships), Greece (in 85 projects and 566 relationships) and Slovenia (in 82 projects and

547 relationships).

17. In comparison with the first and the second evaluation report the programme is more balanced

over the different priority axes.

18. Based on the current high level of cooperation between partners coming from the two future

programme areas, the evaluator considers that 20% flexibility may not be enough to meet

cooperation demand in the future.

Conclusions on the Monitoring System

19. In the 2012 evaluation report, both programme management bodies as well as the projects

indicated that IMIS was working well, and that all important data had been collected from the

projects to monitor their progress.

20. The monitoring system IMIS has been supplemented in 2013 with the Monitoring Wizard. Mostimportant difference between IMIS and the Monitoring Wizard is in terms of establishing links

between project and programme indicators. However, also other improvements are made in

the Wizard (e.g. actual value/target value, no. of contributions).The Wizard also provides more

clarity for the beneficiaries on the interpretation of the indicators.

21. A major improvement according to the evaluator is that the Monitoring Wizard also collects

qualitative information on the nature of projects contribution to the programme objectives.

This means that a project may contribute to programme objectives through one or more of its

output/s and results. In the coming annual report the results on this programme result indicator

will be presented.


9/182


22. In 2011 JTS started to adapt the IMIS towards the integration of the IPA and ENPI Funds. IPA

Funds have been integrated since the 3rd call. For ENPI there are 6 projects which will be

monitored in a separate way.

Conclusions on the Programme Management system

23. The JTS, MA, MC, SCP, CA and AA are in general working well. There is good cooperation

between and within the bodies.

24. There continues to be a difference in the experience of the SCPs. The SCPs could have a

good role in the implementation of the programme.

Conclusions with regard to results: data and methodological limitations

25. It is important to highlight the data and methodological limitations before summarising the

findings on the results achieved in order to interpret correctly the conclusions.

The conclusions are based on the implemented projects under the first call for proposals (30%

of all projects). The additional information on the number, distribution along the AoI and

expected contributions of the contracts concluded under the 2nd, 3rd and 4th call for proposals

allowed making projections on the expectations with regard to achievement of programmes

objectives.The conclusions are based mainly on the data collected through the SEE programme

monitoring system and Final Activity Reports (FARs). While the quantitative data provides

valuable information, its interpretation was hindered by the lack of background and reliable

qualitative information. The diversity of topics and problems covered did not allow going in-

depth into the technical issues, which was exacerbated by the huge number of projects and

beneficiaries. Further, only 65% of the 40 completed projects submitted FARs in time to be

made available for the evaluation. While the common programme and project indicators allow

for a comparison between the PA, the lack of specific thematic indicators per PA made it

impossible to analyse subject specific data.

To overcome the difficulties commented above the evaluation team implemented in-depth

review on the results achieved of 8 projects (2 per PA), which was supported by interviews with

the relevant project managers. The projects are presented in the relevant sections of this report

in order to illustrate the achievements and their contribution to programme objectives and

thematic codes of the EC implementing regulation.

Conclusions with regard to results: achievement of programme objectives

26. From the 4 PA supported under the SEE programme the greatest contribution to achievement

of the planned specific programme objectives so far has been made PA 2 Protection and

improvement of the environment. The contributions under this PA are slightly over the planned

contributions and the implementation so far shows good prospects for achievement of the

planned objectives related to this PA. Thus the SEE programme is expected to contribute tosustainable development and improvement of environment.

The programme made progress and is likely to achieve its specific objective related to

facilitation of innovation and entrepreneurship, knowledge economy and information society

(PA1), despite that the expected contributions under this PA are about 12 % lower than

originally planned. A suboptimal involvement of the public sector was observed in 1 st call

projects, particularly under AoI1.3. However, this was significantly improved following the

targeted AoI1.3 second call and public sector involvement is high in the 3 rdCall project and

positive in the 4thCall.

27. It is likely the projects implemented under PA4 Development of transnational synergies for

sustainable growth areas will contribute to enhancement of regional and transnational


10/182

8


cooperation and integration. The projects implemented so far show potential with this regard as

well as with regard to making SEE regions more attractive (e.g. through urban regeneration

and tourism based on cultural values).

28. The implementation of PA3 Improvement of the accessibility is less advanced compared to the

other PA since most of the projects are still in progress. Therefore conclusions with regard to

the achievement of specific objective related to physical and virtual accessibility could not be

made at the time of this analysis. There is so far little contribution to improved virtual

accessibility. However the ongoing projects have the potential to bring the planned results with

this regard.

29. On the basis of the above it could be expected that the SEE programme would contribute to

improvement of the territorial, economic and social integration process in South East Europe

and to cohesion, stability and competitiveness of the area. The extent to which this will be

achieved depends on how the programme will address the factors that hamper achievement of

results under the specific PA and will improve transnational and regional cooperation. The

Capitalisation Strategy initiated could have positive role in this process.

Conclusions on communication

The main conclusions of the 3rd

report concerning communication are the followings:30. In general it can be said that the recommendations of the previous evaluation reports are taken

into consideration by the JTS, and the communication plan of the programme for 2012 and

2013 integrated some of the recommendations.

31. A significant majority of the stakeholders1 is satisfied with the quality information provided in

time by the JTS in all phases of the programme implementation.

32. The events organized by the JTS were considered to be very useful, and the JTS provided

much more opportunities for the project stakeholders to exchange their experiences and good

practices (annual conference, thematic seminars.)

33. Significant improvement can be observed in the presentation of the results in programme

communication to the general public (e.g. video, developed website, etc), but there is still place

for further developments in creating connections with the media.

34. The capacity, expertise and motivation of the SCPs concerning the communication activities is

still very different in the different member states, and it seems that there is no real chance to

change this situation significantly in the frame of this programme, however it should be done in

the next programming period.

35. The establishment of links with the national, regional and local media was not in the focus in

2012 and 2013 at member state level.

36. Not all the communication indicators are monitored appropriately and it seems unrealistic to

complete the missing data from national and project level.

1Please have a look at sub-chapter 5.3.1. for the concrete values


11/182


Recommendations

Recommendations on project generation and selection procedure

1. Reduce the duration of the project selection process to the possible minimum, i.e. by covering

administrative and contracting tasks in shorter timeframes.

2. Further reduce the subjectivity factor, by continuing the improvement of the work done until

now to concretise selection criteria.

3. To include sustainability as a more determinant and relevant criterion for project selection in

future calls, by allocating it a higher weight in the overall scoring.

4. To give a higher importance to innovation in the territory and its geographical character in the

selection criteria by identifying strategic projects not only in targeted calls, but also in the

traditional bottom up calls for Proposals.

5. To ensure a better geographical balance, by encouraging the most recently accessed countries

to take the lead in (part of) projects, thus increasing the transnational balance of the

programme as a whole.

6. To increase even more capacity building actions for SCPs to ensure a higher geographical

coverage and quality of projects content.

7. To continue providing transparent and detailed information (feedback) regarding eligibilityresults of the selection procedure to project applicants in order to reinforce their capacity and

contribute to the generation of higher quality projects for future calls.

Recommendations on monitoring and implementation

8. It is recommended to continue the already started capitalisation of the programme results and

pay more attention to promote the results in the coming period.

9. It is recommended to draw the attention of the decision makers that with the splitting of the

current SEE programme area into 2 new transnational programmes (Danube, Adriatic-Ionian)

the current high level cooperation between the countries of the two new programme areas will

be reduced significantly, furthermore there is one country (the Former Yugoslav Republic of

Macedonia), which is not included in any of the new programmes. It is highly recommended to

use the flexibility rule at maximum level to ensure the future cooperation as wide as possible.

10. It is recommended for the next programming period to pay special attention to the functioning

of the SEE Contact Points, and ensure that the less experienced SEE Contact Points get the

appropriate support and motivation to be able to catch up with the others, and ensure the

quality of the programme management with it in all participating countries.

11. As far as possible try to see for next programming period how the integration of IPA and ENPI

can be included from the start of the programme.

Recommendations on the indicators

12. It is recommended to draw the attention of the project beneficiaries of those projects, which arecurrently under implementation, to the fulfilment of those 16 output and result indicators, which

are currently behind the target.

13. It is recommended for the next programming period that the targets of project indicators should

be set at the start of the programme, and they should not be an adding up from figures coming

from project applications.


12/182

10


Recommendations on results of programme

14. In the design of the Priority axes for the follow on and/or similar programmes specific measures

should be taken in order to ensure more representation of the private sector in actions for

improving accessibility (PA3) and more involvement of public institutions in facilitation of

innovation and entrepreneurship (PA1).

15. The electronic monitoring system of the next programmes should ensure smooth integration of

IPA and ENPI funds. In addition, the functions for monitoring output and result indicators and

activities should be improved providing for clearer link between the two categories.

16. Collective efforts by the MA, the JTS and the participating countries should be taken for the

next programming period (next programmes) in order to harmonise the procedures in the

different countries with special attention on simplifying the reporting procedures.

Recommendations on communication

17. It is recommended to continue the presentation of the project results to the general public and

make further efforts to do it at national level also (e.g. establish links with national, local media,

such as newspapers, TV channels etc.).

18. It is recommended to continue the already started capitalisation process, ensure the

appropriate interactivity level of the events in order to provide opportunities for the projects toexchange their experiences and create links, synergies among themselves.

19. It is recommended for those SCPs who are currently less active in the national level

communication to give importance to the communication activities in their countries for the rest

of the programme, and ask support from the JTS, if it is needed.

20. It is recommended for the next programming period to reduce the differences among the

countries concerning the quality of the national level communication. Define at the beginning of

the programme the annual minimum level requirements for the SCPs concerning the national

level programme communication activities, and it should be monitored by the JTS / MA on a

regular basis.

21. It is recommended for the next programming period to ensure from the beginning of the

programme the collection of the relevant data about the national level media appearances.


13/182


Abbreviations

Abbreviation and its full name

AA Audit Authority

AC Acceding Country

AF Application Form

CA Certifying Authority

CC Candidate Country

CfP Call for Proposals

DEU Delegation of the European Union

ENPI European Neighbourhood Policy Instrument

EC European Commission

ERDF European Regional Development Fund

FLC First Level ControlGIS Geographical Information System

IMIS Integrated Monitoring and Information System

IPA Instrument for Pre-Accession

IVB Transnational European Territorial Cooperation Programme IVB of 2007-2013 (Interreg)

JTS Joint Technical Secretariat

LP Lead Partner

MA Managing Authority

MC Monitoring Committee

MoU Memorandum of Understanding

MS Member State

OP Operational Programme

PCC Potential Candidate Country

PO Project Officer

PP Project Partner

PPP Public Private Partnership

R&D Research & Development

SCP SEE Contact Point

SEE South East Europe

SME Small and Medium Enterprises

ToR Terms of Reference

Abbreviation of countries and their full name

AL Albania IT Italy

AT Austria MD Moldova

BG Bulgaria ME Montenegro

BiH Bosnia and Herzegovina RO Romania

EL Greece SI Slovenia

MK Former Yugoslav Republic of Macedonia SK Slovakia

HU Hungary SR Serbia

HR Croatia UA Ukraine


14/182


15/182


1 Introduction

1.1 Context

The South East Europe Programme Area is the most diverse, heterogeneous and complextransnational cooperation area in Europe, made up of a broad mix of 16 countries with a total

population of 200 million people. For 14 countries the eligible area is the whole territory of the

country and in 2 countries only certain regions are eligible under the SEE programme 2. These

countries are acting as a bridge between North, South, East and West Europe in the European

transportation network system. The stability, prosperity and security of the region are of significant

interest to the EU. The emergence of new countries and with it the establishment of new frontiers

has changed the patterns of political, economic, social and cultural relationships.

What differentiates the South East Europe Programme is not only the number of countries

participating in it, which makes it the transnational programme with the largest cooperation area,

but also the diversity of the participating countries. This is the only transnational programme area

with such a large number of non-EU countries participating. Out of the 16 participating countries 8

are EU Member States, 6 are candidate and potential candidate countries and 2 are countries

participating in the European Neighbourhood Policy. This is a highly complex programme which

presents challenges such as ensuring good mechanisms to contract partners who receive funding

from different instruments: ERDF (European Regional Development Fund), IPA (Instrument for Pre-

accession Assistance) and potentially ENPI (European Neighbourhood and Partnership

Instrument).

2 In Italythe eligible regions are: Lombardia, Bolzano/Bozen, Trento, Veneto, Friuli-Venezia-Giulia, Emilia Romagna, Umbria,

Marche, Abruzzo, Molise, Puglia, Basilicata.

In Ukraine: Cjermovestka Oblast, Ivano-Frankiviska Oblast, Zakarpatska Oblast and Odessa Oblast


16/182

14 Evaluation of South East Europe Programme 2007-2013

1.2 The evaluation and this third and final evaluation report

The objective of the on-going evaluation of the SEE Programme is to improve its performance. The

Operational Programme has indicated that the evaluation should improve the quality, effectiveness

and consistency of the use of assistance, the strategy and the implementation of the programme.

A first evaluation report has been developed in 2011, a second evaluation report has been

developed in 2012 and this third and last evaluation report is summarising the results and doing a

third analysis in 2013.

This third and final evaluation report focuses on analysing results of the programme (chapter 4.),

summarising and up-dating the information about project selection and monitoring and

implementation (Chapters 2 and 3.), making recommendations for the future (all chapters) and

evaluating the communication activities (Chapter 5.). In the annexes additional information can be

found.

As approach used for the evaluation each year a survey (web-based questionnaire) has been held.

See the separate annex document for the results of the last questionnaire. About 60% of the 122

projects filled the questionnaire (74 people). About 60% of the full JTS and MA filled thequestionnaire. Only 18% of the MC members and 27% of the SCP filled the questionnaire. The

questionnaire was sent to 306 persons (17 JTS/MA staff, 53 MC member, 33 SCP colleagues

(some are both MC member and SCP), 210 beneficiaries). The evaluators got feedback from 101

persons in the survey in 2013 (33%). 10 of them represent the JTS and MA, 8 of them are MC

member, 74 project beneficiaries, and 9 of them represent SCPs. The 9 answers from the SCPs

represent 8 countries from the SEE programme area.

Furthermore interviews have been held each year with the Managing Authority and the Joint

Technical Secretariat. Lastly desk research was an important source of information and the

evaluators participated in most Monitoring Committee meetings.


17/182


2 Project generation and selection

2.1 Scope of the evaluation of the project generation and selection

This chapter presents the findings of the evaluation on project selection and generation. Accordingto the evaluation plan, three questions have been answered:

Q1. Do the project selection criteria effectively support the selection of the best quality transnational

projects that contribute to the objectives of the programme?


Q3. How could the selection process be improved? - Recommendations

2.2 Assessment of project generation and selection process and instruments

2.2.1 Project generation and selection

The selection of high quality transnational projects is strongly related to the capacity of the

programme to generate good project ideas and partnerships and to adopt appropriate evaluation

criteria. Taking into account the current stage of the Programme, the answer to Q1 mainly focuses

on the evolution of the Programmes capacity to select relevant projects by the use of appropriate

selection criteria. Since the structure and approach of each call also determines the use of specific

criteria, in this section the evaluators briefly analyse the calls selection processes and criteria, by

identifying the most relevant characteristics of each call and analysing their evolution over time,

also taking into account the extent to which the Programme has incorporated the recommendations

formulated in the 1stand 2ndevaluation reports. Complementary to previous analyses, the results of

the 4th

call and the two restricted calls launched by the Programme are also covered in this section.

The calls of the Programme

To complete the analyses on the calls for proposals carried out in the previous evaluation reports, it

is necessary to assess the performance of the 4 th one, which was still open at the moment of

finalising the second evaluation report. The one-step procedure and more traditional bottom-up

approach adopted in this last call were oriented at allocating the budget remaining from the

previous calls. From the initial 342 applications received a lower number with respect to the other

calls due to the one-step approach a first set of 37 projects was approved, implying an initial

success rate of 10.8%. Nevertheless, taking into account the under-spending of projects approved

under the 1stand 2ndcalls, a reserve list of projects was foreseen, allowing a later approval of an

additional number of 10+1 projects (which is in total 48 projects under the 4 thcall), thus increasingthe success rate of the 4 thcall up to 14.04%.

When compared to previous calls, this success rate is a decrease to 3 rd (strategic) call but an

increase to 1stand 2ndcalls (all previous calls had a two-step application procedure). The highest

number of proposals was submitted under the area of intervention 4.3, showing an important

interest although on average presenting the lowest quality level. On the other hand, over-

dimensioned budgets were submitted in several cases, as a result of the Programmes end in 2013,

reinforcing the perception of last try opportunity.

Nonetheless, it is important to stress that both the experience gained by the Programmes

Authorities throughout the Programme life cycle, the clear definition and knowledge of what is

expected by the SEE Programme as well as the gradual coaching and higher understanding of the


18/182


programme by project partners made the difference in the selection of relevant and policy

integrated projects during the 4 thcall. The successful projects are characterised by an innovative

nature, a considerable comprehension of the topics, a high relevance for the programme area, a

coherent and tailored logical frame as well as a strong commitment demonstrated by applicants.

Complementary to the four main calls, the Programme also launched two restricted complementary

calls with the objective to achieve a higher commitment of the IPA and ENPI allocated funds. This

was put in practice through the involvement of additional partners from selected countries into

existing partnerships and projects, consequently ensuring an important added value from both a

territorial and a content-wise perspective. The first complementary call was an IPA call oriented to

Montenegro, directed to the existing projects selected under the 2 ndopen call for proposals of SEE,

whose Lead Partners were invited to involve additional partners from Montenegro by adjoining or

incorporating them to their partnerships. The second one, an additional ENPI call for partners

from Moldova, was foreseen as a way to compensate a substantial leftover of ENPI funds for this

country which remained non-allocated to the projects. This call was opened to the Lead Partners

from the 3rd and 4th calls willing to additionally involve Moldavian partners. Both restricted calls

foresaw clear steps that involved different actors and criteria.

The innovative way to manage external funds, the implication of both national and programmes

authorities and a strong focus on the quality of the project idea and partnership has helped to

ensure a high coherence with the Programmes priorities and rules as well as to overcome the

challenges related to the use of different typologies of funds in a complex environment. As a result

of this targeted strategy, IPA funds have been fully integrated into the SEE Programme and ENPI

funds show a good alignment to the existing framework.

To conclude this section, and in a synthetic way, the main characteristics of the different calls and

their related approaches, processes and selection criteria have been summarised in a table that

can be found below. This table shows how the Programme has used different approaches in the 4

calls, evolving to more sophisticated methodologies, such as the two-step procedure and targeted

and restricted calls, as well as improving the assessment criteria definition towards the selection of

more integrated projects.


19/182


Table 2.1 Overview of the calls of the Programme

Items 1st call 2nd call 3rd call 4th call

Process

Steps of

the call

2 steps (first Expression of Interest after

which a selection of applicants is invited to

submit full proposal)

2 steps 2 steps 1 step: single step procedure

whereby all applicants have to

submit the whole package of

information

Approach

of call

Fully open - Bottom up Open targetedBottom up StrategicTop down

Targeted and restricted call

Competitive, fully open

Traditional bottom-upDates May 2008March 2009 December 2009December 2010 June 2010February 2012 October 2011June 2012

Success rate 822 EoIs, 40 projects approved: 4.86% 423 EoIs, 26 approved: 6.14% 49 EoIs, 8 projects approved: 16.32% 342 full applications submitted, 48

projects approved: 14, 04%

Selectioncriteria

Formal

and

Formal check:

Proposal submitted within deadline;

Application submitted in English;

Official application form used;

All required parts and annexes are filled in

and submitted.

Eligibility check:

Partners from a minimum of 3 partner

states, located in the eligible area, of

which at least one from a Member State;

Project implementation has not startedand it wont start prior to the approval of

the AF;

Lower administrative burden

Clear and efficient rules for participation to

the call

Slight improvement in clarity, transparency and

time efficiency

Flexible approach to eligibility check in terms of

administrative documents

Eligibility criteria more clearly defined with

respect to previous calls:

Timely submission of application form;

The project includes all the mandatory activities

as described in the ToR for the call per priority;

Lead Applicant and financing partners are

eligible;Project fulfils minimum requirements for

transnational strategic partnerships (same as

previous calls);

No formal check

Minimum modifications in

comparison with previous calls

Like in previous calls,

networking/exchange of best

practices as stand-alone activities,

as well as proposals with an

academic focus but no

developments of transnational

policies/strategies are not eligible.


20/182



Selectioncriteria

Eligibilit

y check

Project complies in general with the OP

and has impact in the programme area;

Proposed types of activities are eligible;

Beneficiaries have secured co-financing;

Beneficiaries are eligible.

The composition of the partnership is in line

with the prescriptions set out in the ToR for

the call per priority;

The specific partnership requirements

regarding the limitation of number of projects

to participate in as LP/PP are respected;

Financing partners (ERDF/IPA-I)cooperate jointly;

ERDF/IPA-I partners have secured public co-

financing;

ENPI partners have submitted all necessary

documents.

Quality

check

Is the partnership composition well

justified and able to contribute in a

balanced manner to the implementation of

the transnational co-operation project?

Do beneficiaries have the sufficient

institutional and technical capacity to

achieve the project aims?

To what extent is the project expected to

provide a significant contribution to the

overall objectives of the programme?To what extent is the project expected to

provide a significant contribution to the

Priority Axes and Areas of Intervention of

the programme?

Are project deliverables clearly defined

Transnationality of partnership criterion

related to the balance of partnerships

composition

Higher focus on Previous experience vs.

cooperation and partners balanced

involvement

Higher focus on clarity vs. logic conceptual

approach and work plan

Concreteness and usability of project

deliverablesMaintenance of the value for money

criterion

Higher focus on the nature of the problem

tackled by the project

Lower focus on the time perspective vs.

Quality selection criteria are highly coherent with

the ToRs

5 Terms of reference with specific thematic

focus detailed the compulsory activities to be

implemented in order to reach the expected

outputs

Strategic requirements: relevance to the

integration of the SEE programme area;

involvement of key figures in the field and long-

term perspective and impactHigher focus on sustainability, innovation and

geographical character

Minimum modifications in

comparison with previous calls

A minor adjustment can be seen,

adding one specific question on

the sustainability component

Questions related to investments

and equipment costs have been

dropped

Project activities detail is assessed


21/182



and are the assumptions on their use

realistic?

Value for money

Are there synergies and /or

complementarities with other territorial co-

operation projects or other EU funded

policies and programmes?

transnationality impact

Sound viability plan

Innovation component

Sustainability component (political,

institutional and financial)

Comments

Strengths 1stcall: In general terms, the features of the

SEE best quality Project and the quality

selection criteria share a high level of

correspondence.

2nd call: The two step approach allows

focusing on project content and partnership

levels vs. administrative requirements, thus

producing high quality projects

3rd call: Strong territorial marketing during pre-

announcement period

Use of knowledge and expertise from other EU

programmes

High competition at SCP level

Support to territories with limited presence in the

SEE programme

Partially effective key stakeholders involvement

by bottom up approach

4th call: Increased evidence of

strategic awareness

Evidence of in-country stakeholder

development

More concrete and direct

capitalisation

Weakne

sses

The number of projects per country shows a

geographical imbalance in the SEE

programme where most projects that ensure

a high innovation in the territory and most

lead applicants come from very experienced

countries

Long two-step process

A certain level of subjectivity detected

Repetitions of similar requirements in more

than one criterion

Streamline the formal and eligibility checks

under one single categoryGeographical imbalance due to the

experience quality requirement

Need for a higher balance in the relevance

and weight of transferability, sustainability

and innovation as selection criteria/sub-

Strong imbalance in the leadership of projects is

confirmed

No minimum quality threshold per sub-criterion

was defined

Majority of Lead Partners from few

countries

Duration of the selection process

Possibility to interpret this 4th call

as a last trythe last chance to

get access to the programmefunding.


22/182



criteria

Poor communication and information on the

project selection results to applicants

Source: Ecorys on basis of information from SEE programme and previous evaluation reports


23/182


2.2.2 Assessment of supporting tools

Complementary to Q1, this part analyses the main supporting tools that have been developed and

provided by the Programme to project potential beneficiaries and the clearness and transparency of

the selection processes adopted in the different calls.

The supporting tools

In general terms, the assessment made by project beneficiaries with respect to the usefulness of

the available supporting instruments for the generation of quality projects is positive.

The support provided by programme managers, the programme manual, the application packs and

the applicants guidelines are considered particularly useful tools. Nevertheless, a lower grade of

satisfaction can be observed with respect to the usefulness of other instruments, such as the

project idea database, instruction phase seminars focused mainly in financial and administrative

issues and the limited language coverage of official documents that were not always available in all

the languages of the covered countries. Although not being a requirement for any programme

authority and thus being a voluntary initiative of the SEE contact points, it could have providedadditional support to project partners and potential beneficiaries.

The surveys to project beneficiaries, that have been carried out for the purpose of producing the

different evaluation reports3, show that no relevant differences over time can be observed on the

perception of the usefulness, user-friendliness, accessibility, richness on quality information and

time availability of the different tools that have been put at the disposal of project beneficiaries in

the different calls of the Programme. This can be also confirmed on the basis of the information

provided by the consulted beneficiaries of the 4 th call. The table above summarises the

beneficiaries perception on thefollowing:

The Instruction phase seminars organised have successfully assisted applicants on financial

and administrative topics and issues At this respect complementary content-wise and thematic

oriented seminars organised at national level would have increased the usefulness of this tool;

The Project idea database, although representing a tool with a high potential, has only partially

fulfilled the applicants expectations and has been scarcely used, confirming that this tool needs

to be more intensively advertised among potential applicants in order to attract their attention;

The capacity of national structures to accompany potential beneficiaries throughout the process

of project generation is a key aspect that was considered as very effective by potential

beneficiaries. The coaching and steering role of the JTS has been crucial in the different calls,

and especially in the 3rd

one, in order to tailor the projects to the calls requirements and its

expected results;

The efficient communication and exchange of experiences, as well as information and publicityevents, are key factors in all the phases of the programme cycle. At this regard, the strong

territorial marketing during the 4th call pre-announcement period has been certainly recognised;

Meetings between potential Members States and IPA countries' beneficiaries have been of

extreme relevance and utility for both ENPI and IPA calls.

To sum up, there has been an increasing consciousness of the relevance of constant project

support and its correlation with project quality from Programmes Authorities that has resulted in a

more complete and comprehensive selection procedure.

3All Lead Partners have been invited to answer the web-based questionnaire of which 60% of the projects (74 of the 122

projects) have filled out the questionnaire.


24/182


Clarity and transparency of the selection procedure

The beneficiaries perception of transparency is positive, although to some consulted project

beneficiaries of the 4thcall, more information should have been provided by programme managers

to project applicants regarding the evaluation of their proposals. At this respect, it is important to

stress that previous recommendations were already taken into account starting from previous calls,

where a detailed evaluation summary was provided to the applicants on a request basis.

Nevertheless, the automatic provision of the evaluation sheets with comments for each project as

well as the publication of detailed scoring are factors which have still been underlined as necessary

by project applicants of the 4 thcall. The improvement of the already started path would allow project

applicants to learn from previous mistakes and contribute to higher quality proposal in the future.

On the other hand, previous evaluation analysis recommended the introduction of certain elements

of flexibility at the time of carrying out the formal and eligibility checks; capacity building actions for

SCPs in order to improve the quality of the assistance; and strong support to territories which have

a more limited presence in the Programme, especially with respect to projects leadership. In this

sense, positive steps have been adopted at Programme level in order to take into account these

recommendations and to improve clarity and transparency in the selection process, thus:

Regarding transparency, National bodies and JTS are conscious of the need to increasetransparency where reasons for ineligibility should on request - be explained transparently to

lead partners. Moreover, the quality evaluation analysis applied the 4-eyes principle with an

independent expert scoring of projects and in parallel a JTS project manager also made an

assessment. In case of serious scoring gap, a third person (from the JTS) was also involved to

safeguard an objective approach to project selection;

Regarding clarity, improvements have been made by the JTS in the selection approach

adopted, not anticipating the quality assessment but trying to understand if the projects comply

with the Priority Axis, Area of Intervention and the Programme in general. At this respect, more

flexibility was guaranteed under the projects assessment of the 4th

call as a way to overcome

possible constraints related to the complexity of the application form (f lexibility in the correlation

between the assessment grid and the application form was ensured in favour of the

applicant).The corrections and improvements introduced in the selection criteria also

contributed to increase the clarity and transparency of the whole selection process.

2.3 Conclusions and recommendations on project generation and selection

2.3.1 Conclusions

The table below summarises the conclusions and recommendations of the first and second report

and provides also the conclusions and recommendations of this final revaluation report on projectselection and generation. This is explained in more detail in the text after the table.

Table 2.2 Overview of the answers on the evaluation questions

Evaluation

question

Source used 1st& 2

ndEvaluation report Final Evaluation report: answer +

recommendation

Topic 1 Assess project selection procedure and criteria

Q1. Do the

project selection

criteria effectively

support the

selection of the

best quality

Desk research

Interviews

Mostly, high matching

between criteria and best

quality transnational

projects can be observed.

However, still need to

further adjust selection

o Mostly, more flexible approach,

important alleviation of administrative

burden, trends towards the fine tuning of

selection criteria, included more focus on

sustainability and innovation in the territory.

o Recommendations for the future:


25/182


Evaluation

question

Source used 1st& 2

ndEvaluation report Final Evaluation report: answer +

recommendation

transnational

projects?

criteria. High coherence

between 1st,2nd and 4th

call.

To reduce the subjectivity factor by further

working in concretising selection criteria,

reduce even more administrative

requirements; to increase the weight of the

sustainability criterion for project selection

and to give a higher importance to the

geographical balance.

Q2. Is the project

selection

procedure clear

and transparent?

Desk research

Questionnaire

among

projects

Mostly: the 2 step

approach focuses on

quality, but project

generation capacity needs

to be further enhanced

(geographical imbalance,

low flexibility, subjectivity).

Third call with a top down

approach assured ahigher geographical

coverage and the

involvement of relevant

entities.

o Mostly: Increasing evidence of

strategic awareness of project applicants

and better project fit and integration into the

policy transnational framework [project

selection and project generation are

strongly interrelated, therefore a higher

awareness of project applicants is seen as

a prior step for ensuring a higher success

of the project selection]; A positive trendtowards offering more information and

feedback on the non-selected project

proposals has been observed; A more

strategic evaluation analysis has been

observed, with a shift from an

administrative check towards a more

thematic approach.

o Recommendations for future:

To ensure better geographical balance, by

encouraging less experienced countries to

take the lead in projects and provide strong

support during project implementation;

Increase even more capacity building

actions for SCPs to improve geographical

coverage and quality of projects content;

To keep on improving the process of

transparency and communication in order to

ensure the generation of higher quality

projects for future calls.

Q3. How could

the selectionprocess be

improved?

Desk research

Interviews

Please see Annex Recommendations for future: Please see

section 2.3.2 below

Q1. Do the project selection criteria effectively support the selection of the best quality

transnational projects?

A clear trend towards the fine tuning/narrowing of the spectrum of selection requirements to

more concrete assessment criteria can be evidenced;

A steady adoption of a more flexible approach en route to a more in-depth analysis of project

content, transnational partnership development and consistency has been detected;

Gradual alleviation of the administrative burden by the introduction of more flexible approaches

with respect to formal and administrative requirements has been observed.


26/182



Increasing evidence of strategic awareness of project applicants and better project fit and

integration into the policy transnational framework. This is based on the assumption that project

selection and project generation are strongly interrelated. In this case, a higher awareness of

project applicants is seen as a prior step for ensuring a higher success of the project selection.

Although the duration of the procedure has been reduced in the 4th call with respect to the 2nd

one, this has still resulted in a long process;

Despite the partial reduction of the administrative burden (for instance by accepting scanned

documents and faxes), which is judged as a clear improvement, the (sometimes necessary)

administrative requirements may still represent an obstacle for potential quality projects to apply

for financial support;

The geographical and leadership imbalances of projects, although decreased during the

programme life cycle, can still be observed and still need to be reduced.

A trend towards offering more information and feedback on the non-selected project proposals

has been observed by the national bodies and JTS thus increasing the awareness and

knowledge of what worked wrong by the project promoters. This, without any doubt, has

contributed to the enhancement in the quality of future project proposals.

Also, the use of a two-step approach has supported the generation of quality projects in thecontext of the Programme, especially with the third top down and targeted call.

A more strategic evaluation analysis has been observed, with a shift from an administrative

check towards a more thematic approach while safeguarding objectivity in the project selection

procedure through the 6-eye principle.

Q3: How could the selection process be improved?

A number of recommendations are included in the section below.

2.3.2 Recommendations

The following is recommended on project generation and selection:

1. Reduce the duration of the project selection process to the possible minimum, i.e. by covering

administrative and contracting tasks in shorter timeframes as much as possible.

2. Further reduce the subjectivity factor, by continuing the improvement of the work done until now

to concretise selection criteria.

3. To include sustainability as a more determinant and relevant criterion for project selection in

future calls, by allocating it a higher weight in the overall scoring.

4. To give a higher importance to innovation in the territory and its geographical character in the

selection criteria by identifying strategic projects not only in targeted calls, but also in the

traditional bottom up calls for proposals.

5. To ensure a better geographical balance, by encouraging the most recently accessed countriesto take the lead in projects, thus increasing the transnational balance of the programme as a

whole.

6. To increase even more capacity building actions for SCPs to ensure a higher geographical

coverage and quality of projectscontent.

7. To continue providing transparent and detailed information (feedback) regarding eligibility

results of the selection procedure to project applicants in order to reinforce their capacity and

contribute to the generation of higher quality projects for future calls.


27/182


3 Monitoring and implementation

3.1 Scope of the evaluation of monitoring and implementation

In this chapter the performance of the implementation and the monitoring system of the SEEprogramme is evaluated. The objective of the evaluation of the monitoring and implementation is to

evaluate the performance of the implementation and the monitoring system of the SEE programme

though its monitoring indicators and to give ideas to improve the whole indicator system of the SEE

programme.

In this evaluation the focus has therefore been on a number of specific issues:

Performance of the programme through the indicators

Geographical scope of the projects

Thematic scope of the projects

Monitoring system

Programme management system

This chapter will summarise the results of the previous reports and update it to its current situation

for all specific issues mentioned above. This means that for the indicators an updated table of

performance is included and also an analysis about it (paragraph 3.2). For the geographical scope

the results of the previous report are mentioned and for the thematic scope additional research is

done to see which country is most interested in which area of intervention (paragraph 3.3). For the

programme management and monitoring system the previous two reports are up-dated and further

analysis is made (paragraph 3.4).

3.2 Performance of the programme

3.2.1 Summary of previous findings

In the first evaluation report (2011) the evaluators concluded that the assessment of the indicator

system was not possible as the realised indicators were not available during that evaluation period.

In the second evaluation report (2012) it was still difficult to show the performance of the

programme through indicators as either the target was not available (project level indicators), or the

realization/actual value was not given (programme level indicators), or the indicators seemed to be

completed with incorrect approach (programme indicators).

Although a quantitative assessment was possible only limitedly, the evaluators listed someobservations in both the first, and the second evaluation report. In the first evaluation it was

observed that the programme contained a helpful set of indicators, however that targets were not

quantified and that there were ambiguities for the beneficiaries on which indicators they had to fill.

During the second evaluation report the SEE programme had indicators defined at programme and

project level; however for the project indicators still no targets were defined.

3.2.2 Assessment of the indicator system

Currently improvements have been made in the monitoring system. For all indicators there is now a

target available (extracted from the project applications) and the realization was monitored in the

Monitoring Wizard Report (project basis) and reflected in the annual implementation report 2012

(programme basis).

The SEE programme has indicators defined at programme and project level in:


28/182


The SEE Operational Programme, section 4.4 (programme level);

The SEE Programme Manual Version 2.1 May 2010, 2ndCall for project proposals, Annex 2.

(programme level and project level)

The Annual Implementation Report (programme and project level)

IMIS (programme and project level)

The Application Form (project level)

The Progress Report (project level)

The Monitoring Wizard (project and programme level)

These sources all use the same set of indicators. The programme manual for the 2 nd call for

proposals also mentions other indicators (input indicators, indicators for the cross cutting issues

and indicators for the TA). The Annual Implementation Report is reporting on the indicators

mentioned in the Operational Programme. The Annual Implementation Report of 2012 also reports

on the indicators defined for the projects. The Monitoring Wizard was introduced in 2012 and helps

to monitor the projects from the point of view of their link to the programme objectives.

Programme indicators

In the second evaluation report the evaluators concluded that the Operational Programme has aclear description of the indicator system at programme level and that the indicators are specific and

quantified and output and result indicators are interlinked.

Based on the information available in the documents (see annex 2) the following remarks can be

made currently for the programme indicators:

After four calls 122 projects have been approved, which is not far behind the target of 130

implemented projects set at the beginning of the programme. As the budget of the projects is

higher than anticipated, fewer projects were sufficient to allocate all available funds.

Looking at the programme indicators (see annex 2) 55% of the output and result

indicators are on schedule4in achieving their targets(21 indicators have achieved 45-65%

of the target) or are far above the expected figures, 16 indicators have achieved more than 65%

of their target). 45% of the indicators are slightly or largely behind schedule: 18 indicators have

achieved 35-45% of their target and 13 indicators have realised less than 35% of their target.

One can still question if the number of projects is a good indicator, as the number of projects

does not relate to the content. However, because this indicator has also been used in the

Working Document of the EC as an indicator the evaluator suggested maintaining this indicator

(see evaluation report 2012).

In the 2012 evaluation report the evaluators mentioned that the performance of the number of

contributions has been completed by mentioning the number of projects that have been

approved, instead of number of contributions (of which it is expected to obtain on average 3

contributions per projects). The evaluator recommended to specify what a contribution is and tocorrect the performance of the indicator accordingly. In 2012 in the second evaluation report the

SEE programme reported that: Data is reported according to the understanding that 1

approved project equals 1 contribution, which was originally adopted by the Programme

monitoring bodies. This result indicator was further specified and defined in the course of 2012

through an external study aiming at enhancing SEE monitoring system (please refer to section

2.1.6 Qualitative analysis for more details on the study). Therefore, in the next Annual

Implementation Report, the values reported for this result indicator will be overhauled based on

the study feedback. With the introduction of the new Monitoring Wizard there is now a clearer

link between project achievements and the programme objectives as well as a clearer definition

of a contribution together with a methodology to measure the number of contributions.

4As the projects started in 2009 and will run until 2014 one can expect that about 50-60% of the targets should be achieved in

2012. When a 45-65% of the target has been achieved we therefore speak of being on schedule.


29/182


Project indicators

In the Programme Manual for the first call was indicated that the set of indicators would be

developed later. In the Programme Manual of the second call the approach for the indicators was

further elaborated into a set of input, output and result indicators. The programme manual of the

third and fourth call does not contain this information anymore since the indicator system was in the

process of being revised.

In the evaluation report of 2012, the evaluators assessed the indicators set well presented,

comprehensive and limited. However, the evaluators mentioned several points of attention for the

programme. One of the main points was that the indicators are not quantified (are not having a

target) and are not always SMART.

The previous evaluations concluded that more attention should be given to the Lead Partners of the

programme to avoid wrong reporting. Recommendations from the evaluator in this field were:

It would be good to add an explanation of the indicators like it has been done for the indicators

no of successfully implemented projects. For example, when is a project achieving synergy

with an Objective 1 or 2 programme? The text of the programme manual for the second call is clear but merely written for the

programme than for the applicant/ beneficiary. It might be difficult for the beneficiary to

understand what he has to do with this information. For example table 9 Output and result

indicators related to TA is relevant for the programme but not for the applicants/beneficiary as it

concerns Priority Axis 5 Technical Assistance. Also the input indicators are not so relevant for

the applicants/beneficiaries.

It could be confusing for the applicant to have all the different tables with indicators.

During the interviews with JTS members in 2013 it was clarified that the programme took actions on

the above. The programme has provided indicator descriptions and interpretation in the Monitoring

Wizard tool. In the Monitoring Wizards the JTS asked the projects to check, if they misunderstood

the indicators following the provided interpretation. In case of misunderstanding, the indicators were

revised. This did not happen when the target was too optimistic though because that would require

a project modification.

According to the Monitoring Wizard, the output indicators at project level are progressing

rather well. 8 of 22 indicators a far ahead on schedule (more than 65% of the targets already

achieved halfway through programme implementation). 9 indicators are on schedule (45-65% of

targets achieved). 5 are a bit behind schedule (35-45% of the targets realised). The output indicator

values of the projects are behind on number of press conferences, number of guidelines produced,

joint action plans produced, number of individuals that participated in exchange programmes andnumber of promotion concepts (see figure below and annex). This is considering that there are 82

projects still running.

The table below shows to what extent the targets of the indicators have been achieved. Projects of

the first call (which are currently finished) reached their targets quite well. 14 of the 22 indicators

score above 100%, 6 indicators are between the 75% and 100%, and only 2 indicators score below

the 75%. The most problematic indicators are the promotion of concepts (51% of the target) and the

number of joint action plans produced (71%). While the average hits per month on the operations

website and the number of articles/appearances published in the press and in other media were

much more than expected in advance.


30/182


Table 3.1 Project output indicators after four calls (% of the project targets achieved in each call) dark green +

underscore + > 100%, dark red and italic is < 15%

. Description 1st call 2nd

call 3rd

call 4th

call TOTAL

1 No. of articles/appearances published in the press and in other

media (including online media, TV, radio)

191% 56% 24% 14% 89%

2 No. of press conferences 76% 46% 25% 6% 43%

3 Average of hits per month on the operations website 341% 211% 37% 6% 252%4 No. of publications produced (editions, specify: e.g. folder,

newsletter, brochure, report, handbook)

92% 84% 1% 0% 80%

5 No. of transnational events implemented 115% 71% 36% 7% 53%

6 No. of national events implemented 106% 68% 2% 3% 48%

7 No. of regional events implemented 104% 76% 17% 4% 50%

8 No. of study visits organised 184% 79% 58% 1% 90%

9 No. of studies realised 85% 76% 25% 4% 57%

10 No. of guidelines produced 109% 43% 21% 3% 43%

11 No. of management plans developed 83% 68% 15% 14% 56%

12 No. of joint action plans produced 71% 45% 5% 2% 35%13 No. of databases created or improved 107% 93% 6% 2% 63%

14 No. of training events, seminars organised 118% 41% 3% 1% 62%

15 No. of participants involved in training events/seminars 89% 39% 4% 2% 53%

16 No. of individuals that participated in exchange programmes 141% 24% 58% 4% 43%

17 No. of promotion concepts 51% 98% 57% 11% 40%

18 No. of promotion actions 142% 102% 36% 10% 82%

19 No. of services developed 124% 43% 0% 16% 60%

20 No. of small scale infrastructure projects 75% 133% N/A 0% 74%

21 No. of person in charge for administration of projects 132% 99% 100% 51% 88%

22 No. of project meetings held 125% 81% 24% 18% 74%Source: Ecorys on basis of the Monitoring Wizard (July 2013).

The evaluator is of the opinion that for some output indicators targets set by projects were too high,

which complicated reaching the targets (in blue in the table above):

No. of press conferences

No of studies realized

No. of management plans developed

No. of joint action plans developed

No. of participants involved in training events and seminars

No. of individuals that participated in exchange programmes

No. of promotion concepts

While for other output indicator the projects put a lower and more realistic target:

Average of hits per month on the projects website

No. of study visits organized

No. of promotion actions

No. of person in charge for administration of projects (especially in the first call)

According to the Monitoring Wizard the result indicators at project level are merely on

schedule. 7 of 26 indicators a far ahead on schedule (more than 65% of the targets already

achieved whereas the programme is halfway its implementation). 9 indicators are on schedule (45-65% of targets achieved). 6 are a bit behind schedule (35-45% of the targets realised) and only 4


31/182


are far behind schedule (0-35% of targets achieved). Especially the number of individuals reached

directly through dissemination outputs in the co-operation area (33%), the number of strategies

adopted (25%), the number of private market reactions achieved (19%) and the number of

infrastructures of common interest improved (25%) are lagging far behind schedule.

Table 3.2 Project result indicators after four calls (% of the project targets achieved in each call): dark green +

underscore + > 100%, dark red and italic is < 15%

Description 1st call 2nd call 3rd call 4th call TOTAL

1 No. of permanent information sources / channels in

operation (e.g. websites, regular publications)

137% 56% 24% 14% 136%

2 No. of individuals reached directly through

dissemination outputs in the co-operation area

640% 46% 25% 6% 33%

3 No. of administrative actors reached directly through


203% 211% 37% 6% 92%

4 No. of private sector actors reached directly through


675% 84% 1% 0% 543%

5 No. of SME reached directly through dissemination

outputs in the co-operation area

296% 71% 36% 7% 227%

6 No. of advanced tools and methodologies adopted to

increase the projects visibility among experts and

wider communities, the public

80% 68% 2% 3% 54%

7 No. of common positions / agreements formulated 78% 76% 17% 4% 42%

8 No. of common methodologies adopted 88% 79% 58% 1% 59%

9 No. of strategies adopted at governmental level 56% 76% 25% 4% 25%

10 No. of innovative products developed 67% 43% 21% 3% 37%

11 No. of policies /instruments improved or developed 77% 68% 15% 14% 42%

12 No. of common standards established (e.g. through

new guidelines)

102% 45% 5% 2% 53%

13 No. of new tools / instruments developed 87% 93% 6% 2% 53%

14 No. of impact studies on environmental issues

carried out (e.g. in pre-investment projects)

89% 41% 3% 1% 61%

15 No. of pilot actions prepared (first application) 72% 39% 4% 2% 48%

16 No. of pilot actions implemented (first application) 86% 24% 58% 4% 40%

17 No. of permanent exchange programmes established 64% 98% 57% 11% 39%

18 No. of staff members with increased capacity

(awareness / knowledge / skills)

104% 102% 36% 10% 53%

19 No. of advanced tools and methodologies adopted to

improve knowledge management within the

partnership

352% 43% 0% 16% 145%

20 No. of regions proactively promoted 89% 133% N/A 0% 50%

21 No. of common management structures / systems

established

84% 99% 100% 51% 63%

22 No. of individuals benefiting directly from new /

improved services

99% 81% 24% 18% 76%

23 No. of investment proposals developed 98% 0% 0% 0% 42%

24 No. of private market reactions achieved (e.g. private

activities mobilized)

44% 0% 0% 0% 19%

25 No. of investment projects implemented (specify

volume of investment)

116% 83% 32% 22% 107%

26 No. of infrastructures of common interest improved 42% 76% 0% 3% 25%


32/182


Source: Ecorys on basis of Monitoring Wizard (July 2013)

Projects of the first call (which are currently finished) reached their targets on 11 of the 26

indicators. The results of the number of individuals reached directly through dissemination outputs

in the co-operation area (640%) and the number of private sector actors reached directly through

dissemination outputs in the co-operation area (675%) were even six times as much as expected.

The projects had most difficulties with reaching their targets on the number of infrastructure

improved (42%), the number of private market reactions achieved (e.g. private activities mobilized)

(44%), the number of strategies adopted at governmental level (56%), the number of permanent

exchange programmes established (64%) and the number of innovative products developed (67%).

The evaluator is of the opinion that projects set their targets for some result indicators too high,

which complicates reaching the targets (in blue in the table above). While for other result indicators

the projects have set their targets more realistically:

Number of administrative actors reached directly through dissemination outputs in the co-

operation area

Number of staff members with increased capacity (awareness / knowledge / skills) (especially in

the 2nd

call) the number of individuals reached directly through dissemination outputs in the co-operation

area (especially in the1stcall)

the number of private sector actors reached directly through dissemination outputs in the co-

operation area (especially in the1stcall)

3.3 Geographical & Thematic scope of projects

3.3.1 Geographical scope

In comparison with the situation in the first and second evaluation reports of this programme the

geographical balance over the programme area remains largely the same after the fourth call.

Participation of the Member States

The map and table below shows that 38% of the 122 projects have an Italian Lead Partner (46

Lead Partners). Italy is having the strongest involvement both in terms of number of Lead

Partners and of Project Partners (185 project partners). However, if this proportion is related to

the population5 (see table below), Italy is scoring just above average and, in the number of

project partners, even below average.

Slovenia shows the opposite picture. Whereas Slovenia provides the project leadership in only

15 projects, if one relates it to the countrys population, then Slovenia scores far above the

average. Also in terms of number of project partners Slovenia scores far above the average ofthe Member States when relating the numbers of partners to the population.

Austria, Greece, Hungary, Italy and Slovenia are scoring above the average on all aspects both

in number of Lead Partners and Project Partners and also when relating it to the population.

Bulgaria, Romania and Slovakia are scoring below average on number of Lead Partners (both

in numbers and when relating it to the population). However they score average or above

average on number of Project Partners.

The newest Member States Bulgaria and Romania are not strongly involved as Lead Partners;

however both are represented with large numbers of Project Partners. Relating project partners

to the population, both countries score average or above average among the Member States.

Institutions from countries outside the programme area, e.g. Poland, Belgium and Germany,

also take part in SEE projects as observers, 10% or 20% project partners.

5For Italy the population of the regions involved in the SEE region is being used.


33/182


Participation of the IPA and ENPI Countries

Whereas the ERDF budget of the member states is available for all member states, the IPA and

ENPI funding is available per country. So the participation of the IPA and ENPI countries is first of

all influenced by the budget available.

Of the IPA Countries, Serbias participation is the strongest in the SEE Programme area concerning

the number of project partners. When relating it to the population Montenegro is the strongest

participant with over 40 project partners per million inhabitants. Herewith Montenegro is clearly

scoring above average when relating to the population. The Former Yugoslav Republic of

Macedonia and Montenegro score both average of the IPA countries.

For the ENPI countries, obtaining ENPI funds has been an obstacle. By December 2012 the

Republic of Moldova had signed the Financing Agreement, which allowed for an additional call in

2013 and 6 Moldovan project partners have been able to receive ENPI funding (+1 project from the

4thcall). Unfortunately, the Financing Agreement with Ukraine could not be signed on time and thus

financing of Ukrainian partners is only possible through the 10% rule. Therefore only the Republic

of Moldova is receiving ENPI support (and is indicated above average in the figure below).However, when looking at the involvement of the Republic of Moldova and Ukraine as observers

and 10% project partners, Ukraine is more involved (with 24 10% rule project partners and 9

observers in overall 26 projects) than the Republic of Moldova (with 10 10% rule project partners

and 3 observers in overall 11 projects)6. This is also why in the figure below Ukraine has been

indicated as having an average contribution.

Figure 3.1 Geographical distribution of number of LP and PP per country after 4 calls

Source: Ecorys on basis of information from JTS SEE after 4 calls

6Information from the SEE project website, November 2013

0 PP, 24 10%

7 PP,

10 10%

Below&aboveaverag


34/182


Table 3.3 The geographical distribution of Lead and Project Partners after 4 calls, both in actual numbers and

number per mln habitants. For each column: blue& underscore is above average, red & italicis below average

Source: Ecorys on basis of information from JTS SEE after 4 calls

Danube and Adriatic-Ionian relations

The analysis of the relations from the partnership point of view in the macro regions of Danubestrategy area and Adriatic-Ionian area countries showed in the second evaluation report a stronger

relationship within the Danube strategy area than within the Adriatic-Ionian area when looking at

which countries are involved in the projects.

Currently there seems to be more balance in the programme. About one third of the project has

50% or more of the Danube strategy countries involved, which means 7 or more of the 14

Danube strategy countries7. However, also for the Adriatic-Ionian area around one-third of the

projects have 50% or more of the Adriatic-Ionian area countries8involved, which means 4 or

more of the 7 countries.

7The Danube Programme will cover parts of 9 EU countries (Austria; Bulgaria; Croatia; Czech Republic; Germany (Baden-

Wrttemberg and Bavaria) not whole territory; Hungary; Romania; Slovakia; Slovenia) and 5 non-EU countries (Bosnia

and Herzegovina; the Republic of Moldova; Montenegro; Serbia; Ukraine (not whole territory), having the same

geographical scope than the EU Strategy for the Danube Region.8Parts of Italy, Slovenia, Croatia, Bosnia and Herzegovina, Montenegro, Albania and Greece. As a reference the countries of

the Adriatic CBC programme 2007-2013 have been used.

MS LP LP/mln population PP PP/mln population IPA PP PP/mln

population

AT 17 2,0 111 13,1 AL 34 12,0

BG 1 0,1 143 19,6 BiH 32 8,3GR 17 1,5 152 13,5 FYRoM 23 11,2

HU 17 1,7 156 15,7 MN 25 40,1

IT 46 1,5 184 6,1 RS 80 11,0

RO 6 0,3 170 8,0

SK 3 0,6 58 10,7

SI 15 7,3 109 52,9

Total 122 1,142

HR 59 13,4


35/182


Figure 3.2 Geographical distribution of number of LP and PP per country after 4 calls

Source: Ecorys based on information from JTS SEE

In the second evaluation report also a figure with the privilege relations was included after 3 calls.

The figure showed that the country involved in most projects is Romania, secondly

3rd+Evaluation+Report

Documents

Transcript of 3rd+Evaluation+Report