3rd+Evaluation+Report

download 3rd+Evaluation+Report

of 182

Transcript of 3rd+Evaluation+Report

  • 8/9/2019 3rd+Evaluation+Report

    1/182

    Evaluation of South East Europe

    Programme 2007-2013

    Final report

    Client: VTI Hungarian public non-profit company on regional development & town planning

    Rotterdam, Budapest, Sofia, Madrid, 25 November 2013

  • 8/9/2019 3rd+Evaluation+Report

    2/182

  • 8/9/2019 3rd+Evaluation+Report

    3/182

    Evaluation of South East Europe

    Programme 2007-2013

    Final report

    Client: VTI Hungarian Public Nonprofit Company on RegionalDevelopment and town Planning

    Marie-Jose Zondag &Elvira Meurs [NL]

    CsillaKarlik [HU]

    IrenaVladimirova& Daniel Nigohosyan [BG]

    JavierFernandez&ValentinaPatrini [ES]

    Rotterdam, Budapest, Sofia, Madrid, 25 November 2013

  • 8/9/2019 3rd+Evaluation+Report

    4/182

    2

    II22696

    About Ecorys

    At Ecorys we aim to deliver real benefit to society through the work we do. We offer research,

    consultancy and project management, specialising in economic, social and spatial development.

    Focusing on complex market, policy and management issues we provide our clients in the public,

    private and not-for-profit sectors worldwide with a unique perspective and high-value solutions.

    Ecorys remarkable history spans more than 80 years. Our expertise covers economy and

    competitiveness; regions, cities and real estate; energy and water; transport and mobility; social

    policy, education, health and governance. We value our independence, integrity and partnerships.

    Our staff comprises dedicated experts from academia and consultancy, who share best practices

    both within our company and with our partners internationally.

    Ecorys Netherlands has an active CSR policy and is ISO14001 certified (the international standard

    for environmental management systems). Our sustainability goals translate into our company policy

    and practical measures for people, planet and profit, such as using a 100% green electricity tariff,

    purchasing carbon offsets for all our flights, incentivising staff to use public transport and printing onFSC or PEFC certified paper. Our actions have reduced our carbon footprint by an estimated 80%

    since 2007.

    ECORYS Nederland BV

    Watermanweg 44

    3067 GG Rotterdam

    P.O. Box 4175

    3006 AD Rotterdam

    The Netherlands

    T +31 (0)10 453 88 00

    F +31 (0)10 453 07 68

    E [email protected]

    Registration no. 24316726

    W www.ecorys.nl

  • 8/9/2019 3rd+Evaluation+Report

    5/182

    Table of contents

    3Evaluation of South East Europe Programme 2007-2013

    Executive Summary 5

    Conclusions 5

    Recommendations 9

    Abbreviations 11

    1 Introduction 13

    1.1 Context 13

    1.2 The evaluation and this third and final evaluation report 14

    2 Project generation and selection 15

    2.1 Scope of the evaluation of the project generation and selection 15

    2.2 Assessment of project generation and selection process and instruments 15

    2.2.1

    Project generation and selection 15

    2.2.2 Assessment of supporting tools 21

    2.3 Conclusions and recommendations on project generation and selection 22

    2.3.1 Conclusions 22

    2.3.2 Recommendations 24

    3 Monitoring and implementation 25

    3.1 Scope of the evaluation of monitoring and implementation 25

    3.2 Performance of the programme 25

    3.2.1 Summary of previous findings 25

    3.2.2

    Assessment of the indicator system 253.3 Geographical & Thematic scope of projects 30

    3.3.1 Geographical scope 30

    3.3.2 Thematic scope of the projects 35

    3.4 Monitoring system 37

    3.4.1 Summary of previous findings 37

    3.4.2 Monitoring Wizard 38

    3.5 Programme management system 38

    3.6 Conclusions and recommendations on monitoring and implementation 40

    3.6.1 Conclusions 40

    3.6.2 Recommendations 42

    4 Results of the programme 43

    4.1 Scope of evaluation 43

    4.2 Results of the programme 43

    4.2.1 Summary of the findings of the 1st and 2nd evaluation reports 43

    4.2.2 Analysis of results achieved per PA and AoI 43

    4.2.3 PA 1. Facilitation of innovation and entrepreneurship 47

    4.2.4 PA 2. Protection and improvement of the environment 52

    4.2.5 PA 3. Improvement of the accessibility 57

    4.2.6 PA 4. Development of transnational synergies for sustainable growth areas 63

    4.2.7

    Additional analysis with regard to achievement of results 67

    4.3 Evaluation of the added value 70

  • 8/9/2019 3rd+Evaluation+Report

    6/182

    4

    Evaluation of South East Europe Programme 2007-2013

    4.3.1 Transnational added value 70

    4.3.2 Contribution to European Policies 74

    4.4 Future perspectives 80

    4.5

    Conclusions and recommendations on results of the programme and futureperspectives 82

    4.5.1 Conclusions 82

    4.5.2 Recommendations 85

    5 Programme communication 87

    5.1 Scope of the evaluation for communication 87

    5.2 Key results of the 1stand 2ndevaluation report on communication 87

    5.3 Progress in programme communication in 2012 and 2013 88

    5.3.1 Progress in transnational level communication 89

    5.3.2 Progress in national level communication 93

    5.4

    Conclusions and recommendations on communication 95

    5.4.1 Conclusions 95

    5.4.2 Recommendations 96

    Annex 1: Treatment of recommendations 97

    Annex 2: Performance of programme indicators 105

    Annex 3: Results of the programme 109

    Annex 4: Indicators on communication 113

    Annex 5: Questionnaire 117

  • 8/9/2019 3rd+Evaluation+Report

    7/182

    5Evaluation of South East Europe Programme 2007-2013

    Executive Summary

    This executive summary contains the conclusions and recommendations of the third and final

    evaluation report from the ongoing evaluation of the transnational programme for South East

    Europe 2007-2013.The focus of the report is on the results of the programme and the threeevaluation reports. Furthermore the monitoring and implementation, the communication activities

    and the added value of the programme have been evaluated.

    Overall it can be concluded that the transnational programme for South East Europe 2007-2013 is

    functioning well. More details are provided below.

    Conclusions

    Conclusions on the project generation and selection procedure

    Q1. Do the project selection criteria effectively support the selection of the best quality transnational

    projects?

    1. A clear trend towards the fine tuning/narrowing of the spectrum of selection requirements to

    more concrete assessment criteria can be evidenced;

    2. A steady adoption of a flexible approach to formal checks en route to a more in-depth analysis

    of project content, transnational partnership development and consistency has been detected;

    3. Gradual reduction of the weight of the administrative burden by the introduction of more flexible

    approaches to formal and eligibility checks;

    Q2. Is the project selection procedure clear and transparent?

    4. Increasing evidence of strategic awareness of project applicants and better project fitting and

    integration into the policy transnational framework.5. Although the duration of the procedure has been reduced in the 4th call with respect to the 2 nd

    one, this has still resulted in a long process.

    6. Despite the partial reduction of the administrative burden (for instance by accepting scanned

    documents and faxes), which is judged as a positive improvement, the (sometimes necessary)

    administrative requirements may still represent an obstacle for potential quality projects to

    apply for financial support;

    7. The geographical and leadership imbalances of projects, although decreased during the

    programme life cycle, can still be observed and still need to be reduced.

    8. A trend towards offering more information and feedback on the non-selected project proposals

    has been observed by the national bodies and JTS thus increasing the awareness and

    knowledge of what worked wrong with non-successful applications. This, without any doubt,has contributed to the enhancement in the quality of future project proposals.

    9. Also, the use of a two-step approach has supported the generation of quality projects in the

    context of the Programme, especially with the third top down and targeted call.

    10. A more strategic evaluation analysis has been observed, with a shift from an administrative

    check towards a more thematic approach while safeguarding objectivity in the project selection

    procedure through the 6-eye principle.

  • 8/9/2019 3rd+Evaluation+Report

    8/182

    6

    Evaluation of South East Europe Programme 2007-2013

    Conclusions on performance of the programme

    11. The South East Europe Programme is performing well. The ERDF funding has been fully

    committed to 122 projects after 4 calls for proposal. Originally it was foreseen to support 130

    projects of average budget of 1.5 MEUR each. However the specific character of 3rd call

    strategic projects resulted in much higher allocations per project.

    12. During the first and the second evaluation report only a limited quantitative assessment was

    possible. In the meantime improvements have been made in the monitoring system. The

    introduction of the Monitoring Wizard tool ensures that project achievements can now be linked

    to programme objectives. The results will be available in the coming Annual Implementation

    Report.

    13. The Wizard shows that the programme is on schedule with achieving its objectives. 55% of the

    output and result indicators are on schedule in achieving their targets (21 indicators have

    achieved 45-65% of the target) or are far above the expected figures, 16 indicators have

    achieved more than 65% of their target). 45% of the indicators are slightly or largely behind

    schedule: 18 indicators have achieved 35-45% of their target and 13 indicators have realised

    less than 35% of their target.

    Conclusions on the geographical & thematic scope of projects14. In comparison with the situation in the first and second evaluation reports the geographical

    balance of projects over the programme area remains largely the same. All programme

    participating countries are covered.

    15. Of the IPA countries the participation of Croatia and Serbia are the strongest in the SEE

    programme with respectively 59 and 80 project partners.

    16. The country most cooperated with is Romania as they are in most projects (107 projects) and

    with whom is being cooperated most (701 relationships), followed by Hungary, Italy (both in

    102 projects and respectively 668 and 670 relationships), Bulgaria (in 92 projects and 600

    relationships), Greece (in 85 projects and 566 relationships) and Slovenia (in 82 projects and

    547 relationships).

    17. In comparison with the first and the second evaluation report the programme is more balanced

    over the different priority axes.

    18. Based on the current high level of cooperation between partners coming from the two future

    programme areas, the evaluator considers that 20% flexibility may not be enough to meet

    cooperation demand in the future.

    Conclusions on the Monitoring System

    19. In the 2012 evaluation report, both programme management bodies as well as the projects

    indicated that IMIS was working well, and that all important data had been collected from the

    projects to monitor their progress.

    20. The monitoring system IMIS has been supplemented in 2013 with the Monitoring Wizard. Mostimportant difference between IMIS and the Monitoring Wizard is in terms of establishing links

    between project and programme indicators. However, also other improvements are made in

    the Wizard (e.g. actual value/target value, no. of contributions).The Wizard also provides more

    clarity for the beneficiaries on the interpretation of the indicators.

    21. A major improvement according to the evaluator is that the Monitoring Wizard also collects

    qualitative information on the nature of projects contribution to the programme objectives.

    This means that a project may contribute to programme objectives through one or more of its

    output/s and results. In the coming annual report the results on this programme result indicator

    will be presented.

  • 8/9/2019 3rd+Evaluation+Report

    9/182

    7Evaluation of South East Europe Programme 2007-2013

    22. In 2011 JTS started to adapt the IMIS towards the integration of the IPA and ENPI Funds. IPA

    Funds have been integrated since the 3rd call. For ENPI there are 6 projects which will be

    monitored in a separate way.

    Conclusions on the Programme Management system

    23. The JTS, MA, MC, SCP, CA and AA are in general working well. There is good cooperation

    between and within the bodies.

    24. There continues to be a difference in the experience of the SCPs. The SCPs could have a

    good role in the implementation of the programme.

    Conclusions with regard to results: data and methodological limitations

    25. It is important to highlight the data and methodological limitations before summarising the

    findings on the results achieved in order to interpret correctly the conclusions.

    The conclusions are based on the implemented projects under the first call for proposals (30%

    of all projects). The additional information on the number, distribution along the AoI and

    expected contributions of the contracts concluded under the 2nd, 3rd and 4th call for proposals

    allowed making projections on the expectations with regard to achievement of programmes

    objectives.The conclusions are based mainly on the data collected through the SEE programme

    monitoring system and Final Activity Reports (FARs). While the quantitative data provides

    valuable information, its interpretation was hindered by the lack of background and reliable

    qualitative information. The diversity of topics and problems covered did not allow going in-

    depth into the technical issues, which was exacerbated by the huge number of projects and

    beneficiaries. Further, only 65% of the 40 completed projects submitted FARs in time to be

    made available for the evaluation. While the common programme and project indicators allow

    for a comparison between the PA, the lack of specific thematic indicators per PA made it

    impossible to analyse subject specific data.

    To overcome the difficulties commented above the evaluation team implemented in-depth

    review on the results achieved of 8 projects (2 per PA), which was supported by interviews with

    the relevant project managers. The projects are presented in the relevant sections of this report

    in order to illustrate the achievements and their contribution to programme objectives and

    thematic codes of the EC implementing regulation.

    Conclusions with regard to results: achievement of programme objectives

    26. From the 4 PA supported under the SEE programme the greatest contribution to achievement

    of the planned specific programme objectives so far has been made PA 2 Protection and

    improvement of the environment. The contributions under this PA are slightly over the planned

    contributions and the implementation so far shows good prospects for achievement of the

    planned objectives related to this PA. Thus the SEE programme is expected to contribute tosustainable development and improvement of environment.

    The programme made progress and is likely to achieve its specific objective related to

    facilitation of innovation and entrepreneurship, knowledge economy and information society

    (PA1), despite that the expected contributions under this PA are about 12 % lower than

    originally planned. A suboptimal involvement of the public sector was observed in 1 st call

    projects, particularly under AoI1.3. However, this was significantly improved following the

    targeted AoI1.3 second call and public sector involvement is high in the 3 rdCall project and

    positive in the 4thCall.

    27. It is likely the projects implemented under PA4 Development of transnational synergies for

    sustainable growth areas will contribute to enhancement of regional and transnational

  • 8/9/2019 3rd+Evaluation+Report

    10/182

    8

    Evaluation of South East Europe Programme 2007-2013

    cooperation and integration. The projects implemented so far show potential with this regard as

    well as with regard to making SEE regions more attractive (e.g. through urban regeneration

    and tourism based on cultural values).

    28. The implementation of PA3 Improvement of the accessibility is less advanced compared to the

    other PA since most of the projects are still in progress. Therefore conclusions with regard to

    the achievement of specific objective related to physical and virtual accessibility could not be

    made at the time of this analysis. There is so far little contribution to improved virtual

    accessibility. However the ongoing projects have the potential to bring the planned results with

    this regard.

    29. On the basis of the above it could be expected that the SEE programme would contribute to

    improvement of the territorial, economic and social integration process in South East Europe

    and to cohesion, stability and competitiveness of the area. The extent to which this will be

    achieved depends on how the programme will address the factors that hamper achievement of

    results under the specific PA and will improve transnational and regional cooperation. The

    Capitalisation Strategy initiated could have positive role in this process.

    Conclusions on communication

    The main conclusions of the 3rd

    report concerning communication are the followings:30. In general it can be said that the recommendations of the previous evaluation reports are taken

    into consideration by the JTS, and the communication plan of the programme for 2012 and

    2013 integrated some of the recommendations.

    31. A significant majority of the stakeholders1 is satisfied with the quality information provided in

    time by the JTS in all phases of the programme implementation.

    32. The events organized by the JTS were considered to be very useful, and the JTS provided

    much more opportunities for the project stakeholders to exchange their experiences and good

    practices (annual conference, thematic seminars.)

    33. Significant improvement can be observed in the presentation of the results in programme

    communication to the general public (e.g. video, developed website, etc), but there is still place

    for further developments in creating connections with the media.

    34. The capacity, expertise and motivation of the SCPs concerning the communication activities is

    still very different in the different member states, and it seems that there is no real chance to

    change this situation significantly in the frame of this programme, however it should be done in

    the next programming period.

    35. The establishment of links with the national, regional and local media was not in the focus in

    2012 and 2013 at member state level.

    36. Not all the communication indicators are monitored appropriately and it seems unrealistic to

    complete the missing data from national and project level.

    1Please have a look at sub-chapter 5.3.1. for the concrete values

  • 8/9/2019 3rd+Evaluation+Report

    11/182

    9Evaluation of South East Europe Programme 2007-2013

    Recommendations

    Recommendations on project generation and selection procedure

    1. Reduce the duration of the project selection process to the possible minimum, i.e. by covering

    administrative and contracting tasks in shorter timeframes.

    2. Further reduce the subjectivity factor, by continuing the improvement of the work done until

    now to concretise selection criteria.

    3. To include sustainability as a more determinant and relevant criterion for project selection in

    future calls, by allocating it a higher weight in the overall scoring.

    4. To give a higher importance to innovation in the territory and its geographical character in the

    selection criteria by identifying strategic projects not only in targeted calls, but also in the

    traditional bottom up calls for Proposals.

    5. To ensure a better geographical balance, by encouraging the most recently accessed countries

    to take the lead in (part of) projects, thus increasing the transnational balance of the

    programme as a whole.

    6. To increase even more capacity building actions for SCPs to ensure a higher geographical

    coverage and quality of projects content.

    7. To continue providing transparent and detailed information (feedback) regarding eligibilityresults of the selection procedure to project applicants in order to reinforce their capacity and

    contribute to the generation of higher quality projects for future calls.

    Recommendations on monitoring and implementation

    8. It is recommended to continue the already started capitalisation of the programme results and

    pay more attention to promote the results in the coming period.

    9. It is recommended to draw the attention of the decision makers that with the splitting of the

    current SEE programme area into 2 new transnational programmes (Danube, Adriatic-Ionian)

    the current high level cooperation between the countries of the two new programme areas will

    be reduced significantly, furthermore there is one country (the Former Yugoslav Republic of

    Macedonia), which is not included in any of the new programmes. It is highly recommended to

    use the flexibility rule at maximum level to ensure the future cooperation as wide as possible.

    10. It is recommended for the next programming period to pay special attention to the functioning

    of the SEE Contact Points, and ensure that the less experienced SEE Contact Points get the

    appropriate support and motivation to be able to catch up with the others, and ensure the

    quality of the programme management with it in all participating countries.

    11. As far as possible try to see for next programming period how the integration of IPA and ENPI

    can be included from the start of the programme.

    Recommendations on the indicators

    12. It is recommended to draw the attention of the project beneficiaries of those projects, which arecurrently under implementation, to the fulfilment of those 16 output and result indicators, which

    are currently behind the target.

    13. It is recommended for the next programming period that the targets of project indicators should

    be set at the start of the programme, and they should not be an adding up from figures coming

    from project applications.

  • 8/9/2019 3rd+Evaluation+Report

    12/182

    10

    Evaluation of South East Europe Programme 2007-2013

    Recommendations on results of programme

    14. In the design of the Priority axes for the follow on and/or similar programmes specific measures

    should be taken in order to ensure more representation of the private sector in actions for

    improving accessibility (PA3) and more involvement of public institutions in facilitation of

    innovation and entrepreneurship (PA1).

    15. The electronic monitoring system of the next programmes should ensure smooth integration of

    IPA and ENPI funds. In addition, the functions for monitoring output and result indicators and

    activities should be improved providing for clearer link between the two categories.

    16. Collective efforts by the MA, the JTS and the participating countries should be taken for the

    next programming period (next programmes) in order to harmonise the procedures in the

    different countries with special attention on simplifying the reporting procedures.

    Recommendations on communication

    17. It is recommended to continue the presentation of the project results to the general public and

    make further efforts to do it at national level also (e.g. establish links with national, local media,

    such as newspapers, TV channels etc.).

    18. It is recommended to continue the already started capitalisation process, ensure the

    appropriate interactivity level of the events in order to provide opportunities for the projects toexchange their experiences and create links, synergies among themselves.

    19. It is recommended for those SCPs who are currently less active in the national level

    communication to give importance to the communication activities in their countries for the rest

    of the programme, and ask support from the JTS, if it is needed.

    20. It is recommended for the next programming period to reduce the differences among the

    countries concerning the quality of the national level communication. Define at the beginning of

    the programme the annual minimum level requirements for the SCPs concerning the national

    level programme communication activities, and it should be monitored by the JTS / MA on a

    regular basis.

    21. It is recommended for the next programming period to ensure from the beginning of the

    programme the collection of the relevant data about the national level media appearances.

  • 8/9/2019 3rd+Evaluation+Report

    13/182

    11Evaluation of South East Europe Programme 2007-2013

    Abbreviations

    Abbreviation and its full name

    AA Audit Authority

    AC Acceding Country

    AF Application Form

    CA Certifying Authority

    CC Candidate Country

    CfP Call for Proposals

    DEU Delegation of the European Union

    ENPI European Neighbourhood Policy Instrument

    EC European Commission

    ERDF European Regional Development Fund

    FLC First Level ControlGIS Geographical Information System

    IMIS Integrated Monitoring and Information System

    IPA Instrument for Pre-Accession

    IVB Transnational European Territorial Cooperation Programme IVB of 2007-2013 (Interreg)

    JTS Joint Technical Secretariat

    LP Lead Partner

    MA Managing Authority

    MC Monitoring Committee

    MoU Memorandum of Understanding

    MS Member State

    OP Operational Programme

    PCC Potential Candidate Country

    PO Project Officer

    PP Project Partner

    PPP Public Private Partnership

    R&D Research & Development

    SCP SEE Contact Point

    SEE South East Europe

    SME Small and Medium Enterprises

    ToR Terms of Reference

    Abbreviation of countries and their full name

    AL Albania IT Italy

    AT Austria MD Moldova

    BG Bulgaria ME Montenegro

    BiH Bosnia and Herzegovina RO Romania

    EL Greece SI Slovenia

    MK Former Yugoslav Republic of Macedonia SK Slovakia

    HU Hungary SR Serbia

    HR Croatia UA Ukraine

  • 8/9/2019 3rd+Evaluation+Report

    14/182

  • 8/9/2019 3rd+Evaluation+Report

    15/182

    13Evaluation of South East Europe Programme 2007-2013

    1 Introduction

    1.1 Context

    The South East Europe Programme Area is the most diverse, heterogeneous and complextransnational cooperation area in Europe, made up of a broad mix of 16 countries with a total

    population of 200 million people. For 14 countries the eligible area is the whole territory of the

    country and in 2 countries only certain regions are eligible under the SEE programme 2. These

    countries are acting as a bridge between North, South, East and West Europe in the European

    transportation network system. The stability, prosperity and security of the region are of significant

    interest to the EU. The emergence of new countries and with it the establishment of new frontiers

    has changed the patterns of political, economic, social and cultural relationships.

    What differentiates the South East Europe Programme is not only the number of countries

    participating in it, which makes it the transnational programme with the largest cooperation area,

    but also the diversity of the participating countries. This is the only transnational programme area

    with such a large number of non-EU countries participating. Out of the 16 participating countries 8

    are EU Member States, 6 are candidate and potential candidate countries and 2 are countries

    participating in the European Neighbourhood Policy. This is a highly complex programme which

    presents challenges such as ensuring good mechanisms to contract partners who receive funding

    from different instruments: ERDF (European Regional Development Fund), IPA (Instrument for Pre-

    accession Assistance) and potentially ENPI (European Neighbourhood and Partnership

    Instrument).

    2 In Italythe eligible regions are: Lombardia, Bolzano/Bozen, Trento, Veneto, Friuli-Venezia-Giulia, Emilia Romagna, Umbria,

    Marche, Abruzzo, Molise, Puglia, Basilicata.

    In Ukraine: Cjermovestka Oblast, Ivano-Frankiviska Oblast, Zakarpatska Oblast and Odessa Oblast

  • 8/9/2019 3rd+Evaluation+Report

    16/182

    14 Evaluation of South East Europe Programme 2007-2013

    1.2 The evaluation and this third and final evaluation report

    The objective of the on-going evaluation of the SEE Programme is to improve its performance. The

    Operational Programme has indicated that the evaluation should improve the quality, effectiveness

    and consistency of the use of assistance, the strategy and the implementation of the programme.

    A first evaluation report has been developed in 2011, a second evaluation report has been

    developed in 2012 and this third and last evaluation report is summarising the results and doing a

    third analysis in 2013.

    This third and final evaluation report focuses on analysing results of the programme (chapter 4.),

    summarising and up-dating the information about project selection and monitoring and

    implementation (Chapters 2 and 3.), making recommendations for the future (all chapters) and

    evaluating the communication activities (Chapter 5.). In the annexes additional information can be

    found.

    As approach used for the evaluation each year a survey (web-based questionnaire) has been held.

    See the separate annex document for the results of the last questionnaire. About 60% of the 122

    projects filled the questionnaire (74 people). About 60% of the full JTS and MA filled thequestionnaire. Only 18% of the MC members and 27% of the SCP filled the questionnaire. The

    questionnaire was sent to 306 persons (17 JTS/MA staff, 53 MC member, 33 SCP colleagues

    (some are both MC member and SCP), 210 beneficiaries). The evaluators got feedback from 101

    persons in the survey in 2013 (33%). 10 of them represent the JTS and MA, 8 of them are MC

    member, 74 project beneficiaries, and 9 of them represent SCPs. The 9 answers from the SCPs

    represent 8 countries from the SEE programme area.

    Furthermore interviews have been held each year with the Managing Authority and the Joint

    Technical Secretariat. Lastly desk research was an important source of information and the

    evaluators participated in most Monitoring Committee meetings.

  • 8/9/2019 3rd+Evaluation+Report

    17/182

    15Evaluation of South East Europe Programme 2007-2013

    2 Project generation and selection

    2.1 Scope of the evaluation of the project generation and selection

    This chapter presents the findings of the evaluation on project selection and generation. Accordingto the evaluation plan, three questions have been answered:

    Q1. Do the project selection criteria effectively support the selection of the best quality transnational

    projects that contribute to the objectives of the programme?

    Q2. Is the project selection procedure clear and transparent?

    Q3. How could the selection process be improved? - Recommendations

    2.2 Assessment of project generation and selection process and instruments

    2.2.1 Project generation and selection

    The selection of high quality transnational projects is strongly related to the capacity of the

    programme to generate good project ideas and partnerships and to adopt appropriate evaluation

    criteria. Taking into account the current stage of the Programme, the answer to Q1 mainly focuses

    on the evolution of the Programmes capacity to select relevant projects by the use of appropriate

    selection criteria. Since the structure and approach of each call also determines the use of specific

    criteria, in this section the evaluators briefly analyse the calls selection processes and criteria, by

    identifying the most relevant characteristics of each call and analysing their evolution over time,

    also taking into account the extent to which the Programme has incorporated the recommendations

    formulated in the 1stand 2ndevaluation reports. Complementary to previous analyses, the results of

    the 4th

    call and the two restricted calls launched by the Programme are also covered in this section.

    The calls of the Programme

    To complete the analyses on the calls for proposals carried out in the previous evaluation reports, it

    is necessary to assess the performance of the 4 th one, which was still open at the moment of

    finalising the second evaluation report. The one-step procedure and more traditional bottom-up

    approach adopted in this last call were oriented at allocating the budget remaining from the

    previous calls. From the initial 342 applications received a lower number with respect to the other

    calls due to the one-step approach a first set of 37 projects was approved, implying an initial

    success rate of 10.8%. Nevertheless, taking into account the under-spending of projects approved

    under the 1stand 2ndcalls, a reserve list of projects was foreseen, allowing a later approval of an

    additional number of 10+1 projects (which is in total 48 projects under the 4 thcall), thus increasingthe success rate of the 4 thcall up to 14.04%.

    When compared to previous calls, this success rate is a decrease to 3 rd (strategic) call but an

    increase to 1stand 2ndcalls (all previous calls had a two-step application procedure). The highest

    number of proposals was submitted under the area of intervention 4.3, showing an important

    interest although on average presenting the lowest quality level. On the other hand, over-

    dimensioned budgets were submitted in several cases, as a result of the Programmes end in 2013,

    reinforcing the perception of last try opportunity.

    Nonetheless, it is important to stress that both the experience gained by the Programmes

    Authorities throughout the Programme life cycle, the clear definition and knowledge of what is

    expected by the SEE Programme as well as the gradual coaching and higher understanding of the

  • 8/9/2019 3rd+Evaluation+Report

    18/182

    16 Evaluation of South East Europe Programme 2007-2013

    programme by project partners made the difference in the selection of relevant and policy

    integrated projects during the 4 thcall. The successful projects are characterised by an innovative

    nature, a considerable comprehension of the topics, a high relevance for the programme area, a

    coherent and tailored logical frame as well as a strong commitment demonstrated by applicants.

    Complementary to the four main calls, the Programme also launched two restricted complementary

    calls with the objective to achieve a higher commitment of the IPA and ENPI allocated funds. This

    was put in practice through the involvement of additional partners from selected countries into

    existing partnerships and projects, consequently ensuring an important added value from both a

    territorial and a content-wise perspective. The first complementary call was an IPA call oriented to

    Montenegro, directed to the existing projects selected under the 2 ndopen call for proposals of SEE,

    whose Lead Partners were invited to involve additional partners from Montenegro by adjoining or

    incorporating them to their partnerships. The second one, an additional ENPI call for partners

    from Moldova, was foreseen as a way to compensate a substantial leftover of ENPI funds for this

    country which remained non-allocated to the projects. This call was opened to the Lead Partners

    from the 3rd and 4th calls willing to additionally involve Moldavian partners. Both restricted calls

    foresaw clear steps that involved different actors and criteria.

    The innovative way to manage external funds, the implication of both national and programmes

    authorities and a strong focus on the quality of the project idea and partnership has helped to

    ensure a high coherence with the Programmes priorities and rules as well as to overcome the

    challenges related to the use of different typologies of funds in a complex environment. As a result

    of this targeted strategy, IPA funds have been fully integrated into the SEE Programme and ENPI

    funds show a good alignment to the existing framework.

    To conclude this section, and in a synthetic way, the main characteristics of the different calls and

    their related approaches, processes and selection criteria have been summarised in a table that

    can be found below. This table shows how the Programme has used different approaches in the 4

    calls, evolving to more sophisticated methodologies, such as the two-step procedure and targeted

    and restricted calls, as well as improving the assessment criteria definition towards the selection of

    more integrated projects.

  • 8/9/2019 3rd+Evaluation+Report

    19/182

    17Evaluation of South East Europe Programme 2007-2013

    Table 2.1 Overview of the calls of the Programme

    Items 1st call 2nd call 3rd call 4th call

    Process

    Steps of

    the call

    2 steps (first Expression of Interest after

    which a selection of applicants is invited to

    submit full proposal)

    2 steps 2 steps 1 step: single step procedure

    whereby all applicants have to

    submit the whole package of

    information

    Approach

    of call

    Fully open - Bottom up Open targetedBottom up StrategicTop down

    Targeted and restricted call

    Competitive, fully open

    Traditional bottom-upDates May 2008March 2009 December 2009December 2010 June 2010February 2012 October 2011June 2012

    Success rate 822 EoIs, 40 projects approved: 4.86% 423 EoIs, 26 approved: 6.14% 49 EoIs, 8 projects approved: 16.32% 342 full applications submitted, 48

    projects approved: 14, 04%

    Selectioncriteria

    Formal

    and

    Formal check:

    Proposal submitted within deadline;

    Application submitted in English;

    Official application form used;

    All required parts and annexes are filled in

    and submitted.

    Eligibility check:

    Partners from a minimum of 3 partner

    states, located in the eligible area, of

    which at least one from a Member State;

    Project implementation has not startedand it wont start prior to the approval of

    the AF;

    Lower administrative burden

    Clear and efficient rules for participation to

    the call

    Slight improvement in clarity, transparency and

    time efficiency

    Flexible approach to eligibility check in terms of

    administrative documents

    Eligibility criteria more clearly defined with

    respect to previous calls:

    Timely submission of application form;

    The project includes all the mandatory activities

    as described in the ToR for the call per priority;

    Lead Applicant and financing partners are

    eligible;Project fulfils minimum requirements for

    transnational strategic partnerships (same as

    previous calls);

    No formal check

    Minimum modifications in

    comparison with previous calls

    Like in previous calls,

    networking/exchange of best

    practices as stand-alone activities,

    as well as proposals with an

    academic focus but no

    developments of transnational

    policies/strategies are not eligible.

  • 8/9/2019 3rd+Evaluation+Report

    20/182

    8 Evaluation of South East Europe Programme 2007-2013

    Items 1st call 2nd call 3rd call 4th call

    Selectioncriteria

    Eligibilit

    y check

    Project complies in general with the OP

    and has impact in the programme area;

    Proposed types of activities are eligible;

    Beneficiaries have secured co-financing;

    Beneficiaries are eligible.

    The composition of the partnership is in line

    with the prescriptions set out in the ToR for

    the call per priority;

    The specific partnership requirements

    regarding the limitation of number of projects

    to participate in as LP/PP are respected;

    Financing partners (ERDF/IPA-I)cooperate jointly;

    ERDF/IPA-I partners have secured public co-

    financing;

    ENPI partners have submitted all necessary

    documents.

    Quality

    check

    Is the partnership composition well

    justified and able to contribute in a

    balanced manner to the implementation of

    the transnational co-operation project?

    Do beneficiaries have the sufficient

    institutional and technical capacity to

    achieve the project aims?

    To what extent is the project expected to

    provide a significant contribution to the

    overall objectives of the programme?To what extent is the project expected to

    provide a significant contribution to the

    Priority Axes and Areas of Intervention of

    the programme?

    Are project deliverables clearly defined

    Transnationality of partnership criterion

    related to the balance of partnerships

    composition

    Higher focus on Previous experience vs.

    cooperation and partners balanced

    involvement

    Higher focus on clarity vs. logic conceptual

    approach and work plan

    Concreteness and usability of project

    deliverablesMaintenance of the value for money

    criterion

    Higher focus on the nature of the problem

    tackled by the project

    Lower focus on the time perspective vs.

    Quality selection criteria are highly coherent with

    the ToRs

    5 Terms of reference with specific thematic

    focus detailed the compulsory activities to be

    implemented in order to reach the expected

    outputs

    Strategic requirements: relevance to the

    integration of the SEE programme area;

    involvement of key figures in the field and long-

    term perspective and impactHigher focus on sustainability, innovation and

    geographical character

    Minimum modifications in

    comparison with previous calls

    A minor adjustment can be seen,

    adding one specific question on

    the sustainability component

    Questions related to investments

    and equipment costs have been

    dropped

    Project activities detail is assessed

  • 8/9/2019 3rd+Evaluation+Report

    21/182

    19Evaluation of South East Europe Programme 2007-2013

    Items 1st call 2nd call 3rd call 4th call

    and are the assumptions on their use

    realistic?

    Value for money

    Are there synergies and /or

    complementarities with other territorial co-

    operation projects or other EU funded

    policies and programmes?

    transnationality impact

    Sound viability plan

    Innovation component

    Sustainability component (political,

    institutional and financial)

    Comments

    Strengths 1stcall: In general terms, the features of the

    SEE best quality Project and the quality

    selection criteria share a high level of

    correspondence.

    2nd call: The two step approach allows

    focusing on project content and partnership

    levels vs. administrative requirements, thus

    producing high quality projects

    3rd call: Strong territorial marketing during pre-

    announcement period

    Use of knowledge and expertise from other EU

    programmes

    High competition at SCP level

    Support to territories with limited presence in the

    SEE programme

    Partially effective key stakeholders involvement

    by bottom up approach

    4th call: Increased evidence of

    strategic awareness

    Evidence of in-country stakeholder

    development

    More concrete and direct

    capitalisation

    Weakne

    sses

    The number of projects per country shows a

    geographical imbalance in the SEE

    programme where most projects that ensure

    a high innovation in the territory and most

    lead applicants come from very experienced

    countries

    Long two-step process

    A certain level of subjectivity detected

    Repetitions of similar requirements in more

    than one criterion

    Streamline the formal and eligibility checks

    under one single categoryGeographical imbalance due to the

    experience quality requirement

    Need for a higher balance in the relevance

    and weight of transferability, sustainability

    and innovation as selection criteria/sub-

    Strong imbalance in the leadership of projects is

    confirmed

    No minimum quality threshold per sub-criterion

    was defined

    Majority of Lead Partners from few

    countries

    Duration of the selection process

    Possibility to interpret this 4th call

    as a last trythe last chance to

    get access to the programmefunding.

  • 8/9/2019 3rd+Evaluation+Report

    22/182

    0 Evaluation of South East Europe Programme 2007-2013

    Items 1st call 2nd call 3rd call 4th call

    criteria

    Poor communication and information on the

    project selection results to applicants

    Source: Ecorys on basis of information from SEE programme and previous evaluation reports

  • 8/9/2019 3rd+Evaluation+Report

    23/182

    21Evaluation of South East Europe Programme 2007-2013

    2.2.2 Assessment of supporting tools

    Complementary to Q1, this part analyses the main supporting tools that have been developed and

    provided by the Programme to project potential beneficiaries and the clearness and transparency of

    the selection processes adopted in the different calls.

    The supporting tools

    In general terms, the assessment made by project beneficiaries with respect to the usefulness of

    the available supporting instruments for the generation of quality projects is positive.

    The support provided by programme managers, the programme manual, the application packs and

    the applicants guidelines are considered particularly useful tools. Nevertheless, a lower grade of

    satisfaction can be observed with respect to the usefulness of other instruments, such as the

    project idea database, instruction phase seminars focused mainly in financial and administrative

    issues and the limited language coverage of official documents that were not always available in all

    the languages of the covered countries. Although not being a requirement for any programme

    authority and thus being a voluntary initiative of the SEE contact points, it could have providedadditional support to project partners and potential beneficiaries.

    The surveys to project beneficiaries, that have been carried out for the purpose of producing the

    different evaluation reports3, show that no relevant differences over time can be observed on the

    perception of the usefulness, user-friendliness, accessibility, richness on quality information and

    time availability of the different tools that have been put at the disposal of project beneficiaries in

    the different calls of the Programme. This can be also confirmed on the basis of the information

    provided by the consulted beneficiaries of the 4 th call. The table above summarises the

    beneficiaries perception on thefollowing:

    The Instruction phase seminars organised have successfully assisted applicants on financial

    and administrative topics and issues At this respect complementary content-wise and thematic

    oriented seminars organised at national level would have increased the usefulness of this tool;

    The Project idea database, although representing a tool with a high potential, has only partially

    fulfilled the applicants expectations and has been scarcely used, confirming that this tool needs

    to be more intensively advertised among potential applicants in order to attract their attention;

    The capacity of national structures to accompany potential beneficiaries throughout the process

    of project generation is a key aspect that was considered as very effective by potential

    beneficiaries. The coaching and steering role of the JTS has been crucial in the different calls,

    and especially in the 3rd

    one, in order to tailor the projects to the calls requirements and its

    expected results;

    The efficient communication and exchange of experiences, as well as information and publicityevents, are key factors in all the phases of the programme cycle. At this regard, the strong

    territorial marketing during the 4th call pre-announcement period has been certainly recognised;

    Meetings between potential Members States and IPA countries' beneficiaries have been of

    extreme relevance and utility for both ENPI and IPA calls.

    To sum up, there has been an increasing consciousness of the relevance of constant project

    support and its correlation with project quality from Programmes Authorities that has resulted in a

    more complete and comprehensive selection procedure.

    3All Lead Partners have been invited to answer the web-based questionnaire of which 60% of the projects (74 of the 122

    projects) have filled out the questionnaire.

  • 8/9/2019 3rd+Evaluation+Report

    24/182

    22 Evaluation of South East Europe Programme 2007-2013

    Clarity and transparency of the selection procedure

    The beneficiaries perception of transparency is positive, although to some consulted project

    beneficiaries of the 4thcall, more information should have been provided by programme managers

    to project applicants regarding the evaluation of their proposals. At this respect, it is important to

    stress that previous recommendations were already taken into account starting from previous calls,

    where a detailed evaluation summary was provided to the applicants on a request basis.

    Nevertheless, the automatic provision of the evaluation sheets with comments for each project as

    well as the publication of detailed scoring are factors which have still been underlined as necessary

    by project applicants of the 4 thcall. The improvement of the already started path would allow project

    applicants to learn from previous mistakes and contribute to higher quality proposal in the future.

    On the other hand, previous evaluation analysis recommended the introduction of certain elements

    of flexibility at the time of carrying out the formal and eligibility checks; capacity building actions for

    SCPs in order to improve the quality of the assistance; and strong support to territories which have

    a more limited presence in the Programme, especially with respect to projects leadership. In this

    sense, positive steps have been adopted at Programme level in order to take into account these

    recommendations and to improve clarity and transparency in the selection process, thus:

    Regarding transparency, National bodies and JTS are conscious of the need to increasetransparency where reasons for ineligibility should on request - be explained transparently to

    lead partners. Moreover, the quality evaluation analysis applied the 4-eyes principle with an

    independent expert scoring of projects and in parallel a JTS project manager also made an

    assessment. In case of serious scoring gap, a third person (from the JTS) was also involved to

    safeguard an objective approach to project selection;

    Regarding clarity, improvements have been made by the JTS in the selection approach

    adopted, not anticipating the quality assessment but trying to understand if the projects comply

    with the Priority Axis, Area of Intervention and the Programme in general. At this respect, more

    flexibility was guaranteed under the projects assessment of the 4th

    call as a way to overcome

    possible constraints related to the complexity of the application form (f lexibility in the correlation

    between the assessment grid and the application form was ensured in favour of the

    applicant).The corrections and improvements introduced in the selection criteria also

    contributed to increase the clarity and transparency of the whole selection process.

    2.3 Conclusions and recommendations on project generation and selection

    2.3.1 Conclusions

    The table below summarises the conclusions and recommendations of the first and second report

    and provides also the conclusions and recommendations of this final revaluation report on projectselection and generation. This is explained in more detail in the text after the table.

    Table 2.2 Overview of the answers on the evaluation questions

    Evaluation

    question

    Source used 1st& 2

    ndEvaluation report Final Evaluation report: answer +

    recommendation

    Topic 1 Assess project selection procedure and criteria

    Q1. Do the

    project selection

    criteria effectively

    support the

    selection of the

    best quality

    Desk research

    Interviews

    Mostly, high matching

    between criteria and best

    quality transnational

    projects can be observed.

    However, still need to

    further adjust selection

    o Mostly, more flexible approach,

    important alleviation of administrative

    burden, trends towards the fine tuning of

    selection criteria, included more focus on

    sustainability and innovation in the territory.

    o Recommendations for the future:

  • 8/9/2019 3rd+Evaluation+Report

    25/182

    23Evaluation of South East Europe Programme 2007-2013

    Evaluation

    question

    Source used 1st& 2

    ndEvaluation report Final Evaluation report: answer +

    recommendation

    transnational

    projects?

    criteria. High coherence

    between 1st,2nd and 4th

    call.

    To reduce the subjectivity factor by further

    working in concretising selection criteria,

    reduce even more administrative

    requirements; to increase the weight of the

    sustainability criterion for project selection

    and to give a higher importance to the

    geographical balance.

    Q2. Is the project

    selection

    procedure clear

    and transparent?

    Desk research

    Questionnaire

    among

    projects

    Mostly: the 2 step

    approach focuses on

    quality, but project

    generation capacity needs

    to be further enhanced

    (geographical imbalance,

    low flexibility, subjectivity).

    Third call with a top down

    approach assured ahigher geographical

    coverage and the

    involvement of relevant

    entities.

    o Mostly: Increasing evidence of

    strategic awareness of project applicants

    and better project fit and integration into the

    policy transnational framework [project

    selection and project generation are

    strongly interrelated, therefore a higher

    awareness of project applicants is seen as

    a prior step for ensuring a higher success

    of the project selection]; A positive trendtowards offering more information and

    feedback on the non-selected project

    proposals has been observed; A more

    strategic evaluation analysis has been

    observed, with a shift from an

    administrative check towards a more

    thematic approach.

    o Recommendations for future:

    To ensure better geographical balance, by

    encouraging less experienced countries to

    take the lead in projects and provide strong

    support during project implementation;

    Increase even more capacity building

    actions for SCPs to improve geographical

    coverage and quality of projects content;

    To keep on improving the process of

    transparency and communication in order to

    ensure the generation of higher quality

    projects for future calls.

    Q3. How could

    the selectionprocess be

    improved?

    Desk research

    Interviews

    Please see Annex Recommendations for future: Please see

    section 2.3.2 below

    Q1. Do the project selection criteria effectively support the selection of the best quality

    transnational projects?

    A clear trend towards the fine tuning/narrowing of the spectrum of selection requirements to

    more concrete assessment criteria can be evidenced;

    A steady adoption of a more flexible approach en route to a more in-depth analysis of project

    content, transnational partnership development and consistency has been detected;

    Gradual alleviation of the administrative burden by the introduction of more flexible approaches

    with respect to formal and administrative requirements has been observed.

  • 8/9/2019 3rd+Evaluation+Report

    26/182

    24 Evaluation of South East Europe Programme 2007-2013

    Q2. Is the project selection procedure clear and transparent?

    Increasing evidence of strategic awareness of project applicants and better project fit and

    integration into the policy transnational framework. This is based on the assumption that project

    selection and project generation are strongly interrelated. In this case, a higher awareness of

    project applicants is seen as a prior step for ensuring a higher success of the project selection.

    Although the duration of the procedure has been reduced in the 4th call with respect to the 2nd

    one, this has still resulted in a long process;

    Despite the partial reduction of the administrative burden (for instance by accepting scanned

    documents and faxes), which is judged as a clear improvement, the (sometimes necessary)

    administrative requirements may still represent an obstacle for potential quality projects to apply

    for financial support;

    The geographical and leadership imbalances of projects, although decreased during the

    programme life cycle, can still be observed and still need to be reduced.

    A trend towards offering more information and feedback on the non-selected project proposals

    has been observed by the national bodies and JTS thus increasing the awareness and

    knowledge of what worked wrong by the project promoters. This, without any doubt, has

    contributed to the enhancement in the quality of future project proposals.

    Also, the use of a two-step approach has supported the generation of quality projects in thecontext of the Programme, especially with the third top down and targeted call.

    A more strategic evaluation analysis has been observed, with a shift from an administrative

    check towards a more thematic approach while safeguarding objectivity in the project selection

    procedure through the 6-eye principle.

    Q3: How could the selection process be improved?

    A number of recommendations are included in the section below.

    2.3.2 Recommendations

    The following is recommended on project generation and selection:

    1. Reduce the duration of the project selection process to the possible minimum, i.e. by covering

    administrative and contracting tasks in shorter timeframes as much as possible.

    2. Further reduce the subjectivity factor, by continuing the improvement of the work done until now

    to concretise selection criteria.

    3. To include sustainability as a more determinant and relevant criterion for project selection in

    future calls, by allocating it a higher weight in the overall scoring.

    4. To give a higher importance to innovation in the territory and its geographical character in the

    selection criteria by identifying strategic projects not only in targeted calls, but also in the

    traditional bottom up calls for proposals.

    5. To ensure a better geographical balance, by encouraging the most recently accessed countriesto take the lead in projects, thus increasing the transnational balance of the programme as a

    whole.

    6. To increase even more capacity building actions for SCPs to ensure a higher geographical

    coverage and quality of projectscontent.

    7. To continue providing transparent and detailed information (feedback) regarding eligibility

    results of the selection procedure to project applicants in order to reinforce their capacity and

    contribute to the generation of higher quality projects for future calls.

  • 8/9/2019 3rd+Evaluation+Report

    27/182

    25Evaluation of South East Europe Programme 2007-2013

    3 Monitoring and implementation

    3.1 Scope of the evaluation of monitoring and implementation

    In this chapter the performance of the implementation and the monitoring system of the SEEprogramme is evaluated. The objective of the evaluation of the monitoring and implementation is to

    evaluate the performance of the implementation and the monitoring system of the SEE programme

    though its monitoring indicators and to give ideas to improve the whole indicator system of the SEE

    programme.

    In this evaluation the focus has therefore been on a number of specific issues:

    Performance of the programme through the indicators

    Geographical scope of the projects

    Thematic scope of the projects

    Monitoring system

    Programme management system

    This chapter will summarise the results of the previous reports and update it to its current situation

    for all specific issues mentioned above. This means that for the indicators an updated table of

    performance is included and also an analysis about it (paragraph 3.2). For the geographical scope

    the results of the previous report are mentioned and for the thematic scope additional research is

    done to see which country is most interested in which area of intervention (paragraph 3.3). For the

    programme management and monitoring system the previous two reports are up-dated and further

    analysis is made (paragraph 3.4).

    3.2 Performance of the programme

    3.2.1 Summary of previous findings

    In the first evaluation report (2011) the evaluators concluded that the assessment of the indicator

    system was not possible as the realised indicators were not available during that evaluation period.

    In the second evaluation report (2012) it was still difficult to show the performance of the

    programme through indicators as either the target was not available (project level indicators), or the

    realization/actual value was not given (programme level indicators), or the indicators seemed to be

    completed with incorrect approach (programme indicators).

    Although a quantitative assessment was possible only limitedly, the evaluators listed someobservations in both the first, and the second evaluation report. In the first evaluation it was

    observed that the programme contained a helpful set of indicators, however that targets were not

    quantified and that there were ambiguities for the beneficiaries on which indicators they had to fill.

    During the second evaluation report the SEE programme had indicators defined at programme and

    project level; however for the project indicators still no targets were defined.

    3.2.2 Assessment of the indicator system

    Currently improvements have been made in the monitoring system. For all indicators there is now a

    target available (extracted from the project applications) and the realization was monitored in the

    Monitoring Wizard Report (project basis) and reflected in the annual implementation report 2012

    (programme basis).

    The SEE programme has indicators defined at programme and project level in:

  • 8/9/2019 3rd+Evaluation+Report

    28/182

    26 Evaluation of South East Europe Programme 2007-2013

    The SEE Operational Programme, section 4.4 (programme level);

    The SEE Programme Manual Version 2.1 May 2010, 2ndCall for project proposals, Annex 2.

    (programme level and project level)

    The Annual Implementation Report (programme and project level)

    IMIS (programme and project level)

    The Application Form (project level)

    The Progress Report (project level)

    The Monitoring Wizard (project and programme level)

    These sources all use the same set of indicators. The programme manual for the 2 nd call for

    proposals also mentions other indicators (input indicators, indicators for the cross cutting issues

    and indicators for the TA). The Annual Implementation Report is reporting on the indicators

    mentioned in the Operational Programme. The Annual Implementation Report of 2012 also reports

    on the indicators defined for the projects. The Monitoring Wizard was introduced in 2012 and helps

    to monitor the projects from the point of view of their link to the programme objectives.

    Programme indicators

    In the second evaluation report the evaluators concluded that the Operational Programme has aclear description of the indicator system at programme level and that the indicators are specific and

    quantified and output and result indicators are interlinked.

    Based on the information available in the documents (see annex 2) the following remarks can be

    made currently for the programme indicators:

    After four calls 122 projects have been approved, which is not far behind the target of 130

    implemented projects set at the beginning of the programme. As the budget of the projects is

    higher than anticipated, fewer projects were sufficient to allocate all available funds.

    Looking at the programme indicators (see annex 2) 55% of the output and result

    indicators are on schedule4in achieving their targets(21 indicators have achieved 45-65%

    of the target) or are far above the expected figures, 16 indicators have achieved more than 65%

    of their target). 45% of the indicators are slightly or largely behind schedule: 18 indicators have

    achieved 35-45% of their target and 13 indicators have realised less than 35% of their target.

    One can still question if the number of projects is a good indicator, as the number of projects

    does not relate to the content. However, because this indicator has also been used in the

    Working Document of the EC as an indicator the evaluator suggested maintaining this indicator

    (see evaluation report 2012).

    In the 2012 evaluation report the evaluators mentioned that the performance of the number of

    contributions has been completed by mentioning the number of projects that have been

    approved, instead of number of contributions (of which it is expected to obtain on average 3

    contributions per projects). The evaluator recommended to specify what a contribution is and tocorrect the performance of the indicator accordingly. In 2012 in the second evaluation report the

    SEE programme reported that: Data is reported according to the understanding that 1

    approved project equals 1 contribution, which was originally adopted by the Programme

    monitoring bodies. This result indicator was further specified and defined in the course of 2012

    through an external study aiming at enhancing SEE monitoring system (please refer to section

    2.1.6 Qualitative analysis for more details on the study). Therefore, in the next Annual

    Implementation Report, the values reported for this result indicator will be overhauled based on

    the study feedback. With the introduction of the new Monitoring Wizard there is now a clearer

    link between project achievements and the programme objectives as well as a clearer definition

    of a contribution together with a methodology to measure the number of contributions.

    4As the projects started in 2009 and will run until 2014 one can expect that about 50-60% of the targets should be achieved in

    2012. When a 45-65% of the target has been achieved we therefore speak of being on schedule.

  • 8/9/2019 3rd+Evaluation+Report

    29/182

    27Evaluation of South East Europe Programme 2007-2013

    Project indicators

    In the Programme Manual for the first call was indicated that the set of indicators would be

    developed later. In the Programme Manual of the second call the approach for the indicators was

    further elaborated into a set of input, output and result indicators. The programme manual of the

    third and fourth call does not contain this information anymore since the indicator system was in the

    process of being revised.

    In the evaluation report of 2012, the evaluators assessed the indicators set well presented,

    comprehensive and limited. However, the evaluators mentioned several points of attention for the

    programme. One of the main points was that the indicators are not quantified (are not having a

    target) and are not always SMART.

    The previous evaluations concluded that more attention should be given to the Lead Partners of the

    programme to avoid wrong reporting. Recommendations from the evaluator in this field were:

    It would be good to add an explanation of the indicators like it has been done for the indicators

    no of successfully implemented projects. For example, when is a project achieving synergy

    with an Objective 1 or 2 programme? The text of the programme manual for the second call is clear but merely written for the

    programme than for the applicant/ beneficiary. It might be difficult for the beneficiary to

    understand what he has to do with this information. For example table 9 Output and result

    indicators related to TA is relevant for the programme but not for the applicants/beneficiary as it

    concerns Priority Axis 5 Technical Assistance. Also the input indicators are not so relevant for

    the applicants/beneficiaries.

    It could be confusing for the applicant to have all the different tables with indicators.

    During the interviews with JTS members in 2013 it was clarified that the programme took actions on

    the above. The programme has provided indicator descriptions and interpretation in the Monitoring

    Wizard tool. In the Monitoring Wizards the JTS asked the projects to check, if they misunderstood

    the indicators following the provided interpretation. In case of misunderstanding, the indicators were

    revised. This did not happen when the target was too optimistic though because that would require

    a project modification.

    According to the Monitoring Wizard, the output indicators at project level are progressing

    rather well. 8 of 22 indicators a far ahead on schedule (more than 65% of the targets already

    achieved halfway through programme implementation). 9 indicators are on schedule (45-65% of

    targets achieved). 5 are a bit behind schedule (35-45% of the targets realised). The output indicator

    values of the projects are behind on number of press conferences, number of guidelines produced,

    joint action plans produced, number of individuals that participated in exchange programmes andnumber of promotion concepts (see figure below and annex). This is considering that there are 82

    projects still running.

    The table below shows to what extent the targets of the indicators have been achieved. Projects of

    the first call (which are currently finished) reached their targets quite well. 14 of the 22 indicators

    score above 100%, 6 indicators are between the 75% and 100%, and only 2 indicators score below

    the 75%. The most problematic indicators are the promotion of concepts (51% of the target) and the

    number of joint action plans produced (71%). While the average hits per month on the operations

    website and the number of articles/appearances published in the press and in other media were

    much more than expected in advance.

  • 8/9/2019 3rd+Evaluation+Report

    30/182

    28 Evaluation of South East Europe Programme 2007-2013

    Table 3.1 Project output indicators after four calls (% of the project targets achieved in each call) dark green +

    underscore + > 100%, dark red and italic is < 15%

    . Description 1st call 2nd

    call 3rd

    call 4th

    call TOTAL

    1 No. of articles/appearances published in the press and in other

    media (including online media, TV, radio)

    191% 56% 24% 14% 89%

    2 No. of press conferences 76% 46% 25% 6% 43%

    3 Average of hits per month on the operations website 341% 211% 37% 6% 252%4 No. of publications produced (editions, specify: e.g. folder,

    newsletter, brochure, report, handbook)

    92% 84% 1% 0% 80%

    5 No. of transnational events implemented 115% 71% 36% 7% 53%

    6 No. of national events implemented 106% 68% 2% 3% 48%

    7 No. of regional events implemented 104% 76% 17% 4% 50%

    8 No. of study visits organised 184% 79% 58% 1% 90%

    9 No. of studies realised 85% 76% 25% 4% 57%

    10 No. of guidelines produced 109% 43% 21% 3% 43%

    11 No. of management plans developed 83% 68% 15% 14% 56%

    12 No. of joint action plans produced 71% 45% 5% 2% 35%13 No. of databases created or improved 107% 93% 6% 2% 63%

    14 No. of training events, seminars organised 118% 41% 3% 1% 62%

    15 No. of participants involved in training events/seminars 89% 39% 4% 2% 53%

    16 No. of individuals that participated in exchange programmes 141% 24% 58% 4% 43%

    17 No. of promotion concepts 51% 98% 57% 11% 40%

    18 No. of promotion actions 142% 102% 36% 10% 82%

    19 No. of services developed 124% 43% 0% 16% 60%

    20 No. of small scale infrastructure projects 75% 133% N/A 0% 74%

    21 No. of person in charge for administration of projects 132% 99% 100% 51% 88%

    22 No. of project meetings held 125% 81% 24% 18% 74%Source: Ecorys on basis of the Monitoring Wizard (July 2013).

    The evaluator is of the opinion that for some output indicators targets set by projects were too high,

    which complicated reaching the targets (in blue in the table above):

    No. of press conferences

    No of studies realized

    No. of management plans developed

    No. of joint action plans developed

    No. of participants involved in training events and seminars

    No. of individuals that participated in exchange programmes

    No. of promotion concepts

    While for other output indicator the projects put a lower and more realistic target:

    Average of hits per month on the projects website

    No. of study visits organized

    No. of promotion actions

    No. of person in charge for administration of projects (especially in the first call)

    According to the Monitoring Wizard the result indicators at project level are merely on

    schedule. 7 of 26 indicators a far ahead on schedule (more than 65% of the targets already

    achieved whereas the programme is halfway its implementation). 9 indicators are on schedule (45-65% of targets achieved). 6 are a bit behind schedule (35-45% of the targets realised) and only 4

  • 8/9/2019 3rd+Evaluation+Report

    31/182

    29Evaluation of South East Europe Programme 2007-2013

    are far behind schedule (0-35% of targets achieved). Especially the number of individuals reached

    directly through dissemination outputs in the co-operation area (33%), the number of strategies

    adopted (25%), the number of private market reactions achieved (19%) and the number of

    infrastructures of common interest improved (25%) are lagging far behind schedule.

    Table 3.2 Project result indicators after four calls (% of the project targets achieved in each call): dark green +

    underscore + > 100%, dark red and italic is < 15%

    Description 1st call 2nd call 3rd call 4th call TOTAL

    1 No. of permanent information sources / channels in

    operation (e.g. websites, regular publications)

    137% 56% 24% 14% 136%

    2 No. of individuals reached directly through

    dissemination outputs in the co-operation area

    640% 46% 25% 6% 33%

    3 No. of administrative actors reached directly through

    dissemination outputs in the co-operation area

    203% 211% 37% 6% 92%

    4 No. of private sector actors reached directly through

    dissemination outputs in the co-operation area

    675% 84% 1% 0% 543%

    5 No. of SME reached directly through dissemination

    outputs in the co-operation area

    296% 71% 36% 7% 227%

    6 No. of advanced tools and methodologies adopted to

    increase the projects visibility among experts and

    wider communities, the public

    80% 68% 2% 3% 54%

    7 No. of common positions / agreements formulated 78% 76% 17% 4% 42%

    8 No. of common methodologies adopted 88% 79% 58% 1% 59%

    9 No. of strategies adopted at governmental level 56% 76% 25% 4% 25%

    10 No. of innovative products developed 67% 43% 21% 3% 37%

    11 No. of policies /instruments improved or developed 77% 68% 15% 14% 42%

    12 No. of common standards established (e.g. through

    new guidelines)

    102% 45% 5% 2% 53%

    13 No. of new tools / instruments developed 87% 93% 6% 2% 53%

    14 No. of impact studies on environmental issues

    carried out (e.g. in pre-investment projects)

    89% 41% 3% 1% 61%

    15 No. of pilot actions prepared (first application) 72% 39% 4% 2% 48%

    16 No. of pilot actions implemented (first application) 86% 24% 58% 4% 40%

    17 No. of permanent exchange programmes established 64% 98% 57% 11% 39%

    18 No. of staff members with increased capacity

    (awareness / knowledge / skills)

    104% 102% 36% 10% 53%

    19 No. of advanced tools and methodologies adopted to

    improve knowledge management within the

    partnership

    352% 43% 0% 16% 145%

    20 No. of regions proactively promoted 89% 133% N/A 0% 50%

    21 No. of common management structures / systems

    established

    84% 99% 100% 51% 63%

    22 No. of individuals benefiting directly from new /

    improved services

    99% 81% 24% 18% 76%

    23 No. of investment proposals developed 98% 0% 0% 0% 42%

    24 No. of private market reactions achieved (e.g. private

    activities mobilized)

    44% 0% 0% 0% 19%

    25 No. of investment projects implemented (specify

    volume of investment)

    116% 83% 32% 22% 107%

    26 No. of infrastructures of common interest improved 42% 76% 0% 3% 25%

  • 8/9/2019 3rd+Evaluation+Report

    32/182

    30 Evaluation of South East Europe Programme 2007-2013

    Source: Ecorys on basis of Monitoring Wizard (July 2013)

    Projects of the first call (which are currently finished) reached their targets on 11 of the 26

    indicators. The results of the number of individuals reached directly through dissemination outputs

    in the co-operation area (640%) and the number of private sector actors reached directly through

    dissemination outputs in the co-operation area (675%) were even six times as much as expected.

    The projects had most difficulties with reaching their targets on the number of infrastructure

    improved (42%), the number of private market reactions achieved (e.g. private activities mobilized)

    (44%), the number of strategies adopted at governmental level (56%), the number of permanent

    exchange programmes established (64%) and the number of innovative products developed (67%).

    The evaluator is of the opinion that projects set their targets for some result indicators too high,

    which complicates reaching the targets (in blue in the table above). While for other result indicators

    the projects have set their targets more realistically:

    Number of administrative actors reached directly through dissemination outputs in the co-

    operation area

    Number of staff members with increased capacity (awareness / knowledge / skills) (especially in

    the 2nd

    call) the number of individuals reached directly through dissemination outputs in the co-operation

    area (especially in the1stcall)

    the number of private sector actors reached directly through dissemination outputs in the co-

    operation area (especially in the1stcall)

    3.3 Geographical & Thematic scope of projects

    3.3.1 Geographical scope

    In comparison with the situation in the first and second evaluation reports of this programme the

    geographical balance over the programme area remains largely the same after the fourth call.

    Participation of the Member States

    The map and table below shows that 38% of the 122 projects have an Italian Lead Partner (46

    Lead Partners). Italy is having the strongest involvement both in terms of number of Lead

    Partners and of Project Partners (185 project partners). However, if this proportion is related to

    the population5 (see table below), Italy is scoring just above average and, in the number of

    project partners, even below average.

    Slovenia shows the opposite picture. Whereas Slovenia provides the project leadership in only

    15 projects, if one relates it to the countrys population, then Slovenia scores far above the

    average. Also in terms of number of project partners Slovenia scores far above the average ofthe Member States when relating the numbers of partners to the population.

    Austria, Greece, Hungary, Italy and Slovenia are scoring above the average on all aspects both

    in number of Lead Partners and Project Partners and also when relating it to the population.

    Bulgaria, Romania and Slovakia are scoring below average on number of Lead Partners (both

    in numbers and when relating it to the population). However they score average or above

    average on number of Project Partners.

    The newest Member States Bulgaria and Romania are not strongly involved as Lead Partners;

    however both are represented with large numbers of Project Partners. Relating project partners

    to the population, both countries score average or above average among the Member States.

    Institutions from countries outside the programme area, e.g. Poland, Belgium and Germany,

    also take part in SEE projects as observers, 10% or 20% project partners.

    5For Italy the population of the regions involved in the SEE region is being used.

  • 8/9/2019 3rd+Evaluation+Report

    33/182

    31Evaluation of South East Europe Programme 2007-2013

    Participation of the IPA and ENPI Countries

    Whereas the ERDF budget of the member states is available for all member states, the IPA and

    ENPI funding is available per country. So the participation of the IPA and ENPI countries is first of

    all influenced by the budget available.

    Of the IPA Countries, Serbias participation is the strongest in the SEE Programme area concerning

    the number of project partners. When relating it to the population Montenegro is the strongest

    participant with over 40 project partners per million inhabitants. Herewith Montenegro is clearly

    scoring above average when relating to the population. The Former Yugoslav Republic of

    Macedonia and Montenegro score both average of the IPA countries.

    For the ENPI countries, obtaining ENPI funds has been an obstacle. By December 2012 the

    Republic of Moldova had signed the Financing Agreement, which allowed for an additional call in

    2013 and 6 Moldovan project partners have been able to receive ENPI funding (+1 project from the

    4thcall). Unfortunately, the Financing Agreement with Ukraine could not be signed on time and thus

    financing of Ukrainian partners is only possible through the 10% rule. Therefore only the Republic

    of Moldova is receiving ENPI support (and is indicated above average in the figure below).However, when looking at the involvement of the Republic of Moldova and Ukraine as observers

    and 10% project partners, Ukraine is more involved (with 24 10% rule project partners and 9

    observers in overall 26 projects) than the Republic of Moldova (with 10 10% rule project partners

    and 3 observers in overall 11 projects)6. This is also why in the figure below Ukraine has been

    indicated as having an average contribution.

    Figure 3.1 Geographical distribution of number of LP and PP per country after 4 calls

    Source: Ecorys on basis of information from JTS SEE after 4 calls

    6Information from the SEE project website, November 2013

    0 PP, 24 10%

    7 PP,

    10 10%

    Below&aboveaverag

  • 8/9/2019 3rd+Evaluation+Report

    34/182

    32 Evaluation of South East Europe Programme 2007-2013

    Table 3.3 The geographical distribution of Lead and Project Partners after 4 calls, both in actual numbers and

    number per mln habitants. For each column: blue& underscore is above average, red & italicis below average

    Source: Ecorys on basis of information from JTS SEE after 4 calls

    Danube and Adriatic-Ionian relations

    The analysis of the relations from the partnership point of view in the macro regions of Danubestrategy area and Adriatic-Ionian area countries showed in the second evaluation report a stronger

    relationship within the Danube strategy area than within the Adriatic-Ionian area when looking at

    which countries are involved in the projects.

    Currently there seems to be more balance in the programme. About one third of the project has

    50% or more of the Danube strategy countries involved, which means 7 or more of the 14

    Danube strategy countries7. However, also for the Adriatic-Ionian area around one-third of the

    projects have 50% or more of the Adriatic-Ionian area countries8involved, which means 4 or

    more of the 7 countries.

    7The Danube Programme will cover parts of 9 EU countries (Austria; Bulgaria; Croatia; Czech Republic; Germany (Baden-

    Wrttemberg and Bavaria) not whole territory; Hungary; Romania; Slovakia; Slovenia) and 5 non-EU countries (Bosnia

    and Herzegovina; the Republic of Moldova; Montenegro; Serbia; Ukraine (not whole territory), having the same

    geographical scope than the EU Strategy for the Danube Region.8Parts of Italy, Slovenia, Croatia, Bosnia and Herzegovina, Montenegro, Albania and Greece. As a reference the countries of

    the Adriatic CBC programme 2007-2013 have been used.

    MS LP LP/mln population PP PP/mln population IPA PP PP/mln

    population

    AT 17 2,0 111 13,1 AL 34 12,0

    BG 1 0,1 143 19,6 BiH 32 8,3GR 17 1,5 152 13,5 FYRoM 23 11,2

    HU 17 1,7 156 15,7 MN 25 40,1

    IT 46 1,5 184 6,1 RS 80 11,0

    RO 6 0,3 170 8,0

    SK 3 0,6 58 10,7

    SI 15 7,3 109 52,9

    Total 122 1,142

    HR 59 13,4

  • 8/9/2019 3rd+Evaluation+Report

    35/182

    33Evaluation of South East Europe Programme 2007-2013

    Figure 3.2 Geographical distribution of number of LP and PP per country after 4 calls

    Source: Ecorys based on information from JTS SEE

    In the second evaluation report also a figure with the privilege relations was included after 3 calls.

    The figure showed that the country involved in most projects is Romania, secondly