Monitoring & Evaluation, 12-16 Dec 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın...
-
Upload
madeleine-dean -
Category
Documents
-
view
218 -
download
2
Transcript of Monitoring & Evaluation, 12-16 Dec 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın...
Monitoring & Evaluation, 12-16 Dec 2011
1
Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına Yönelik Teknik Yardım
Technical Assistance on Institutional Building for the Implementation of RCOP in Turkey
This project is co-financed by the European Union and the Republic of Turkey
MONITORING & EVALUATIONMONITORING & EVALUATIONfor RCOPfor RCOP
Ankara, 12 – 16 December 2011Ankara, 12 – 16 December 2011
Monitoring & Evaluation, 12-16 Dec 2011 2
This project is co-financed by the European Union and the Republic of Turkey
SECTION I.MONITORING VS VALUATION
• Clarification of definitions
Monitoring & Evaluation, 12-16 Dec 2011 3
This project is co-financed by the European Union and the Republic of Turkey
Distinction between evaluation, Distinction between evaluation, monitoring and auditmonitoring and audit
• Evaluation • Assessment of the efficiency, effectiveness, impact, relevance and sustainability of aid policies and actions
• Monitoring • Ongoing analysis of programme/project progress towards achieving planned results with the purpose of improving management decision making
• Audit • Assessment of the legality and regularity of project expenditure and income i.e.
compliance with laws and regulations and with applicable contractual rules and criteria;
whether project funds have been used efficiently and economically i.e. in accordance with sound financial management;
whether project funds have been used effectively i.e. for purposes intended.
Primarily a financial and financial management focus, with the focus of effectiveness being on project results.
Monitoring & Evaluation, 12-16 Dec 2011 4
This project is co-financed by the European Union and the Republic of Turkey
Evaluation-monitoring-project Evaluation-monitoring-project managementmanagement
• Monitoring: to be aware of the present situation• Evaluation: to rationally connect the present situation with
the desired future situation• Project management: operations and transactions
(contracting, approval of reports, payments…) but activities leading to these rely on monitoring and to (interim) evaluation outcomes
Potential overlaps of monitoring and project management activities
Segregation of duties, clear definition of responsibilities Cooperation
Monitoring & Evaluation, 12-16 Dec 2011 5
This project is co-financed by the European Union and the Republic of Turkey
Distinction between and Distinction between and evaluation monitoringevaluation monitoring
Monitoring•What? Monitoring is an integral part of a day-to-day management.•How? Monitoring embodies the regular tracking of inputs, activities, outputs, reach, outcomes, and impacts of development activities at the project program, sector and national levels•Why?Monitoring provides information by which management can identify and solve implementation problems and assess progress towards project's objectivesEvaluation•What? Evaluation is an assessment that refers to design, implementation and results of completed or on-going project / program / policy. •How? Evaluation should be systematic and objective. Key criteria to be used are: relevance, developmental efficiency, effectiveness, impact and sustainability.•Why?Evaluation should provide credible and useful information to enable the incorporation of lessons
Monitoring & Evaluation, 12-16 Dec 2011 6
This project is co-financed by the European Union and the Republic of Turkey
Distinction between and Distinction between and evaluation monitoringevaluation monitoring
Monitoring•focused on daily management issues•typical questions: “How many?” "When?” “How?” “For how much?” •assess whether activities are implemented effectively and efficientlyEvaluation •addresses strategic questions: “So what?”(impact and sustainability) and “Why?” (relevancy)•analysis is deeper and seeks for actual cause-results relationships•perception of “big picture"
Monitoring & Evaluation, 12-16 Dec 2011 7
This project is co-financed by the European Union and the Republic of Turkey
Distinction between and Distinction between and evaluation monitoringevaluation monitoring
Monitoring•usually means a system•data collected and analyzed more or less frequently•according to a predefine timetable (Performance Measurement Plan)•regularity and continuity of data collection•methodology used to analyze itEvaluation•flexibility in specifying, which aspects of the program should be assessed, when and how.
Monitoring & Evaluation, 12-16 Dec 2011 8
This project is co-financed by the European Union and the Republic of Turkey
Distinction between and Distinction between and evaluation monitoringevaluation monitoring
Monitoring•Part of modern project management•Generates useful information for project manager
• where are bottlenecks?• how are we doing towards our objectives?• are expenses under control?
•Utility is the primary featureEvaluation•Target groups: donors, planners, assistance recipients and wider public
• Have we achieved our goals? • Are our results sustainable?• Have we learned anything for the future?
•Focus on • transparency of evaluator’s approach • revealing cause-effect relationships between subsequent
layers of analysis.
Monitoring & Evaluation, 12-16 Dec 2011 9
This project is co-financed by the European Union and the Republic of Turkey
Distinction between and Distinction between and evaluation monitoringevaluation monitoring
Monitoring•Project management needs rapid assessment methods
• fast feedback• not very expensive
Evaluation•Scrupulous research methodologies
• representative surveys• comprehensive quantitative analyzes
Monitoring & Evaluation, 12-16 Dec 2011 10
This project is co-financed by the European Union and the Republic of Turkey
M&E – CM&E – Comparative characteristicsomparative characteristics
Characteristics Evaluation Monitoring
Subject:usually focused on strategic aspects
addresses operational management issues
Character:incidental, flexible subject & methods
continuous, regular, systematic
Primary client:stakeholders and external audience
programme management
Approach: objectivity, transparency utility
Methodology:rigorous research methodologies, sophisticated tools
rapid appraisal methods
Primary focus:focus on relevancy, outcomes, impact and sustainability
focus on operational efficiency and effectiveness
Objectives:to check outcomes / impact, verify developmental hypothesis
to identify and resolve implementation problems
to document successes and lessons learned
to assess progress towards objectives
Monitoring & Evaluation, 12-16 Dec 2011 11
This project is co-financed by the European Union and the Republic of Turkey
M&E system diagrammeM&E system diagramme
INTERVENTION
LOGIC SOURCES OF DATA M&E TOOLS FOLLOW-UP
OVERALLOBJECTIVES EXTERNAL REPORTS AND DATA
FINAL EVALUATION, SURVEY,STATISTICAL ANALYSIS
FINAL EVALUATION REPORTPROGRAMMING
BENEFICIARIES OPINIONMID-TERM EVALUATIONBENEFICIARIES SURVEY
MID-TERM EVAL. REPORTREVISING CONTRACTS
PROJECTPURPOSE
EXTERNAL EXPERTS' OPINIONSTRUCTURED INTERVIEW,
FOCUS GROUPSQUARTERLY MONITORING
REPORT
RESULTSQUANTITAIVE DATA ON
PROJECT PROGRESS PIPELINE ANALYSIS MANAGEMENT MEETINGS
IMPLEMENTERS' PERIODIC
REPORTS REVIEW OF PERIODIC REPORTS REPORT APPROVAL PROCESS
ACTIVITIES
PRODUCTS DELIVERED BY
CONTRACTORSREVIEW OF PRODUCTS
AND SERVICES PRODUCT REVIEW REPORT
MEANS &RESOURCES
DIRECT OBSERVATIONS SITE VISITS SITE VISIT REPORT
Monitoring & Evaluation, 12-16 Dec 2011 12
This project is co-financed by the European Union and the Republic of Turkey
SECTION II.EVALUATION
• EU Regulations on evaluation• Introductory guide on basic notions
Monitoring & Evaluation, 12-16 Dec 2011 13
This project is co-financed by the European Union and the Republic of Turkey
COUNCIL REGULATION COUNCIL REGULATION 1085/2006 establishing IPA1085/2006 establishing IPA
Article 22 Evaluation•Commission shall regularly evaluate
• the results and efficiency of policies and programmes
• effectiveness of programming whether the objectives have been met
•Objectives of Evaluation• to enable the COM to formulate recommendations• results shall feed back into programme design
and resource allocation
Monitoring & Evaluation, 12-16 Dec 2011 14
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
• Title II Common rules for implementation/Chapter V Evaluation and monitoring
• Article 57 Evaluation• Objectives: to improve the quality of
effectiveness and consistency of the assistance strategy and implementation of the programmes.
• Subject: Multi-annual indicative planning documents (ex ante evaluation)
• Forms: Strategic Ex ante Interim Ex post
Monitoring & Evaluation, 12-16 Dec 2011 15
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Article 57 Evaluation (cont’d)•Interim:
At least one during implementation Specifically when the monitoring reveals significant
departure from the goals initially set.•Ex post
the responsibility of the Commission Shall include identifiable IPA component-specific results
•The results of ex ante and interim evaluation shall be taken into account in the programming
Monitoring & Evaluation, 12-16 Dec 2011 16
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
CHAPTER III/Implementation/Section 1 Framework for implementation and principlesArticle 78 Implementation principles in the event of participation in Community programmes and agencies (as amended by COM REGULATION 80/2010)•All evaluations shall be carried out by the COM prior to the conferral of management powers on the beneficiary country, •After the conferral
BC is responsible for interim evaluation; COM’s rights to perform any ad-hoc evaluations remains; ex-ante and ex-post remain with COM BC’s right to carry out ex-ante and ex-post as it deems
necessary.
Monitoring & Evaluation, 12-16 Dec 2011 17
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Title III Regional development and human resources development/Chapter III Implementation/Section 3 Evaluation and monitoring Article 166 Evaluation•ex ante
• Beneficiary countries - under the responsibility of the operating structure.• For each operational programme separately. • If duly justified and in agreement with the COM BC may carry out a single ex ante
evaluation covering more than one operational programme.• Shall aim to
• optimise the allocation of budgetary resources under operational programmes• improve programming quality• identify and appraise
• the disparities, gaps and potential for development, • the goals to be achieved, • the results expected, • the quantified targets,• The coherence of the strategy proposed • and the quality of the procedures for implementation, monitoring,
evaluation• Annexed to OP
Monitoring & Evaluation, 12-16 Dec 2011 18
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Article 166 Evaluation (cont’d)•Interim
• During the programming period• Beneficiary countries• Linked to the monitoring of OPs, in particular where
• this reveals a significant departure from the goals initially set or
• proposals are made for the revision of OPs• Results shall be sent to the sectoral monitoring committee for
the OP and to COM• Evaluations shall be carried out by experts or bodies, internal
or external, functionally independent of the authorities• The results shall be published
Monitoring & Evaluation, 12-16 Dec 2011 19
This project is co-financed by the European Union and the Republic of Turkey
Purpose and principlesPurpose and principles
Logical elements of purpose of evaluation:•Assessment;•Systematic and objective;•Ongoing or completed project, programme or policy;•Design, implementation and results;•Determine the relevance and fulfillment of objectives, developmental efficiency, effectiveness, impact and sustainability;•Provide information that is credible and useful;•Enabling the incorporation of lessons learned into the decision-making process of both recipients and donors;
Monitoring & Evaluation, 12-16 Dec 2011 20
This project is co-financed by the European Union and the Republic of Turkey
Purpose and principlesPurpose and principles
Principles of the approach to evaluation:•Impartiality and independence of the evaluation process from the programming and implementation functions;•Credibility of the evaluation, through use of appropriately skilled and independent experts and the transparency of the evaluation process, including wide dissemination of results;•Participation of stakeholders in the evaluation process, to ensure different perspectives and views are taken into account; •Usefulness of the evaluation findings and recommendations, through timely presentation of relevant, clear and concise information to decision makers.
Monitoring & Evaluation, 12-16 Dec 2011 21
This project is co-financed by the European Union and the Republic of Turkey
Evaluation typesEvaluation types
Ex anteArticle 2 of the Financial Regulation•Use of budget appropriations
• principles of sound financial management• quantified objectives must be identified• „mobilisation of Community resources must be
preceded by an evaluation to ensure that the resultant benefits are in proportion to the resources applied.”
Monitoring & Evaluation, 12-16 Dec 2011 22
This project is co-financed by the European Union and the Republic of Turkey
Evaluation typesEvaluation types
Ex ante•Implementation Rules for FR – ex ante evaluation shall identify:
• need to be met in the short or long term;• objectives to be realised;• results expected and the indicators needed to measure
them;• added value of Community involvement;• risks, including fraud;• lessons learned from similar experiences in the past;• volume of appropriations, human resources and
administrative expenditures;• monitoring system to be set up.
Monitoring & Evaluation, 12-16 Dec 2011 23
This project is co-financed by the European Union and the Republic of Turkey
Evaluation typesEvaluation types
Ex ante•Process that supports the preparation of interventions - related to programming ~ quality assurance•Purpose: to gather and analyse information
• define objectives, • ensure that these objectives can be met,• that the instruments used are cost-effective, • reliable later evaluation will be possible
•Independent from planners
Monitoring & Evaluation, 12-16 Dec 2011 24
This project is co-financed by the European Union and the Republic of Turkey
Evaluation typesEvaluation types
Interim•Assessment of the quality of programme implementation based on the first outputs and results
• relevance of the adopted strategy;• newly occured factors having an impact on the
implementation;• objectives have been defined accurately;• indicators are relevant;• so-far effectiveness and extrapolation;• management quality of the project implementation;• reliable collected data, including the monitoring system
•Independent from implementors
Monitoring & Evaluation, 12-16 Dec 2011 25
This project is co-financed by the European Union and the Republic of Turkey
Evaluation typesEvaluation types
• Ex post• After programme completion• Examining long-lasting effects
• impact (more) visible• verification to what extent objectives have been
achieved • Sustainability – estimated character• Anticipated and unexpected effects• Source of information for future programming
excercises
Monitoring & Evaluation, 12-16 Dec 2011 26
This project is co-financed by the European Union and the Republic of Turkey
Major tasks for an Evaluation UnitMajor tasks for an Evaluation Unit
• Identifying the need for an evaluation and selecting the topics/themes to be evaluated;
• Designing the evaluation, including preparing the Terms of Reference;
• Drafting tender documents for the evaluation study and selecting the contractor according to the established rules;
• Briefing the contractor and the parties involved, and supporting the evaluation mission;
• Ensuring the production of a high quality evaluation report and of the dissemination of evaluation findings and recommendations.
Monitoring & Evaluation, 12-16 Dec 2011 27
This project is co-financed by the European Union and the Republic of Turkey
Tools and key documentsTools and key documents
Primary tools available to support a project evaluation:•Terms of reference for the evaluation mission;•The project’s Logframe matrix - to help assess what has been achieved against plan; •Programme planning documents; •Monitoring reports (internal and external); •IPA Interim Evaluation Reports;•IPA Monitoring reports•Minutes of SMCs and other organisations involved in monitoring
Monitoring & Evaluation, 12-16 Dec 2011 28
This project is co-financed by the European Union and the Republic of Turkey
Key documents producedKey documents produced
• Terms of Reference for the evaluation mission,• Final Evaluation Mission Report
• should mirror the structure of the main evaluation criteria
• taking into account • the nature of the project, • the stage at which the evaluation is carried out, • the users for whom the report is prepared.
Monitoring & Evaluation, 12-16 Dec 2011 29
This project is co-financed by the European Union and the Republic of Turkey
Evaluation CriteriaEvaluation Criteria
Relevance•@ significance; pertinence•To what extent do we do the right things ? Does it make sense?•~ to the identified problems or real needs to be addressed •The appropriateness of project objectives to
• problems that it was supposed to address• physical and policy environment within which it operated
•The extent to which the objectives of a programme/project are consistent with beneficiaries’ needs, country needs, global priorities.
Monitoring & Evaluation, 12-16 Dec 2011 30
This project is co-financed by the European Union and the Republic of Turkey
Evaluation CriteriaEvaluation Criteria
• Relevance (cont’d)• Relates primarily to programme design
• extent to which correctly address the identified problems or real needs. (Needs to be kept under review throughout the life of the project in case changes occur)
• logic and completeness of the project planning process• internal logic and coherence of the project design• at two points in time: when the project was
designed, and at the time of the evaluation. • Focus
• identification of real problems or needs• correct beneficiaries
Monitoring & Evaluation, 12-16 Dec 2011 31
This project is co-financed by the European Union and the Republic of Turkey
Evaluation CriteriaEvaluation Criteria
Efficiency•How well the various activities transformed the available resources into the intended results (outputs)
• quantity, quality and timeliness.• value-for-money: similar at lower cost in the same time
•Focus: the quality of day-to-day management, i.e:• budget• personnel, information, property, etc;• risk and change• relations/co-ordination with actors• time (deadlines)
Monitoring & Evaluation, 12-16 Dec 2011 32
This project is co-financed by the European Union and the Republic of Turkey
Evaluation CriteriaEvaluation Criteria
Effectiveness•Contribution made by results to achievement of the Project Purpose•Benefits accruing to target groups•Did we achieve our objectives? To what extent did our outputs produce the desired outcomes?•By how far the intended beneficiaries really benefited from the products or services it made available?•Focus:
• whether the planned benefits have been delivered and received;
• appropriateness of the indicators of benefit used;• how unplanned results may have affected the benefits received;
Monitoring & Evaluation, 12-16 Dec 2011 33
This project is co-financed by the European Union and the Republic of Turkey
Evaluation CriteriaEvaluation Criteria
Impact•Effect of the project on its wider environment, and its contribution to the wider policy or sector objectives (as summarised in the project’s Overall Objective).•Outcome: relationship between the project’s purpose and overall objectives•Benefits received by the target beneficiaries had a wider overall effect on larger group•Focus:
• to what extent the planned overall objectives have been achieved, • how far that was directly due to the project;• unplanned impacts’ effects on overall impact;
•Ex: higher standard of living, increased food security, democratic rule of law
Monitoring & Evaluation, 12-16 Dec 2011 34
This project is co-financed by the European Union and the Republic of Turkey
Evaluation CriteriaEvaluation Criteria
Sustainability•Continuation of benefits, effects generated by a programme/project after its termination.•Likelihood of benefits produced to continue to flow after external funding has ended,•Focus:
• Ownership• Policy support and the responsibility of the beneficiary institutions• Institutional capacity, commitment• Economic and financial factors, socio-cultural aspects, gender equality• Technology, environmental aspects• Financial sustainability
•Ex: a micro-credit scheme that is generating enough money for the scheme to operate, cover risks and develop its staff
Monitoring & Evaluation, 12-16 Dec 2011 35
This project is co-financed by the European Union and the Republic of Turkey
A Quick reflectionA Quick reflection
• Appropriateness is not about money. It is about what was achieved against what was needed.
• Effectiveness is not about money. It is about how much it was achieved.
• Efficiency is about how much was achieved and how much money was spent.
• Sustainability is not about money. It is about the embedment of what was achieved in the given context (natural, societal, economic…)
Monitoring & Evaluation, 12-16 Dec 2011 36
This project is co-financed by the European Union and the Republic of Turkey
Decision optionsDecision options
Look out:•one may be very efficient but not effective!•or very effective but not relevant!•to do things right is fine, but to do the right things is for sure far more important!
Monitoring & Evaluation, 12-16 Dec 2011 37
This project is co-financed by the European Union and the Republic of Turkey
Decision optionsDecision options
• Continue project implementation as planned• Re-orient/restructure the project• To stop the project (mid-term evaluation);• Modify the design of future projects or
programmes in light of lessons learned (ex-post evaluation)
• Modify policies, co-operation strategies, and subsequent programming or identification exercises (sector, thematic or cross-sector evaluations)
Monitoring & Evaluation, 12-16 Dec 2011 38
This project is co-financed by the European Union and the Republic of Turkey
Performance RatingsPerformance Ratings
• Performance rating in two steps:1. Numerical rating on each criteria for each component of the sector
from –2 to +2:2. Conversion of numerical overall rating into a qualitative rating.
• Qualitative ratings• « Highly satisfactory » The programmes reviewed are expected
to achieve or exceed all the objectives set during their lifetime; most numerical ratings in the range of 1 to +2
• « Satisfactory » The programmes reviewed are expected to largely achieve the objectives; most numerical ratings in the range of 0 to 1
• « Unsatisfactory » The programmes reviewed are not expected to achieve most of the objectives; most numerical ratings in the range of –1 to 0
• « Highly unsatisfactory » The programmes reviewed are not expected to achieve any of the objectives; most numerical ratings in the range of -1 to –2
Monitoring & Evaluation, 12-16 Dec 2011 39
This project is co-financed by the European Union and the Republic of Turkey
Quality Assessment of IE ReportsQuality Assessment of IE Reports
• Criteria – Rate – Remarks• A. General: Does the report design appropriately fit the
evaluation?• B. Sound sectoral overview: to what extent are the sector
composition and priorities appropriately described?• C1. Sound analysis: to what extent are the facts and data
adequately analysed?• C2. Sound analysis: to what extent have the indicators of
achievement been adequately considered and have they been used properly where was possible?
• D. Robust Findings in the implementation evaluation: do the Conclusions follow logically from, and are they justified by, the data described in the Sectoral Overview?
Monitoring & Evaluation, 12-16 Dec 2011 40
This project is co-financed by the European Union and the Republic of Turkey
Quality Assessment of IE ReportsQuality Assessment of IE Reports
• E. Impartial conclusions: does the report provide value judgements based upon the five evaluation criteria of relevance, efficiency, effectiveness, sustainability and impact?
• F. Useful recommendations: to what extent do the Recommendations follow logically from the Conclusions? Are they operational? Do they clearly address the monitoring sector and are they targeted to the different stakeholders?
• G. The executive summary: to what extent is the executive summary a synthesis and does it meet the requirements set out in the template guidelines?
• H. Annexes: to what extent do the Annexes support the analysis in the main text?
• I. Overall style, structure and text design: within the template’s framework, to what extent is the text easily readable and accessible to the various categories of readers so that the main messages are easily detectable?
Monitoring & Evaluation, 12-16 Dec 2011 41
This project is co-financed by the European Union and the Republic of Turkey
Quality Assessment of IE ReportsQuality Assessment of IE Reports
Verbal Unacceptable
Poor
Sufficient/ adequate
Good
Excellent
Single category
-2 -1 0 1 2
Entire report
<=-15 -14 to -6
-5 to 5 6 to 14
>=15Taking into account the contextual constraints on the evaluation, the overall quality rating of the report is considered to be:
Monitoring & Evaluation, 12-16 Dec 2011 42
This project is co-financed by the European Union and the Republic of Turkey
Why Results Are IgnoredWhy Results Are Ignored??
• Implications may not be presented in a way that nonresearchers can understand.
• Results sometimes contradict deeply held beliefs.
• Vested interest in a program.
Monitoring & Evaluation, 12-16 Dec 2011 43
This project is co-financed by the European Union and the Republic of Turkey
SECTION III.SECTION III.MONITORINGMONITORING
• EU Regulations on monitoring• Monitoring system and its evaluation
Monitoring & Evaluation, 12-16 Dec 2011 44
This project is co-financed by the European Union and the Republic of Turkey
MONITORINGMONITORING
• @ follow up , controlling • A continuing observation • uses systematic collection of relevant and selected data • to provide management and the main stakeholders of a
programme/project with indications• of the extent of progress and achievement of
objectives• of process and impact.
Monitoring & Evaluation, 12-16 Dec 2011 45
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Title II Common rules for implementation/Chapter V Evaluation and monitoring Article 58 Monitoring in the case of decentralised management (IPA Monitoring Committee)
•BC in agreement with NIPAC and COM•to ensure coherence and coordination in the implementation of the IPA components•meeting the objectives set out in the financing agreements and MIIFP•based on the elements given by the SMCs
Monitoring & Evaluation, 12-16 Dec 2011 46
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
IPA Monitoring Committee – proposals •to COM, NIPAC and NAO for:
• any actions to ensure the coherence and • crosscomponent corrective measures needed to
ensure the achievement•to SMCs for:
• decisions on any corrective measures to ensure the achievements of programme
Monitoring & Evaluation, 12-16 Dec 2011 47
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
IPA Monitoring Committee – organisation •Internal rules of procedure
• in compliance with a monitoring committee mandate established by the COM
• within the institutional, legal and financial framework of the BC
•Members: Commission, NIPAC, NAO, OS, strategic coordinator•COM and NIPAC co-chair•Meetings at least once a year•Intermediate meetings on a thematic basis.
Monitoring & Evaluation, 12-16 Dec 2011 48
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Article 59 Sectoral monitoring committees in the case of decentralised management
•Assist IPA Monitoring Committee•Attached to programmes or components.•May include representatives of civil society,•Objective:
• effectiveness and quality of the implementation• in accordance with the related sectoral and/or financing
agreements.•Proposals
• to the COM and NIPAC, cc. NAO • for decisions on any corrective measures to ensure the
achievements of programme objectives
Monitoring & Evaluation, 12-16 Dec 2011 49
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Sectoral monitoring committee - reports•to IPA monitoring committee on
• progress • by priority axis/ measures or operations;• results achieved, financial implementation
indicators• any aspects of the functioning of the management
and control systems raised by the audit authority, NAO or the competent accrediting officer
Monitoring & Evaluation, 12-16 Dec 2011 50
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Article 61 Annual and final reports on implementation
•Sectoral annual report and sectoral final report by OS• Sectoral annual report: financial year• Sectoral final report: whole period of
implementation•SMC examination to be sent to NIPAC, NAO and COM
Monitoring & Evaluation, 12-16 Dec 2011 51
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Annual and final reports on implementation•NIPAC sends annual and final reports based on synthesis of sectoral reports to COM and NAO•by 31 August each year•progress made in relation to the priorities of MIIFP and OPs•financial implementation of Community assistance.
Monitoring & Evaluation, 12-16 Dec 2011 52
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Title III Regional development and human resources development/Chapter III Implementation/Section 3 Evaluation and monitoring
Article 167 Sectoral monitoring committee - Organisation •To be established by OS for each programme•single SMC may be set up for several programmes within the same component •meeting at least twice a year, at the initiative of BC or COM
Monitoring & Evaluation, 12-16 Dec 2011 53
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Sectoral monitoring committee – Organisation (cont’d) •Internal rules of procedure
• in compliance with a monitoring committee mandate established by the COM
• within the institutional, legal and financial framework of the BC• In agreement with the OS and the IPA monitoring committee
•COM and Head of OS co-chair•composition to be decided by OS, in agreement with COM
• Members: COM, NIPAC, strategic coordinator for RD and HR components, OS
• Civil society and socio-economic partners where appropriate• European Investment Bank in advisory role where EIB makes a
contribution
Monitoring & Evaluation, 12-16 Dec 2011 54
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Sectoral monitoring committee – Tasks •Consider/approve/revise criteria for selecting within six months of the entry into force of the financing agreement•review progress in OP targets based on OS reporting•examine (by reference to the indicators)
• results of implementation, achievement of the targets priority axis and measures
• interim evaluations• sectoral annual and final reports• relevant parts and recommendations of annual audit activity report• any proposal to amend the financing agreement
•propose to OS any revision or examination of the programme (incl. its financial management)
Monitoring & Evaluation, 12-16 Dec 2011 55
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Article 169 Sectoral annual and final reports on implementation
•OS to COM and NIPAC after examination by SMC• sectoral annual report by 30 June each year• sectoral final report six months after the final date of eligibility of
expenditure•COM informs NIPAC and OS on admissibility of SAR within 10 working days from the date of its receipt – formal•COM informs NIPAC and OS on its opinion on admissible
• SAR report within two months from the date of receipt• SFR within a maximum of five months from the date of receipt
Monitoring & Evaluation, 12-16 Dec 2011 56
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Sectoral annual and final reports - content•quantitative and qualitative progress in
• OP, priority axes, measures, operations• against quantified targets
•financial implementation of OP, priority axis, measure:• total expenditure paid out by the final beneficiaries included
in NF’s payment applications • total expenditure committed and paid out by NF
accompanied by computerised forms listing the operations, • total payments received from the COM
•breakdown of allocation by categories as listed in FA
Monitoring & Evaluation, 12-16 Dec 2011 57
This project is co-financed by the European Union and the Republic of Turkey
COM REGULATION 718/2007COM REGULATION 718/2007implementing 1085/2006implementing 1085/2006
Sectoral annual and final reports – content (cont’d)•steps taken by OS or SMC to ensure quality and effectiveness
• monitoring and evaluation measures, incl. data collection arrangements;
• summary of problems and subsequent measures;• use of technical assistance.
•information and publicity activities;•progress and financing of major projects.
Monitoring & Evaluation, 12-16 Dec 2011 58
This project is co-financed by the European Union and the Republic of Turkey
Monitoring conceptsMonitoring concepts
Activity and/or result oriented monitoring?•Activity based monitoring
• identify activities/deliverables and deadlines for each• activities to be completed in a timely and appropriate
manner•Result based monitoring
• Results to be achieved•Analogy to project management and monitoring
• Activities must be completed – legal and financial consequences
• Results must be achieved – effectiveness aspect
Monitoring & Evaluation, 12-16 Dec 2011 59
This project is co-financed by the European Union and the Republic of Turkey
ROM and Internal monitoring in EUROM and Internal monitoring in EU
ROM Internal Monitoring
Objectives To provide independent assessment of project performance, with focus on ‘results’
To provide advice and recommendations to project stakeholders
To generate aggregate data for reporting
To support effective and
timely decision making
by project managers
To promote accountability for resource use and
achievement of results
Responsible
COM Project implementing
partners/contractors
Monitoring & Evaluation, 12-16 Dec 2011 60
This project is co-financed by the European Union and the Republic of Turkey
ROM and Internal monitoring in EUROM and Internal monitoring in EU
ROM Internal Monitoring
Method Short-visits to project sites by independent experts, on a periodic
basis
Analysis of project records and interviews with stakeholders
Standardised assessment
Project plans, ongoing
data collection, analysis
of data, progress reports
Consultation with
stakeholders
Participation in review meetings
Monitoring & Evaluation, 12-16 Dec 2011 61
This project is co-financed by the European Union and the Republic of Turkey
Key principles for information flowKey principles for information flow
• Results information needs at the project, program, and policy levels results information must move both horizontally and vertically• Horizontal: what information is being collected by their own
organization and by other• Vertical: bureaucratical and hierarchical burdens
need to coordinate and collaborate in sharing performance information
• Responsibility at each level needs to be clear for• What data are collected (source)• When data are collected (frequency)• How data are collected (methodology)• Who collects data• Who reports data• For whom data are collected
Monitoring & Evaluation, 12-16 Dec 2011 62
This project is co-financed by the European Union and the Republic of Turkey
Needs of Monitoring SystemNeeds of Monitoring System
Ownership
•Demand part : who needs the information?•Needs to be identified•Critical issue: when levels of need and data collecting /maintenance do not fit•Ownership vs enforcement
• Legal• Financial• Technical
Monitoring & Evaluation, 12-16 Dec 2011 63
This project is co-financed by the European Union and the Republic of Turkey
Needs of Monitoring SystemNeeds of Monitoring System
Management
•Who, how, and where the system will be managed•Data collection overlaps, duplication of data•Time lags in receiving data•Data maintenance
• Responsible levels• Continuous development and trainings
Monitoring & Evaluation, 12-16 Dec 2011 64
This project is co-financed by the European Union and the Republic of Turkey
Needs of Monitoring SystemNeeds of Monitoring System
Credibility
•Report all data—both good and bad• Deliberate non-reporting of information demonstrating failure• Political pressure
•Reliability: the extent to which the data collection approach is stable and consistent across time and space - measurement of the indicators is conducted the same way every time•Validity: indicators should measure actual and intended performance levels•Timeliness
• Frequency (how often are data collected?)• Currency (how recently have data been collected?)• Relevance (are data available frequently enough to support
management decisions?)
Monitoring & Evaluation, 12-16 Dec 2011 65
This project is co-financed by the European Union and the Republic of Turkey
Design of a monitoring systemDesign of a monitoring system
1. Analyse project objectives to clarify project design
2. Review implementation procedures to determine information needs at the different levels of the project management structure
3. Review indicators4. Design reporting schemes5. Prepare an implementation plan for the
monitoring system
Monitoring & Evaluation, 12-16 Dec 2011 66
This project is co-financed by the European Union and the Republic of Turkey
Analyse project objectivesAnalyse project objectives
• Time factor since original design: chaning project environment and actors
• Logframe approach• analysis of existing situation in light of future desired
situation• planning operational details• Logframe matrix
• A possible tool: project start-up workshop• all stakeholders• review project documents and key assumptions• revise project objectives
Monitoring & Evaluation, 12-16 Dec 2011 67
This project is co-financed by the European Union and the Republic of Turkey
Review implementation proceduresReview implementation procedures
• Level of detail of information need and frequency of reporting vary according to the level of management
• Define information needs to the different levels of the management structure
• From project administrators to IPAMC• Workflow to be defined – procedures manual• Problem: users don’t know in advance what they need request
more information than needed users are not aware of what information is available
• Continual review of users requirements through:• planning and review meetings: what is lacking or redundant• systematic dialogue with users on content and format
Monitoring & Evaluation, 12-16 Dec 2011 68
This project is co-financed by the European Union and the Republic of Turkey
Review indicatorsReview indicators
To avoid:•selection of too many indicators•selection of overly-complex indicators •over-concentration on progress indicators
Monitoring & Evaluation, 12-16 Dec 2011 69
This project is co-financed by the European Union and the Republic of Turkey
Design reporting schemesDesign reporting schemes
• Reports• Based on project administrators’ needs on
daily/weekly/monthly basis• Based on the needs of the hierarchy• Incidental, Interim, Final as per contract
• Meetings• ‘monthly’, ‘Project Review/Implementation Meetings’
• EUD/NAC office vs CA, OBs• Informal, but methodology shall be determined in advance (all
the projects, only problematic ones)• Who represents the organisation?
• Steering Committees
Monitoring & Evaluation, 12-16 Dec 2011 70
This project is co-financed by the European Union and the Republic of Turkey
SECTION IV.SECTION IV.MANAGEMENT INFORMATION MANAGEMENT INFORMATION
SYSTEMSYSTEM
• Main concepts and hints• An example: the Hungarian MIS
Monitoring & Evaluation, 12-16 Dec 2011 71
This project is co-financed by the European Union and the Republic of Turkey
MISMIS
• All projects (service, supply, works, grant schemes): Similar characteristics
• Follow the same rules: PRAG rules and local• Efficient use of limited resources: No need to
rewrite again and again• Harmonization and Standardization of Monitoring:
Common to all
Monitoring & Evaluation, 12-16 Dec 2011 72
This project is co-financed by the European Union and the Republic of Turkey
MISMIS
Objectives
•Projects achieved their objectives•Project implementations follow the rules•Project progressing as planned and performing well•Problems detected earlier
Functions• Follow-up of implementation• Management control tool• Support to beneficiaries• Immediate data availability for the hierarchy and
public
Monitoring & Evaluation, 12-16 Dec 2011 73
This project is co-financed by the European Union and the Republic of Turkey
MISMIS
• Common communication platform and channel• Enforces obeying the rules• Provides monitoring system integrity• Creates standards for reporting and data entry• Verifies the declarations; Reports and Visits• Improve the quality of monitoring activities
Monitoring & Evaluation, 12-16 Dec 2011 74
This project is co-financed by the European Union and the Republic of Turkey
MISMIS
Progress
•Ensuring project to produce expected outputs:• with planned activities• within the contract budget and duration
• Monitoring the Progress on:• Financial Progress• Technical Progress• Risk Assessment• Performance indicators
Monitoring & Evaluation, 12-16 Dec 2011 75
This project is co-financed by the European Union and the Republic of Turkey
MISMIS
Characteristics of the system•Built with Participatory Approach•Comprehensive and decentralized•Open to all actors•Easily Accessible (Web-based)•Transparent System (Any data cannot be deleted or changed without permission)•Up-to-date data collection from the source•Different user levels according to responsibilities including project owners
Monitoring & Evaluation, 12-16 Dec 2011 76
This project is co-financed by the European Union and the Republic of Turkey
MISMIS
Some points of consideration•Capacity issues
• Participatory approach during development and testing
• Database maintenance• Need to assign staff specific to MIS (not IT)
•Source code issue• For further upgrades• Public procurement consequences
Monitoring & Evaluation, 12-16 Dec 2011 77
This project is co-financed by the European Union and the Republic of Turkey
EMIR – the Hungarian modelEMIR – the Hungarian model
www.emir.gov.hu