Workshop II Project Management...1 UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 1 Workshop...

1

UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 1

Workshop IIProject Management

UNITAR-HIROSHIMA FELLOWSHIP FOR AFGHANISTAN 2007Introduction to Project Management 15–17 August 2007, Dehradun, India

Presented by: Jobaid Kabir, Ph.D.


Part 7. Monitoring and Evaluation

2


Monitoring and Evaluation

Monitoring and Evaluation basicsPurpose of Monitoring and EvaluationStakeholder Participation in Monitoring and Evaluation Planning and Managing an Evaluation


Steps of Planning and Managing an Evaluation1. Planning Evaluations 2. Defining Evaluation Questions

and Measurement Standards 3. Data Collection Process 4. Managing the Evaluation Process 5. Communicating and Using

Evaluation Results 6. Evaluation Standards

3


In planning evaluation activities decide on…

WHY: purpose, users of evaluation findings and howWHAT: objectives and the questions to addressHOW: data sources & collection methods WHO: will do evaluations, required expertise, need for external consultants, stakeholder involvementWHEN: timing of evaluations to help decision makingRESOURCES: budget needs


Evaluation Analytical Process

Evaluation Objectives

Evaluation Objectives

Overall Conclusions and lessons learned

Overall Conclusions and lessons learned

Overall Recommendations

Overall Recommendations

QuestionsQuestions Detailed Conclusions

Detailed Conclusions

Detailed Recommendations

Detailed Recommendations

Data Collection Methods

Data Collection Methods FindingsFindings Specific DecisionsSpecific Decisions

AnalysisSynthesisInterpretationAssessment




Evaluation Purpose

Evaluation Purpose

Source: Danida, 1999.

4


Tools of Monitoring and Evaluation

Performance indicatorsThe logical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation


Performance IndicatorsMeasures:

InputsProcessesOutputsOutcomes

Enables:Tracking progressDemonstrate resultsTake corrective action

Stakeholder participation improves their understanding and allows them to use indicators for decision-making

5


Use of Performance Indicators

Setting targets and assessing progress toward achieving themIdentifying problems through early warning system to take corrective actionTells whether an in-depth evaluation is needed


Advantages of Performance Indicators

Effective means to measure progress towards objectivesFacilitates benchmarking:

With different organizational unitsWith districtsOver time

6


Disadvantages of Performance Indicators

Poorly defined indicators are not good measures of successTendency to define too many indicators, or those without accessible data sources, making system costly, impractical and likely to be underutilizedOften a trade-off between picking the optimal or desired indicators and having to accept the indicators which can be measured using existing data


Cost for Performance Indicators

Ranges from low to highDepends on:

Number of indicators collectedFrequencyQuality of information sought

7


Skills Needs for Performance Indicators

Well trained people for defining practical indicatorsMIS skills are required for implementing performance monitoring systems


Time Needed for Performance Indicators

Depends on:Extent of participatory processProgram complexity

May take 6–12 months

8



Performance indicatorsLogical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation


The Logical Framework (LogFrame) Approach

Helps clarify objectives of a project, program, or policyA vehicle for engaging partners in clarifying objectives and designing activitiesServes as a useful tool to review progress and take corrective action

9


LogFrame Approach (Cont.)

Helps identify expected causal links between inputs, processes, outputs, outcomes and impactLeads to identification of performance indicators at each stage in this chain, as well as risks which might impede the attainment of the objectives


Contents of a logical framework

External factors realized to obtain planned outputs on schedule?Actions outside the control of the project necessary for project inception

What are sources of information?

Indicators should be included in all activities.

Activities: Undertaken to accomplish outputs.

Factors outside the control of the project which, if not present, may limit progress from outputs to achievements of project purpose?

What sources of information?

Quantity of outputs and by when will they be produced? (Quantity, quality, time)

Outputs: Produced by the project in order to achieve project purpose

Necessary external factors if achievements of project purpose is to help reaching project goal?

What sources of information exist or can be provided cost-effectively? Does provision for collection need to be made under inputs-outputs?

Quantitative measures of impacts and benefits. (Estimated time)

Purpose:

Necessary external factors for sustaining objectives?

What sources of information exist, or can be provided cost-effectively?

Quantitative judgment of objectives achieved? (Estimated time)

Goal:

AssumptionsVerificationIndicatorsSummary

10


LogFrame UseImprove quality of project and program designs by requiring the specification of clear objectives, the use of performance indicators and assessment of risksSummarize design of complex activitiesAssist the preparation of detailed operational plansProvide objective basis for activity review, monitoring, and evaluation


Advantages of LogFrameEnsures that decision-makers ask fundamental questions and analyze assumptions and risksEngages stakeholders in the planning and monitoring processWhen used dynamically, it is an effective management tool to guide implementation, monitoring and evaluation

11


Disadvantages of LogFrame

If managed rigidly, stifles creativity and innovationIf not updated during implementation, it can be a static tool that does not reflect changing conditionsTraining and follow-up are often required


Cost for LogFrameLow to mediumDepends on extent and depth of participatory process

12


Skills Needs for LogFrameFacilitation skillAdvanced facilitation skills for participatory planning and management


Time needed for LogFrame• Depends on scope and depth of

participatory process• Several days to several months

13





Theory-Based EvaluationAllows in-depth understanding of the workings of a program or activity—the “program theory” or “program logic.” Need not assume simple linear cause-and-effect relationships

14


Theory-Based Evaluation (Cont.)

By mapping causal factors and how they interact, it helps decide

Which steps should be monitoredHow well they are in fact borne out

Allows identifying critical success factors


Theory-Based Evaluation (Cont.)

When data shows that these factors have not been achieved, the conclusion is that the program is less likely to be successful in achieving its objectives

15


Example of Theory-Based Evaluation

Success of a program to improve literacy levels by increasing the number of teachers might depend on:

Availability of classrooms and textbooksLikely reactions of parents, school administration and childrenSkills and morale of teachers Districts where extra teachers are to be locatedReliability of funding


Theory-Based Evaluation Use

Mapping design of complex activitiesImproving planning and management

16


Advantages of Theory-Based Evaluation

Provides early feedback about what is or is not working and whyAllows early correction of problems as they emergeAssists identification of unintended side-effects of the programHelps prioritize issues to investigate in greater depth by using focused data collection or sophisticated M&E techniquesProvides basis to assess the likely impacts of programs


Disadvantages of Theory-Based Evaluation

Can become overly complex if the scale of activities is large or if an exhaustive list of factors and assumptions is assembledStakeholders might disagree about which determining factors they judge important, which can be time-consuming to address

17


Cost for Theory-Based Evaluation

Medium to highDepends on

Depth of analysisDepth of data collection


Skills Needs for Theory-Based Evaluation

Facilitation skillAnalytical skill

18


Time needed for Theory-Based Evaluation

Depends onDepth of the analysisDuration of the program or activityDepth of M&E work undertaken




19


Formal SurveysCan be used to collect standardized information from a carefully selected sample populationSurveys often collect comparable information for a relatively large population in target groups


Formal Surveys UseProvides baseline data against which performance of the strategy, program or project can be comparedCompares different groups at a given point in timeCompares changes over time in the same group

20


Formal Surveys Use (Cont.)

Compares actual conditions with the targets established in a program or project designDescribes conditions in a particular community or groupProvides a key input to a formal evaluation of the impact of a program or project


Advantages of Formal Surveys

Findings from the sample of people interviewed can be applied to the wider target group or populationQuantitative estimates can be made for the size and distribution of impacts

21


Disadvantages of Formal Surveys

Results are often not available for a long periodData processing and analysis can be a major bottleneck for the large surveysExpensive and time-consumingSome kinds of information are difficult to obtain through formal interviews


Cost for Formal SurveysRanges from roughly $30–60 per household in the United States Costs will be significantly higher if there is no master sampling frame for population to be surveyed

22


Skills Needs for Formal Surveys

Sound technical and analytical skills for sample and questionnaire design, data analysis and processing


Time needed for Formal Surveys

Depends on sample sizeMay require 6 months to 2 years

23





Rapid Appraisal MethodsA quick, low-cost way to gather the views and feedback of beneficiaries and other stakeholders, in order to respond to decision-makers’ needs for information

24


Rapid Appraisal Methods Use

Provide rapid information for management decision-making for projects or programsProvides qualitative understanding of

complex socioeconomic changeshighly interactive social situations, or people’s values and motivations

Provide context and interpretation for quantitative data collected by more formal methods


Advantages of Rapid Appraisal Methods

Low costCan be conducted quicklyProvides flexibility to explore new ideas

25


Disadvantages of Rapid Appraisal Methods

Findings usually relate to specific communities or localities—thus difficult to generalizeLess valid, reliable and credible than formal surveys


Cost for Rapid Appraisal Methods

Low to medium, depending on the scale of methods adopted

26


Skills Needs for Rapid Appraisal Methods

Non-directive interviewingGroup facilitationField observationStatistical skills


Time needed for Rapid Appraisal Methods

Four to six weeksDepends on size and location of the population interviewed and the number of sites observed

27


Rapid Appraisal MethodsKey informant interviewFocus group discussionCommunity group interviewDirect observationMini-survey


Rapid Appraisal MethodsKey informant interview

Series of open-ended questions posed to individuals selected for their knowledge and experience in a topic of interestInterviews are qualitative, in-depth and semi-structured

28


Rapid Appraisal MethodsFocus group discussion

Facilitated discussion among 8–12 carefully selected participants with similar backgroundsParticipants might be beneficiaries or program staffThe facilitator uses a discussion guideNote-takers record comments and observations


Rapid Appraisal MethodsCommunity group interview

Series of questions and facilitated discussion in a meeting open to all community membersThe interviewer follows a carefully prepared questionnaire

29


Rapid Appraisal MethodsDirect observation

Use of a detailed observation form to record what is seen and heard at a program siteInformation may be about ongoing activities, processes, discussions, social interactions and observable results


Rapid Appraisal MethodsMini-survey

A structured questionnaire with a limited number of close-ended questions that is administered to 50–75 peopleSelection of respondents may be random or ‘purposive’ (interviewing stakeholders at locations such as a clinic for a health care survey)

30





Participatory MethodsProvide active involvement in decision-making for those with a stake in a project, program, or strategy and generate a sense of ownership in the M&E results and recommendations

31


Participatory MethodsUse

Learn about local conditions and local people’s perspectives and priorities to design more responsive and sustainable interventionsIdentify problems and trouble-shooting problems during implementationEvaluate a project, program, or policyProvide knowledge and skills to empower public


Advantages of Participatory Methods

Examines relevant issues by involving key players in the design processEstablishes partnerships and local ownership of projectsEnhances local learning, management capacity and skillsProvides timely and reliable information for decision-making

32


Disadvantages of Participatory Methods

Sometimes regarded as less objectiveTime-consuming if key stakeholders are involved in a meaningful wayPotential for domination and misuse by some stakeholders to further their own interests


Cost for Participatory Methods

Low to mediumCosts vary greatly, depending on scope and depth of application and on how local resource contributions are valued

33


Skills Needs for Participatory Methods

Facilitation skill


Time needed for Participatory Methods

Varies greatly, depending on scope and depth of application

34


Commonly Used Participatory Tools

Stakeholder analysisParticipatory rural appraisal Beneficiary assessmentParticipatory monitoring and evaluation


Commonly Used Participatory ToolsStakeholder analysis

The starting point of most participatory work and social assessmentsUsed to develop an understanding of the power relationships, influence and interests of various people involved in an activity and to determine who should participate and when.

35


Commonly Used Participatory ToolsParticipatory rural appraisal

A planning approach focused on sharing learning between local people, both urban and rural, and outsidersEnables development managers and local people to assess and plan appropriate interventions collaboratively often using visual techniques so that non-literate people can participate


Commonly Used Participatory ToolsBeneficiary assessment

Involves systematic consultation with project beneficiaries and other stakeholders to identify and design development initiatives, signal constraints to participation, and provide feedback to improve services and activities.

36


Commonly Used Participatory ToolsParticipatory M&E

Involves stakeholders at different levels working together to identify problems, collect and analyze information and generate recommendations




37


Public Expenditure Tracking Surveys (PETS)

Tracks flow of public funds and determine the extent to which resources actually reach the target groupsExamines the manner, quantity, and timing of releases of resources to different levels of government, particularly to the units responsible for the delivery of social services


PETS (Cont.)

Often implemented as part of larger service delivery and facility surveys which focus on the quality of service, characteristics of the facilities, their management, incentive structures, etc.

38


PETS UseDiagnosing problems in service delivery quantitativelyProviding evidence on delays, “leakage” and corruption


Advantages of PETSSupports the pursuit of accountability when little financial information is availableImproves management by pinpointing bureaucratic bottlenecks in the flow of funds for service delivery

39


Disadvantages of PETSGovernment agencies may be reluctant to open their accounting booksCost could be substantial


Cost for PETSCan be high until national capacities to conduct them have been established

40


Skills Needs for PETSSound technical and analytical skills for sample and questionnaire design, data analysis and processing, and good understanding of sector to be assessed


Time needed for PETSThree to six months

41





Cost-Benefit & Cost-EffectivenessAnalysis

Cost-benefit and cost-effectiveness analysis are tools for assessing whether or not the costs of an activity can be justified by the outcomes and impactsCost-benefit analysis measures both inputs and outputs in monetary termsCost-effectiveness analysis estimates inputs in monetary terms and outcomes in non-monetary quantitative terms (such as improvements in student reading scores)

42


Cost-Benefit & Cost-EffectivenessAnalysis Use

Informing decision makers about the most efficient allocation of resourcesIdentifying projects that offer the highest rate of return on investment


Advantages of Cost-Benefit & Cost-EffectivenessAnalysis

Good quality approach for estimating program and project efficiencyMakes explicit economic assumptions that might otherwise remain implicit or overlooked at the design stageUseful for convincing policy-makers and funding agencies that the benefits justify the activity

43


Disadvantages of Cost-Benefit & Cost-EffectivenessAnalysis

Fairly technical, requiring adequate financial and human resourcesRequired data may not be available and projected results may be highly dependent on assumptions madeResults must be interpreted with care where benefits are difficult to quantify


Cost for Cost-Benefit & Cost-Effectiveness Analysis

Varies greatly, depending on scope of analysis and availability of data

44


Skill Needs for Cost-Benefit & Cost-EffectivenessAnalysis

Require technical skill in economic analysis and availability of relevant economic and cost data


Time needed for Cost-Benefit & Cost-EffectivenessAnalysis

Varies greatly depending on scope of analysis and availability of data

45





Impact EvaluationSystematic identification of the effects – positive or negative, intended or not – on individual households, institutions and the environment caused by program or project

46


Impact Evaluation (Cont.)Helps understand the extent to which activities reach the poor and the magnitude of their effects on people’s welfare


Impact Evaluation (Cont.)For large scale sample surveys project populations and control groups are compared before and after at several points during program intervention

47


Impact Evaluation (Cont.)For small-scale rapid assessment and participatory appraisals estimates of impact are obtained from combining group interviews, key informants, case studies and available secondary data


Impact Evaluation UseMeasures outcomes and impacts of an activity and distinguishes these from the influence of other external factorsHelps clarify whether costs for an activity are justified

48


Impact Evaluation Use Cont.)

Tells decision makers whether to expand, modify or eliminate projects, programs or policiesDraws lessons for improving design and management of future activities


Impact Evaluation Use Cont.)

Compares effectiveness of alternative interventionsStrengthens accountability for results

49


Advantages of Impact Evaluation (Cont.)

Provides estimates of the magnitude of outcomes and impacts for different demographic groups, regions or over timeSystematic analysis and rigor can give managers and policy-makers added confidence in decision-making


Advantages of Impact Evaluation

Provides answers to central development questions:

To what extent are we making a difference?What are the results on the ground? How can we do better?

50


Disadvantages of Impact Evaluation

Some approaches are very expensive and time-consumingReduced utility when decision-makers need information quicklyDifficult to identifying an appropriate alternative


Cost for Impact EvaluationNumber of World Bank impact evaluations have ranged from $200,000-$900,000 depending on program size, complexity and data collectionRapid impact evaluations can be conducted for as little as $10,000-$20,000

51


Training Needs for Impact Evaluation

Strong technical skills in:Social science research designManagementAnalysis and reportingQuantitative and qualitative research skills


Time needed for Impact Evaluation

Can take up to 2 years or moreRapid assessment evaluations can often be conducted in less than 6 months.

52


Examples Impact Evaluation1. Randomized pre-test post-test

evaluation2. Quasi-experimental design with

before and after comparisons of project and control populations

3. Ex-post comparison of project and non-equivalent control group

4. Rapid assessment ex-post impact evaluations


1. Randomized pre-test post-test evaluation• Design:

Subjects (families, schools, communities etc) are randomly assigned to project and control groupsQuestionnaires or other data collection instruments are applied to groups before and after the project interventionAdditional observations may be made during project implementation

53


1. Randomized pre-test post-test evaluation

Example:Water supply and sanitation services where the demand exceeds supplyBeneficiaries are selected by lottery


2. Quasi-experimental design with before and after comparisons of project and control populations• Design:

Where randomization is not possible, a control group is selected which matches the characteristics of the project group as closely as possibleWhere projects are implemented in phases, participants selected for subsequent phases can be used as the control for the first phase project group

54


2. Quasi-experimental design with before and after comparisons of project and control populations

Example:These models have been applied in World Bank low-cost housing programs in El Salvador, Zambia, Senegal and the Philippines



• Design:Data collected on project beneficiaries and a non-equivalent control group is selected as Model Data collected only after project implementationMultivariate analysis is often used to statistically control for differences in the attributes of the two groups

55



• Example:Assessing the impacts of micro-credit programs in BangladeshVillages where micro-credit programs were operating were compared with similar villages without these credit programs


4. Rapid assessment ex-post impact evaluations• Design:

Some evaluations only study groups affected by the project while others include matched control groupsParticipatory methods can be used to allow groups to identify changes resulting from the project, who has benefited and who has not, and what were the project’s strengths and weaknesses

56


4. Rapid assessment ex-post impact evaluations (Cont.)• Design:

Triangulation is used to compare the group information with the opinions of key informants and information available from secondary sourcesCase studies on individuals or groups may be produced to provide more in-depth understanding of the processes of change


4. Rapid assessment ex-post impact evaluations• Example:

Assessing community managed water supply projects in Indonesia

Workshop II Project Management...1 UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 1 Workshop...

Documents

Transcript of Workshop II Project Management...1 UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 1 Workshop...