1

Workshop Introduction to project evaluations

Inka Píbilová inka@evaluace.com

10 February 2014

Climbing Kilimanjaro (5,895 m) in 1 year

2 Source: Suzanne Field

EC Project cycle management

Programming

Identification

Formulation Financing

Implementation and monitoring

Evaluation

3 http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Review logical framework

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 4

Quantitative - SMARTER

Specific / Simple (to understand, collect)

Measurable

Attainable/Available at cceptable costs

Relevant to project / stakeholders

Time-bound

Evaluate/Engaging

Reevaluate/Recordable

Use both quantitative and qualitative indicators

http://www.smarttoolkit.net/?q=node/391 http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Qualitative - SPICED

Subjective Participatory Interpreted and communicable Cross-checked and compared Empowering Diverse / disaggregated (by gender)

Compare using trends (increase), thresholds (min. 30%), targets (strategy by 12/Y1)

Min. 30 % of participants initiate a project aiming to address a local issue.

Reasons why participants have (not) implemented a project to address a local issue

5

Monitoring x evaluation x audit

Evaluation • Assessment of project efficiency, effectiveness, impact, relevance and sustainability for the purpose of learning and accountability to stakeholders Monitoring • Ongoing analysis of project progress towards achieving planned results with the purpose of improving management decision making Audit • Assessment of (i) the legality and regularity of project expenditure and income i.e. compliance with laws and regulations and with applicable contractual rules and criteria; (ii) whether project funds have been used efficiently and economically i.e. in accordance with sound financial management;; and (iii) whether project funds have been used effectively i.e. for purposes intended. • Primarily a financial and financial management focus, with the focus of effectiveness being on project results.

6 http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

7

Evaluation principles for a HORRIBLE evaluation

Source: Participants of the workshop

8

Evaluation principles for GREAT evaluations

Source: Participants of the workshop

• EC http://ec.europa.eu/europeaid/

– Impartiality and independence of the evaluation process from the programming and implementation functions;

– Credibility of the evaluation, through use of appropriately skilled and independent experts and the transparency of the evaluation process, including wide dissemination of results;

– Participation of stakeholders in the evaluation process, to ensure different perspectives and views are taken into account; and

– Usefulness of the evaluation findings and recommendations, through timely presentation of relevant, clear and concise information to decision makers.

• OECD DAC key norms, standards, criteria http://www.oecd.org/development/evaluation/dcdndep/41612905.pdf

• UNDP Evaluation policy http://web.undp.org/evaluation/policy.htm

• IPDET Handbook on Evaluation Ethics, Politics, Standards, Principles http://dmeforpeace.org/sites/default/files/M14_NA.pdf

• Patton, Michael Quinn.(2008) Utilization-Focused Evaluation: 4th edition

9

Evaluation standards and principles

10 Source: Participants of the workshop

• Background (project, context, stakeholders)

• Purpose

• Use of outputs

• Scope and focus

• Evaluation criteria and questions

• Methodology

• Timetable

• Budget

• Human resources – responsibilities, expertise required

Guidelines:

• UNDP Handbook (p. 194 - 200) ttp://web.undp.org/evaluation/handbook/

• EC/EuropeAid Project Cycle Management Guidelines (p. 126 - 136) http://ec.europa.eu/europeaid/multimedia/publications/documents/tools/europeaid_adm_pcm_guidelines_2004_en.pdf

• New Zealand Guide http://dmeforpeace.org/sites/default/files/NZAID_ToR%20Guideline.pdf

11

Evaluation Terms of Reference - structure

Problem Tree

Example: River Pollution

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 12

Stakeholder Analysis Matrix

Example: River Pollution

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 13

Stakeholder Analysis – Power / Interest / Attitude In

flu

en

ce

/Po

we

r

KEEP SATISFIED INFLUENCE & ENGAGE

KEEP INFORMED MONITOR

Interest

Policy makers

CSOs Activists

Attitude

14

Background (project, context)

- project name, identification

- history of the project, objectives, results, key activities, progress over time (add logical framework if you wish – you get more tailored proposals)

- organisational, social and political context in which the evaluation occurs

- main stakeholders involved in the project incl. target groups, beneficiaries, partners, donors

- focus and scope of the evaluation – which project components, geographical area, time period, target groups etc.

Our project:

15

Evaluation Terms of Reference - example

The rationale and purpose of the evaluation

- Why the evaluation is being undertaken (what do you want to get out of it) including for accountability, learning, improvement

- Why now

Use of outputs

- How it will benefit the different stakeholders

- To whom, when and how the findings will be reported (debriefing, presentation, report, videos, posters – printed or on-line…)

Our project:

16

Evaluation Terms of Reference - example

• OECD/DAC evaluation criteria (similar to the EC/EuropeAid)

– using project logical framework

• Questions, concerns and values of stakeholders

• Previous research / evaluations

• Guidelines / Evaluation Tools such as Kirkpatrick Model

• Experts

17

Methodology - sources of evaluation questions

Source: Road to Results

How to measure learning outcomes?

http://leanlearning.wikispaces.com/learning_analytics 18

OECD/DAC Evaluation criteria

19 http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Relevance – are we doing the right things?

The appropriateness of project objectives to the problems that it was supposed to address, and to the physical and policy environment within which it operated. The extent to which the project is suited to the priorities and policies of the target group, recipient and donor.

It should include and including an assessment of the quality of project preparation and design – i.e. the logic and completeness of the project planning process, and the internal logic and coherence of the project design.

Potential evaluation questions:

• To what extent are the objectives of the programme still valid?

• Are the activities and outputs of the programme consistent with the overall goal and the attainment of its objectives?

• Are the activities and outputs of the programme consistent with the intended impacts and effects?

20

OECD/DAC Evaluation Criteria

Source: OCEED/DAC, http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Effectiveness – are we doing things right?

An assessment of the contribution made by results to achievement of the Project Purpose, and how Assumptions have affected project achievements.

This should include specific assessment of the benefits accruing to target groups, including women and men and identified vulnerable groups such as children, the elderly and disabled.

Potential evaluation questions:

• To what extent were the objectives achieved / are likely to be achieved?

• What were the major factors influencing the achievement or non-achievement of the objectives?

21

OECD/DAC Evaluation Criteria

Source: OCEED/DAC, http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Efficiency – is the project worthwhile?

The fact that the project results have been achieved at reasonable cost, i.e. how wellinputs/means have been converted into activities, in terms of quality, quantity and time, and the quality of the results achieved.

The project shou use the least costly resources possible in order to achieve the desired results.

This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted.

Potential evaluation questions:

• Were activities cost-efficient?

• Were objectives achieved on time?

• Was the programme or project implemented in the most efficient way compared to alternatives?

22

OECD/DAC Evaluation Criteria

Source: OCEED/DAC, http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Impact – what changes has the project achieved / contributed to?

The effect of the project on its wider environment, and its contribution to the wider policy or sector objectives (as summarised in the project’s Overall Objective).

The positive and negative changes produced by a project, directly or indirectly, intended or unintended, positive and negative.

This involves the main impacts and effects resulting from the project on the local social, economic, environmental and other development indicators. It must also include the positive and negative impact of external factors.

Potential evaluation questions:

• What has happened as a result of the programme or project?

• What real difference has the activity made to the beneficiaries?

• How many people have been affected?

23

OECD/DAC Evaluation Criteria

Source: OCEED/DAC, http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Sustainability – wil the changes (nebefits for target group) last?

An assessment of the likelihood of benefits produced by the project to continue to flow after external funding has ended, and with particular reference to factors of ownership by beneficiaries, policy support, economic and financial factors, socio-cultural aspects, gender equality, appropriate technology, environmental aspects, and institutional and management capacity.

Sustainability is concerned with measuring whether the benefits of a project are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially (and socially) sustainable.

Potential evaluation questions:

• To what extent did the benefits of a programme or project continue after donor funding ceased?

• What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?

24

OECD/DAC Evaluation Criteria

Source: OCEED/DAC, http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

• What „is“

• Describe project (inputs, activities, outputs) or process

• Simple: Who, what, where, when, how, how many …

• Often used to gather opinions from target groups.

Examples:

• What is the project objective from the perspectives of different stakeholders?

• What were the reasons for joining the program?

• How many persons were reached?

• How was the project implemented?

Evaluation design to answer them:

One-shot, before-and-after, time series, (long-term) panel, case studies…

25

Evaluation Questions - Descriptive

Source: Road to Results

• Compare what „is “ with what „should be“ (target)

Examples:

• Do the project activities address the needs of stakeholders?

• To what extent has the project achieved the result/objective indicators?

• Have min. 5.000 persons been reached through the campaign?

• Has the number of schools involved in Global Learning increased? (baseline!)

Evaluation design to answer them:

One-shot, before-and-after, time series, (long-term) panel, case studies…

26

Evaluation Questions - Normative

Source: Road to Results

• Determine what difference the project makes - what change has it brought

• Often refer to outcome, impact

• Compare indicators before and after, with and without the project (graph)

• Careful about attribution x contribution!

– Can you say that the project achieved this or has contributed to this?

– Are there any alternative explanations (external factors) for achievements?

Examples:

• As a result of the training, have teachers incorporated Global Learning in their lesson plans?

27

Evaluation Questions – Cause-and-Effect

Time

Goal

without

with

T=0 T=1

Evaluation design to answer them: Experimental (control group), quasi experimental (compare group), nonexperimental (causual tracing, case study, story harvesting, outcome mapping…) Source: Road to Results

• Evaluation criteria and questions

Design, Data Our project – Question Baseline data Indicators Sources Collection Methods

28

Evaluation Terms of Reference – Evaluation Matrix

• Avoid questions on multiple issues – separate these

– Has the methodology been developed and disseminated to min. 5.000 teachers?

• You can develop subquestions for a particular question on an issue

– What concerns have teachers while introducing Global Learning at school ? (descriptive)

– Has the project addressed these concerns of the teachers? (normative)

– Has the number of teachers using Global Learning increased as a result of the project? (cause-and-effect)

• Set a realistic number of questions!

29

Tips for evaluation questions

Source: Road to Results

Would the evaluation question… Q1 Q2 Q3..

• be in line with the evaluation purpose?

• be of interest to stakeholders?

• reduce present uncertainty?

• yield important information?

• be of continuing (not fl eeting) interest?

• be critical to the evaluation´s scope and comprehensiveness? (or nice to have)

• have an impact on the course of events?

• be answerable given the fi nancial and human resources, time, methods, and technology available?

• Be reasonable to ask given the project cycle? (Questions about impact, for example, are best answered after the project has been fully operational for a few years)

30

Select a few, most important questions…how?

Source: Road to Results, Kusters et al.: Making evaluations Matter

Sampling random x non-random (quota, typical case, snow-ball…), sample size!

Participatory – community meetings, mapping, transect walks..

31

Data Collection Methods - Quantitative x Qualitative x Mixed

Source: Road to Results / IPDET © 2009, adopted

Conversation with

Concerned Individuals

Community Interviews

Field Visits

Reviews of

Official Records

Participant Observations:

Diaries, journals, self-

reported checklists

Key Informant Interviews

Expert (panel) judgement

Delhi technique

Focus Group

Interviews

Panel Surveys

Censuses

Field Experiments

Informal/Less Structured Methods Formal/More Structured Methods

One-Time Surveys

Direct Observations

Surveys

Timeline and resources Te

rms

of

Ref

eren

ce –

Ob

ject

ives

, sc

op

e, s

take

ho

lde

rs, q

ues

tio

ns,

b

ud

get,

sch

edu

le, o

utp

uts

, use

.

Desk study Interviews Surveys Focus groups Case studies

Pre

limin

ary

fin

din

gs &

co

ncl

usi

on

s

Final debriefing of all partners

Communication with the Project Partners

Draft evaluation report commented by all partners Fi

nal

eva

luat

ion

rep

ort

Inception phase

1-3 months

Field research

1-3 months

Reporting phase

1-2 months

Init

ial b

rief

ing

and

ince

pti

on

Evaluator Selection

32

or mixed - participatory ! External evaluation is not necessarily independent! (Who pays the evaluator? Who checks the quality?)

! Even external evaluation consumes time of the project team!

! It depends WHO the evaluator is

Source: UNDP Handbook, Making evaluations matter

33

Internal evaluators or external evaluators

₊ May have a better understanding of the projetct, context, policies

₊ Develop organisational capacities ₊ Higher ownership of

recommendations by the organisation ₊ Usually cheaper ₋ May not be able to see alternative

perspectives, solutions ₋ Influenced more by the implementing

organisation (want to keep their jobs) ₋ May be less credible to stakeholders ₋ May be time-consuming

₊ May bring a new perspective or special (technical, evaluation) expertise

₊ More independent from the implementer – may facilitate better between stakeholders (across hierarchies, in case of mistrust)

₊ Usually perceived as more credible ₋ May not be able to comprehend fully

the project due to time/other contrains

₋ Usually more expensive

• Timetable

• Budget

• Human resources – responsibilities, expertise required

Our project:

34

Evaluation Terms of Reference - example

Evaluability

- Are clear why we do the evaluation?

- Do we have an (updated) logical framework?

- Do we have sufficient (baseline, monitoring) data available?

- Do we have accessible reliable information sources?

- Do we have sufficient funds for an internal/external evaluation? Will the evaluation be cost-effective, will it bring reasonable benefits vs. costs?

- Is it likely that it will be used to improve actions in future? Can stakeholders influence the evaluation decisions? Will they accept and use the findings? Is there a strong leadership to put the recommendations in practice?

- Are there no major factors hindering the evaluation? Are staff members or other stakeholders overloaded due to other priorities? Are there any tendencies that would affect impartiality?

Source: UNDP Handbook, Road to Results, Making evaluations matter

35

Are we ready?

Qualitative Data Analysis – frequency and relations

36

Topics Quotes Findings

Parents decide on topics I think the process of deciding would be valuable.

There was a strong feeling that parents should be more involved in the choice of topics

//// //// //// //// //// //// //// ////

Cover a couple of topics per session

Sometimes we just got into a topic and then it was time to leave or move to something else. We need more time to discuss.

Many participants (38 or 52 interviewed) thought there should be more time for discussion.

//// //// //// ///

Not enough time spent on each topic

//// //// //// ///

Evaluation question: Were parents satisfied with the training?

Source: IPDET © 2009

Develop categories, check reliability, analyse, interpret, share draft, report

Quantitative Data Analysis

37 Source: IPDET © 2009, docs. Google.com

• Clean data • Use mode (most frequent), median (middle value), mean (average), range

(lowest to highest value), statistical methods (correlations…) • Produce tables, graphs etc

Data Analysis and Interpreting

38

• Needs to be clear before data collection

• Methodology incl. reliability and limits is a part of report

• Categorize and triangulate findings

• Use quotes, examples, graphs

• Distinguiish between findings (evidence) and interpreting (conclusions)

• Do not generalize findings from 3 respondents to the whole sector!

• Have a short summary for those who cannot read the whole report

Source: Road to Results, evaluace.com

Reporting: as per the expected use by each stakeholder

39

Case study

Reporting – example of quantitative data

40 Source: Inka Pibilova

the Project A

Project A

Project A

Project A

Project A

Reporting – example of quantitative data

41 Source: Naviga4: Prevention and early detection of women with cancer, Georgia, MFA CR

No. of women reached

2

20

48

31

303

476

3244

1033

Léčba prekancerózy

Léčba rakoviny

Prekanceróza

Rakovina

Vyšetřeny v onkocentrech

Odeslané do onkocenter

Vyšetřeny v mobilních…

Účast na osvětových akcích

Vyléčeno ?

Participation in awareness raising

Checked-up in mobile clinics

Sent to oncocentres

Checked-up in oncocentre

Diagnosed with cancer

Diagnosed with pre-cancer

Treated with cancer

Treated with pre-cancer

Cured

Reporting – example of qualitative data

42 Source: Inka Pibilova

Project A

Finding: Most volunteers expressed doubts about the Programme, which partially impacted external communication and advocacy. They believed the donor should better clarify the program objectives – if the programme is to primarily serve the communities in the South or the young professionals from Europe.

“I believe the development sector

needs well trained and well managed

professionals, not volunteers sent

with a weak/unclear mandate to 'do

something'. This may end up doing

more harm than good." Volunteer

Reporting – example of qualitative data

43 Source: Inka Pibilova

Project A Too expensive

treatment...

...incorrect

Radiation

inaccessible

Chemistry teacher and mother, 26 years old Cervical precancerosis Incorrect treatment and relapse

Reporting – example of qualitative data

• Examples of evaluation reports (see bottom of evaluation section) and other tools at http://www.evaluace.com/ - or contact inka@evaluace.com

• Road to Results https://openknowledge.worldbank.org/handle/10986/2699

• EPDET development evaluation training 31 August – 6 September 2014 in Slovakia – check http://www.dww.cz/index.php?page=epdet

• www.Betterevaluation.org

• EC evaluation guidelines http://ec.europa.eu/europeaid/evaluation/methodology/tools/too_en.htm

• UNDP Handbook (p. 194 - 200) ttp://web.undp.org/evaluation/handbook/

• OECD DAC key norms, standards, criteria http://www.oecd.org/development/evaluation/dcdndep/41612905.pdf

• RISK, UK: How do we know it is working? http://www.developmenteducationreview.com/issue11-review1

• The Most Significant Change Guide: http://www.mande.co.uk/docs/MSCGuide.pdf

44

Where to learn more?