Comprehensive evaluation of the European Union ......Comprehensive evaluation of the European Union...

Comprehensive evaluation of the European Union humanitarian aid, 2012-2016

October, 2017 1

Comprehensive evaluation of

the European Union humanitarian aid, 2012-2016

Annexes to the Draft final report

Volume 1


October, 2017 2

Table of Contents

Contents

Annex 1 Cost-effectiveness assessment framework ............................................ 5

A1.1 Introduction ...................................................................................... 5

A1.1.1 Structure .................................................................................... 5 A1.1.2 Methodological notes ................................................................... 5

A1.2 Cost-effectiveness of ECHO as a donor ................................................. 7 A1.3 Cost-effectiveness of ECHO-funded actions ..........................................12 A1.4 Additional descriptive indicators ..........................................................14

Annex 2 Full list of interviewed stakeholders .....................................................15 Annex 3 Methodological notes on the sources and use of the quantitative data .....18

A3.1 EVA data .........................................................................................18

A3.1.1 Source of EVA data .....................................................................18 A3.1.2 Content of EVA data ...................................................................19

A3.2 UN OCHA FTS funding data ................................................................20

A3.2.1 Source of FTS funding data ..........................................................20 A3.2.2 Content of FTS data ....................................................................20 A3.2.3 Comparison of ECHO’s EVA data and UN OCHA FTS data .................21

A3.3 Appeals data ....................................................................................24

A3.3.1 Introduction to the appeals process ..............................................24 A3.3.2 Data on the Appeals and Response Plans .......................................25 A3.3.3 Limitation to the FTS response plan data extracted .........................27 A3.3.4 Appeals coverage and funding gap ...............................................28

Annex 4 Profile of respondents to the three surveys ..........................................29

A4.1 Respondents to the ECHO field survey .................................................29 A4.2 Respondents to the framework partner survey......................................30 A4.3 Respondents to the local partner survey ..............................................33

Annex 5 Meta-evaluation ................................................................................35

A5.1 Introduction .....................................................................................35 A5.2 Findings of the meta-analysis by evaluation questions ...........................36

A5.2.1 Relevance .................................................................................36 A5.2.2 Coherence .................................................................................53 A5.2.3 EU added value ..........................................................................71 A5.2.4 Effectiveness .............................................................................76 The extent to which ECHO funded actions effectively promote the visibility of

ECHO activities ...........................................................................................87 A5.2.5 Efficiency ..................................................................................88 A5.2.6 Sustainability .............................................................................96

Annex 6 Field reports ...................................................................................... 1 Annex 7 Rapid evaluations .............................................................................. 1 Annex 8 Relevance ......................................................................................... 1

A8.1 Supporting evidence EQ1: Needs-based approach in ECHO ..................... 1

A8.1.1 Review of the needs assessment process in ECHO ........................... 1 A8.1.2 Level of needs, ECHO funding, global funding, humanitarian appeals . 3


October, 2017 3

A8.2 Supporting evidence for EQ2: ECHO’s choice of partners ......................... 9 A8.3 Supporting evidence on EQ3: Appropriate, comprehensive and context-adapted strategies .........................................................................................17 A8.4 Supporting evidence on EQ4: Field network .........................................19

Annex 9 Coherence .......................................................................................31

A9.1 Survey results ..................................................................................31 A9.2 Activation of the UCPM in context where ECHO was covering the same crisis from a humanitarian aid perspective ........................................................32

Annex 10 EU added value ..........................................................................35 Annex 11 Effectiveness .............................................................................41

Figure 1. Local implementing partners: In your view what has been ECHO’s role in… 42

Annex 12 Efficiency ..................................................................................44

A12.1 Statistics on use of different transfer modalities ....................................44 A12.2 Statistics on portfolio levels by TA for different parts of the world ...........46 A12.3 ECHO budget funding sources ............................................................46 A12.4 Indicators for cost-effectiveness analysis .............................................48 A12.5 Survey results ..................................................................................50 A12.6 Available benchmarks for TCTRs .........................................................53

Table 1. Average TCTRs in ECHO funded projects 2011-2014 .......................53

Annex 13 Prospective evaluation ................................................................55

A13.1 List of EU commitments for the World Humanitarian Summit ..................55

Annex 14 OPC analysis ..............................................................................58

A14.1 Introduction ..................................................................................58 A14.1.1 Purpose and scope of the open public consultation .................58 A14.1.2 Structure of the report ............................................................58 A14.2 Overview of responses to the open public consultation .................58 A14.2.1 Number and distribution of replies received ............................58 A14.2.2 Distribution by level of familiarity with DG ECHO humanitarian aid activities ...............................................................................................59 A14.3 Analysis of the survey responses ...................................................59 A14.3.1 DG ECHO performance 2012-2016 ...........................................59 A14.3.1.1 Relevance ................................................................................60 A14.3.1.2 Coherence ...............................................................................61 A14.3.1.3 EU added value ........................................................................66 A14.3.1.4 Effectiveness ...........................................................................68 A14.3.1.5 Sustainability ..........................................................................69 A14.3.1.6 Visibility ..................................................................................71 A14.3.2 Follow-up to DG ECHO's international commitments and global humanitarian challenges .............................................................................71

Annex 15 Terms of reference .....................................................................76 1. General context ......................................................................................76 2. EU Humanitarian Aid ...............................................................................78

2.1. Legal and strategic framework ................................................................78 2.2. Objectives of EU humanitarian aid operations ...........................................81 2.3. The EU intervention in the humanitarian field ...........................................81 2.4. Implementation and financial framework .................................................83 2.5. Evaluations and related exercises ...........................................................84

Overall exercises .........................................................................................84 Recently completed geographic and thematic evaluations ................................84


October, 2017 4

Recently launched evaluations (on-going) ......................................................85

2.6. Cost-effectiveness guidance for ECHO evaluations ....................................85

3. Purpose and scope of the evaluation ..........................................................85

3.1. Purpose ...............................................................................................85 3.2. General scope ......................................................................................86 3.3. Evaluation questions .............................................................................87 3.4. Supplementary tasks.............................................................................88

4. Management and supervision of the evaluation ...........................................89 5. Specific requirements ..............................................................................89

5.1. Methodology ........................................................................................89 5.2. Evaluation team ...................................................................................90

6. Content of the Offer ................................................................................91 7. Budget of the contract .............................................................................91 8. Timetable ...............................................................................................92 9. Phases of the evaluation ..........................................................................92

9.1. Inception Phase ....................................................................................93 9.2. Desk phase ..........................................................................................93 9.3. Field Phase ..........................................................................................94 9.4. Synthesis Phase ...................................................................................95 9.5. Dissemination and Follow-up Phase .........................................................96

10. Award ....................................................................................................97 Annex 1: Cost-effectiveness guidance for ECHO evaluations ..................................98 Annex 2: The final report ...................................................................................98


October, 2017 5

Annex 1 Cost-effectiveness assessment framework

A1.1 Introduction

A1.1.1 Structure

The general EQ for this framework is: “To what extent did ECHO achieve cost-

effectiveness in its response?”

In line with ADE guidance, the framework assesses the following two dimensions:

1- Cost-effectiveness of ECHO as a donor

2- Cost-effectiveness of ECHO funded actions.

This framework closely follows the ADE guidance; we have not tried to re-invent the

wheel although some improvements have been made to the ADE guidance.

Specifically, we have added decision rules for each indicator to make the process of

drawing evaluative judgement clear, transparent and as objective as possible.

The framework is focused on key aspects directly linked to cost-effectiveness – coming

from relevance and efficiency sections. Other aspects also feed into cost effectiveness:

‘Effectiveness’ itself, coordination/ coherence, configuration of field network. These are

covered under separate evaluation questions of the evaluation framework.

For easier reference, the evaluation framework from the inception report is pasted in

annex and those questions covered by CEA framework are highlighted.

We also included a table with descriptive background indicators which are not

associated to any decision rules (and are therefore not part of the CEA framework per

se but are nevertheless there to help the team have a better understanding of the

issues).

A1.1.2 Methodological notes

As our evaluation is a comprehensive, strategic one, we have more indicators related

to the first part ‘cost-effectiveness of ECHO as a donor’.

The analysis is also mostly based on level 1 tools1 rather than level 3 tools2 – as

current evaluation budget and timescales do not permit intensive data collection and

analysis foreseen for level 3 analysis. Moreover, past evaluations do not allow us to

carry out a deeper analysis at the intervention level (CEA-related issues are only

covered to a limited extent and mostly qualitatively).

More specifically, the following list indicates which aspects - related to ‘cost-

effectiveness of ECHO funded actions’ - are not covered in the CEA framework and for

which reasons:

Cost-effectiveness or cost benefit ratio (CER): calculated by using the total costs as the numerator and the quantitative benefits as the denominator.

Producing CER even by sector would be too difficult. Aside the from usual

1

Level one tools require low to medium additional effort with a focus on qualitative evidence (and quantitative evidence where possible and easy to access) through document review and interviews that can be easily added to existing evaluation interview schedules or scanning for new information within a broader review of documents. 2

Level three tools require the most effort, time and skills as the data required for the analysis is not generated routinely in ECHO management and monitoring systems. It should not be expected for all evaluations to use level three tools unless dedicated resources are provided, however these tools may be of particular importance in specific contexts or may become more relevant over time with management and monitoring system changes that more readily enable the analysis.


October, 2017 6

problems of lack of indicators on which to base the analysis and attribution

issues, the main issues are that:

Not all benefits can be monetised and summed up: Projects even within the same sector produce multiple outcomes including non-quantifiable ones which

cannot be aggregated.

Only the health sector has its routinely used summary measure of health outcome, namely quality-adjusted life year (QALY) and even QALYs (which

incorporate the impact of interventions on both the quantity and quality of life)

have several limitations which matter a lot in HA context e.g. the fact that a

QALY is a QALY regardless of who accrues it (equity aspect - the distribution of

health is neglected)3.

Other sectors do not have a single, measurable outcome indicator.

Such CER ratios are not very helpful in absence of benchmarks: Comparable benchmarks often do not exist due to specificity of crisis situations and donor

interventions: there is lack of comparability of interventions in general

(differences in timing, programme design and location -e.g. more accessible,

less accessible-, scale, programmes with start-up costs vs on-going projects). .

Alpha ratio: value of transfers reaching beneficiaries. Calculating the alpha ratio for a larger sample than the Inspire study would have implied level 3 analysis

which is not feasible within the scope of the assignment – e.g. it would have

implied completing a fiche for each project based on financial and narrative

reports submitted to ECHO (like we did for project mapping but in more detail

onbudget data of partners) but also collecting additional data provided by

partners in response to questions of clarification.

Cost per beneficiary or cost per unit of result is calculated by dividing the budget of a programme or an output by the number of beneficiaries or units of

result. Calculating these ratios was not possible in the present case.

Project mapping illustrated that even for key results, costing data is not available in a format to calculate cost per unit.

Interpreting cost per beneficiary or cost per unit of result implies making some comparisons: examples of possible comparisons include cash transfers v. food

aid to deliver an amount of calories or nutritional value; funding an intervention

through one NGO v. another NGO; the cost of an intervention v. the average

cost of similar interventions in that context. Interventions even within the same

sector are not comparable when looking at the global portfolio of ECHO which

comprises interventions implemented in very different contexts (see earlier

point on lack of comparability of interventions in general).

Overall number of beneficiaries: an aggregate figure (in total or by sector) cannot be retrieved from the information systems because of double counting,

e.g.

- The same project has activities in various sectors: summing counts of beneficiaries per sector could actually imply counting all the same

beneficiaries again if they benefit from interventions in several sectors

(double counting of the beneficiaries).

- The same beneficiaries can also benefit from a follow up project in the same year

3 Due to its limitations and ethical issues, this approach was not adopted in the sector specific health

evaluation


October, 2017 7

- Also difficult to give a limit to the definition of a beneficiary. There can be indirect beneficiaries to some projects too. Ex: building of public

infrastructure.

A1.2 Cost-effectiveness of ECHO as a donor

[Relevance section]

Judgement criterion: Available evidence suggests that ECHO’s budget allocations

are based on needs (including gaps lefts by other donors) rather than political

considerations

Indicator Indicator

type

Decision Rule(s) Source of

evidence

The extent to which ECHO’s

strategic programming (GGOPHAs / HIPs) is underpinned by

a robust and impartial needs assessment

Qualitative INFORM, FCA and IAFs are robust and based on reliable

methodologies and data

There is clear link between ECHO funding allocations to specific crises and needs while

taking account of gaps left by other donors

Countries/crises with the most needs and vulnerabilities have been allocated more funding as compared to countries with fewer needs

GGOPHA review

Review of outputs

of INFORM, FCA and IAF and HIPs

Interviews with ECHO staff and

experts

The extent to

which stakeholders believe that ECHO’s budget

allocations are based on needs

Qualitative The general opinion across

different categories of stakeholders interviewed is that ECHO’s budget allocations are

based on needs while taking account of gaps left by other donors

Stakeholder

interviews

% of Framework partners who

strongly agree/ agree with the following statement: “ECHO funding allocations to

specific regions/ countries are based needs assessment rather than other factors

such as political considerations”

Quantitative At least 75% strongly agree/ agree

Q.21 of online survey of

framework partners

ECHO funding

allocations to specific crises/ sectors overtime

Quantitative Echo funding to specific sectors

and crises over time has evolved with needs of the sector/ crises

ECHO funding

allocations as per EVA / contracted amounts per HIPs

http://dgecho-partners-helpdesk.eu/financing_decisions/dgecho_strategy/start


October, 2017 8

Indicator Indicator

type

Decision Rule(s) Source of

evidence

% of ECHO funding to

different regions over time

Share of funding has evolved to reflect changing needs of

regions

HOPE / EVA database

/ contracted amounts per HIPs

% of ECHO funding allocated to forgotten crises

Quantitative ECHO's allocation to forgotten crises is reasonable in relation to needs.

HOPE / EVA database

Forgotten crisis index

ECHO funding allocated to forgotten crises as

a % of funding to forgotten crises

Quantitative ECHO is one of the main donors to forgotten crises

Forgotten crisis index

OCHA FTS

ECHO funding to specific sectors/ crises as a share of total needs

Quantitative A significant share of ECHO funding (ECHO total funding) is allocated to sectors/ crises with largest needs.

OCHA FTS appeals data

ECHO funding patterns take into account the gaps

left by other donors as measured by ECHO funding to specific sectors/

crises as a share

of total funding

Quantitative List of crises / sectors where ECHO is a major donor corresponds to the list of crises

/ sectors with largest gaps

OCHA FTS appeals data

OCHA FTS

NB: for the indicators under this Judgement criterion, the basis for our decision rules

will need to be fine-tuned after follow-up interviews with IAF experts at DG ECHO and

once we confirm with DG ECHO whether OCHA FTS / OCHA FTS appeals data can be

used at all.

Where ECHO funding needs to be put in perspective with what other donors are doing

at the global level, there is no other option but to use OCHA FTS for both ECHO

funding and other funding and stick to their definitions (of sectors for instance).

Despite the limitation related to some (ECHO) funding not being captured as

contributing to an appeal, FTS OCHA appeals data are the only source of information

available to give an overview of needs and how those are filled across sectors with the

same indicator

[Efficiency section]

Judgment criterion: ECHO balanced cost in relation to effectiveness and timeliness

in making strategic choices about its portfolio of assistance (i.e. including about its

partners, sectors, approaches, geographical locations, beneficiaries and transfer

modalities)


October, 2017 9

Indicator Indicator

type

Decision Rules Source of

evidence

Extent to which ECHO has a clear view of costs throughout the delivery

chain

Qualitative ECHO has a clear view of costs for each project

Inspire report

% of funding implemented via local implementing partners

Quantitative

% of funding implemented via local implementing partners from ECHO falls between the benchmarks of global average of 2% and target of 25%

Inspire study

% of projects with different transfer

modalities

Quantitative

Cash / in kind / vouchers

a growing share if echo funding is distributed in form of cash.

ECHO C&V database

Inspire

study

Adoption of innovations and/ or best practices driving cost effectiveness e.g. Cash vouchers, multi sectoral approaches, consortiums, multi annual funding

Qualitative ECHO pioneered or adopted quickly new approaches

Meta-evaluation

Interviews

Other evidence of ECHO

weighing up the costs and benefits of choices made

Qualitative Number of assessments

available about the choices made: e.g. funding different types of partners (NGOs versus UN), consortia versus individual

partners, different transfer modalities, and the balance between different approaches

(preparedness versus response) – especially in protracted crises

Depth and breadth of these assessments

Even in the absence of formal

assessments, the general opinion across ECHO is that there is strategic thinking about the choices made: e.g. funding different types of partners (NGOs versus UN), consortia versus individual partners,

different transfer modalities, and the balance between different approaches (preparedness versus response) – especially in protracted crises

Previous evaluation of ECHO

funded actions reflect upon choices made

Interviews

with ECHO staff (HQ and field)

Field missions

Meta-evaluation


October, 2017 10

Indicator Indicator

type


evidence

There are feedback and learning mechanisms (examples of efficient

delivery are identified and disseminated)

Qualitative There is evidence of some mechanisms and tools being in place (e.g. meetings,

workshops, trainings, community of practice, documents) to share results.

There are other qualitative elements confirming dissemination efforts

Literature review (monitoring

reports & evaluation)

Interviews with ECHO HQ and field officers

Online

survey with ECHO field

officers

% of Framework partners who strongly agree/ agree with the following statement: ECHO funding allocations take efficiency

and cost-effectiveness criteria into account in an appropriate way

Quantitative

At least 75% strongly agree/ agree

Q.21 of online survey of framework partners

Evidence of flexibility mechanisms being in place to find extra funding to beef up initial budget when needs emerge

Qualitative Mapping of channels available show various flexibility mechanisms exist to ensure timeliness of ECHO funding actions): funding available

through the channel, speed at

which it can be mobilized, cases where it has been used

Mapping of the decision making processes

Interview with HQ

Evidence of ECHO response being timely

Quantitative and

qualitative

Field interviews suggest ECHO response is timely

At least 75% of local implementing partners agreeing ECHO’s response is comprehensive, appropriate and timely given the identified needs and context in my country (Q24)

At least 75% of FPA partners

think ECHO’s speed of response to emerging crises facilitated effective delivery (Q31)

Interviews

Online survey local implementing partners

Online survey FPA

partners

Evidence of ECHO funding being predictable

Quantitative and qualitative

Whether the duration of ECHO funding cycles have generated any inefficiencies e.g. staff turnover etc

At least 75% of FPA partners

Desk research

Interviews


October, 2017 11

Indicator Indicator

type


evidence

think ECHO’s 12-18 month project funding cycles facilitated effective delivery

(Q31)

Online survey FPA partners

Evidence of funding allocations being revised when needs arise

Quantitative

Comparison of annexes to initial worldwide decisions vs revised worldwide decisions to assess revisions made if any

Desk research

Judgment criterion: ECHO took appropriate actions to ensure cost-effectiveness

throughout the project cycle

Indicator Indicator

type


evidence

The extent to which cost data and quantitative indicators of efficiency are available and used to drive efficiency

% sampled projects referring to quantitative indicators of efficiency, including:

Admin costs as % of

budget

"cost per beneficiary (value in $,€/beneficiary)”

"cost per output/outcome (value in $,€)"

Quantitative Majority of projects reviewed refer to them

Project mapping

Share of projects where there is any evidence of

ECHO considering cost-effectiveness, efficiency, vfm issues in project design and implementation; and nature of this evidence

Quantitative and

qualitative

Majority of projects reviewed have strong

evidence

Project mapping

Evidence of ECHO considering economy,

efficiency and cost-effectiveness in selecting partner proposals / negotiating contracts and monitoring partners

Qualitative Selection process includes criteria related the

economy, efficiency and cost-effectiveness.

ECHO officers examine economy, efficiency and cost-effectiveness issues during project

Review of the selection

process

Review of monitoring process

Interviews with


October, 2017 12

Indicator Indicator

type


evidence

implementation

There are other qualitative

elements confirming these are duly considered in the selection/ monitoring process

ECHO / field and partners

A1.3 Cost-effectiveness of ECHO-funded actions

Judgement Criterion: Humanitarian actions funded by ECHO were efficient / cost-

effective

Indicator Indic

ator

type


evidence

Alpha ratio Quantitative

Substantial share of ECHO reached the final beneficiary as per summary of Inspire study report which had the objective to answer the question, ‘how much funding

reaches the beneficiary?’

Inspire study

Previous thematic and geographical evaluations concluded that ECHO-funded actions are efficient / cost-effective

Qualitative

Majority of evaluations made this conclusion.

Meta-evaluation

ECHO staff share the view that projects are generally efficient / cost-effective– or

have an understanding of the factors influencing efficiency and take these into account

Qualitative

Majority of ECHO staff share this view

ECHO Interviews

Field

missions

ECHO sectoral staff have an understanding of main cost drivers for their sectors, cost per beneficiary for

selected KRI

Qualitative

Majority of ECHO staff have this understanding

Rapid evaluations

Judgement Criterion: partners systematically consider cost-effectiveness, efficiency,

vfm issues in project design and implementation


October, 2017 13

Indicator Indi

cato

r

type


evidence

Share of projects where there is any evidence of partners considering cost-effectiveness, efficiency, vfm issues in project design and implementation; and nature of this evidence

Qualitative and quantitative

Majority of projects mapped have strong evidence

Project mapping

Results of the multiple-answer question (Q38. What actions are typically taken by your organisation to ensure value for money in the delivery of ECHO funded actions?) indicate a range of actions are typically taken to ensure vfm

Quantitative

A majority of FPA respondents tick at least three actions

Online survey with FPA and local implementation partners

Judgement Criterion: Obstacles to / success factors for cost-effectiveness are clear

and appropriate adaptation strategies are put in place as a response (e.g. measures

mitigating the obstacles, adapting the intervention)

Indicator Indi

cato

r

type


evidence

Survey respondents have a shared understanding of main obstacles to / success factors for cost-effectiveness as per Answer to question: Based on your experience of implementing ECHO funded actions, please indicate the extent to which the following factors impede or facilitate their effective delivery.

Quantitative

Key items are selected by majority of respondents

Online survey with FPA and local implementation partners

Qualitative feedback from fieldwork shows the facts impeding / fostering cost-effectiveness are well understood and appropriate adaptation strategies are put in place as a response

Qualitative

Interviewees predominantly have this understanding

Fieldwork interviews with other relevant actors

Survey respondents and interviewees point towards the same factors identified as

Qualitative

Interviewees and survey respondents predominantly have this awareness

Meta-evaluation


October, 2017 14

Indicator Indi

cato

r

type


evidence

influencing cost-effectiveness in previous ECHO evaluations

% projects reviewed incorporating lessons learned

Quantitative

More than 75% do so project mapping

A1.4 Additional descriptive indicators

Indicator Indicator type Source of evidence

Average grant size Quantitative HOPE / EVA

% of projects which get funding several years in a row

% of ECHO funding to UN, NGOs and IOs Quantitative HOPE / EVA

% of funding to each partner Quantitative HOPE / EVA

Number of partners Quantitative HOPE / EVA

Value / Share of assistance reaching partners

Share of 23.02 chapter in total ECHO budget

Share of 23.02 chapter (minus cost field network, ECHO flight/ contribution to UNHAS) in total ECHO budget

Quantitative Analysis of ECHO budget

% of projects using different approaches (preparedness/prevention versus response)

Share of budget line reserved for response (23 02 01) vs preparedness/prevention (23 02 02): out of total intervention related budget (23 02 chapter)

Quantitative Analysis of ECHO budget


October, 2017 15

Annex 2 Full list of interviewed stakeholders

Table A2.1 Scoping interviews

Category Organisation

ECHO staff HQ ECHO - A3

ECHO staff HQ ECHO - B1



ECHO staff HQ ECHO - C1




ECHO staff HQ ECHO - D1

ECHO staff HQ ECHO - D4

Other EU entities DEVCO - C,G,H-4

Other EU entities DEVCO - C1

Other EU entities NEAR - A5

Other EU entities NEAR - A3

Table A2.2 Sectoral and thematic interviews




ECHO staff HQ ECHO – C1

ECHO staff field ECHO field








ECHO framework partners FAO

ECHO framework partners Solidarites International Association

ECHO framework partners UNHCR

ECHO framework partners DRC

ECHO framework partners DRC


October, 2017 16

ECHO framework partners IOM

ECHO framework partners IOM

ECHO framework partners UNICEF



ECHO framework partners CIRC

Global forums and clusters Food Security Cluster

Global forums and clusters Global WASH Cluster

Global forums and clusters Global WASH Cluster

Global forums and clusters Global Shelter Cluster - IFRC

Global forums and clusters Global Shelter Cluster - IFRC

Global forums and clusters Global Shelter Cluster - REACH

Global forums and clusters Global protection cluster

Global forums and clusters Global protection cluster

Table A2.3 Third round of interviews


ECHO HQ ECHO – D1

ECHO HQ ECHO

ECHO HQ ECHO

ECHO HQ ECHO – D

ECHO HQ ECHO – B1

ECHO HQ ECHO – C2

ECHO HQ ECHO – C3

ECHO HQ ECHO – D4

ECHO HQ ECHO

ECHO HQ ECHO – D1

ECHO HQ ECHO – D3

Other EU DEVCO

Other EU DEVCO

Other EU DEVCO

Framework partners World Vision International

Framework partners CARE



Framework partners WFP

Framework partners ICRC


October, 2017 17

Framework partners MSF

Framework partners UN OCHA


Other NGO Voice

EU Member States Austria

EU Member States Estonia

EU Member States Sweden, Ministry for Foreign Affairs

EU Member States Swedish International Development Cooperation

Agency

EU Member States Belgium, FPS Foreign Affairs, International Trade and

Development Cooperation

EU Member States UK, DFID & Foreign & Commonwealth Office

Academics/Think tanks ALNAP

Academics/Think tanks Falk School of Sustainability and Environment,

Chatham University

Table A2.4 Participants to the validation workshop


DG ECHO ECHO – A3

DG ECHO ECHO – D1

DG ECHO ECHO – C1

Other Voice

Other Voice


Framework partners World Food Programme

Framework partners World Food Programme

Framework partners UNHCR



Framework partners Save the Children

Framework partners Action Contre la Faim

Framework partners Norwegian Refugee Council

Framework partners Danish Refugee Council

Academics/Think tanks ALNAP

EU Member States UK Department for International Development

EU Member States Belgian Ministry of Foreign Affairs


October, 2017 18

Annex 3 Methodological notes on the sources and use of the

quantitative data

Three main sources of quantitative data (EVA data on ECHO funding, FTS data on all

donors’ funding and appeals data) have been used in this report– essentially in the

relevance section. Important notes on these sources are detailed below. Three main

sources of quantitative data (EVA data on ECHO funding, FTS data on all donors’

funding and appeals data) have been used in this report– essentially in the relevance

section. Important notes on these sources are detailed below.

Before turning to that though, the box below clarifies the different angles from which

one looked at EU Humanitarian aid.

EU Humanitarian aid from different angles

The evaluation team used different sources of data on EU humanitarian aid funding to

serve different purposes and analysis. A clear reference to the exact source is

provided in all cases.

The main source used in the evaluation is the EVA database extracted by DG ECHO

from HOPE. It shows contracted amounts (i.e. amounts of contracts which have been

signed by the framework partners). The main reason to use EVA is that it allows to do

a granular analysis of the data, e.g. by sector.

In some cases, the evaluation presents committed amounts as per DG ECHO

budgetary data (i.e. amounts available to DG ECHO as per Budget Execution

summaries or Excel files directly provided by DG ECHO). The main advantage of this

source of information is that it allows for an analysis of the source of funding and for

making a distinction between different lines of the EU Budget (e.g. initial budget,

reinforcements from the EAR, Heading 4 redeployments, reinforcements from the EDF)

from other sources (e.g. external assigned revenues).

The third source of information used in the evaluation concerns the allocated

amounts as per the HIPs documents (i.e. amounts allocated to specific HIPs). This

has been used to analyse DG ECHO funding by objective (see e.g. Error! Reference

source not found.) or to have a first indication of 2017 trend as 2017 data on

contracted amounts is not yet available.

The evaluation team also used data extracted from the FTS database to put the EU

humanitarian aid funding in perspectives of the global humanitarian aid funding. The

limitation associated with the use of the FTS data are detailed below.

In general, committed amounts (if all sources are taken into account) are equal or

superior to allocated amounts, themselves equal or superior to contracted amounts.

Provided that sufficient time lag has elapsed for all amounts to be contracted and

provided that the budget is implemented in full (all resources are contracted), the

three indicators will have similar values.

A3.1 EVA data

A3.1.1 Source of EVA data

EVA dataset is managed by ECHO and was first extracted on April 19, 2017. A second

extraction was made on December 18, 2017 to make sure 2016 data was as up-to-

date as possible given the timeframe of this study (2016 resources continue to be

contracted in N+1). All graphs presenting EVA data only have been updated with the

most recent extraction. Other more sophisticated analyses which have been made on

EVA data as well as other sources of data (e.g. FTS data or IAF assessments) have not


October, 2017 19

been updated due to the time and resources it would have involved. Below each

figure, the data of extraction is clearly indicated.

A3.1.2 Content of EVA data

The EVA dataset contains ECHO’s contracted amounts per action and per result.

Because each action result is allocated to one sector, this breakdown by result allows

to do an analysis by sector without double-counting. The total contracted amount per

action simply is the sum of the contracted amount per action result. The dataset

contains other key variables as presented below:

Year. Under EVA, there are two options to present the year: The contract year refers to the year the project contract was signed and the budget was

transferred to the partner. The financial year refers to the budget year, in other

words, the year of the HIP decisions of DG ECHO. The financial year is the year

which is used as a basis for the analysis presented in this evaluation.

Beneficiary country. The beneficiary country generally refers to the country where the project is executed. In the case of multi-country projects, the single

country which appears in the dataset as the beneficiary country is the first

which appears in HOPE contract line and should correspond to the main

beneficiary country. There is no way from existing ECHO information systems to

breakdown the country allocation of these multi-country projects. This can

slightly bias country allocations (see Table A3.1for some key statistics on

multicountry projects)

Table A3.1 Statistics on multi-country projects

2012 2013 2014 2015 2016 Grand Total

Value share 20% 29% 23% 7% 3% 14%

Number of projects 67 66 84 53 36 306

Source: ICF, 2017. Analysis of ECHO EVA database, data extracted on 19 April 2017.

Sectors. There are 15 sectors, including since 2016 the multi-purpose cash transfer (for cash transfer designed to cover needs in several sectors)4.

- Child protection - Coordination - Disaster Risk Reduction / Disaster Preparedness - Education in emergencies - Food security and livelihoods - Gender - Health - Mine actions - Multi-purpose cash transfer - Nutrition - Protection - Shelter and settlements - Support to operations - WASH

4 See ECHO, 2017. Single Form Guidelines. Available at: http://dgecho-partners-

helpdesk.eu/_media/single_form_guidelines_until_21_june_2016.pdf


October, 2017 20

EVA contains information on subsectors as well. However one result can cover

several sub-sectors, so that analysis of funding per sub-sector is not possible

based on EVA without double counting5.

Number of beneficiaries: EVA contains estimated total number of direct beneficiaries targeted by the action and, where available, estimated total

number of direct beneficiaries targeted by the result, knowing that the sum of

estimated number of individuals of all results can exceed total number of direct

beneficiaries at the action level if the beneficiaries benefit from more than one

result of the project.

Partner and Partner Category. These two indicators specify the short name of ECHO’s partner and its category (International Organisation, NGO, UN agency

or other (=GIZ))

Region groupings. The countries have been grouped into regions based on HIP groupings.

A3.2 UN OCHA FTS funding data

A3.2.1 Source of FTS funding data

The Financial Tracking System is managed by UN OCHA6. The FTS website aims at

tracking humanitarian funding flows, at different levels, from the donors to the

implementing partners. It also tracks bilateral contributions and private funding. The

FTS dataset is fed by the reports on funding flows and pledges (regarding

humanitarian funding flows) and plans (regarding the appeals and humanitarian

response plans), provided voluntarily by donors and recipient organizations. FTS is

constantly updated. The fist dataset was extracted on 17th of October and the latest

dataset on 30th of October 2017.

A3.2.2 Content of FTS data

Thanks to FTS’s flow model, the data can be extracted from different perspective

(from the perspective of the organization, the emergency, the location, the project,

the plan or the sector). For the scope of this evaluation we extracted data from the

perspective of the organization (donor) or the location (recipient country) for the years

2012 to 2016.

Year. Year refers to the usage year, i.e. calendar year the resources were intended to be used, and not when resources were allocated. FTS relies on flow

model and thus usage year can be defined as from the perspective of the

source of the flow or destination of the flow. We display destination usage year

as it is the default option on FTS.

Beneficiary country. The reported “destination location” refers to the execution country of the action. FTS categories include one “Not specified” category and

one “Multiple locations (shared)” category. Taken together these two categories

represent 9% of total funding for the year 2016.

Sectors. FTS categorizes the actions into 20 different sectors.

- Camp Coordination / Management - Child Protection - Coordination and support services

5 For some sectors, e.g. protection, DG ECHO provided the team with its own weighted estimates of funding by sub-

sector, correcting for double-counting. 6 Data available on the FTS website: https://fts.unocha.org

https://fts.unocha.org/


October, 2017 21

- Early Recovery - Education - Emergency Shelter and NFI - Emergency Telecommunications - Food Security - Gender Based Violence - Health - Logistics - Mine Action - Protection - Water Sanitation Hygiene - Nutrition - Multi-sector (Multi-sector' refers to projects and activities with no one

dominant sector and often applies to UNHCR assistance for refugees)

- Multiple Sectors (shared) - All other funding - Other - Not specified

Number of beneficiaries: This is not available from FTS.

Donor: FTS contains data on all donors. To calculate ECHO’s funding share, data on all donors combined and ECHO only was extracted.

A3.2.3 Comparison of ECHO’s EVA data and UN OCHA FTS data

The purpose of this section is (i) to illustrate the differences across the two databases

and explain the origin of these and (ii) highlight what the strategy has been as part of

this study to present consistent data.

A3.2.3.1 Main differences

A naïve view would be that, since information on ECHO funding is directly transmitted

from EDRIS to FTS, differences would be minimal. In reality, when OCHA FTS receives

the data, they recode it to make it fit their classification system. The differences in

definitions highlighted in 0 and A3.2.2 result in significant discrepancies across

databases.

Differences across years

Total funding by year according to the two data sources do not fully match. Different

factors can explain this discrepancy.

Table A3.2 Comparison of ECHO’s funding to humanitarian aid from 2012 to 2016

according to FTS and EVA

2012 2013 2014 2015 2016

FTS Total ECHO funding € 1,235,706,840 € 1,314,304,793 € 1,315,362,107 € 1,623,402,332 € 1,850,089,069

EVA Total ECHO funding (financial year)

€ 1.246.934.217 € 1.247.707.070 € 1.157.282.841 € 1.553.777.523 € 1.913.940.114

EVA Total ECHO funding (contract year)

€ 1.216.455.572 € 1.259.848.373 € 1.232.052.341 € 1.327.894.096 € 2.034.823.170


October, 2017 22

Source: ICF, 2017. Analysis of UN OCHA FTS database, data extracted on 17 October 2017.

And, ICF, 2017. Analysis of ECHO EVA database, data extracted on 19 April 2017.

Table A3.3 FTS data converted to EUR using the following Xrates:

2012 2013 2014 2015 2016

Historical annual Xrate: USD/EUR 0.77833812 0.75294512 0.75272819 0.90129642 0.90342143

Years. Where in FTS the years correspond to the usage year, i.e. calendar year the resources were intended to be used, and not when resources were

allocated. In EVA the years correspond to the financial year (year of the HIP

decision) or the contract year (the year the project contract was signed and the

budget was transferred to the partner).

Scope of funding covered in both datasets.

- Larger scope in EVA: In EVA, the externally assigned revenues7 are included. While in FTS, external assigned revenues are separately counted.

In addition, EVA includes ECHO’s contribution to the Facility for Refugees in

Turkey while the Facility for Refugees in Turkey is excluded in FTS because

ECHO’s contribution can’t be dissociated from that of other donors.

- Larger scope in FTS: The provision of emergency support within the EU has been excluded from the EVA dataset as it was not within the scope of this

evaluation. However, this funding is not similarly subtracted in the FTS

database.

Use of different exchange rates for converting funding data. ECHO funding data is converted to USD in FTS and has been converted back to EUR here using

historical annual exchange rate.

Differences in top 10 countries

Looking at more disaggregated data, e.g. top 10 countries, differences are even larger

(see Table A3.4), especially when looking at single years (Table A3.5Table A3.5). In

addition to the above factors, other factors come into play.

Multicountry projects. In FTS, multicountry projects are shown under a specific “Not specified” category and one “Multiple locations (shared)” category.

Table A3.4 Comparison of the top 10 funded countries from 2012 to 2016

EVA Total 2012-2016 - financial year

EVA Total 2012-2016 - contract year

FTS Total 2012-2016

Syrian Arab Republic € 814.829.914 € 809.329.914 € 747.891.790

Turkey € 606.300.000 € 540.650.000

South Sudan Republic € 533.260.000 € 533.260.000 € 683.029.622

Iraq € 333.055.000 € 327.305.000 € 371.360.178

Niger € 277.744.981 € 273.544.981

Somalia € 255.450.000 € 256.150.000 € 328.937.932

Democratic Republic Of Congo

€ 241.403.672 € 241.403.672 € 393.033.887

Lebanon € 240.420.000 € 240.420.000 € 399.948.666

Jordan € 236.206.758 € 236.206.758 € 340.699.953

7 external assigned revenues are collected from and assigned by various donors to a specific programme or

action


October, 2017 23

Yemen € 227.300.000 € 230.005.202

Not Specified € 475.202.281

Ethiopia € 402.709.535

Chad € 310.918.985

Source: ICF, 2017. Analysis of UN OCHA FTS database, data extracted on 17 October 2017. And, ICF, 2017. Analysis of ECHO EVA database, data extracted on 19 April 2017.

Table A3.5 Comparison of the top 10 funded countries for 2016

EVA 2016 - financial year

EVA 2016 - contract year FTS 2016

Turkey € 543,650,000 € 479,500,000 € 57,070,027

Syrian Arab Republic € 162,949,965 € 160,549,965 € 203,410,073

Iraq € 157,850,000 € 153,100,000 € 141,066,051

South Sudan Republic € 122,700,000 € 142,700,000 € 142,928,477

Lebanon € 84,550,000 € 85,800,000 € 67,233,573

Jordan € 69,400,000 € 69,400,000 € 60,711,300

Yemen € 67,300,000 € 60,800,000 € 163,498,846

Nigeria € 52,800,000 € 55,779,000 € 56,644,466

Niger € 47,869,103

Democratic Republic Of Congo € 38,850,000

Yemen

€ 53,100,000

Somalia

€ 48,550,000

Greece

€ 198,882,216

Not Specified

€ 75,123,734

Source: ICF, 2017. Analysis of UN OCHA FTS database, data extracted on 17 October 2017.

And, ICF, 2017. Analysis of ECHO EVA database, data extracted on 19 April 2017.

Sectors. Analysis by sector is the least comparable type of analysis, as shown in Table A3.6 below.

Table A3.6 Differences in top 10 sectors

EVA 2012- 2016 - contract year FTS 2012-2016

Sector Total Amount (EUR)

Sector Total Amount (EUR)

Food security and livelihoods € 2,160,815,944

(blank) € 2,169,165,271

Health € 888,202,139 Health € 1,155,165,775

Shelter and settlements € 732,928,291

Multi-sector € 1,116,268,072

Nutrition € 728,417,699 Food Security € 1,109,137,283

WASH € 708,317,188 Coordination and

support services € 634,720,263

Protection € 649,154,471 Water Sanitation

Hygiene € 388,113,978

Multi-purpose cash transfer € 339,296,832

Emergency Shelter and NFI € 348,879,717

Disaster Risk Reduction / Disaster € 289,523,594

Protection € 210,024,099


October, 2017 24

Preparedness

Coordination € 224,617,577 Early Recovery € 155,506,179

Support to operations € 209,028,081 Education € 39,446,124

Education in emergencies € 76,101,991

Mine Action € 4,349,642

Child protection € 52,645,834 Nutrition € 2,967,927

Mine actions € 10,163,960 Total € 7,333,744,330

UNKNOWN € 1,501,304

Gender € 358,646

Total € 7,071,073,552

Source: ICF, 2017. Analysis of UN OCHA FTS database, data extracted on 17 October 2017. And, ICF, 2017. Analysis of ECHO EVA database, data extracted on 19 April 2017.

Additional factors explaining the differences for sectors are that (I) EVA allows to

breakdown the funding of a single project under different sectors, which is not the

case of FTS (all funding allocated to a single, main sector). And, (II) unlike EVA, FTS

data includes a large multi-sector and not specified category – to which some

protection funding certainly goes to - but this will not be included into FTS protection

data.

Strategy used

EVA contains more granular information to analyse ECHO funding. Wherever ECHO

funding only was necessary for the analysis, the preferred source of information was

EVA.

However, to put ECHO data in perspective with what other donors are doing, there is

no other option but to use FTS data. Since definitions (e.g. of years and sectors) are

different across data sources, where ECHO funding needs to be put in perspective with

what other donors are doing at the global level, there is no other option but to use

OCHA FTS for both ECHO funding and other donors’ funding and stick to FTS

definitions (e.g. of years and sectors) – in order to compare like with like.

A3.3 Appeals data

Appeals data from FTS was used to put ECHO funding into perspective with global

needs in monetary terms.

A3.3.1 Introduction to the appeals process8

Humanitarian Response Plans (HRPs), Flash Appeals and Regional Response Plans

(RRPs) reflects the needs of affected population during humanitarian crises or

emergencies. The HRPs and Flash Appeals are elaborated by the Humanitarian

Country Teams (HCT) which brings together UN agencies, NGO’s and other actors. The

Regional Response Plan (RRPs) are regional refugee response plans, coordinated by

UNHCR and its partners. For all types of appeals, the teams elaborate a plan that

includes the country or context strategy and the objectives; it details how the strategy

will be implemented and how much funding is required. The planning process usually

takes place once a year, between September and December and cover a one-year

horizon. However, plans can be issued at any time in the year. The online

planning/project system allows for the participating agencies to upload their projects

and funding requests and to update them during the course of the plan. By the end of

each calendar year, the Global Humanitarian Overview publishes known and

anticipated requirements for the year to come.

8 See also: https://fts.unocha.org/content/guide-funding-response-plans-and-appeals


October, 2017 25

A3.3.2 Data on the Appeals and Response Plans

FTS releases the data on overall finalised requirements for the year in late January.

Data based on the list of appeals made between 2012 and 2016, was retrieved from

the FTS archived website9. This list includes HRPs, Flash Appeals and RRPs. The final

list is slightly modified compared to the initially extracted list. Additional data needed

to be retrieved while overlooking some appeals as detailed below.

Additions compared to FTS list

Country breakdowns for RRPs. FTS does not break down the funding needs of the Regional Response Plans per country, but provides only the total for the

region. In order to study the funding needs per country within the region,

country breakdowns were retrieved based on own desk research (breakdowns

are available online in RRP documentation). Please note the sum of the country

breakdowns as per Regional Response Plans available online did not always

match total regional funding requirements provided by FTS. In these cases,

information from RRP prevailed since the detailed country breakdowns were

needed for the analysis.

List of the RRPs for which country breakdowns have been retrieved through

desk research:

- Burundi Regional Refugee Response Plan (DR Congo, Rwanda, Tanzania and Uganda), 2015 and 2016

- Central African Republic Regional Refugee Response Plan (Cameroon, Chad, Republic of Congo, Democratic Republic of the Congo), 2015 and 2016

- Ebola Virus Outbreak - Overview of Needs and Requirements (inter-agency plan for Guinea, Liberia, Sierra Leone, Region) - October 2014 - June 2015

- Nigeria regional refugee response plan, 2015 and 2016 - South Sudan regional refugee response plan 2014 to 2016 - Revised Syria Regional Response Plan 2012 - Syria regional refugee and resilience plan (3RP), 2013 to 2016 - Yemen Regional Refugee and Migrant Response Plan, 2015 and 2016

Addition of missing RRPs. Some HRP data were absent from the FTS dataset exports and were also added manually through desk research:

- Korea DPR Overview of Needs and Assistance 2012 - Myanmar - Kachin Response Plan, 2012 and 2013 - DPR Korea Needs and Priorities 2015 and 2016 - Zimbabwe 2013

Omissions compared to FTS list if no country breakdown available.

The Central Emergency Response Fund (CERF), is a central fund managed by the UN

for global emergency response to sudden crisis10. Although this fund processes its

appeals via the UN system, and is referred in FTS data extractions, we did not take

these into account because they have no ex-ante breakdown of needs per country.

The same goes for appeals labelled as "West Africa Regional Contribution 2015" and

"Miscellaneous 2015 (GMT and HND)" for which no country breakdown could be found

and which we are not taken into account in the study.

Figure A3.1 below gives an overview of the types of appeals that were made for each

country, per year.

9 https://ftsarchive.unocha.org/pageloader.aspx?page=searchProject-

ListSelectAppeals&CQ=cq021117194517RM9XLRdII 10

http://www.unocha.org/cerf/about-us/cerf-at-a-glance

https://ftsarchive.unocha.org/pageloader.aspx?page=searchProject-ListSelectAppeals&CQ=cq021117194517RM9XLRdIIhttps://ftsarchive.unocha.org/pageloader.aspx?page=searchProject-ListSelectAppeals&CQ=cq021117194517RM9XLRdII


October, 2017 26

Figure A3.1 List of countries and respective appeals from 2012 to 2016

Source: ICF, 2017. Analysis of UN OCHA FTS database, data extracted on 29 September

2017.


October, 2017 27

A3.3.3 Limitation to the FTS response plan data extracted

Incomplete picture of needs.

Not all agencies and humanitarian actors participate in UN coordinated appeals

process. It is the case for ICRC and MSF who work independently. Hence, their funding

requirements are not included in the HRPs, Flash Appeals or RRPs and are not counted

as needs.

Lack of comparability of appeals data.

Scale of appeals requirements vary for various reasons (“valid” and “non valid ones”):

Valid reasons: different scales of need, different cost of delivering assistance per se.

Non-valid reasons (undermining comparability): data being inflated voluntarily (appeals data stemming out of politicized processes)11, differences in scope

with some appeals covering development components and others excluding it.

Scope sometimes includes development component. For example, the Syria 3RP

includes a “‘refugee component’ and a ‘resilience component’, with the latter

accounting for between a quarter and a third of the total amount requested.”12

The fact that differences in appeals requirements is not fully driven by differences in needs is well illustrated in the average cost of response per

person which varies widely based on data from UN OCHA FTS and UN-

coordinated appeals: US$801 per person for South Sudan, US$36 per person

for Nigeria13.

Possibility of double counting between RRPs and HRPs for countries covered by both types of appeals. For countries covered by both RRPs and HRPs during a

same year, it cannot be excluded that the funding requirements of the RRP are

not overlapping with the Humanitarian Response Plans: the same requirements

might be counted twice. For instance, Central African Republic was covered in

2014 by both an HRP and a RRP (Ebola Virus Outbreak, regional response

plan), however it cannot be excluded that the HRP did not integrate any funding

needs for projects related to the Ebola Virus Outbreak. The countries potentially

subject to this double counting are listed below:

- Burkina Faso, 2014 - Cameroon, 2014, 2015 and 2016 - Central African Republic, 2014 and 2015 - Democratic Republic of the Congo, 2014, 2015 and 2016 - Gambia, 2014 - Mali, 2014 - Mauritania, 2014 - Senegal, 2014 - Yemen, 2015 and 2015 - Chad, 2015 - Djibouti, 2015 and 2016 - Niger, 2015 and 2016

11

Olin, E., & von Schreeb, J. , 2014. Funding Based on Needs? A Study on the Use of Needs Assessment Data by a Major Humanitarian Health Assistance Donor in its Decisions to Allocate Funds. PLoS Currents, 6, ecurrents.dis.d05f908b179343c8b4705cf44c15dbe9. http://doi.org/10.1371/currents.dis.d05f908b179343c8b4705cf44c15dbe9 12

Development International, 2015. The 2015 UN-coordinated appeals: An ambitious plan to meet growing humanitarian needs. Available at: http://devinit.org/wp-content/uploads/2015/04/The-2015-UN-coordinated-appeals-F.pdf [Consulted on 9 November 2017] 13

Development International, 2015. Op. Cit.


October, 2017 28

- Sudan, 2015 and 2016 - Somalia, 2016

A3.3.4 Appeals coverage and funding gap

FTS tracks funding flows in relation to the projects listed in the HRPs, Flash appeals

and RRPs. Each funding flow is thus labelled as either “Inside response plans/appeals”

or “Outside response plans/appeals”. Only funding flows labelled as “Inside response

plans/appeals” do count when FTS present data on needs coverage to date and

funding gap still needed for each appeal.

In contrary to this approach, we calculated, under the relevance section, funding gap

as the difference between appeals requirements and any type of funding flow

received. One reason for this is that even if recorded outside the appeals, the funding

flows targeting the same crises still serve to alleviate the same needs. In addition, DG

ECHO’s position as a contributor to appeals is generally low (funding not recorded

against the appeal or ECHO partners not participating in the appeal).


October, 2017 29

Annex 4 Profile of respondents to the three surveys

A4.1 Respondents to the ECHO field survey

Geographical distribution: responses came from 129 members based in 34

different DG ECHO field offices. Significant share of respondents were based in Kenya

(15 respondents representing 12%), Jordan (11 or 9%), Turkey (9 or 7%) and

Senegal (8 or 6%), reflecting the presence of regional office in these countries.

Table A4.1 Number of respondents per country (n=129)

Source: ICF, 2017. Survey: ECHO framework partners. Completed from 12.06.2017 to

14.07.2017

Level of experience: The respondents to the survey have different level of

experience with ECHO field network. The largest share of respondents (43%) have

been with the network for more than five years. Only 12% or 15 respondents have

been with the field network for a limited time of period (less than one year).

Table A4.2 Number of years spent as part of ECHO field network (n=129)


14.07.2017

Role in the field: The respondents represent a wide range of professional roles within

the network. The majority of respondents are Technical Assistants (24%), Programme

Assistants (17%), Heads of Country Officers (15%), Programme Officers (13%) or

Sectorial Experts (10%).


October, 2017 30

Table A4.3 Specific role within ECHO field network (n=127)


14.07.2017

Type of activities undertaken: The respondents were involved in a variety of

functions in their role, demonstrating a confidence in responding to the number of

questions in the survey. The three most visible functions of respondents were:

monitoring of projects (77% of respondents were to a very large or large extent

involved in this function), monitoring of humanitarian situation on the ground (77%)

and providing technical advice to HQ on monitoring projects (69%).

Table A4.4 Functions where respondent is most often involved (n=varies from 126

to 129)


14.07.2017

A4.2 Respondents to the framework partner survey

Among the 361 respondents to the survey, 89% are from framework partners and

11% are from UN agencies.


October, 2017 31

Table A4.5 Share of framework partners (n=361)

Source: ICF, 2017. Survey: ECHO framework partners. Completed from 07.06.2017 to 04.08.2017

The most represented framework partners in the survey are Oxfam (6% of

respondents), CARE (6%), FAO (4%) and Save the Children (4%). In total, 88

different framework partners organisation are represented in the survey.

The 88 surveyed framework partners are based in 80 different countries. A third

(36%) of the respondents are based in Europe, mainly in Germany, France, Spain, UK,

Belgium, Netherlands, and Switzerland. The other 64% of the respondents'

organisations are based in the field; Pakistan (5% of respondents), Jordan (3%),

Turkey (3%) Nepal (3%).

The majority of the framework partners were involved in 2 to 5 ECHO funded projects

between 2012 and 2016. Roughly half of the partners had closed all their actions by

the time of the survey.

Table A4.6 In how many ECHO funded humanitarian aid actions were you involved

between 2012 and 2016? (n=361)


04.08.2017

Table A4.7 Have the actions you have been involved in been finalised? (n=361)


October, 2017 32


04.08.2017

On average, the partners work in 3 different sectors. Food Security and Livelihoods

(51%) and WASH (47%) are the main sectors in which framework partners are active.

Disaster Risk Reduction and Preparedness is supported by 43% of the respondents'

organisations. And the Health sector is supported by 37% of the respondents'

organisations.

Table A4.8 Sector of activity of the framework partners (n=361)


04.08.2017. Note: This is a multiple answer question, hence the total does not equal

100%.

ECHO framework partners are active in all regions worldwide, however they are not

spread evenly. Asia and Oceania is the most covered region where almost a third of

the surveyed ECHO framework partners are active in various sectors. The Middle East

is covered by 28% of the surveyed ECHO framework partners. The Great Lakes region

and Central Africa count only 11% of framework partners each.

Table A4.9 Region of activity of framework partners (n=361)


04.08.2017. Note: This is a multiple answer question, hence the total does not equal 100%.


October, 2017 33

A4.3 Respondents to the local partner survey

103 respondents took part to the local partner's survey, representing 74 different local

partners. CARE (8% of the respondents), Islamic Relief (8%) and the Red Cross (8%)

are the most represented local partners in this survey.

The partners are based in 49 different countries, Pakistan (12%), Ethiopia (8%) Nepal

(7%) and Sudan (7%) were the countries were most respondents were located.

Table A4.10 Share of respondents per country (n=103)

Source: ICF, 2017. Survey: ECHO local implementing partners. Completed from 07.07.2017 to 11.08.2017.

The majority of the local partners were involved in 2 to 5 ECHO funded actions

between 2012 and 2016. And for most of the partners all the actions were closed by

the time of the survey.

Table A4.11 In how many ECHO funded humanitarian aid actions were you involved

between 2012 and 2016? (n=103)



October, 2017 34

Table A4.12 Have the actions you have been involved in been finalised? (n=103)


WASH (40% of active local partners) and Food security and Livelihoods (40% of active

local partners) are the main sectors addressed by local partners. 35% of the local

partners are active in the Disaster Risk Reduction and Preparedness sector.

Table A4.13 Sector of activity of the local partners (n=103)

Source: ICF, 2017. Survey: ECHO local implementing partners. Completed from 07.07.2017 to

11.08.2017.


October, 2017 35

Annex 5 Meta-evaluation

A5.1 Introduction

Table A5.1 Evaluations reviewed as part of the meta-evaluation

Type of

evaluation

Publication

month-year

Name of the evaluation

Thematic Sept -17 Evaluation of the European Commission's interventions

in the Humanitarian Health sector, 2014-2016

Thematic forthcoming EU Approach to building resilience to withstand food

crises in African Drylands (Horn and Sahel), 2007-2015

– joint DEVCO and ECHO evaluation

Thematic Oct-16 Evaluation of ECHO’s actions in the Field of Protection

and Education of Children in Emergency and Crisis

Situations, 2008-2015

Thematic Jan-16 Evaluation of the use of different Transfer Modalities in

ECHO Humanitarian Aid actions (2011 – 2014)

Thematic Mar-15 Evaluation of DG ECHO’s Approach to Communication

under the Humanitarian Implementation Plans (HIP

between 2010 and 2013)

Thematic Oct-14 Mid-term evaluation of Enhanced Response Capacity

Funding

Thematic Sep-14 Joint Evaluation of Drought Risk Reduction in the Horn

of Africa and DIPECHO Central Asia and South Caucasus

(2009 ‐ 2013)

Thematic Jun-14 Evaluation of the implementation of the European

Consensus on Humanitarian Aid, 2008-2012

Thematic Aug-13 Evaluation of the European Commission’s Humanitarian

Action in the Shelter Sector

Thematic Jul-13 Food security and nutrition: Evaluation of European

Commission integrated approach of food security and

nutrition in humanitarian context (2009-2012)

Thematic Mar-13 Evaluation of DG ECHO's Fleet Management

Thematic Jan-13 Working directly with local NGOs: Evaluation of the

potential effectiveness and efficiency gains of working

directly with local NGOs in the humanitarian

interventions of the Commission

Thematic Jul-12 Evaluation and review of humanitarian access strategies

in DG ECHO funded interventions

Thematic Jun-12 Need Analysis, Review and Design of DG ECHO’s

Training in Disaster Risk Reduction and Climate Change

Adaptation

Thematic Jun-12 Evaluation of DG ECHO’s Legal Framework for Funding

of Humanitarian Actions (FPA 2008)

Thematic May-12 Review of existing practices to ensure participation of

Disaster-affected communities in Humanitarian aid


October, 2017 36

Type of

evaluation

Publication

month-year

Name of the evaluation

operations

Thematic Mar-12 Evaluation and review of DG ECHO financed

livelihood interventions in humanitarian crises

Geographic Oct-17 Evaluation of the ECHO interventions in India and

Nepal, 2013-2017

Geographic Dec-16 Evaluation of DG ECHO’s Response to the Humanitarian

Crises in Sudan and South Sudan (2011-2015)

Geographic Oct-16 Evaluation of DG ECHO's Actions on Building Resilience

in the LAC Region

Geographic Jun-16 Evaluation of the ECHO intervention in Pakistan

Geographic Jun-16 Evaluation of the ECHO response to the Syrian Crisis

(2012 – 2014)

Geographic Feb-16 Evaluation of ECHO’s interventions in the Sahel (2010-

2014)

Geographic Sep-15 Evaluation of the DG ECHO Actions in Coastal West

Africa 2008 – 2014

Geographic Dec-13 Evaluation of the European Commission's Humanitarian

and Disaster Risk Reduction Activities (DIPECHO) in

Indonesia

Geographic Nov-12 Evaluation of European Commission’s Humanitarian

Activities in Colombia

Geographic Mar-12 Evaluation of DG ECHO's interventions in the occupied

Palestinian territory and Lebanon

Thematic Jun-06 Evaluation of the European Commission’s Directorate

General for Humanitarian Aid (DG ECHO), 2000-2005

A5.2 Findings of the meta-analysis by evaluation questions

A5.2.1 Relevance

EQ1: To what extent do ECHO budget allocations consider the needs, actions

of other donors and the EU objectives on humanitarian aid?

JC1.1 ECHO’s budget allocations are based on needs, actions of other donors and ECHO objectives

I1.1.1 Evidence that ECHO’s budget allocations are based on needs

Literature and document review e.g. outputs of INFORM. FCA, IAF

Review of ECHO’s approach to needs assessment

Interviews: ECHO HQ, ECHO field, other key donors to understand their approaches to needs assessment

Surveys: ECHO field, FPA partners

Expert opinion

OPC


October, 2017 37

I1.1.2 Evidence that ECHO’s budget allocations

considered the actions of other donors

Document review

Interviews: ECHO HQ, ECHO field


I1.1.3 Proportion of ECHO humanitarian aid funding allocated to each sector and

compared to needs

Literature review

I1.1.4 Evidence that ECHO’s budget size was commensurate to ECHO objectives and expected results

Literature review

Interviews: ECHO HQ, ECHO field


I1.1.5 Evidence of changes in budget allocation over the evaluation period (across sectors / geographies)

Document review (e.g. review of HIPs) and HOPE data analysis

Extent to which the evaluation question has been addressed in the evaluations

The 27 evaluations reviewed were somewhat inconsistent in presenting information on the extent to which ECHO’s allocations considered the needs, actions of other donors and EU objectives on humanitarian aid. While comprehensive information was provided by some reports, others were less informative in this regard. This made comparison across sectors and geographies challenging. The review showed that not all the indicators identified in ICF’s

analytical framework were fully addressed by the 27 evaluations under observation.

Evidence that ECHO’s budget allocations are based on needs: The inclusion of information on the use of needs assessment (both at ECHO and project level) in the

evaluation reviewed was very inconsistent (i.e. not all the evaluations reviewed referred to the needs assessment aspect). Moreover, none of the evaluations looked at the DG ECHO needs assessment process for the programming of HIPs;

Evidence that ECHO’s budget allocations considered the actions of other donors – very limited information identified in the evaluations reviewed;

Proportion of ECHO humanitarian aid funding allocated to each sector and compared to needs – very limited information identified in the evaluations reviewed;

Evidence that ECHO’s budget size was commensurate to ECHO objectives and expected results: insufficient information identified to be able to provide a sound analysis;

Evidence of changes in budget allocation over the evaluation period (across sectors / geographies) – addressed by the evaluations reviewed to a moderate extent, analysis provided in section 0 below.

Extent to which ECHO’s budget allocations were based on needs

The evaluations reviewed suggested that, overall, ECHO’s budget allocations were

based on needs at the level of the actions funded by ECHO, i.e. the needs as identified

through assessments carried out at field/project level. DG ECHO was highly supportive

of assessments and consistently encouraged partners and other local actors to conduct

their own analyses. However, as further elaborated below, at the central programming

level, as part of the HIPs and other strategic decision-making by ECHO, it was found

that the budget allocations did not always fully address the needs, showing insufficient

coverage of themes and, in some cases, a disconnect with specific local needs. Not

enough information was however available to be able to draw firm conclusions on the

extent to which this “gap” might have contributed to reducing the overall relevance of

ECHO’s budget allocations to the needs of stakeholders on the ground.


October, 2017 38

At the level of ECHO funded actions, needs assessments were regularly carried out by

implementing partners in the context of the interventions developed on the ground

(field/project level). The review of evaluations, however, showed that the quality of

such assessments varied substantially, even within the same country. In Indonesia,

for example, “all partners conducted area-specific risk assessments as a basis for

designing their projects, some more comprehensively than others”.

Differences in quality were found also across themes/sectors. For example, within the

food security and nutrition sectors, some partners have developed sophisticated ways

of analysing causes and assessing nutrition problems. “Overall, partners provide good

information on food availability and food access but insufficient information on food

intake and food utilisation, despite the relevance of these elements. Partners’ analyses

of causes of undernutrition were sometimes cursory, with the implicit assumption that

food access in itself will ensure adequate nutrition”. The evaluation indicated that, in

particular in contexts where nutrition is clearly in focus, there is a need to look more

deeply into the causes of malnutrition, to shed light on types of interventions that

might be most effective.

Similarly, within the education in emergencies sector, needs assessments varied in the

level of detail and methods used (for instance, in use of participatory approaches, the

involvement of children, etc.). The specific consideration of age and gender-based

needs (and adapting activities accordingly) also varied according to partners.

In the health sector, differences in the quality of assessments conducted at

field/project level were mainly linked to the partners’ capacity to conduct proper needs

analysis. The thematic evaluation noted that while some of ECHO’s partners had

strong standards and tools to conduct needs assessments and invested in abiding to

quality standards others were altogether absent from the field and conducted such

assessments remotely or purely based on secondary data. The quality of needs

assessment conducted by partners was also influenced by factors inherent to

humanitarian contexts (shortcomings in existing data, security and access issues,

need to react fast), which made establishing proper causality and designing

interventions to address the correct factors of influence, complex.

In this context, the evaluation of the HA consensus also indicated a need for a more

consis

Comprehensive evaluation of the European Union ......Comprehensive evaluation of the European Union...

Documents

Transcript of Comprehensive evaluation of the European Union ......Comprehensive evaluation of the European Union...