Oxfam Australia DRR/DRM Program Evaluation

Oxfam Australia DRR/DRM Program Evaluation

1

Oxfam Australia

Humanitarian Partnership Agreement (HPA Disaster Risk Reduction

(DRR) and Disaster Risk Management (DRM) Program, 2011-14

End of Program Evaluation

August 2014

Author: Peter Chamberlain


2

Abbreviations and Acronyms

ANCP Australian NGO Cooperation Program

AusAID Australian Agency for International Development (now part of DFAT)

CCA Climate Change Adaption

DAC Development Assistance Committee (of the Organisation of Development

Cooperation and Development)

DFAT Australian Department of Foreign Affairs and Trade

DIPECHO Disaster Preparedness – European Commission Humanitarian Office

DPT Disaster Preparedness Team (Indonesia)

DRM Disaster Risk Management

DRR Disaster Risk Reduction

ERT Emergency Response Training

GN Grama Niladhari (sub-district, Sri Lanka)

HPA Humanitarian Partnership Agreement

HSU Humanitarian Support Unit

KAP Knowledge Attitudes and Practices

OXFAM Oxfam Australia

OI Oxfam International

PCVA Participatory Capacity and Vulnerability Assessment

PRIME Preparedness, Response and Influence; a Model of Emergency (Indonesia)


3

Contents Abbreviations and Acronyms ................................................................................................. 2

1. Executive Summary ....................................................................................................... 4

2. Introduction .................................................................................................................... 6

3. Methodology .................................................................................................................. 7

4. Project Summary ........................................................................................................... 8

4.1 Solomon Islands ..................................................................................................... 8

4.2 Sri Lanka ................................................................................................................ 8

4.3 South Africa ............................................................................................................ 9

4.4 Indonesia .............................................................................................................. 10

5. Findings ....................................................................................................................... 10

5.1 Program Design .................................................................................................... 10

5.2 Who benefitted? .................................................................................................... 11

5.3 Efficiency and Effectiveness ................................................................................. 12

5.4 Relevance ............................................................................................................. 12

5.5 The Parameters of DRR........................................................................................ 12

5.6 Partnership ........................................................................................................... 13

5.7 Engaging Other Stakeholders ............................................................................... 13

5.8 Gender and Cross Cutting Themes ....................................................................... 14

5.9 Climate Change Adaption ..................................................................................... 14

5.10 Increased Use of Technology................................................................................ 15

5.11 Exit Strategies ....................................................................................................... 15

5.12 Sustainability ......................................................................................................... 16

6. Conclusion ................................................................................................................... 16

Annex 1: List of Participants ................................................................................................ 17

Annex 2: Workshop Timetable ............................................................................................ 18

Annex 3: Terms of Reference ............................................................................................. 19


4

1. Executive Summary

In 2011, as a member of the Australian Department of Foreign Affairs and Trade (DFAT)’s

Humanitarian Partnership Agreement (HPA), Oxfam Australia was invited to submit a three

year Disaster Risk Reduction (DRR) and Disaster Risk Management (DRM) proposal with a

total budget of AUD$1.5m. The DRR Program set out to develop context specific models in

the four regions in which Oxfam is directly programming, and the Solomon Islands, Sri

Lanka, Indonesia and South Africa programs were selected to implement projects. The DRM

component aimed to strengthen the DRR implementing capacity of Oxfam and its partners

by the development of tools and training materials and by the provision of technical support

to the projects.

As the project nears completion, a two day workshop has been held in Melbourne in July

2014. The purpose of the workshop was to share experiences, capture impact and review

effectiveness with findings anticipated to inform future program design and to strengthen

Oxfam's DRR/Resilience work in line with 2020 strategic commitments. This report reflects

the findings of this workshop, supplemented by information from project reports and case

studies.

There is agreement that the program has given considerable impetus to the development of

DRR within Oxfam. In 2011 isolated DRR projects existed in a number of countries, but there

was inconsistent documentation and no clear approach or plan within the agency. DRR is

now well understood by regional and country programs and there is a cadre of staff and

partners who are familiar with the approach. Tools such as the Participatory Capacity and

Vulnerability Analysis (PCVA) have been developed and adopted – and adapted to the

needs of the very different working environments which exist in the four countries covered by

the project.

There is evidence to suggest that the models developed have been successful and

influential. The Indonesian and Sri Lankan models have been adopted and replicated by

local governments: the Solomon Islands has successfully accessed further DRR funding and

the South Africa project also received additional funding for its innovative urban DRR from

another Oxfam affiliate.

The program design was appropriate – but the need for greater consultation in relation to the

design and for ongoing technical support was noted.

The importance of selecting appropriate DRR partners is a key learning: it would be fair to

say that three of the four projects experienced difficulties with the partners initially selected.

The intention to integrate Climate Change Adaption (CCA) into the DRR models has been

difficult to achieve. Country teams lacked the expertise in this area and it proved difficult to

engage with the high level, strategic expertise which Oxfam has at head office level. In

some countries, hard pressed communities simply do not see climate change as a priority

issue in their day-to day-lives. There is evidence that gender and cross cutting themes such

as disability inclusion have been addressed, without being a major focus of the program.


5

Participants at the workshop indicated a number of directions for future DRR and resilience

work within Oxfam Australia. Better engagement with a number of stakeholders was

recommended, including other agencies concerned with DRR, academic institutions and the

private sector. There is also considerable interest in better utilising technology which is

readily available in many countries – such as mobile phone apps and messaging services,

computer programs and other information technology.

As Oxfam Australia enters another period of structural change, it is important that DRR is not

forgotten or deprioritised. This program has provided the agency with DRR coordination,

innovation and a network over the past three years. The agency should continue to build on

these achievements.

The workshop was a rich source of ideas and suggestions. These are summarised in the

following recommendations:

Recommendation 1: Multi-regional programs require a high level of communication

and coordination in design and implementation phases. These processes must be

resourced accordingly.

Recommendation 2: The current flexible definition of DRR by Oxfam is appropriate. It

should not however, be expanded to incorporate conflict related disasters.

Recommendation 3: Oxfam must recognise that partner assessment and selection for

DRR are critical. Project planning should include resources for training and capacity

building in skills relevant to the implementation of the project.

Recommendation 4: Oxfam should adhere to existing guidelines and conduct a risk

mapping exercise, which also includes a power analysis, prior to the commencement

of new DRR projects.

Recommendation 5: Oxfam will considers how to provide expertise and support to

private sector engagement initiatives in relation to DRR projects.

Recommendation 6: Oxfam should strengthen its DRR work through practical

linkages to universities in Australia and the countries in which it operates.

Recommendation 7: Oxfam should work to provide practical CCA training and

support at the project and community levels.

Recommendation 8: Oxfam should explore options for better use of technology in

DRR using its existing resources and in collaboration with other actors. Areas such

as early warning, disaster mapping and appropriate communications should be

considered.

Recommendation 9: Oxfam must develop its plans for continuing its strategic

commitment to DRR. It should clarify its vision and strategy beyond the current

projects and ensure that necessary technical support continues to be provided for

DRR within the agency.


6

2. Introduction

Oxfam Australia successfully tendered for AusAID’s1 Humanitarian Partnership Agreement

(HPA) in late 2010. Selected agencies were asked to submit concept notes for Disaster Risk

Reduction (DRR) and Disaster Risk Management (DRM) projects. If both concept notes

were accepted, a budget of up to $1.5 million per agency - to be used over three years - was

made available. AusAID defined DRR as risk reduction work undertaken specifically with

communities, whilst DRM was defined as the capacity building of agencies and their partners

who would deliver the DRR component. There was considerable flexibility in how the funds

were to be allocated: for example there were no geographical specifications. AusAID also

encouraged HPA agencies to consider linkages with Climate Change Adaption (CCA).

In early 2011 when the concept papers were developed, Oxfam Australia had an

organisational commitment to DRR, but was somewhat uncertain of its approach to DRR,

although policies and approach documents existed within the Oxfam confederation. The

concept note references DRR work in nine countries – but there was no common model.

The DRR design recognised this situation and split funds between the four regions in which

Oxfam Australia works (Southern Africa, South Asia, East Asia, and Pacific) with the idea of

developing replicable DRR models in each. South Africa, Sri Lanka, Indonesia and the

Solomon Islands were the countries selected by the regional units. Overall project

objectives were decided:

Goal: Reduce vulnerability and enhance the resilience of communities to disasters and

climate change

Objective 1: Appropriate context specific project models are developed and trialled, and

lessons learned are documented and shared

Objective 2: Selected communities and partners are empowered to contextualise, identify,

analyse, evaluate and treat short term disaster and long term climate change risks

Objective 3: Target government agencies have increased awareness of DRR/CCA and, if

appropriate, are supported to develop their own DRR/CCA and response plans

Objective 4: Target government agencies and service providers are held accountable to

deliver community entitlement and services to targeted communities

The DRM project was designed to support these projects and develop common tools and

training materials. The development of a Participatory Capacities and Vulnerabilities

Analysis (PCVA) and the incorporation of DRR into the existing Emergency Response

Training (ERT) package were seen as key components, along with a technical support

position for the initial phase. The DRM project shared the goal of the DRR project and had

the following objectives.

Objective 1: To strengthen existing capacity development mechanisms to incorporate DRR

and CCA

Objective 2: To apply and adapt DRR and CCA mechanisms to targeted programs.

1 Subsumed by the Department of Foreign Affairs and Trade (DFAT) in November 2013


7

The program was launched with a workshop held in Lombok, Indonesia, which provided an

opportunity for participants from the target countries to meet, share information about the

projects and learn more about DRR concepts and tools and established a model of change.

As the DRR/DRM program nears completion this report has been commissioned to assess

the results and to reflect on the lessons for Oxfam Australia. Section 3 provides a brief

account of the methodology. Section 4 provides a short summary of each project. Section 5

documents the main findings of the final project workshop held on 28-29 July 2014.

3. Methodology This report has drawn upon a number of documentary resources, such as concept papers,

donor reporting and case studies, but its primary focus has been on documenting the

reflections and lessons learned at the completion workshop held in Melbourne on 28-29 July

2014. Participants included at least two field staff from each country – and two

representatives of partner organisations from South Africa. A number of Melbourne based

staff, including Humanitarian Program Coordinators, attended subject to availability. A full

list of participants can be found in Annex 1 and the workshop timetable is provided in Annex

2.

The Terms of Reference for the exercise (see Annex 3) incorporate both the DAC criteria

and Oxfam Australia’s Seven Core Questions2 for monitoring evaluation and learning. Based

on the Oxfam Minimum Standards for Evaluation, it also specifies the need to assess the

following areas:

project design

effectiveness

Oxfam and partner approaches

relevance

accountability

value for money

sustainability

The workshop timetable was designed to address these issues, so the structure and content

of the report reflect the workshop outputs. The report does not provide a detailed account of

the progress and outcomes of each constituent project, which can be found in the formal

reporting on the program.

2 1. What significant changes have occurred in women’s men’s boys and girl’s lives and to what extent are these likely to be

sustained? 2. How far has greater equity been achieved between women and men, boys and girls, and other groups? 3. What changes in policies, practices, ideas, beliefs and attitudes have occurred in specific institutions, groups and individuals? 4. How effectively and appropriately have those we seek to benefit been involved at relevant stages through the process? 5. How

effectively and appropriately have we worked with others and involved them in relevant stages through the process? 6. How effectively and efficiently have our resources been used? 7. To what degree have we learnt from this experience and shared the learning with others and ourselves? What will we now do differently, or what will we do more of? See “Oxfam Seven Core

Questions Toolkit”, July 2012


8

4. Project Summary

4.1 Solomon Islands The project has worked in three diverse communities in different parts of the country.

Maraone is a inaccessible inland village in Makira province, ten hours walk from the

provincial capital. Nifiloli is a coastal community in Temotu province in the far east of the

country. Kolosori is a peri-urban settlement in Isabel province. The country has a high

exposure to natural hazards and low lying areas – such as Temotu - are being particularly

affected by climate change.

The Solomon Islands Red Cross (SIRC) was identified as the main partner organisation

because it had a strong track record of DRR implementation in the country. The project

articulated “the super approach” designed to bring government, civil society and community

stakeholders together and incorporating the lessons from previous projects. It has

successfully worked with the three communities to implement PCVAs and Community Action

Plans.

A number of problems were encountered. The extreme remoteness of the project sites

made access difficult and unsuitable for local level replication. SIRC struggled to find staff

trained in the necessary DRR skills and to adapt them to the needs of different communities.

The partnership more generally, experienced challenges, which resulted in Oxfam’s decision

to implement directly in year three, while continuing to build SIRC’s capacity. The need to

respond to disasters such as the Temotu tsunami and the recent Guadalcanal floods has

also affected the project. Whilst the project has engaged extensively with local and national

level governments and disaster management authorities, it has been found that they

generally lack resources. For example the government is implementing an extensive risk

assessment process without the necessary resources to implement the action plans based

upon them.

The project has helped clarify the Solomon’s DRR model and helped to establish it as a core

theme of the country program. Additional external funding has now been obtained for

integrated DRR/resilience work through the Cargill Foundation and a small DRR advocacy

project, funded through the Australian NGO Cooperation Program (ANCP), is planned to

influence the government’s approach.

4.2 Sri Lanka The Sri Lanka project has taken advantage of a surge of interest in DRR following the

catastrophic tsunami in December 2004, which has led to the creation of a National Council

for Disaster Management (2005) and a national disaster management policy (2013). Oxfam

originally planned to work in post- conflict northern areas of the country, but government

inflexibility on methods of implementation caused the project to change its focus to

Batticaloa District in the east of the country. Batticaloa is one of the districts most severely

affected by natural hazards. Typically these hazards include floods, cyclones, landslides

and lightning strikes.

Oxfam has worked in nine villages in five Grama Niladhari (GN) Divisions. Working with local

communities and grass roots partners, GN Divisions have conducted PCVAs, established

GN Disaster Management Committees and developed action plans. Other agencies have


9

contributed – for example the Sri Lanka Red Cross has provided first aid training to

committee members.

The project has not been without challenges: the government is more interested in the

“hardware” of disaster response than the “software” of a community based DRR approach.

It has also proved difficult for the project to make progress on climate change adaption as

communities and local governments have much more pressing and immediate concerns to

contend with. The project also aspires to collaborate more with the private sector – but there

is a lack of knowledge of how to do this – and a degree of suspicion about the role of the

private sector amongst NGO staff. Finally, the project has had to deal with misappropriation

of funds by one partner which resulted in minor implementation delays.

Overall, the model has attracted considerable interest within Sri Lanka and the government

has undertaken to replicate the model in 19 more GN Divisions. There is improving

government engagement with DRR at divisional and national levels and Oxfam has now

been invited to participate in the National Disaster Management Committee. This project will

be utilised to leverage and expand the existing model. This will complement future work

funded by DFAT through the HPA mechanism, as well as through DIPECHO and ANCP.

4.3 South Africa South Africa is defined as a middle income country with around 62% living in urban areas

and less than one officially declared disaster per year – but with serious issues of social

inequality and consequently, vulnerability to shocks. More than 40% of the population is

estimated to be unemployed and there are a high number of unregistered refugees. Oxfam

has supported five established local partners in very different contexts: Refugee Social

Services focuses on refugees living in urban settings: Project Empower with peri urban

areas of Durban; Sophokama works in informal settlements in Eastern Cape and Kwazulu

Regional Christian Council and Tholulwazi Uzivikele work in often remote rural areas.

In engaging with DRR for the first time the South Africa program has needed to define what

constitutes DRR in the country. It has found many gaps in government support for the most

vulnerable, such as refugees and impoverished communities. The project has found a niche

in DRR for “micro-disasters” such as fires, floods etc. which often affect a small number of

people. The considerable differences in the context faced by each partner mean the project

has not developed a single model – but rather learned how to be effective in each setting.

The project has however followed the standard steps of introducing communities to DRR,

conducting a PCVA and developing an action plan. Influencing government at a local or

national level has proved challenging within the timescale of the project but innovative ways

of influencing important stakeholders have been developed. For example, a photography

initiative encourages community members to document their living conditions and a resulting

exhibition has been used to influence groups with influence – such as the architects who

design urban housing.

As a result of the project, DRR has been accepted as an important element in programming

in South Africa and there is interest in adopting some of the processes used – even amongst

partners not formally supported by the project. The innovative nature of the work –

particularly urban DRR - led to some additional funding being provided by Oxfam GB.


10

4.4 Indonesia Oxfam’s program in Indonesia differed from the others because it already had a large, well

known DRR program. The PRIME program was established as part of the response to the

2004 Indian Ocean Tsunami which devastated Aceh and became one of the key

components of the country program. The relatively limited funds available were used to

innovate: to use the existing methodology in a different context (Western Sumatra) and to

strengthen components in the program such as CCA and gender.

Western Sumatra is particularly vulnerable to natural hazards, particularly earthquakes, and

the government has recently established Local Disaster Management Authorities – but these

are weak and under resourced. The project set out to develop local DRR structures in 22

villages and to use these as models for replication elsewhere.

Communities followed the PCVA process and developed action plans. They also formed

Disaster Preparedness Teams (DPTs), which in turn were linked to form DPT forums.

Knowledge Attitude and Practise (KAP) surveys show important changes as a result of the

project. The perception that disasters are an “act of God” that cannot be foreseen or

prepared for has been substantially eroded, with a 50% increase in awareness of disaster

risk in their area – and 96.5% of people who now believe that it is important. 69.5% of

families have now developed their own disaster plan. This success has led to strong local

government interest in the model which is now being replicated in 11 villages and 7 schools.

The district level forums have also inspired the creation of a journalists’ forum dedicated to

spreading information about disasters in the area.

The initial target of 22 villages was perhaps too high, and the project has learned the

importance of selecting the right partners: some of those initially involved were felt to be

focused on disaster response rather than DRR and somewhat autocratic in their dealings

with communities. The project would also have liked to have gone beyond its focus on

identifying and preparing for risks to look at building resilience. Attempts at developing links

with the private sector did not produce significant results and also require further attention.

5. Findings

5.1 Program Design The design of Oxfam’s DRR/DRM activities was different to those of other HPA partners,

which largely allocated the funds to one or two existing projects, often managed by other

affiliates in their confederations. The allocation of four relatively small grants clearly carried

risks in terms of the possibility of relatively high operating costs and dissipated impact.

However participants generally felt that the results had justified this approach. One

participant noted the huge increase in the level of understanding and engagement with DRR

within Oxfam – something for which the DRR/DRM program has been a catalyst. The

proceedings of the end of project workshop - compared with the launch workshop almost

three years ago - suggest that a critical mass of staff familiar with DRR and committed to its

integration has been created.

Although the final workshop was generally agreed to be useful, some participants argued

that greater learning and more effective sharing of ideas could have been achieved better by

meetings earlier in the project cycle.


11

The project objective of creating regional DRR “models” may have been simplistic: within

regions - and countries the program has learned that tools and approaches need to be

tailored to the specific circumstances – but Oxfam is now unquestionably better placed to do

this.

Arguably objectives 3 and 4 of the DRR project - strengthening government DRR/CCA and

holding them to account for this – are too ambitious, but demonstrable success has been

achieved by the country projects – most notably with the replication of the project approach

by local governments in Sri Lanka and Indonesia.

The replication of these approaches, plus the success in leveraging additional funds in the

Solomons and South Africa mean that a good case can be made to say that the approach

has been cost-effective.

Some elements of the design could have been improved. The initial idea that the need for a

DRR coordinator would cease after one year when the necessary tools had been developed

was unrealistic – indeed the need for a central resource and focal point for DRR will outlive

the project.

The initial design which required all projects to conform to common objectives without

sufficient communication or consultation was also unsatisfactory and led to considerable

frustration.

The tools developed for the DRM component were generally adopted and used successfully

by the projects – although the Indonesia project noted that tools - such as the PCVA - were

not significantly different from the Oxfam GB versions, which they were already using.

Recommendation 1: Multi-regional programs require a high level of communication

and coordination in design and implementation phases. These processes must be

resourced accordingly.

5.2 Who benefitted? According to program reports the number of community members who have directly or

indirectly benefitted is 44,457. The numbers of direct beneficiaries is relatively small for most

of the projects. This in part reflects the small and isolated communities with which projects

are working – for example in the Solomons where the targeted communities numbered

between 125 and 500 individuals. It is also indicative of the “pilot” approach which focused

more on developing and documenting replicable models, rather than maximising beneficiary

numbers in the short term. It should also be recognised that there is a trade-off between

quantity and quality; the Indonesia project reached more than 23,000 direct and indirect

beneficiaries in 22 communities, but project staff believe that the impact might have been

greater if they had focused on fewer communities.

Local civil society and government partners have also clearly benefitted. The civil society

organisations involved in the project have strengthened their skills to the level where they

have been able to deliver the projects. Exposure to the DRR model has had further benefits:

“Even partners who do not have DRR programs are doing DRR work.”3 The replication of

DRR models by local governments in Sri Lanka and Indonesia is indicative of increased local

3 OXFAM South Africa Program, Case Study 17, Redeveloping Disaster Risk Reduction (2014), p35)


12

government capacity, and the Oxfam’s invitation to join the Sri Lankan governments’

National Disaster Management Committee suggests that opportunities to support and

influence national structures may be possible.

Finally, the project has undoubtedly benefitted Oxfam and its staff. The agency now has a

much more coherent approach to DRR, together with a cadre of trained staff and a range of

tested tools and training materials.

5.3 Efficiency and Effectiveness As discussed in the discussion of the project design (see section 5.1 above) the project has

made good progress towards its objectives, even if more work is needed in several areas

(e.g. strengthening and influencing government, CCA). The encouraging signs that the

program has led to replication and to further funding also suggest that the DRR models

developed do represent good value for money.

The program has created a good foundation for further DRR work within Oxfam: the

development of contextualised models, the availability of DRR tools and training materials

and a series of case studies which document the learning from the projects. To get the best

value from the investment, Oxfam will need to continue to focus on DRR and to capture

learnings from current and future projects.

5.4 Relevance The relevance of DRR is unquestionable. Internally, both the Oxfam and OI strategic plans

contain commitments to DRR. Indonesia, Sri Lanka and the Solomons have all been

stricken by large and small “natural” disasters in recent years, leading to formal government

commitments to DRR. In each case this commitment is not matched by a clear

understanding – particularly at a local level – of what the commitment means and how it can

be implemented. In these circumstances Oxfam’s DRR interventions have been relevant –

and influential. South Africa is less prone to such disasters – but in developing and

documenting approaches to “micro disasters” in a highly unequal society - particularly urban

and peri-urban contexts, the project is certainly relevant to Oxfam’s role and mandate.

5.5 The Parameters of DRR The workshop spent some time looking at the parameters of the current approach. One

question is whether Oxfam’s DRR framework should include conflict related disasters –

rather than restricting its focus to natural hazards. There is no doubt about the relevance of

conflict: all four target countries have experienced war, or at least serious social tensions in

the last two decades. Participants were broadly in agreement on the importance of

addressing conflict related disasters, but felt this required more investment and attention

than is feasible within the ambit of the current program. It is clear that this will need to be

further considered in future, but most contributors felt that to extend the DRR model to

include conflict would risk overcomplicating - and potentially diminishing - its impact.

The South Africa Program also noted that it had struggled with definitional issues: are the

“micro-disasters” such as fire or wiring hazards in urban tenements really part of the DRR

paradigm? All that can be said is that DRR tools and approaches have been found to be

effective in dealing with them.


13

The relationship of DRR and resilience within Oxfam may also need clarification. Amongst

DRR practitioners there is recognition of the need to go beyond risk management and to

develop community resilience, particularly in relation to livelihoods, so DRR and resilience

should be complimentary if there is good internal communication around these issues.

Recommendation 2: The current flexible definition of DRR by Oxfam is appropriate. It

should not however be expanded to incorporate conflict related disasters.

5.6 Partnership Selecting and developing the right implementing partners was recognised as a key issue by

all countries. Problems with partners were common - but of different types. In the Solomon

Islands the partner selected had unrivalled DRR credentials but had its own agenda of

expanding its presence in to new and remote areas where its comparative advantage was

lessened. The partner also had a disaster response mandate – and when disasters did

occur, staff and resources were required elsewhere. In Indonesia partners with strong

disaster response credentials were selected – but in at least one case their approach to

communities was too autocratic for community based DRR. Sri Lanka was forced to change

locations when the demands of government in the North were realised – and also had to

deal with corruption in one local partner.

No simple or common solution was identified, but participants agreed that a long term and

strategic approach was necessary in selecting appropriate DRR partners. Careful partner

assessment is necessary to understand their strengths, weaknesses and motivation, and

ensure that proper selection occurs. In most cases ideal partners may be difficult to find –

and DRR work will need to be accompanied by capacity building and a more strategic

approach which also considers the option for direct delivery if project timeframe and

resources are limited. Participants also mentioned the importance of improving ways of

working with Red Cross national societies, given their access and mandate for preparedness

and response, and their ability to access communities and work at multiple levels.

Recommendation 3: Oxfam must recognise that partner assessment and selection for

DRR are critical. Project planning should include resources for training and capacity

building in skills relevant to the implementation of the project.

5.7 Engaging Other Stakeholders Three types of stakeholders were discussed at the workshop. First, it is unlikely that Oxfam

alone will be implementing DRR within a given country. Participants noted that they were

unaware of the activities of other HPA partners – some of which turned out to be active in

DRR. It was suggested that a DRR mapping exercise would be useful in the early stages of

future projects.

Recommendation 4: Oxfam should adhere to existing guidelines and conduct a

mapping exercise, which also includes a power analysis, prior to the commencement

of new DRR projects.

Workshop participants were also aware of the potential of working with private sector

partners: three of the four countries participating in the program are classified as “middle

income” countries and all have dynamic economies. Some caution is clearly necessary:

there are many examples of unscrupulous companies – particularly those focused on


14

resource extraction – which Oxfam should not legitimise by any sort of partnership.

However, collaboration with many others could be beneficial. Currently country program

staff are likely to be reluctant to engage with the private sector and, in any case lack the

necessary skills to do so. Workshop participants gave examples of unsuccessful attempts at

collaboration: in Indonesia, companies were approached at a local level but were unable to

make any meaningful commitment without reference to their national hierarchy.

Recommendation 5: Oxfam should consider how to provide expertise and support to

private sector engagement initiatives in relation to DRR projects.

There is evidence that academic institutions are becoming increasingly engaged in DRR: in

Indonesia for example, the government has identified the twelve main disaster risks and is

allocating support to an equal number of universities to work on one of these risks each.

Workshop participants were keen to develop links with academic institutions in Australia and

in their countries of operation and they proposed a number of ways that this could be done:

Use students to help document DRR projects

Ask university faculties to provide technical support in specific areas (e.g.

incorporating CCA in DRR)

Oxfam to contribute to academic courses to influence the debate on DRR

It should be recognised that there is an opportunity cost to participation in academic

engagement for busy program staff, but these ideas deserve further consideration.

Recommendation 6: Oxfam should strengthen its DRR work through practical

linkages to universities in Australia and the countries in which it operates.

5.8 Gender and Cross Cutting Themes Gender disaggregated data is provided for all projects and activities. This shows that

beneficiary numbers are almost equal between men and women and that there has been a

concerted effort to ensure that women participate in disaster management bodies. For

example 44% of Disaster Management Committee members in Sri Lanka are women: in

Indonesia 60% of those participating in community risk analysis and action plans are

women. It was suggested that to go further, future projects could include specific gender

objectives and make greater use of gender standards which already exist.

Generic child protection policies apply to all of Oxfam’s work. There are also good examples

of the inclusion of people living with disabilities. For example in Sri Lanka their houses are

identified and marked so that they can be assisted in case evacuation is necessary. Specific

collaboration with specialist agencies is also being considered. For example, the Sri Lanka

program may prepare an MoU with Handicap International to strengthen its work in this area.

It was agreed that none of the projects are likely to have had any significant environmental

impact.

5.9 Climate Change Adaption Including CCA in Oxfam’s DRR approach was clearly part of the program design, but

progress has been somewhat limited. In some countries, communities and authorities have

not seen it as a high priority and local staff lacked the expertise and tools to convince them


15

that it should be taken more seriously. In countries which are immediately and tangibly

affected by climate change – such as the Solomon Islands, this is less problematic.

Accessing appropriate technical support has proved difficult, although participants spoke

positively of a DRR/CCA learning event conducted last year. Field staff said that while

Oxfam has considerable high level CCA expertise, they had struggled to access simple,

practical support in incorporating CCA at a project and community level.

It was suggested that the existing PCVA does not place sufficient emphasis on CCA.

Participants noted that DRR and CCA are not vastly different concepts – but the benefits of

integration needed to be made clearer.

Recommendation 7: Oxfam should work to provide targeted and practical CCA

training and support at the project and community levels.

5.10 Increased Use of Technology Workshop participants felt that Oxfam does not currently use available technology in an

effective manner and some of the most enthusiastic debate centred on innovations which

could strengthen DRR. Oxfam is increasingly working in countries with high rates of mobile

phone usage and increasing technological capacity. Ideas for better use of technology

include:

Encouraging insurance companies to speed up claims by using Google Earth to

verify crop damage – rather than wait for physical verification (Sri Lanka)

Using smart phone apps to collect and share disaster information – an approach

already trialled in the recent Guadalcanal floods in the Solomon Islands

Persuading mobile phone networks to disseminate disaster warnings (Sri Lanka)

GIS systems for risk mapping

Better use of walkie-talkies in addition to mobile phones for better collective

communication among stakeholders (Indonesia)

The impetus for better technological use could come from a number of sources. It was noted

that Oxfam’s own IT staff often have considerable capability which is often under-utilised.

Increased engagement with academic institutions and the private sector also have potential

for collaboration on applying available technology in DRR.

Recommendation 8: Oxfam should explore options for better use of technology in

DRR using its existing resources and in collaboration with other actors. Areas such

as early warning, disaster mapping and appropriate communications should be

considered.

5.11 Exit Strategies The key to sustainable outcomes is the strength of community organisations and our DRR

partners. To be successful, capacity benchmarks need to be established and regular

assessments need to be conducted to measure progress towards them. The importance of

appropriate messaging about the project duration from early in the project cycle is also

important. A particular issue with the type of “pilot” approach is that it implies some sort of

follow up or expansion of activities. In the Solomon further DRR funding from another donor


16

will ensure that this occurs – but future DRR activities are less clear in the other three

countries.

5.12 Sustainability The DRR/DRM program has made considerable progress over the last three years but the

future of these activities within the agency is unclear at its conclusion. There should be

greater clarity on how Oxfam will meet its strategic commitment to DRR and how it will be

resourced. It is important that the progress made in the development of DRR models is not

lost through a lack of clear plans or funding. Workshop participants also agreed that for

DRR to expand beyond the current projects, a position is necessary to provide ongoing

technical support and to ensure learning is shared across projects and countries.

Recommendation 9: Oxfam must develop its plans for continuing its strategic

commitment to DRR. It should clarify its vision and strategy beyond the current

projects and ensure that necessary technical support continues to be provided for

DRR within the agency.

6. Conclusion The DRR/DRM Program has had considerable success in developing and institutionalising

DRR within Oxfam Australia. There is now a cadre of skilled staff within each region and a

set of training materials, tools and approaches which simply did not exist three years ago.

Documentation of the component projects through reports and case studies mean that much

of the learning from the program is accessible to those outside the project.

Replication of project approaches in Sri Lanka and Indonesia and the additional DRR funds

which have been obtained in the Solomon Islands and South Africa are evidence that the

merits of this work are being recognised outside the agency.

Weaknesses in design and partner selection have largely been recognised and overcome.

Gaps remain: there are improvements which can be made in incorporating CCA and in

adaption of available technology and the potential of academic and private sector

partnerships are largely unrealised.

However the real value of this challenging and innovative program will be in how Oxfam

Australia chooses to build upon it and develop DRR within the agency.


17

Annex 1: List of Participants HPA DRR REVIEW WORKSHOP

JULY 28-29

CONFIRMED PARTICIPANTS

International

Mohammed Riyas - Oxfam in Sri Lanka Rasiah Sivaskaran – Oxfam in Sri Lanka

Lorima Tuke – Oxfam in Solomon Islands

Katie Greenwood – Oxfam in Solomon Islands

Ade Reno Sudiarno – Oxfam in Indonesia

Petrasa Wacana – Oxfam in Indonesia

Inger Harber – Oxfam South Africa

Mxolisi Nyuswa – KwaZulu Regional Christian Council, South Africa

Yasmin Rajah – Refugee Social Services, South Africa

Melbourne Office

Andee Davidson – Pacific

Charmaine Consul Goncalves – Pacific

Evan Davies – Pacific

Juhi Sonrexa - Pacific

Jo Podlesak – East Asia

Peter Ikin – East Asia

Rachida Hunting – East Asia

Remy Kinna – East Asia

Farooq Dar – South Asia

Sophie Ford – South Asia

Katia Rotar – Africa Maud Mukova-Moses - Africa Richard Simpson – Africa

Kerry Farrance – Africa

Emma Renowden – HSU

Louise Mooney – HSU

Meaghan Barry – HSU

Meg Quartermaine – HSU

Praphulla Shrestha - HSU Jayne Pilkington – Program Quality Manoja Wickramarathne – Program Quality James Riturban – Program Quality


18

Annex 2: Workshop Timetable Oxfam Humanitarian Partnership Agreement (HPA) Disaster Risk Reduction (DRR)

Review Workshop Agenda Monday 28 – Tuesday 29 July 2014

Purple Room, Multicultural Hub, 506 Elizabeth Street, Melbourne 3000 Objectives:

1. Evaluate overall program impact 2. Capture and share results from each location 3. Identify key lessons on design, approach and implementation to strengthen Oxfam's DRR/Resilience

work in line with 2020 strategic commitments.

Day 1: Internal Session, Monday 28th July 8:30 - 9:00 Welcome and Introductions 9:00 - 10:15 Indonesia Presentation 10:15 - 11:30 Solomon Islands Presentation 11:30 - 11:50 Morning break 11:50 - 13:00 South Africa Presentation 13:00 - 14:00 Lunch break 14:00 - 15:10 Sri Lanka Presentation 15:10 - 15.30 Afternoon break 15:30 - 16:45 Global Learning Café: Group Work 16:45 - 17:00 Summary Day 2: Internal Session, Tuesday 29th July 8:30 - 9:00 Recap of Day 1 9:00 - 10:45 Global Learning Café: Group Work 10:45 - 11:00 Morning break 11:00 - 12:30 Global Learning Café: Group Work 12:30 - 13:30 Lunch break 13:30 - 15:30 Plenary Discussion 15:30 - 16.00 Afternoon break 16:00 - 17:00 Plenary Discussion, Evaluation & Close

19

Annex 3: Terms of Reference

PROJECT TITLE

End Program Impact Evaluation of Oxfam’s Humanitarian Partnership Agreement (HPA) Disaster Risk Reduction (DRR) and Disaster Risk Management (DRM) Program Consultancy. Responsible to: Emma Renowden, DRR Coordinator Accountable to: Meg Quartermaine, Humanitarian Support Manager

PURPOSE

To co-facilitate a workshop then complete an internal impact evaluation of Oxfam’s HPA DRR/DRM Program 2011-2014.

BACKGROUND

Oxfam’s three year HPA DRR/DRM program, funded by the Australian Government, commenced in July 2011 and has built on past experience to further develop successful integrated DRR and CCA models in South Africa, Sri Lanka, Solomon Islands and Indonesia. Goal: Reduce vulnerability and enhance the resilience of communities to disasters and climate change Objective 1: Appropriate context specific project models are developed and trialed, and lessons learned are documented and shared Objective 2: Selected communities and partners are empowered to contextualise, identify, analyse, evaluate and treat short term disaster and long term climate change risks Objective 3: Target government agencies have increased awareness of DRR/CCA and, if appropriate, are supported to develop their own DRR/CCA and response plans Objective 4: Target government agencies and service providers are held accountable to deliver community entitlement and services to targeted communities Duration: July 2011 - June 2014 Funding: AUD $1.5 million Beneficiaries: 44,457 (11,492 direct, 32,965 indirect) - Solomon Islands - 913 direct (517 men, 396 women, of whom 609 were children); Sri Lanka - 4703 (1377 men, 1399 women, 1934 children); South Africa - 5234 direct (2074 men, 2974 women, 186 children) and 9941 indirect; Indonesia - 642 direct beneficiaries (299 men, 343 women) and 23,024 indirect (11,213 men, 11,811 women). Locations: Solomon Islands - Makira, Temoutu and Isabel provinces; Sri Lanka - Batticaloa district; South Africa - KwaZulu Natal & Eastern Cape (urban and rural locations); Indonesia -Agam and Pariaman districts, West Sumatra Civil Society Partners: Solomon Islands Red Cross (SIRC), Lanka Evangelical Alliance Development Society* (LEADS), Community Development Organisation (CDO), KwaZulu Regional Christian Council (KRCC), Refugee Social Services (RSS), Project Empower (PE), Sophakama, Tholulwazi Uzivikele (TU), Wahana Lingkungan Hidup (Walhi) West Sumatra*, JEMARI Sakato, Persatuan Keluarga Berencana Indonesia (PKBI) West Sumatra

SCOPE

As part of Oxfam’s internal commitment to program quality, staff from the four countries plus relevant head office staff will come together for a two day workshop to evaluate what worked well and what could have been done

20

better. The workshop will be a valuable opportunity to share experiences, capture impact and review effectiveness with the findings anticipated to inform an evaluation report. The broad aims of the workshop will be to:

Evaluate overall program impact

Share and capture results from each location

Identify key lessons to strengthen Oxfam's DRR/Resilience work in line with 2020 strategic commitments. The consultant will be responsible for: 1. Review of relevant background documents, specifically annual and final reports, case studies; and input into workshop agenda planning.

2. Document and analyse of workshop findings from key program staff, including what had gone well and what that can be improved.

3. Write Evaluation Report based on an agreed format, which includes recommendations for future design. The consultant will be required to attend and help co-facilitate the workshop. Afterwards, the consultant will work autonomously on the report with final submission to the DRR Coordinator in Melbourne. Support and overall management will be provided by the DRR Coordinator with input from the HSU Operations Manager as required. Accountability for the work after the impact evaluation is complete will rest with the DRR Coordinator and relevant Program Quality staff, who will ensure that proper dissemination across Oxfam and utilisation for input into the Oxfam Strategic Plan.

APPROACH

The impact evaluation report will be determined through document review (final reports) and findings from a two day workshop which will involve key country office and Melbourne based program staff. The process of undertaking the impact evaluation should be gender-sensitive, culturally-sensitive and participatory. Guidance about how to ensure evaluations are participatory and gender-sensitive is provided below and in the 7 Core Questions Toolkit. The consultant will use methods involving document and content analysis to consolidate findings from stakeholder consultations and program reports. This information will form the basis of an analysis of progress and achievements against the program logical framework and will cover the period up to date. The consultant will draw from such analysis signs of success or failure and provide recommendations regarding specific actions that should be taken to take corrective measures to improve implementation/impact. The maximum length of the final report (in English) should be 20 pages (excludes cover and appendices) and structured along the outline indicated below: 1. Title Page 2. Abbreviations/acronyms list 3. Executive Summary – this should (1) provide a short project introduction (2) explain briefly the evaluation methodology (3) summary of the finding (4) recommendations. 4. Table of Contents 5. Project detail 6. Evaluation Methodology – DAC criteria, Oxfam 7 Core Questions 7. Evaluation Findings 8. Recommendations At minimum the evaluation workshop and report should assess the following. While it may not be possible to respond to all criteria, it is important to be explicit about which criteria are prioritised. The criteria below align with internationally recognised evaluation criteria. Effectiveness: Program contributions to significant and sustained changes. This includes the strength of

assumptions about the changes that will or have resulted from the program/project (outcomes and/or impacts) as well as an assessment of the processes that have enabled the change. Also consider changes in the lives of

21

women, men, boys and girls and other marginalized groups like people with disabilities including changes in equity, accessibility, changes in policies and practices, changes in ideas, attitudes, beliefs in specific institutions, groups and individuals and civil society strengthening.

Oxfam and partner contributions: what roles have Oxfam and program partners’ played? How have Oxfam and partners contributed to outcomes and change processes?

Relevance: The extent to which participant priorities and needs are reflected and an assessment of how effectively communities, partners and stakeholders have been involved in all stages of the program cycle. In what ways do Oxfam strategies coincide with/reflect the political context at the local or national levels?

Accountability: Oxfam’s accountability to communities (women, men, girls and boys and other marginalised groups), partners and stakeholders. How effectively has Oxfam worked with others (partners, allies, authorities, others) and involved them in all stages of the process? Also consider the effectiveness of Oxfam and partner/ally rights based approaches and gender empowerment strategies.

Value for money: Consider the program’s economy, efficiency, effectiveness and its equity. How effectively and efficiently have Oxfam and community, partner and stakeholder resources been used to contribute to program design, implementation and outcomes/impact? Who has benefited from program and how? (see VFM tools under Question 6 in the Seven Core Questions toolkit – ‘Practical Application’ document)

Sustainability: In what ways has the program/project design supported long term sustainability? To what extent has a financial and/or program transfer strategy been developed to ensure continuation or consolidation of the program?

The consultant will provide a brief summary of key recommendations that have emerged from the evaluation that will be useful for developing Oxfam’s project strategies and approaches in the future. Recommendations may be subdivided into those related to (a) project design, and (b) project management and should indicate clearly for whom the recommendation is intended.

TIMEFRAME

Task Method Deliverable Due Date

Preliminary planning meetings with DRR Coordinator

Document review Discussion on agenda

Agenda 28 May -

Co-facilitate workshop and capture key findings

Attend workshop with key program staff

Document key workshop findings

28-29 July

Submit final report Written format, max 20 pages Final Evaluation Report 8 August

LIAISON

Key Oxfam Australia liaison point will be the DRR Coordinator. Communication by telephone, email and face-to-face will be provided on draft and final deliverables throughout the agreed duration. LOCATION Home based with visits to Oxfam Australia, 132 Leicester Street Carlton, VIC 3053, as required.

REQUIRED KNOWLEDGE & EXPERIENCE

Knowledge of DRR/CCA and Oxfam’s approach

Understanding and knowledge of both the development and humanitarian sectors

Proven evaluation, analysis and report writing skills

Cultural and gender sensitivity and participatory approach CONSULTANCY OUTPUTS

22

Final Impact Evaluation Report of the HPA DRR DRM Program which incorporates guidance from Oxfam’s 7 Core Questions Tool Kit, Oxfam Australia Guidance Note for Final Evaluations, Oxfam Policy on Program Evaluation. The Impact Evaluation Report will be based on the OECD/DAC evaluation criteria of relevance, effectiveness, efficiency, impact, and sustainability. Other criteria such as accountability to beneficiaries, partnership, quality, appropriateness, replicability and scalability may also be taken into account

Oxfam Australia DRR/DRM Program Evaluation

Documents

Transcript of Oxfam Australia DRR/DRM Program Evaluation