EXECUTIVE SUMMARY - Home - Department of...

INDEPENDENT REVIEWof two remote service delivery

and community development programsin Papua New Guinea

by Alison Heywoodwith support from Tom Seta (Tanorama),

Colin Wiltshire and Alison George

(Pacific Performance & Quality section, AusAID)

13 August 2012

Aid Activity SummaryThis review incorporates lessons from two distinct AusAID activities in Papua New Guinea:

Aid Activity Name Kokoda Development Program

AidWorks initiative INH843 – Kokoda Track Initiative (Activity 11A608)

Commencement date March 2008 Completion date June 2011 (Phase I)

Total Australian $ 4.2 million

Total other $ n/a

Delivery organisation(s)

This program is managed by a small group within AusAID across all sectors. This team engages with numerous PNG government agencies based on areas of designated responsibilities across service delivery sectors.

Country/Region Papua New Guinea

Primary Sector Health, Education, Community Development

Aid Activity Name Integrated Community Development Program

AidWorks initiative INJ153 – CARE Integrated Community Development Program

Commencement date June 2009 Completion date March 2013

Total Australian $ 6.7 million

Total other $ n/a

Delivery organisation(s)

CARE International

Country/Region Papua New Guinea

Primary Sector Health, Education, Community Development

Independent Review

AcknowledgmentsThanks to all those involved in this review for the support and willing assistance provided during the review period. In particular, thank you to Morea Arua (Central Provincial Administration) and McGill Taimbaru (Oro Provincial Administration) for their participation in the review mission along the Kokoda Track. Their perspectives and feedback are greatly appreciated and have enhanced the comprehensiveness of the review.

Thank-you to all national, provincial and local government representatives in Papua New Guinea for participating in interviews and providing thoughtful feedback.

Particular thanks go to staff at PNG Post for their assistance in facilitating field visits and all AusAID stakeholders for their participation, input and commitment to the process.

To CARE International staff, your comments and cooperation at each stage of the review process was gratefully received.

Thanks also to Linda Kelly, monitoring and evaluation specialist, for your feedback at all stages of this review.

AbbreviationsAusAID Australian Agency for International Development MDG Millennium Development Goals

BEDP Basic Education Development Program MOU Memorandum of Understanding

BOM Board of Management MTDS Medium Term Development Strategy

CBO Community-Based Organisation NEFC National Economic and Fiscal Commission

CBSC Capacity Building Service Centre NGO Non-Government Organisation

CD Capacity Development PCMC Provincial Coordinating and Monitoring Committee

CHW Community Health Worker PER Provincial Expenditure Review

DEC Department of Environment and Conservation PGK Papua New Guinea Kina

DGTP Democratic Governance Transition Program PMT Provincial Management Team

DPLGA Department of Provincial & Local Government Affairs PNG Papua New Guinea

DSIP District Services Improvement Program PPII Provincial Performance Improvement Initiative

EHP Eastern Highlands Province PRS Poverty Reduction Strategy

FBO Faith-Based Organisation SLIP School Learning Improvement Program

GoPNG Government of Papua New Guinea SNP Sub-National Program

HDI Human Development Index SNS Sub-National Strategy

ICDP Integrated Community Development Program TOR Terms of Reference

JU Joint Understanding TSSP Transport Sector Support Program

KDP Kokoda Development Program UNDP United Nations Development Program

KI Kokoda Initiative VBA Village Birth Attendant

KTA Kokoda Track Authority VHV Village Health Volunteer

KTF Kokoda Track Foundation WDC Ward Development Committee

LLG Local Level Government

Independent Review

Table of Contents

EXECUTIVE SUMMARY................................................................................................................1

RECOMMENDATIONS.................................................................................................................. 5

INTRODUCTION..........................................................................................................................9CONTEXT...................................................................................................................................................9ACTIVITY BACKGROUND.............................................................................................................................12

Kokoda Development Program........................................................................................................13Integrated Community Development Program...............................................................................14

REVIEW OBJECTIVES AND QUESTIONS..........................................................................................................15REVIEW SCOPE, METHODS AND LIMITATIONS................................................................................................15REVIEW TEAM..........................................................................................................................................16

REVIEW FINDINGS..................................................................................................................... 17EFFECTIVENESS.........................................................................................................................................17

Program achievements....................................................................................................................17Factors that contribute to effectiveness..........................................................................................18Barriers to achieving outcomes........................................................................................................20Are programs on track?...................................................................................................................24Appropriateness of program objectives...........................................................................................24

EFFICIENCY..............................................................................................................................................27Cost of remote service delivery........................................................................................................27Factors that contribute to efficiency................................................................................................27Barriers to achieving value for money.............................................................................................29Implications of scale-up on efficiency..............................................................................................31

SUSTAINABILITY........................................................................................................................................33Factors that have promoted sustainability......................................................................................33Factors to consider for sustainability...............................................................................................35Improving sustainability outcomes..................................................................................................37

PROGRAM DELIVERY MECHANISMS..............................................................................................................43Direct management by AusAID – the KDP example.........................................................................43NGO management model – the ICDP example................................................................................44Features of strong program delivery mechanisms..........................................................................45

CONCLUSION............................................................................................................................ 47

ANNEXES.................................................................................................................................. 50

Independent Review

EXECUTIVE SUMMARYThe Kokoda Development Program (KDP) and the Integrated Community Development Program (ICDP) are addressing the needs of populations in remote parts of PNG. The KDP is occurring along the Kokoda track corridor in Central and Oro Provinces. The ICDP is occurring in the Obura-Wonenara district of the Eastern Highlands Province. Both programs operate in particularly challenging environments where road networks are extremely deteriorated and limited, the terrain is rugged and the weather extreme. Service delivery is hampered by the high cost of transportation and irregular air services. These two areas are characteristic of many parts of PNG.

KDP’s objective is sustainable development of people and communities along the Kokoda Track corridor, including improved economic opportunities and livelihoods, health, education, transport and related services.

The intended impact of the ICDP is that the most economically, socially and politically marginalised communities, living in extremely disadvantaged districts of PNG, will have sustainable and measurable improvements in their well-being.

The review has been undertaken to inform AusAID senior management decisions around ongoing support to these initiatives, and to future programming where remote service delivery may be required. These two programs were selected as examples of donor-supported programs occurring in very remote areas of PNG, ICDP being a governance/community development program and KDP being a service delivery program bringing services up to minimum standards. Lessons are intended to provide guidance on how service delivery can be improved in remote and disadvantaged areas of PNG, and identify strengths and weaknesses of different implementing, intervention and development models to improve remote service delivery.

Effectiveness

Both programs have made significant gains in terms of achievements against their original objectives, filling a gap by delivering a program of priority services (education and health in particular) for their respective catchment areas that Provincial governments are currently unable to provide. In addition, the ICDP has supported governance and ward planning initiatives to underpin priority service delivery and enhance likely sustainability of these efforts, building capacity for communities to demand services.

In the KDP, activities in the health sector have included improvements to infrastructure including construction of waiting houses, provision of clinical equipment to minimum standards and various trainings. Village Health Volunteer (VHV) training has been undertaken to support the work of health personnel and improve community health through health promotion activities. A Community Health Worker (CHW) undergraduate program to train local people to work in the health sector has six individuals now graduating. Some key achievements as a result of these activities include: improved staffing levels at the hubs; increases in supervised deliveries in health facilities; implementation of integrated 6-weekly health patrols; increased numbers of children being immunised; and improved infection control practice.

Independent Review page 1 of 50

A major education infrastructure program has resulted in new school approvals and registrations. Teacher qualifications have been upgraded and new elementary teachers are being trained. Boards of Management (BOM) have been established and have received School Learning Improvement Program (SLIP) training. A key achievement of the education activities has been significant increases in enrolment of children, with female enrolments higher than national figures.

Communities have had opportunities for various trainings, and women in particular have benefited, taking on leadership roles and having greater confidence to speak out publicly.

In the ICDP under the LEAD component ward planning is occurring for the first time ever, with strong participation by women. Early outcomes in Yelia and Lamari LLGs include: 35/60 possible ward plans developed and implementation has commenced; and 62/286 WDC members are women.

The LEARN component focuses on strengthening the education and law and justice sectors with the following achievements so far: Literacy teachers have been trained and five child and three adult literacy programs have

been established in the Simogu area, with balanced enrolments of girls and boys in the child literacy programs.

Three out of five child literacy schools in Simogu cluster have been formally upgraded to elementary schools.

Elementary school education has been strengthened resulting in registration of schools in seven wards in Wonenara and Simogu.

Five village court networks have been established in Yelia LLG.

Some early outcomes as a result of EARN activities include: improvements in coffee and potato quantity and quality; the ability to grow African yam as a drought-resistant crops; and improved access to protein and an additional income source as a result of support to fish

farming, with expanded numbers and size of initial fish farms.

Early outcomes of support under the LIVE component include: increases in assisted deliveries; Village Birth Attendant (VBA) referrals occurring to health facilities for delivery; improved hygiene related behaviours; and Twelve communities now certified as ‘healthy villages’.

These two programs are on track to achieve most of their objectives. KDP is on track to meeting its retained objectives (see initial objectives outlined in the JU) with the exception of health infrastructure in Naduri. ICDP is likewise mostly on track with the exception of Gadsup-Tairora LLG where the security situation has halted CARE activities in that location. Also, the increased income for coffee growers is yet to be realised (although expectations have been raised), with the high cost of getting coffee to markets a key barrier.

While significant achievements have been made, some challenges to effectiveness include: absence of sufficient attention to initiatives that will support the flow of funds and capacity

development to districts and LLGs to deliver services; cost of providing/supporting services to remote areas; cost of access to markets; compromised communications and coordination with key players;


challenges to completing infrastructure programs in a timely fashion; absence of robust designs, and functional monitoring and reporting mechanisms to inform

progress; and inadequate timeframe for these programs to achieve what was intended.

Efficiency

Efficiencies have been achieved by taking a whole-of-system approach to service delivery, where priorities in all sectors are addressed concurrently, acknowledging the complementarity and dependence between the sectors. This approach contributes significantly to cost efficiencies given the remoteness and usually small populations that are involved. Greater efficiency has also been achieved with both programs having the flexibility to change direction, meaning that less effective approaches can be substituted with more effective ones.

But a number of factors have compromised efficiency and value for money, including: fragmented management arrangements where there is no single point of accountability,

where a team is ‘scattered’ – not under one roof – challenging efficient management and implementation;

lack of attention to both supply and demand aspects of service delivery, either by the program itself or concurrently by another entity or mechanism. In both these programs the most important supply side issue not being addressed adequately (if at all) is the flow of government funds to districts and LLGs; expectations are being raised in communities that services would be introduced and strengthened without anything in place to support this crucial activity. AusAID is well placed to address this shortcoming; and

poor quality of some inputs which has compromised effectiveness, for example poor quality sanitation and ‘one-off’ trainings without follow-up (KDP).

Sustainability

Some features of each activity suggest sustainability of limited aspects is likely. For health and education both programs have worked within government processes, and training has followed standard government guidelines, meaning that staff will be appointed to positions, and schools will be registered. Sub-national government counterparts are interested in progressing program initiatives in other remote areas of their provinces, especially if provided with evidence that program initiatives are making a difference. Where comprehensive approaches to capacity development have been undertaken, early success is evident, particularly in some income generating initiatives of the ICDP, e.g. fish farming. Empowering women in both programs is giving women a voice and in the KDP particularly confidence to deal with difficult situations.

Despite these limited indications of longer term success, there are many complexities to achieving sustainable solutions. Programs are not considering adequately three key factors that impact significantly on sustainability:

1. Availability of government funds to deliver services.2. Appropriate allocation of funds for service delivery.3. Developing local capacity to deliver services.

Addressing the allocation of funds and implementation capacity requires a two-pronged approach. A long-term approach to fix the allocation and flow of funds is needed, working with and improving provincial and local facility level systems. At the same time, improvements can be made in service delivery by focusing on implementation capacity, an approach that will not


only enable the longer-term goal of appropriate allocation of funds but will also have immediate, motivational impacts.

Improving government systems will be achieved if there is:(i) better engagement with provincial, district and local level government;(ii) assistance to Ward Planning; and(iii) effort to tackle the major human resources constraints.

Improving implementation capacity is more likely by:(i) developing community demand and support for services;(ii) following through to outcomes; and(iii) strengthening volunteer networks.

The review revealed a government preference for exit strategies (sustainability strategies) to be planned well in advance and phased, with stakeholders jointly and early on identifying a timeline along which different responsibilities would be taken on by the government.

Where limitations to sustainability can be addressed, they should be addressed before any expansion of activities to new areas.

Program management arrangements

Two different management arrangements are in place in these two programs: KDP is managed by a small group within AusAID across all sectors; and ICDP is managed by CARE, and INGO. Both models can be equally effective. The key factors that contribute to effective implementation are: a cohesive management framework with one point of accountability, having in place good robust monitoring arrangements that are tied to a sound program design document, clear delineation of roles and responsibilities of management and implementing team, establishing strong relationship between the program leadership and government stakeholders.

While some of the lessons learned from these two programs and the subsequent recommendations are not new, their significance is heightened when they are applied to remote area programs. The cost and the extra effort needed to support service provision in remote areas make it just that much harder to be responsive, and thus easier to put these things in the ‘too hard basket’.

Key Lessons

A whole-of-system approach is an efficient and effective approach in small, remote populations, considering all needs, including sectoral needs, under the one program umbrella.

Flexibility in programming is critical if a program is to remain relevant to the needs of beneficiaries, allowing a program to be responsive to emerging needs and adopt more effective approaches.

Community engagement and buy-in is critical to identify needs and generate community demand for better services. This can identify where government support is needed and where communities themselves can implement programs, so that it is not totally reliant on external support.

Support to ward planning gives communities the opportunity to engage in the government process that should see more funds flow down to their communities to meet their needs.


Importantly it allows communities to identify where they can have a role in making improvements in their community (self-reliance). Such support needs to be wary of raising expectations as to what the government might deliver. Re-visiting the ward planning template could help to make the requests more realistic.

Continuous learning and reflection on successes and failures, combined with flexibility to respond / change direction, increases the likelihood of achieving objectives.

Well-designed strategies to engage women and develop their capacity to participate in community discussions and decision making are needed.

Working within government processes enhances success. Supporting strategies to increase the opportunity for people from the local areas to

become service providers (health, education, law and justice) increases sustainability. Programs need to address the contextual challenges of volunteer programs (eg. where

other opportunities compete with participation in volunteer programs) and tailor activities accordingly to increase the likelihood of success.

Capacity development through training is unsuccessful where it is provided as a one-off. Such training is more successful where it is supported with follow-up training and support.

Irrespective of the type of program (community development, service delivery) and the management model, robust design and M&E are essential to know what the program intends to do, measure its progress and know whether intended outcomes have been achieved.

Communications are paramount to sharing program experiences with key stakeholders. This needs to be considered/articulated at the start of the program (in the design), and managed by the implementers.

All elements of supply and demand need to be examined and addressed at commencement (either by the program itself, or by another entity) to increase the likelihood of sustainability.

Greater attention is needed to identify the factors likely to impede sustainability of remote area programs.

RECOMMENDATIONS

For ICDP and KDP:

Effectiveness

A. Improve M&E systems and frameworks to enable regular collation of accurate data, improve reporting and facilitate provision of information on progress to stakeholders. Programs need this for ongoing monitoring and analysis of what is working, or not working, and to support rapid responses if needed. In particular, phase II of the KDP should:(i) develop a robust ‘design’ and M&E logframe;(ii) ensure objectives and outcomes are clearly articulated in the design, with indicators

identified for their measurement; and(iii) clearly delineate roles and responsibilities of the AusAID program manager and

implementation personnel.B. Improve communications by:

(i) continuing engagement through other mechanisms as appropriate (for example, KDP should continue to improve coordination by attending monthly PMC meetings for the KI, to share progress and present projected annual roll-out of activities for KDP);


(ii) developing a communications plan for each program that describes the processes for ongoing communications and regular sharing of information with stakeholders (including beneficiaries) regarding progress; and

(iii) updating and enforcing protocols for entry to program sites that maximise communications with LLGs, members of parliament, etc.

Efficiency

C. Ensure due processes are followed to get the best quality of work done on the ground. D. In relation to sanitation in the KDP program:

(i) engage the services of an independent sanitation engineer to assess the quality of work done in the current KDP program and make recommendations to improve the current state of latrines if appropriate;

(ii) before any further latrines are installed, seek an independent analysis of the range of options for hygienic, value-for-money latrines; and

(iii) identify quality assurance measures to be applied to all future latrine construction.E. If a less experienced local NGO is engaged to implement the program in a new area, CARE

should ensure that significant support can be provided to that NGO.

Sustainability

F. Before expanding, both programs should meet the following prerequisites:(i) programs are ‘fine-tuned’ to improve implementation;(ii) limitations to sustainability are addressed and unfinished work completed;(iii) provincial Administrations are engaged and involved in scale-up / expansion;(iv) provincial government buy-in has been independently verified; and(v) communications should be improved to ensure the lessons on what works, and what

does not work, are available and well-understood.G. Consider more realistic timeframes to meet objectives before expanding activities to new

areas and ensure that ongoing dialogue with provincial government and communities specifically addresses this issue.

H. KDP should consider more actively targeting women to work in health and education, given that they are less likely to work as porters.

I. KDP should ensure better supervision and coordination of VHVs to improve the likelihood of VHVs continuing to work in the community beyond activities for which an allowance is paid.

For AusAID:

Effectiveness

1. Ensure all designs undergo rigorous peer review, irrespective of pressure to implement quickly. If not feasible, AusAID should place strong emphasis in the first year of implementation on developing a sound program logic and M&E framework.

2. Ensure less experienced staff are provided with adequate support, training and supervision to develop program logics and M&E frameworks of an acceptable standard, to manage implementation, and to maintain robust financial and monitoring systems.

3. In consultation with partner governments and implementers, identify underlying critical barriers to effectiveness at the design stage and jointly discuss how they will be addressed.


4. Ensure M&E systems and frameworks facilitate regular collation of accurate data, reporting and easy provision of information on progress to stakeholders.

Efficiency

5. Support flexibility and institute efficient mechanisms for greater responsiveness.6. Consider whole-of-system approaches and improve coordination between sectoral

programs for activities in remote areas.7. Avoid fragmented management arrangements, ensuring all programs have clear lines of

reporting and accountability.

Sustainability

8. Require a detailed analysis of the three major factors that affect sustainability (availability of government funds to deliver services; appropriate allocation of funds for service delivery; local capacity to deliver services) is included in every design (or delivery strategy).

9. Use a supply/demand framework to guide designs (including risk analyses) and identify whether elements that are not the responsibility of the proposed program are being addressed elsewhere.

10. Jointly agree ‘exit strategies’ with partner governments, with implementation clearly outlined in design documents.

Program delivery mechanisms

11. Ensure AusAID management models adhere to streamlined contracting arrangements.12. Regardless of the management model, ensure all programs have independent M&E

expertise to support implementation.

Further suggestions for AusAID:I. Improve communications, including:

(i) devise strategies to achieve ongoing dialogue with partner government and identifying likely topics (i.e. risks to the program) at the concept / design stage;

(ii) institute annual meetings between AusAID and relevant provincial governments to discuss performance / outcomes, constraints to progress, remedial actions, etc; and

(iii) engage with provinces and ensure program managers can participate in appropriate provincial fora to share progress with key stakeholders.— This may be the PCMC, PMT or whatever is in place; most provinces have some

mechanism via which this can occur.— This can be facilitated by AusAID where there is an SNS provincial presence.— Where such avenues for communication are not functional, the challenge may

persist – therefore programs need to be creative in their efforts and approaches should be articulated in design documents.

II. Through the PNG transport program, undertake an analysis of the potential to engage with the private sector to overcome barriers for another national flight service provider to enter (or re-enter) the market.

III. Use lessons from the past more effectively (potentially through a ‘lessons learned database’). In particular, AusAID should develop a list of factors that are known (lessons from many evaluations and research) to contribute to sustainability and include this in design guidance.


IV. Given a potential conflict between the need to focus on poverty and the likelihood of sustainability, AusAID and partner governments should further examine and reconcile this complex issue.

V. Address the funding flow issue by:(i) considering this issue as a pre-requisite to working in remote areas in a province;(ii) seeking provincial agreement to include this dimension as part of a total package of

any remote services program;(iii) identifying the most effective way to make this happen; and(iv) consider limiting efforts to provinces where government engagement at all levels is

assured.


INTRODUCTION

Context“The fundamental purpose of Australian aid is to help people overcome poverty.”1 To do this the aid program is guided by five core strategic goals, reinforced by ten individual development objectives.2

Papua New Guinea (PNG) is classified as a low middle-income country with USD 2,084 GDP per capita (PPP).3 The country has a total land area of 462,840 square kilometres, of which roughly 50 per cent is mountainous. Many areas of the country are still inaccessible by road. Close to 87 per cent of its population (approximately 6.5 million) live in rural areas.

The PNG UN Development Assistance Framework (PNGUNDAF) (2003-2007) defines poverty within a contextual framework of a society with strong traditional, cultural and social values and obligations, and where development challenges continue to provide new and diverse opportunities for all people to improve their standard of living. The desire to improve standards of living throughout PNG is manifested at various levels: individual, community, society and institutional. Against this background poverty is described as being:

a result of weak governance, weak social support systems, inefficient use of natural resources, the lack of economic and financial growth opportunities, a poorly maintained infrastructure network and the inefficient delivery of and lack of access to basic services.

The indicators and characteristics of poverty that emerge out of this discussion include:4

lack of employment/cash; lack of land; absence of basic infrastructure including proper health, education, living conditions, a safe

regular water supply and communications; breakdown of family unit; gender inequality; and fear of crime.

Analysis of the incidence and severity of poverty in PNG is hampered by the absence of current or reliable data. The 1996 National Household Income and Expenditure Survey provides the most reliable quantitative data on poverty – measured by patterns of consumption and nutrition – and using this indicator that survey put the poverty rate at 37.5 per cent (but the rural rate of 41 per cent was 2.5 times higher than the urban rate of 16 per cent). 5 PNG ranks at 145 out of 177 countries on the UN human development indices.

1 An Effective Aid Program for Australia: Making a real difference—Delivering real results, AusAID, 2011, available at http://www.ausaid.gov.au/publications/pages/5621_9774_1073_3040_2380.aspx2 Ibid.3 2009 Human Development Report4 See also The Impact of Foreign Aid on Poverty and Human Well-Being in Papua New Guinea, Simon Feeny, Asia-Pacific Development Journal, Vol. 10, No. 2, December 20035 Papua New Guinea – Australia Development Cooperation Strategy 2006-2010, AusAID, 2006; Priorities of the Poor in Papua New Guinea, Asian Development Bank, 2002; The World Bank’s PNG Poverty Assessment estimated poverty in PNG at 54% in 2003, while UNICEF data suggests 36% of the population are below the international poverty line of USD1.25/day (http://www.unicef.org/infobycountry/papuang_statistics.html#91)


http://www.unicef.org/infobycountry/papuang_statistics.html#91

http://www.ausaid.gov.au/publications/pages/5621_9774_1073_3040_2380.aspx

There is no accepted definition of ‘remote’ in PNG. For the purpose of this review, we consider ‘remote’ areas to be a subset of rural locations – generally places that can only be accessed by foot or by air and are time consuming and/or expensive to access. Areas that do not have a reliable, accessible road (for example, a road that is inaccessible for much of the year due to weather) or where the road is not connected to public and private services (usually the provincial capital) are considered remote. The two activities that formed this review clearly involve remote communities.

While difficult to put a figure on actual populations living in remote areas, with approximately 87 per cent of the population living in rural areas, and around 50 per cent of the country comprised of road-inaccessible mountainous terrain, remoteness could be considered the norm in PNG.

An added dimension to the discussion of the needs of populations characterised by poverty and remoteness is the political/governance context of PNG where under the Organic Law responsibility rests with provinces to ensure a minimum level of services are provided to their populations.

The Department of Provincial and Local Government Affairs (DPLGA) developed a Provincial Performance Improvement Initiative (PPII) to improve capacity of local government staff to undertake their functions. A handbook outlines the provincial and local level governments’ (LLG) service delivery functions and responsibilities6 now expected in the different sectors. Despite this, provision of services has not improved, and is significantly less than it was 20 years ago. This includes both the actual services themselves (for example, the number of schools and health centres) as well as the quality of those services (for example, trained staff). Remote populations suffer disproportionately.

Service provision in PNG is not just the responsibility of government. Priority services (such as health and education) throughout PNG are provided by government, faith-based organisations (FBOs), and civil society organisations (CSOs).7 However, in the two activities considered in this review (Obura-Wonenara District in the Eastern Highlands Province and the Kokoda Track catchment area across Central and Oro Provinces), the government provides the majority of services. Consistent with the ‘design’ of each activity, this review focused (in terms of sustainability) on what service delivery aspects of these two activities provincial governments could take on. An assessment of what FBOs and CSOs would be able to take on is beyond the scope of this review.

Given that basic health and education are largely a provincial and local government functions in these two areas, improvements in governance at these levels could have a direct impact on the quality of services delivered. The AusAID funded PPII intended, initially, to strengthen the corporate management of provincial administrations and the relations between these administrations and elected leaders, improve national agency support for provincial administrations, and improve provincial and district administrations’ accountability for service delivery. The anticipated outcomes from improvements in these areas are better service delivery and more effective economic policy and planning.

Two AusAID-supported programs that are the focus of this review are addressing needs of populations in remote parts of PNG: the Integrated Community Development Project (ICDP) and the Kokoda Development Program (KDP). The operating environment of these programs is

6 The Handbook to the Determination of Service Delivery Functions and Responsibilities, Oct 20097 Annex 1 details the health and education service providers for the ICDP and KDP catchment areas.


particularly challenging. Road networks are deteriorated and limited in reach in both the Obura-Wonenara and most of the Kokoda Track catchment area. Terrain is rugged and the weather ranges from torrential rains to periods of drought. Most bridges from Popondetta to Kokoda (Oro Province) were damaged in Cyclone Guba in 2007, and have not yet been rebuilt.

There are very few income-earning opportunities for people living in the south of Obura-Wonenara District. The district has very limited road access, especially in the south. While the Highlands Highway and other roads provide good transport access in the north of the district, in the south there are only a few poor quality roads which mean that people in the Simbari and lower Lamari are very isolated from a service centre.

Similarly, most of the Kokoda track catchment area is only accessible by foot or plane, with many parts requiring more than one day travel by foot to reach either Post Moresby or Kokoda. There is a good road connecting Port Moresby with Sogeri, Depo and Vesilogo within about 45 minutes. Conversely, the road that connects Kokoda with Popondetta is quite poor, takes a few hours, with many bridges still in disrepair following Cyclone Guba in 2007.

Service delivery and livelihood opportunities are hampered by the high cost of transportation and irregular air services. This difficult operating environment is by no means unique in PNG and as such the findings from this review will have relevance to a range of activities across the country, insofar as those activities incorporate service delivery and opportunities for improved lifestyles.

The Kokoda Development Program is operating in a particularly unique political situation. The PNG-Australia Joint Understanding (JU) on the Kokoda Track and Owen Stanley Ranges does not cover entire local-level governments (LLGs) let alone Districts and Provinces. This ultimately effects sub-national level government buy-in and poses great risks for sustainability.


Map with areas for ICDP and KDP indicated:

Activity BackgroundThe KDP and the ICDP are two programs operating in quite different circumstances, and using different modalities, to address service delivery needs in some of the most remote parts of PNG, where access is generally by air or walking tracks, with road access only to some areas.

Both programs aim to improve economic opportunities and livelihoods (for KDP through service delivery) and basic services. The ICDP has a greater focus on building the capacity of government and civil society to deliver and advocate for services. A brief description of the programs follows.

The KDP is one component of the PNG-Australia Joint Understanding (JU) on the Kokoda Track and Owen Stanley Ranges. The ICDP concept was developed by CARE based on its existing ANCP funded activities. CARE submitted this concept to AusAID and in dialogue with AusAID developed it into a program design which it would manage in consultation with AusAID as a partner.


Kokoda Development Program

The KDP’s geographic focus is on the Kokoda Track Corridor (population of about 8,000), incorporating parts of Central Province and Oro Province. The KDP was designed to be a holistic community-based service delivery program, with investments in developing community agreements with clear expectations for community participation. Emphasis was placed on a participatory community approach.

The KDP was informed by Needs Assessment missions conducted by AusAID and the Government of PNG (GoPNG) in 2008-09.

KDP’s objective is sustainable development of people and communities along the Kokoda Track corridor, including improved economic opportunities and livelihoods, health, education, transport and related services. The four intended outcomes are:

1. Increased access to safe water, improved sanitation and environmental health for local communities and trekkers;

2. Functioning elementary and primary schools and improved access to secondary, technical and vocational education;

3. Functioning aid posts providing primary healthcare services, with the Kokoda Rural Hospital providing secondary health services and support to aid posts; and

4. Improved access to markets and services, through a flight subsidy scheme.

The program was built around existing AusAID sector programs. It takes a two pronged approach to improving services (health, education, water & sanitation, community access):

1. Direct service delivery interventions that include:o improvements to basic infrastructure (new facilities, renovations/maintenance to

existing facilities, basic operating equipment);o human resource improvements (staff training, supported supervision, improved staff

continuity); ando regular supplies to health (e.g. drugs) and education (e.g. books) services.

2. Indirect service delivery interventions (improvements to institutions and organisations responsible for aspects of service delivery) that include:o community strengthening (to increase self-reliance and ability to demand better

services); ando provincial support so that Provincial Government can better undertake their service

delivery role. This is achieved by (where possible and practical) including provincial, district and local level administration staff in the implementation of KDP activities and providing support for their supervision processes.


Integrated Community Development Program

The ICDP is implemented in Obura-Wonenara District in Eastern Highlands Province (EHP). It is one of the most remote districts in PNG and has a population of around 47,000. The program encompasses all 81 Wards within the three LLGs of this district. The program has been informed by a baseline study that describes the extent of poverty in this area and status of some key indicators for the populations of interest.

The ICDP integrates short-term practical improvements with longer-term systemic changes. It takes a long-term view to developing relationships and trust with leaders, officials and community members.

The intended impact of the ICDP is that the most economically, socially and politically marginalised communities, living in extremely disadvantaged districts of PNG, will have sustainable and measurable improvements in their well-being.

The program has four components:1. LEAD: strengthened local level organisations and processes;2. LEARN: strengthened learning environment and opportunities (basic education and

informal adult education);3. EARN: enhanced community livelihoods and enhanced access (subsistence and income

earning livelihoods); and4. LIVE: improved physical and social environment (primary health care, HIV/AIDS

prevention, and law & justice).

The most important of these is the LEAD component, which underpins the other three components. This governance component recognises the importance of aligning with national planning processes and entrenching behaviour change. It is the key driver for sustained change and the major transformative change CARE can facilitate. The intended outcomes are:1. strengthened and better informed communities working in partnership with government

and non-government organisations to identify, prioritise and address development needs;2. improved access to basic services, with greater government and civil society capacity to

deliver and advocate for these services; and3. improved and more secure and self-reliant livelihoods.

The program aims to achieve these outcomes by: building the capacity of, and partnerships between, LLG, non-government organisations

(NGOs), community-based organisations (CBOs), faith-based organisations (FBOs) and communities;

working with and strengthening existing structures such as District Administrations, LLGs, and Ward Development Committees (WDCs);

identifying and work with ‘change agents’ (e.g. Village Health Volunteers, model farmers, WDC members) to support the mobilisation of community members;

ensuring full community participation that enables women, men and youth to work together to address priority concerns;

creating demand for good governance (internally and externally) by supporting communities to participate in district level planning processes; and

working with local administrators, FBOs and CBOs to address constraints to service delivery.


Review Objectives and QuestionsThe purpose of this review is to inform senior management decisions around ongoing support to these initiatives and future programming where remote service delivery may be required. The review will generate lessons on how service delivery can be improved in remote and disadvantaged areas of PNG and identify strengths and weaknesses of different mechanisms used (including the most appropriate implementing partner, intervention model and development model) for improving remote service delivery. Lessons from this review should inform future programming, both in terms of the remaining and future years of the above programs, and also more broadly for the PNG and Pacific Program.

Key questions for this review also relate to the effectiveness, efficiency and sustainability of these remote service delivery activities. The Terms of Reference (TOR) and Review Plan are at Annex 2 and 3.

Review Scope, Methods and LimitationsThe focus of the review was on higher-level findings that would generate lessons for AusAID. This was to be supported by evidence and findings from the KDP and ICDP.

The review has taken a mixed methods approach that has included: document review and analysis, field visits and stakeholder interviews, limited primary data collection, and review of project and other data, including government data. Where project data has not been sufficient, the review has sought further information from the implementers. Data from these sources were triangulated and verified where possible. Discrepancies in information have been noted. Quantitative data has been compared with qualitative data available from project reports and discussions/interviews in the field from implementers, partners and beneficiaries to confirm reliability. Qualitative data collected in group discussions and interviews has been examined for themes, and has been compared and confirmed with data collected with similar groups across project sites.

Field visits included: ICDP: 2½ weeks in Goroka and Obura-Wonenara District (covering the district office, and

three out of a possible 13 clusters – Andakombe, Simogu, Ubo); KDP: 2 days each in Oro and Central provincial offices, 2½ weeks along the Kokoda Track

catchment area, including district offices and facilities at Sogeri and Kokoda, and six program villages (Kovelo, Naduri, Kagi, Manari, Vesilogo and Depo – a total of eight out of 16 locations); and

1½ weeks in Port Moresby.

Site visits and consultations were undertaken with schools and health facilities as well as with communities (refer Annex 4).

Key limitations of this review included8: A lack of robust designs, logframes, monitoring and evaluation (M&E) frameworks that

should contain baseline data on key, SMART9 indicators, and Theory of Change diagrams. These are critical to guiding reviews. The KDP has no definitive design document.

8 The limitations of this review are described in further detail at Annex 3.9 SMART: Specific, Measurable, Achievable, Realistic and Timely


This is a short, focused assignment of two relatively large programs with multiple stakeholders. The time allocated to this assignment has defined both the scope and the depth of the review. Some locations have not been visited and therefore consultations are limited by this.

Unavailability of some key provincial and local level government stakeholders for consultation.

Some but not all the elements of efficiency can be addressed. As with many donor programs these two programs have not been set up to undertake a basic cost-effectiveness analysis.

AusAID personnel associated with both programs have changed over the period since their commencement, with some new personnel in place only very recently. Some corporate knowledge has therefore been lost presenting a challenge to the review to get a detailed history and understanding of the program.

Poor/unavailable government data (although this is a problem across PNG, and not restricted to remote areas).

The review team compensated for some of these limitations by: undertaking initial consultations with program staff to discuss the extent to which the sites

to be visited were representative of the total program; selecting locations for field visits to give a good spread of strong and weak performance; working with each program to agree on expected outcomes and discuss appropriate

indicators for measuring progress against stated outputs and objectives; and undertaking telephone consultations with some individuals associated with each program

in the past but now working in other roles.

People consulted are found at Annex 4. Documents consulted are at Annex 5.

Review TeamThe independent review team was led by Dr. Alison Heywood, Director of Heywood Public Health Group. Dr Heywood has extensive experience in reviews and evaluations in the Pacific and Asia regions. She lived and worked in the health sector in PNG for 11 years, and has been engaging in short term assignments in PNG for the last ten years. She has been supported during part of the review by Mr Tom Seta, Tanorama, whose main area of expertise is in the education and governance sectors in PNG. Excellent guidance and support has been provided from AusAID Canberra-based evaluation officers, Colin Wiltshire and Alison George.


REVIEW FINDINGSThese findings are grouped around the four key areas outlined in the review Terms of Reference: Effectiveness, Efficiency, Sustainability and Program Delivery Mechanisms.

From the outset, the review team would like to acknowledge that both activities have achieved some remarkable results in a challenging environment, which are briefly described here. Further detail on achievements for KDP and ICDP are at Annexes 6 and 7, respectively.

Effectiveness

Program achievements

In the KDP, activities in the health sector have included improving infrastructure (heath facilities, staff housing, and other facilities), provision of clinical equipment to minimum standards, and various trainings. Village Health Volunteer (VHV) training has been undertaken to support the work of health personnel and improve community health through health promotion activities. A Community Health Worker (CHW) undergraduate program established in partnership with the Kokoda Track Authority (KTF) to train local people to work in the health sector has six individuals now graduating. These activities have at this early stage resulted in: improved staffing levels at the hubs; increases in supervised deliveries in health facilities; implementation of integrated 6-weekly health patrols; increased numbers of children being immunised; and improved infection control practice.

A major education infrastructure program has resulted in new school approvals and registrations, and new elementary teachers are being trained. Boards of Management (BOM) have been established and have received School Learning Improvement Program (SLIP) training, although not all BOMs are functioning effectively. Teacher qualifications have been upgraded. As a result, enrolments of children have increased significantly, with female enrolments higher than national figures.

Communities have had opportunities for various trainings10, and women in particular have benefited (with some qualification) to the extent that they are taking on leadership roles and have greater confidence to speak out publicly.

In the ICDP under the LEAD component ward planning is occurring for the first time ever, with strong participation by women. Early outcomes in Yelia and Lamari LLGs include: 35/60 possible ward plans developed and implementation has commenced; and 62/286 WDC members are women.

The LEARN component focuses on strengthening the education and law and justice sectors with the following achievements so far: Literacy teachers have been trained and five child and three adult literacy programs have

been established in the Simogu area; 299 girls and 284 boys are enrolled in the Simogu child literacy program.

10 This training was for the purpose of a community engagement strategy, not for economic improvement.


Three out of five child literacy schools in Simogu cluster have been formally upgraded to elementary schools.

Elementary school education has been strengthened resulting in registration of schools in seven wards in Wonenara and Simogu.

Five village court networks have been established in Yelia LLG.

Some early outcomes as a result of EARN activities include: improvements in coffee and potato quantity and quality; the ability to grow African yam as a drought –resistant crop; and improved access to protein and an additional income source as a result of support to fish

farming, with expanded numbers and size of initial fish farms.

Early outcomes of support under the LIVE component include: increases in assisted deliveries; Village Birth Attendant (VBA) referrals occurring to health facilities for delivery; improved hygiene related behaviours; 12 communities now certified as ‘healthy villages’; and Improved community-built walking tracks.

Factors that contribute to effectiveness

Both programs are reportedly filling a gap by delivering a program of services for their respective catchment areas that Provincial governments are currently unable to provide. Both programs are focused on GoPNG minimum standards and ensuring that GoPNG procedures and processes are being adhered to.

Both programs have been engaging with provincial governments, both at the design stage and ongoing as implementation occurs, although with varying degrees of success. Working with government is partly successful – provincial sectors involvement, MOUs, as evidence. This engagement is essential if long term sustainability is to be realised. The review exercise has prompted the ICDP to examine further strategies to improve some aspects of this engagement.

Both programs have generated enormous enthusiasm in communities for their children to receive an education (see text box). This was a repeatedly expressed priority by beneficiaries of both programs. The process of developing ward plans in ICDP has been one factor that has contributed this enthusiasm alongside the efforts of the communities to build the schools, and in the KDP involvement of communities in the initial audits which has highlighted this critical absence of this opportunity for children.

Workforce issues common throughout PNG (an aging health workforce; irregular staff attendance; lack of supervision) have been faced by both programs. However, both programs are demonstrating initiative in attempts to attract local community people into the workforce to increase retention rates. The VHV and VBA programs are contributing to this effort.


Education story from ICDP:“The children are brighter now”

“The best change has been education for our children. Before, the eldest child used to care for younger children in the family, do all the cleaning, take on the parenting role while mother and father went to the garden. When the school started here it was nice to see the children going to school and mixing with other children; their physical appearance changed because they were not locked away looking after younger children – they look brighter now!”

Mother in Obura-WonenaraFull story is at Annex 8.

Both programs exhibit a culture of continuous learning – small scale analysis of initiatives to identify successes or to reflect and respond to lessons learned. More is needed in KDP especially to give feedback to government about the success of initiatives, for example, the difference that integrated patrols have made to health indicators.11

Positive features of the ICDP are the successful inclusion of women throughout all elements of the program, and development of many key partnerships in delivery of the program (Annex 7-E). Community enthusiasm has been generated and engagement is clearly occurring. Much of this can be attributed to ICDP establishing strong relationships with these communities through field staff being on the ground more often than not, in addition to the engagement occurring through the ward planning process.

The VHV and VBA programs of the ICDP have had greater success developing a cohesive network of volunteers than the VHV program of the KDP. This seems to be due to the greater isolation of these communities from the outside world, plus a program of training that has brought these people together and given them an identity and status e.g. they are identified by their own T-shirts, they meet and re-fresh routinely. While the KDP volunteers have been provided with T-shirts, bags and first aid kits, they are under pressure from their families to engage in trekking where they can earn up to PGK80 per day as a porter. Many consequently appear to only engage in “volunteer” activities for which there is a payment (for example, health patrols). VHV coordinators and some health workers expressed concern to the review about this. However, to their credit the KDP health team is going to trial some incentive schemes that will not only achieve positive health outcomes but may attract volunteers to be more active in the program.

Being responsive to local needs/priorities has been facilitated by a clear AusAID directive, and has been a positive feature of both programs. Easy access to funds has facilitated flexibility to initiate activities/change direction as needed. Both programs have been able to do this. ICDP expanded to all Wards in Yelia LLG when it was realised that for ward planning to be effective all wards had to participate. The Maternal and Infant Health initiative (MIH) has been identified to fill an as yet unmet need.12 KDP was given a clear mandate to be flexible and change direction if indicated.

Community buy-in through initial community consultation on needs, and ongoing engagement throughout implementation has occurred in both programs. Ongoing engagement has been quite strong in the ICDP, but could possibly be strengthened in the KDP. This would mean identifying areas of discontent, even though KDP input (for example one-off trainings) into some elements of the program has finished. Stronger engagement to identify these issues is part of the process of continual learning that could inform donors of areas of weakness.

Barriers to achieving outcomes

Quality of program design and M&E11 NB: while integrated patrols are a recurrent government program, they have not been carried out for many years. The KDP enabled the government health workers to conduct patrols by providing funds directly to health facilities.12 However, it should be noted that the review recommends that this initiative (if pursued/supported by AusAID) must be part of the ICDP, not a separate or parallel program - it clearly contains some elements of the current LIVE component.


Program outcomes and objectives are only acceptable if they are supported by clear measures of their achievement (valid13, SMART indicators), best described in a definitive design document which clearly presents the program logic from inputs to outputs to expected outcomes. This is most clearly presented in a summary logframe, and can be supported by a Theory of Change diagram. This is appropriate whether this is a community development program or a service delivery program. It addresses what a program aims to achieve, how it will be achieved and how such achievement will be measured.

Objectives (outcomes) for the KDP have been extracted from the Kokoda Initiative (KI), and are considered appropriate for this environment. However, at the time of the review KDP did not have a detailed KDP design document in which objectives were articulated together with indicators to measure their achievement. A Theory of Change (TOC) had never been developed, and KDP management and implementation team were unable to complete an acceptable TOC for the review team. Documentation that informed the program was activity and outputs focused. There was no M&E framework with indicators at any level to guide monitoring of the program’s rollout (the “HOW” factor) and inform measuring of results (the “SO WHAT” factor). As Phase 1 has rolled out tangible outcome indicators have been identified by the implementation team, for example increased school enrolments and improved immunisation numbers, to name a few, but these have not been used to more clearly articulate a results-focused design. It will be important that the design for Phase 2 clearly articulates objectives and outcomes, and identifies indicators for their measurement.14

These comments are significant given that some of the initial activities of the KDP were poorly conceptualised/planned without ever ensuring (unstated) outcomes were achieved, for example community training activities. It is also significant given that the process (not described) of achieving outcomes would largely determine if the objectives in the first place were appropriate and likely to be achieved.

At first glance ICDP objectives15 are mostly appropriate, and a design document describes how the program will go about meeting these objectives. However, the indicators presented to measure these are neither valid nor SMART. For example: “Increased funding to local level plans” is not an indicator of “Strengthened local level organisations and processes” (Objective 1). It’s an indicator of a higher level outcome that the program hopes will be achieved should there be “Strengthened local level organisations and processes”. At the Impact level, the “Indicators” are not indicators. Indicators stated at that level are also repeated at the Objective level for various components. There is no hierarchical progression from lower level to higher level achievements/indicators. The logic of indicators is seriously flawed. The same criticism also applies to the stated “Outcome” level indicators.

A Theory of Change diagram is presented, but it is confusing. It is disappointing that the AusAID peer review process did not identify these limitations, particularly given AusAID’s focus on measuring/achieving results in the Aid program.16

13 Validity is the degree to which the indicator measures what it is intended to measure (and not something else).14 It is understood that the design is now being completed for Phase 2 under the SNS; however, the KDP ‘design’ document under JU2 made available to the review team still focuses on activities and outputs and does not constitute a detailed design for the KDP. It would be unlikely to satisfy a peer review process.15 ICDP Terminology is not always clear and makes discussion sometimes confusing; we refer here to ICDP Impact, Outcome and Objective statements as all encompassing the term “Objective”. Stated correctly (using accepted program logic terminology) they would be: Goal, Purpose, and Objective. 16 Aid Effectiveness Review, AusAID, 2011; Repositioning the PNG Aid Programme - Implementation Priorities 2011-2015, AusAID, 2011.


CARE is aware of the shortcomings of their M&E framework and has placed emphasis on strengthening this in further iterations of their design.

Both programs have focused heavily on reporting on inputs and outputs, but less so on progress towards achieving outcomes/objectives. Monitoring adherence of the program to achieve what was intended becomes difficult when there is no sound reference point, i.e. an M&E framework. It is essential that this is improved to get government interested – “this is what we said we would achieve, and this is how we are progressing”. AusAID funded programs must do this in a way that demonstrates best practice.

Challenges to timely implementation of infrastructure programs

For the KDP, achieving improvements to school infrastructure has been challenged on many fronts and has contributed to significant delays. Infrastructure projects were determined through community consultations, contracts were drawn up for the work and responsibilities were clear (supplies, manual labour, etc.). Nevertheless, there were still challenges faced, including land disputes, community disputes over teachers, difficulties keeping chain saw operators working. In addition, in “along the track” communities most active men are engaged in the trekking industry from April to November, reducing the amount of time they have to contribute to community activities such as construction.

While the KDP followed-up on these issues on each visit, the team emphasised that resolution of these problems was up to the community, including the BOMs who have responsibility to manage these issues. The review process highlighted the extent of the delays, the inaccuracy of progress data, and disagreement within the team about the current status of infrastructure. Much of this would not have been an issue had there been sound M&E systems in place and regular team meetings that would bring the delays to the attention of the KDP leadership and management for follow up.

Communications

What is evident from the review is that critical conversations between the government, AusAID and the implementers have fallen short of the mark, despite the various efforts of both programs to share information on progress. These conversations need to occur at different levels for different reasons. This has been more challenging for the KDP given it cuts across two provinces, and is occurring in an environment with many players, increasing the need for greater communication and coordination of effort.17 It has been challenging for ICDP given the administrative and political complexity at the very highest level of government in the EHP.

At a strategic level dialogue is needed at the highest level in provincial government that discusses the whole program of activities needed in remote areas, identifying where the government has a critical role to play to ensure that a comprehensive approach is taken to address all the key elements for success. This is where government buy-in can be attained. The process and frequency for this dialogue can be determined at the start of a new program, but can be strengthened in current programs. This dialogue may be a formal annual event. However, it is important that dialogue is ongoing particularly in relation to the key constraint of provincial funding flows. The appropriate platform from which this occurs regularly may already exist in a province (preferred), or may need to be created.

17 The KDP however have attended a PCMC with Central Province along with KTA, and DEC have requested to attend the next PCMC meeting. In addition, KDP has successfully pushed for KI PMC meetings to include provincial representatives at one meeting per quarter, at which provincial issues will be the main agenda item.


“We could involve them with the PCMC committee – education is the first sub-committee meeting this year – we could now put them in that meeting.”

Central Provincial government representative

There is a critical role for the local members of parliament in each of the projects. As they have the greatest influence on provincial budgets and district funds, engagement with them is essential, either by appointing them into some of the committee structures or creating regular feedback during input opportunities. Their engagement must be more strategic than just around funds which will be a factor in sustainability.

At a programmatic level avenues that facilitate communications between and with stakeholders have to improve. This is needed so that all stakeholders know what is being achieved, understand the role they each have in making a difference to the outcome of these programs, and know when there might be opportunities to join the program in the field and participate in those more fully than has occurred to date (for example, school inspections and training).

AusAID has a role in supporting ongoing dialogue with the provincial administration, particularly where this is proving challenging for program implementers (e.g. ICDP). This also applies when going into the field, in an effort to diminish the repeated comments of not knowing what a program is doing.

Still at a programmatic level there must be improved efforts to achieve greater coordination of the different stakeholders’ activities. This particularly applies to the KDP, and for future remote area programs should there be a number of actors “doing their thing” on the ground. While there are current mechanisms that should support this coordination (for example the KI PMC) it does not appear to be achieving the desired outcome.

Responses are needed both from AusAID and by program managers/implementers. A well-developed Communications Strategy understood and adhered to by all stakeholders is essential. This is largely absent from both programs. Such a strategy will have a number of dimensions to it, aimed at different target groups: There needs to be an intense and innovative

program of work with government for them to understand, and remember, how the program is rolling out. These programs need to ask: what must we do to get government interested, attending briefings, remembering what we say we are doing? Some avenues for this now exist, but are not necessarily working (or available in every province), for example the newly established Provincial Coordinating and Monitoring Committee (PCMC) in Central Province. Existing avenues need to be used more effectively or other avenues found.

The implementation team must play its part during field visits to promote their presence and the activities of the program. It was the impression of the review that this was working quite well for ICDP with CARE field workers in the field more often than not.

Funding to support service implementation

Increased government funding through the LEAD Component of ICDP is unlikely to be achieved without addressing the key blockages to funding flows. This is discussed in detail in the sustainability discussion below.

Cost of accessing markets and providing services in remote areas


It will be difficult to realise the expected ICDP EARN outcome of increased incomes to families, particularly in the very remote areas of the district, unless roads and bridges and/or the cost of flights to get access to markets are addressed. The ability to fully engage in livelihood initiatives and reap the benefits of improved incomes has not yet happened 18, even though some income has been realised through sales of some products at local markets. In this respect the design of the program has not adequately addressed the context of the environment in which it is working. These are enormous barriers to achieving the stated objectives. Those communities that have better access, for example Ubo, are far less affected by such challenges. The program has to raise these issues at the highest level of the provincial government to seek support that the ICDP cannot provide.

The cost of getting services to remote, geographically challenging environments is high. This includes the cost of transport (air, freight), the challenge of bad or non-existent roads, and the absence of regular scheduled air services, to name a few. This impacts on: overall cost to the program if the implementation team are to be out in the field regularly; the ability of key stakeholders to be part of the program, for example to do supervision,

inspections, etc, and to take on their sectoral responsibilities in the longer term; and the ability to retain staff in these areas due to the cost to staff to get to centres for services,

banking, etc.

AusAID addressed this initially in KDP by subsidising regular air services into the area (a service that was cancelled after six months jointly by the operator who withdrew from the area and by AusAID, following a serious crash). This initiative very early on appeared to be having the desired effect of increasing traffic in and out of the area. A precedent has been set.

ICDP is occurring in an area where road access could potentially be improved, but the program is not successfully engaging with politicians to get this addressed, in a way that could lead to more appropriate use of some sources of funding (for example the District Services Improvement Program (DSIP)). This is hampered by difficulty engaging with the very top levels of government. Given the situation with coffee spoiling very quickly in storage houses in Andakombe, for example, if there is a solution it must be identified sooner rather than later.

Issues of access are best analysed before moving into new locations, and the reality of achieving desired outcomes due to these barriers discussed openly with beneficiaries during community consultations.

Inadequate timeframe

The timeframe for ICDP will likely be too short, even with the proposed one year extension, to address complex issues of funding and longer-term access to markets and services in EHP alone. These include the need for provincial government (the Provincial Administrator, Governor) to come to the party and the need to address the critical lack of capacity at all levels of government to match their enthusiasm with action. AusAID and CARE should consider a more realistic timeframe to meet objectives before expanding the activity to new provinces.

Are programs on track?

Taking into account the above discussion on effectiveness, it is the view of the review team that KDP is on track to meeting its retained objectives (outlined in the JU) with the exception of

18 Work is now underway to experiment with approaches to coffee production to value-add, such as decrease weight and increase quality may yet reverse the situation at time of review.


infrastructure. It is expected that current projects will be completed by December 2012, with the exception of Naduri Community Health project works (see Annex 6-A).

ICDP is likewise mostly on track with the exception of Gadsup-Tairora LLG where the security situation has halted CARE activity in that location. Also, the increased income for coffee growers is yet to be realised (although expectations have been raised), with the high cost of getting coffee to markets a key barrier.

Appropriateness of program objectives

The foci of both programs are in line with GoPNG’s plans and priorities, including Vision 2050, which includes empowering the people through improved education and life-skills, enhancing the level of service delivery and basic infrastructure, and increasing trade volume. The GoPNG Medium Term Development Strategy (MTDS) together with its Poverty Reduction Strategy (PRS) also identify the government’s responsibility to provide a number of core functions and basic services, improve governance, rural development, poverty reduction, and increase empowerment through human resource development.

As articulated in ‘Making a Real Difference’ – the Australian Government’s response to the Aid Effectiveness Review – AusAID is increasing its focus on poverty reduction. As such, a focus on getting services to the most disadvantaged communities is in line with AusAID’s mandate. In PNG these are by definition remote communities.19

Both programs are also clearly in line with more recent statements about Australian support to PNG20 calling for a greater focus on producing results in the education and health sectors in particular, in addition to maintaining funding to the transport and law and justice sectors, and taking a medium to long term view to achieving tangible lasting results. Importantly, this statement stipulates that results need to be demonstrated at the output and outcome level, and that Australian assistance be provided through avenues that will deliver services directly, at the same time encouraging more bottom-up approaches to service delivery through targeted community based interventions.

The approach both programs take of working with communities to improve service delivery directly supports Australia’s Democratic Governance Strategy21 which is underpinned by two key principles:

Principle 1: Support state and civil society to work in partnership (with one another) in order to strengthen their complementary roles and responsibilities, thereby facilitating political stability, service delivery, and sustainable economic development.

Principle 2: Work at as local a level as possible in supporting civil society, the sub-national levels of the PNG state and, where relevant, private enterprise, to articulate demand for democratic processes and institutions, and strengthen their capacities to meet that demand and become drivers of change.

The Strategy further states that there is:

19 Allen J, Bourke R M and Gibson J ‘Poor rural places in Papua New Guinea’, Asia Pacific Viewpoint, (2005) Vol. 46, No. 2, pp 201-217.20 Repositioning the PNG Aid Programme - Implementation Priorities 2011-2015, AusAID, 2011.21 Democratic Governance Strategy in Papua New Guinea, AusAID, August 2007 (updated 2009)


… shared governance, in which civil society, democratically elected local government and the private sector, together with the national government (the state) participate in public matters including policy making and implementation, resolution of public problems and the allocation of public resources. It involves working directly with communities as citizens, engaging with the ways in which they can both influence development and call for accountability.

Further:AusAID’s Democratic Governance Strategy’s approach in PNG involves civil society and the state both demanding and supplying better governance that includes demonstrating democratic values such as participation, legitimacy, equality, transparency, accountability and responsiveness. Support for democratic governance will facilitate relationships and constructive interactions between democratically managed social agencies – public and private, profit and not-for-profit – all which reinforce each others’ competences in order to deliver services effectively and alleviate poverty.

Both programs’ objectives reflect community needs. Early information collected to inform the nature of the programs ensured that the needs of beneficiaries, including communities, would be addressed. For the KDP, the initial and subsequent audits22 sought input from communities on service delivery needs, as well as what was needed to reinstate government services not being implemented by the provinces. The broad implementation plan clearly identified community engagement and agreements as part of the program. This has been adhered to throughout implementation. For ICDP, the location and objectives were informed by evidence from PNG poverty studies indicating that Obura-Wonenara is one of the poorest populations in PNG. The ICDP baseline study confirmed this and was used to inform how CARE would work with government to improve services and livelihood opportunities. Ongoing community consultation has ensured the needs of beneficiaries are being addressed in both programs.23

22 Joint Papua New Guinea – Australia Kokoda Development Program Needs Assessment Mission May 2008 Report; Naoro Catchment Audit Mission Report: Health Nov 2008; Naoro Basin: Mini Audit Report – Kairuku and Hiri Districts 2008; Kokoda Development Program Audit Mission Report of Alola, Isurava, Hoi and Kovelo Villages 17th – 24th February 200923 However, the review noted potential problems with initial KDP activities (through DGTP) where community consultation with women (had it been undertaken as part of a comprehensive training package and including a sound monitoring process) would have identified that women’s needs were not being met and activities were not reaching their logical conclusion. ‘Training’ was used as part of a community engagement strategy – to gain trust and buy-in from communities. They designed as one-offs (with follow-up 6 months later to make sure things had happened). KTA then implemented a livelihoods program, and were provided with details of training provided by KDP.


Recommendations for ICDP and KDP to improve effectiveness:A. Improve M&E systems and frameworks to enable regular collation of accurate data,

improve reporting and facilitate provision of information on progress to stakeholders. Programs need this for ongoing monitoring and analysis of what is working, or not working, and to support rapid responses if needed. In particular, phase II of the KDP should:(i) develop a robust ‘design’ and M&E logframe;(ii) ensure objectives and outcomes are clearly articulated in the design, with indicators

identified for their measurement; and(iii) clearly delineate roles and responsibilities of the AusAID program manager and

implementation personnel.B. Improve communications by:

(i) continuing engagement through other mechanisms as appropriate (for example, KDP should continue to improve coordination by attending monthly PMC meetings for the KI, to share progress and present projected annual roll-out of activities for KDP);

(ii) developing a communications plan for each program that describes the processes for ongoing communications and regular sharing of information with stakeholders (including beneficiaries) regarding progress; and

(iii) updating and enforcing protocols for entry to program sites that maximise communications with LLGs, members of parliament, etc.

Recommendations for AusAID on effectiveness:1. Ensure all designs undergo rigorous peer review, irrespective of pressure to implement

quickly. If not feasible, AusAID should place strong emphasis in the first year of implementation on developing a sound program logic and M&E framework.

2. Ensure less experienced staff are provided with adequate support, training and supervision to develop program logics and M&E frameworks of an acceptable standard, to manage implementation, and to maintain robust financial and monitoring systems.

3. In consultation with partner governments and implementers, identify underlying critical barriers to effectiveness at the design stage and jointly discuss how they will be addressed.

4. Ensure M&E systems and frameworks facilitate regular collation of accurate data, reporting and easy provision of information on progress to stakeholders.

Further suggestions for AusAID:I. Improve communications, including:

(i) devise strategies to achieve ongoing dialogue with partner government and identifying likely topics (i.e. risks to the program) at the concept / design stage;

(ii) institute annual meetings between AusAID and relevant provincial governments to discuss performance / outcomes, constraints to progress, remedial actions, etc; and

(iii) engage with provinces and ensure program managers can participate in appropriate provincial fora to share progress with key stakeholders.— This may be the PCMC, PMT or whatever is in place; most provinces have some

mechanism via which this can occur.— This can be facilitated by AusAID where there is an SNS provincial presence.— Where such avenues for communication are not functional, the challenge may

persist – therefore programs need to be creative in their efforts and approaches should be articulated in design documents.

II. Through the PNG transport program, undertake an analysis of the potential to engage with the private sector to overcome barriers for another national flight service provider to enter (or re-enter) the market.


EfficiencyEfficiency can be defined as a “measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results”24. The TOR asked: How were these programs designed for optimal value for money? What does it cost to deliver remote services? What are the opportunity costs to these expensive, geographically targeted programs? What are the implications of scale up on efficiency?

Cost of remote service deliveryThe cost of remote service delivery will always be more expensive (transport, freight, security) compared to supporting programs in areas that are easy to access. While cost data related to freight/travel and security are difficult to tease out for these two programs, both programs estimate that approximately 20 per cent of the total budget of these programs could be attributed to these costs. Ongoing support to remote services including staff support, for example, inspections and delivery of supplies, will always cost more.

Factors that contribute to efficiency

A whole-of-system approach

ICDP and KDP took a ‘whole-of-system’ approach to their programs. This approach results in priorities across all sectors being identified for support, not just priorities in one sector (for example in education, health or livelihoods alone). It goes further than an ‘integrated’ community development approach25, but is not at odds with that approach.

This was deemed to be an effective and efficient approach to adopt when working within small remote populations, and one that could contribute to greater coordination of sectoral efforts. There are a number of reasons for supporting this approach. It maximises overall effectiveness in all sectors and acknowledges the complementarity and dependence between the sectors – better health means greater participation in educational opportunities and income generating activities; improved transport infrastructure is essential if the objectives of income generating activities are going to be realised; increased income means greater opportunity to afford other essential services, such as health and education services. The benefits, for example, from a very strong education program will be thwarted if not matched with strong health and income generation interventions.

24 Organisation for Economic Cooperation and Development – Development Assistance Committee (DAC) terminology. See www.oecd.org/dac/evaluation.25 This approach is in line with the Integrated Community Development policy of GoPNG. As stated in the AusAID DCD Policy of July 2007, Integrated Community Development is development that: is owned and driven by the community; empowers people to drive their own development; addresses the needs of families in a holistic way; and is inclusive, addressing the needs of vulnerable groups i.e. women, children, youth, the elderly and the disabled.


Income generating opportunities for families supports education opportunities: After so many years we now have an elementary school in Simogu for the first time, and my children are going to school. They have not had this opportunity before. But even through education is now free under our government, we have to pay ‘project’ fees to the school. If my family doesn’t have the opportunity to get income then we won’t be able to pay the fees. My children won’t be able to get educated.”

Mother, Simogu village, Yelia LLG (ICDP)

http://www.oecd.org/dac/evaluation

“… I have one car … I need a program for all the sectors, then look at all (sectors) using the one car is efficient. Don’t want one sector becoming independent. Collective effort is better – there is better use of our resources, some feed off others, prompt others to work, complete tasks … put all our resources together.”

KDP, Kokoda LLG

The whole-of-system approach contributes to efficiencies of scale, given the remoteness and usually small populations, where the cost of access to these communities suggests it is best to make the most of the development effort by combining sectoral interventions. Separate sectoral programs in these relatively small communities are likely to result in fragmented programs and likely dissatisfaction by beneficiaries, and state and non-state stakeholders/partners who could be pulled in different directions where separate programs are being implemented, stretching limited human resources and jeopardising commitment to the process.

Overall this approach was also was deemed to be one that could contribute to greater coordination of sectoral efforts. It should be noted that the KDP had some challenges to a coordinated effort in the early stages due to fragmented management arrangements. These challenges have since been rectified with new streamlined management now in place.

In short, it is considered by the review team that a whole-of-system program approach is more likely to achieve the goal of “…maximum impact, effectiveness and value-for-money.”26

Flexibility

As a program rolls out it often becomes apparent that a different approach is required to achieve outcomes. Programs need flexibility to change direction. Both programs have been able to modify original elements of programming to achieve greater effectiveness, and not waste time pursuing avenues that could be considered inefficient and counterproductive. The discretion to do this was not obstructed by the often convoluted processes imposed by donors to make changes that are outside of the planning cycle.

KDP regularly re-visited its plan for implementation and was supported by AusAID to change direction if the need was justified. ICDP saw its program design as one that would evolve as the program rolled out, and was iterative in nature as new needs emerged. Both these approaches are not uncommon in AusAID programs for some years now, and are characterised by annual planning processes where components of programs are re-visited and evaluated in terms of continuing relevance and what is appropriate for continuing support.

26 Repositioning the PNG Aid Programme - Implementation Priorities 2011-2015, AusAID, 2011.


Barriers to achieving value for money

Fragmented management arrangements

Optimal value for money was more likely with the management and implementation arrangements of the ICDP, and less likely in the KDP with its initial management and implementation arrangements. ICDP came under the management umbrella of CARE and created a cohesive program with strong coordination of efforts in the field. Getting to the field is expensive and uncoordinated arrangements can only add to the cost of implementation.27

KDP’s initial fragmented management arrangement impacted adversely on the effective coordination of implementation efforts. It posed extra unnecessary burden on the program manager to consolidate reporting upwards when individual team members were reporting separately (and differently on expenditure of funds) to their respective contractors.

Addressing supply and demand in the service delivery equation

For an intervention/program to be successful both demand and supply side28 elements of the service delivery system need to be addressed adequately.29,30 The programs to date address some elements on both sides of that equation, but not all. Table 1 below identifies some agreed elements of the supply/demand framework. It is a useful checklist for program designs to use. The risk of course is that if all elements of this supply/demand framework are not simultaneously being addressed then the likely impact of the program itself will be compromised, negating any value for money. This does not mean to say that the program itself must do it all. But using this framework allows the donor and implementer to jointly identify whether the program itself, or some other entity, is taking responsibility for each of these elements.

Table 1: Supply and demand side elements to be addressed

Supply side Demand sideCategory Addressed

in design (yes, no)

Responsibility (program,

other)

Category Addressed in design (yes, no)

Responsibility (program,

other)Service delivery Demand side financingWorkforce User behaviourInformation systems Participatory approachesAccess to essential supplies

Demand side accountability

Financing Human rights/equityLeadership/governance Multiple

sector/stakeholders

Neither program looked fully at all elements in this framework, to identify where there are gaps that will impact on the likely success of the program, and how/if those gaps are to be

27 KDP recognised this problem early on and required that any charter flights be approved at team meetings, to improve coordination in this regard.28 Definitions: Supply side - service delivery inputs such as human resources and supplies provided on the basis of formal sectoral planning by technical planners and managers. Demand Side - behaviour and inputs of the recipients or intended recipients of these efforts: individuals, households and communities.29 Everybody’s Business: Framework for Action, WHO, 2007.30 Standing H, ‘Understanding the ‘demand side’ in service delivery: Definitions, frameworks and tools from the health sector’, London (2004) DFID Health Systems Resource Centre


addressed. At design stage AusAID had not considered this. One significant gap is supply side financing, i.e. the flow of government funds. No amount of effort toward any of the other elements will realise the expected results if the flow of funds does not occur.

Technical issues – quality of inputs

The KDP had a mandate to demonstrate tangible outputs quickly. Hasty implementation was paramount. This has impacted on technical quality of inputs, and weaknesses have been compounded by the fragmented management arrangements whereby there was not one key point of accountability.

Just one area that is considered to have been a casualty as a result of this is sanitation efforts of the program. The sub-contract was issued quickly, and paid in full before work was completed and assessed. The KDP does not have sanitation expertise in its team to make adequate assessment of the quality of latrines before a final payment should have been made to the sub-contractor. The review saw many constructions that would have contributed to health problems rather than diminish them.

One-off trainings were conducted under a community engagement strategy. The KDP recognised that people along the track had long been promised things which had never been developed, causing distrust. Training opportunities identified in the initial audit reports were undertaken as a means of building trust with communities. Nevertheless, such one-off training without any follow-up is contrary to the best wisdom of capacity development and should not have been supported. Dissatisfaction of beneficiaries was expressed clearly to the review team.

These issues must be considered and addressed in further programs, and are summarised for the record: Risk of advance payments coupled with contractors failing to perform; Absence of performance guarantees; Absence of systems and necessary technical expertise for contract monitoring; Lack of accepted contract administration practice; and Insistence on best practice for all capacity development (training) initiatives.

Program design and weak monitoring systems

A detailed design document describes the process of achieving outcomes. A monitoring system supports assessment of progress toward those outcomes. Continual assessment and reflection on progress allow the implementers to identify how they are travelling toward meeting their goals. A good test of the efficiency of these systems is the speed with which a program can provide summary evidence of progress to a review team. Both programs under review had to go to some considerable effort to do this, and the KDP team itself could not reach agreement in one instance.

While the individual programs that made up the KDP each went through design processes and had M&E frameworks, this did not negate the need for a robust design process and M&E framework for the KDP itself, which would have enabled greater coordination of inputs and reporting on outputs and outcomes.


Implications of scale-up on efficiencyScale-up may have some impact on efficiency. ICDP is looking to expand, with CARE implementing in a second province and a local NGO implementing under the guidance of CARE in a third.31 There are some risks to efficiency. One relates to the extent to which the ICDP ‘model’ has been developed and fine-tuned. Hasty expansion before this has occurred would be unwise. CARE will need to explain clearly how it is taking the lessons of Obura-Wonenara to these new areas. Once this is demonstrated, scaling up may not reduce efficiency where key approaches, materials and staffing have been established. However, CARE may run the risk of spreading itself too thinly. Implementation by a less experienced local NGO, while being proposed for valid reasons (i.e. to improve sustainability by supporting a local organisation and to work in remote areas that are difficult particularly in terms of security issues), will need significant support from CARE.

ICDP scale-up should not happen without an ‘independent’ assessment (by an independent consultant, or the SNS team) of Provincial Government buy-in. The true commitment from the Provincial Administration needs to be ascertained at the start.

KDP is already planning expansion to new “off-track” communities.

Maintaining or improving efficiencies with expansion will only occur if the following prerequisites are met:

(i) Programs are refined to improve implementation. Expansion should only occur when the models have been ‘fine-tuned’.

(ii) Engage Provincial Administrations so they can be involved in the scale-up/expansion.32

(iii) Improve communication to ensure the lessons on what works and what does not work are available and well-understood.

KDP should prioritise finishing incomplete work along the Kokoda track before expanding to other areas to ensure the KDP team is not spread too thinly and is able to complete quality construction and renovation work, follow-up on training and apply lessons for achieving this level of success to new areas.

A robust ‘design’ and M&E logframe should also be developed for the new phase. Improved coordination with other key stakeholders (KTA, KTF, DEC) would contribute to better value for money and efficiencies.

31 This expansion is based on the amended design of November 2011 and is a reduction in the rate of expansion from the original design, which stated it would work in four districts.32 Both programs are demonstrating an improved approach to this with proposed scale-up, for example arrangements for ‘audit’ visits in new areas of Oro was agreed between the KDP and Oro Provincial government in February 2012.


Recommendations for ICDP and KDP to improve efficiency:C. Ensure due processes are followed to get the best quality of work done on the ground.D. In relation to sanitation in the KDP program:

(i) engage the services of an independent sanitation engineer to assess the quality of work done in the current KDP program and make recommendations to improve the current state of latrines if appropriate;

(ii) before any further latrines are installed, seek an independent analysis of the range of options for hygienic, value-for-money latrines; and

(iii) identify quality assurance measures to be applied to all future latrine construction.E. If a less experienced local NGO is engaged to implement the program in a new area,

CARE should ensure that significant support can be provided to that NGO.

Recommendations for AusAID on efficiency:5. Support flexibility and institute efficient mechanisms for greater responsiveness.6. Consider whole-of-system approaches and improve coordination between sectoral

programs for activities in remote areas.7. Avoid fragmented management arrangements, ensuring all programs have clear lines of

reporting and accountability.


SustainabilityIt is too early to make definitive statements about the likely continuation of benefits from these activities. Nevertheless, some features of each activity suggest sustainability of limited aspects is likely.

Factors that have promoted sustainability

Working within government guidelines and procedures

Both programs are working within government guidelines and procedures (e.g. registration of schools, teachers and health workers; staff appointments). This ensures that these improvements sit firmly within government and will not be impacted on once the intervention ceases. Likewise, all training has followed standard government guidelines.

Expressed government willingness to progress initiatives

In both activities, there is an expressed willingness by sub-national government counterparts to progress program initiatives. In Obura-Wonenara, the district government expressed a desire to see the outcomes of the Ward planning initiatives. The Central and Oro Provincial Administrations and local level governments also indicated that they would like to apply lessons learned from the KDP in other remote areas.

Comprehensive approach to capacity development

Both programs have adopted a comprehensive approach to developing capacity. Training has generally not been a one-off single training activity33, but rather has been provided in modules to reflect the different elements of capacity needed for beneficiaries to be fully skilled to undertake new or renewed tasks, with some evidence already of success (see text box, Annex 6-B (f) and (g); and E, and Annex 7-C).

33 With the exception of some trainings under the Community component of the KDP – discussed further below.


Fish Farming:“Fish at the front door means nutrition and income”

Fish farmers indicate they are confident they can pursue fish farming without further external assistance. This success can be attributed to CARE supporting a comprehensive set of activities to develop fish farming capacity, including: providing three different levels of training; follow-up to trainings; field studies to assess effectiveness; and providing start-up fingerlings.

Partners with specific expertise in aquaculture (Backyard Farmers) were engaged to provide training and support, and worked alongside ICDP in these capacity development activities.

Before, eating fish only happened for the people who knew how to catch fish from the 2 big creeks around. Fish were not easily available to many in the past. I was in the Village when I heard one day that fish training was going to be conducted in Andakombe station. CARE PNG asked community members to give names, and so I gave my name and began working on the fish pond. When the first training occurred in the community, fish fingerlings were brought in and distributed. Now I have 2 big fish ponds with thousands of fish in them. I am expanding and I have dug a third big fish pond. I will be stocking fish fingerling in a few years from now. The big fish ponds are full of big and small fish.

- Papu’s story (full story at Annex 9)

Support under ICDP for growing better or new crops has been successful (e.g. coffee, potatoes, African yam), although challenges are particularly apparent for coffee growers – see text box.

There is also evidence in both programs that some (though not all) capacity development activities are being followed up to understand the impact and identify problem areas, and respond as needed to redress limitations.

Engagement with women

Finally, both activities have also ensured engagement with women in communities. The ICDP has clearly included and empowered women, evidenced by including women in every WDC (through a voting process), giving women a voice in decision-making. Women actively participated in community discussions with the review team, including in mixed groups.

The KDP has also supported activities to empower women. Women in Kokoda Track villages stated that their husbands were happy for them to speak in public gatherings. There were some ‘one-off’ training activities for women - by design and implemented through DGTP - not including any follow-up and support. Consequently some of these early activities resulted in unfulfilled outcomes. The women nevertheless expressed that they felt more empowered to get things done for the community, and demonstrated exceptional resilience and persistence in the face of resistance from local-level government representatives (see text box on following page). This in itself demonstrates an achievement. However, it also indicates poor conceptualisation and design in the first instance where accepted approaches to capacity development have not been adopted i.e. follow-up and support.


Coffee Rehabilitation Program:Under ICDP’s EARN Component, the Coffee Rehabilitation Program provided training (coffee marketing and value chain study) and coffee mobilisation activities. Highlands Coffee Exporters and Kumatega Coffee Development Services partnered with ICDP to deliver these activities. This comprehensive program enabled >80 coffee farming households in the Simogu cluster (comprising five wards) to increase yield and improve quality of coffee cherries from old coffee trees.

Training in coffee tree pruning improved efficiency. Distribution of coffee pulpers drastically decreased processing time and increased the quantity processed for marketing. Simogu women agreed that what once took them weeks or months now takes them only hours. The training also taught participating farmers (men and women) how to establish drainage systems for coffee gardens to reduce soil erosion and allow coffee trees to grow better.

Access to markets

Getting coffee beans to market remains a major challenge for farmers due to high freight costs, limited air services and unfavourable flight schedules. The challenge is heightened where access is by plane only (e.g. Andakombe, Simogu).

A few Simogu farmers have been transporting very small amounts of coffee by foot to Okapa then by PMVs to Goroka, but the majority of coffee still remains in community storage areas, where it will likely deteriorate.

Without better access to markets, the benefits from producing greater yields of coffee will not be realised, expectations will have been raised, and frustration will be ongoing.

Factors to consider for sustainability

Despite these successes, greater consideration for these types of programs needs to be given to the three major factors that influence sustainability:

1. Availability of government funds to deliver services.2. Appropriate allocation of funds for service delivery.3. Local capacity to deliver services.

Capacity to deliver Medium Term Development Strategy (MTDS) priorities34 is detailed in the Provincial Expenditure Review (PER)35:

34 MTDS sectors: health, education, infrastructure maintenance, agriculture and village courts.35 “Step Two: the Ripple Effect” The 20102 Provincial Expenditure Review, NEFC, 2011.


Inclusion of women:“There is a lot of talent on the track not being tapped into by KDP”

While District Women Facilitators (DWFs) worked with communities to do needs assessments and carried out awareness to maximise community participation and ownership, some felt there was more that women could contribute.

While absence of follow-through on some aspects of training could adversely affect sustainability, it was noted that the Leadership and Organisational (governance) training resulted in some women feeling more empowered to get things working to the advantage of the community. They demonstrated empowerment to the extent that they did not give up dealing with a very important financial issue despite challenges from male (government) leaders while being supported by community chiefs and husbands.

Identification of women to participate in these trainings meant that strong women, some with backgrounds in the national government Community Development department, were given this opportunity and were pursuing opportunities to benefit the community.

Women were equally vocal in community meetings held by the review, and had strong views which they expressed freely.

The lesson learned from this is that capacity development initiatives (training) need to be robust enough to withstand the challenges that confront women when they try to use new skills. The responsibility is with the program to ensure the design of those capacity development efforts is such that beneficiaries will have been supported beyond the actual training events. This should have been factored in to the initial design of these activities.

Availability of government funds – ‘the funding gap’

In the provinces of interest (Oro, Central and Eastern Highlands), major funding gaps exist. Central, EHP and Oro have a 30-50 per cent shortfall of funds needed to deliver a minimum standard of services. However, functional grants have been increasing in each budget since 2008 for Oro and Central Provinces and are now up to around 60 per cent of what is required. If this increase continues it is foreseeable that by the KDP is complete, around 80 per cent or more will be provided through the budget.

While historical data on allocation of funds to the provinces indicates a situation that may be improving, there is no evidence that funds at the provincial level will flow to the district, LLG and community level. This binding constraint to better service delivery has not been adequately addressed in either program. ICDP assumed that funds would flow (an expected outcome of the LEAD component) and KDP did not address the issue. However, KDP are now considering strategies to address the provincial funding systems and improve the funding flow.

Allocation of funds – ‘the priority gap’

The priority gap indicates the extent to which priority areas are being addressed. Based on 2010 spending, EHP, Central and Oro have a priority gap between 65-80%, indicating that all three provinces perform poorly in terms of allocating funds to priority areas.

Priority spending gaps can be disaggregated down by sector:

EHP Central Oro

Education 80% 50% 50%

Health 60% 40% 70%

These data indicate that these provinces are not in a position to allocate appropriate resources to deliver a level of services that KDP or ICDP are able to deliver.

The KDP is a particularly unique program, given its ongoing political importance to the Australian government. It is extremely unlikely that Central and Oro provinces will divert scarce resources to this area, which is considered to be relatively well-off. One interviewee noted:

“ … the area is not poor, and if we don’t do it the Australian government will.”

The long-term focus and support given to this area has undoubtedly resulted in local complacency, apathy and donor dependence. The KDP team are aware of this and are considering strategies to improve this situation.

Local capacity to implement

ICDP is working in one district of the EHP with a total population of about 47,000. It has 22 staff to implement its program with an average budget of AUD 1.6 million/year. By comparison the provincial Community Development Division – the division that would be responsible for this program - has two staff and an annual budget of AUD 385,000.36

36 These figures are rough estimates for the purpose of making broad comparisons.


A similar comparison of the KDP to provincial budgets and staffing is not possible given the nature of KDP. The KDP includes seven staff, three of whom are dedicated to one-off tasks (infrastructure, logistics and a receptionist) which would not need to be continued. The cost for one-off items, such as transporting building materials, would constitute a large portion of the budget, and recurrent funds required would be much lower.

KDP and ICDP are resourced generously to deliver their respective programs in a limited number of districts (although the review notes that KDP includes one-off ‘kick-start’ expensive activities). By comparison, the provinces of interest have far less capacity to fulfil their responsibilities to the whole province, let alone take on these districts. Even if resourcing levels were comparable in government, a further consideration has to be the motivation level of staff to implement – an important component of capacity.

Given these provinces are failing to allocate their budget on key service delivery activities, and basic services do not adequately reach even accessible areas, it is highly unlikely that they have the capacity to take full responsibility for neglected, much more difficult to reach, remote areas. This has important implications for AusAID, as continuing with programs like the ICDP and KDP clearly address AusAID’s focus on poverty37, but will likely fall short on sustainability. In continuing to address the acute poverty in these remote areas, AusAID must recognise that the services delivered through these interventions are far beyond what provincial governments could hope to deliver. This runs the risk of setting unrealistic expectations.

Improving sustainability outcomes

Leaving the first factor (availability of government funds to deliver services) aside, addressing the allocation of funds and implementation capacity requires a two-pronged approach. Firstly, a long-term approach to fix the allocation and flow of funds is required, working with and improving provincial and local facility level systems. At the same time, improvements can be made in service delivery by focusing on implementation capacity. This second prong is not only an enabling factor that will support the longer-term goal of appropriate allocation of funds; it will also have immediate, motivational impacts.

First prong: improving government systems

Better engagement with provincial, district and local level government

Engagement with government has not been sufficiently successful in either program. Greater government buy-in is required if these programs are to continue without ongoing external support. This poor engagement is not always through want of trying.

At the time of review ICDP had gone to great efforts but ultimately had not achieved key (top level) support of the Provincial Administration. This was particularly challenging due to the political complexity of the EHP. Sector Divisions by contrast are fully supportive and engaged, as are Districts and LLGs, even if capacity can’t match enthusiasm. The linkages with Sector Divisions has not been formalised in the establishment of a project steering committee. ICDP had intended to present the program formally through the Provincial Management Team meeting; however this team has not met for the past two years. Alternative strategies have been attempted to finalise the process of forming the committee, but with no success (from ICDP 6 monthly report, June 2011).38

37 As set out in the Government’s response to the Aid Effectiveness Review.38 It is understood that since the review field work and review feedback to ICDP some progress is now being made.


“When we need help, we go straight to [KDP], we don’t go to our government”

Teacher along track

“Dispela area mipela callim no gavmen area …”

Community rep (Naduri)

“… all stakeholders must be partners together.”

Kokoda District Official

In the KDP, representatives at all levels of government complained of not being involved or informed of the program’s activities. The review was informed by the KDP of their significant efforts to be inclusive of government as the program has rolled out.39 KDP has protocols to guide staff when visiting communities40, although KDP staff stated these aren’t adhered to routinely. The disparity between feedback from government

and KDP to the review was difficult to explain in the short timeframe of this review, but it may have stemmed from the initial lack of inclusion of government in defining the nature of the program, contributing to an undercurrent of dissatisfaction that had the opportunity to be aired during this review.

Where adequate engagement with the provincial government is not possible, AusAID should consider whether the intervention should proceed in that province, and consider alternate locations – there are plenty of remote, poor areas of PNG, and other provincial governments may be more open to effective engagement. This engagement should occur at the start to establish this crucial relationship.

In the absence of adequate engagement with provincial government, there may be other alternatives to sustaining service delivery at the end of a donor program. There are examples elsewhere in PNG of private companies supporting provision of some basic services, particularly in resource rich areas.41 The likelihood of a private entity emerging and taking over in the Obura-Wonenara District is very slim, and there is as yet no other feasible money stream to support the program.

Communities along the Kokoda Track are in a different position, given the important legacy this area holds for Australia, the frequency of trekkers, and ongoing support from organisations such as Rotary, Lions, the KTF, etc. While there are weaknesses in the way support is provided by these latter groups, the KI could take a more active role in coordinating these other efforts. The Australian government will likely maintain a strong interest in this part of the country, and there is much philanthropic work for these communities. Better coordination of this work would provide longer-lasting benefits (but beware the apathy it creates).

Assistance in Ward Planning (at the Ward, LLG and District levels)

Koiari LLG has produced a ward plan for 2010 to 2014. The last Kokoda District Plan expired in 2007. Assistance for Ward planning is provided by KTA under the KI, although this appears to be only for the purposes of the KTA program, and not for the broader purposes of developing capacity for overall planning. The KI should address this issue and consider what role the KDP might have vis-à-vis the KTA in supporting Ward Planning.

39 Includes: meetings with provincial health; health reports on activities sent to provincial health Advisor and district health office; annual leadership and management workshops with the provincial and district education staff both provinces; limited participation in patrols and training in education; registration and staffing of facilities (health and education) is done by the provinces – government by definition is involved.40 Protocol includes: letters, radio calls, notices on community notice boards, visits to Standards Officer, LLG Executive Manager, District officers, though visits aren’t always adhered to (KDP Implementation team interview).41 Though this is sometimes only in the immediate vicinity of the resource activity to benefit company staff and does not necessarily extend to communities beyond this boundary.


While some excellent work on Ward planning has commenced under the ICDP, these Ward Plans fail to identify where funding will come from (although activities that don’t require funding sources are identified). The plans are thus aspirational and not realistic, but the ward planning process itself has raised expectations that are unlikely to be realised. These limitations could be partly addressed by modifying the ward planning template (developed under SPSN) and adding a column that identifies funding sources/responsibilities. The viability of ward planning is discussed further at Annex 10.

Address major human resources constraints

Workforce management is complex, and in PNG it is weak. The workforce in all sectors is ageing. Supervision and inspections are limited at best, with negative implications for professional conduct and performance. Staff are often absent.

These challenges are amplified where those trained (in health and education) are not from the area, and are thus not prepared for the reality of remote appointments. Both programs have focused efforts on training local people to improve staff performance and retention. A particular challenge with remote locations is that salaries can’t be accessed easily. The individual often needs to leave the workplace to visit the nearest town to collect their pay.

Trekking companies offer excellent wages for porters – an opportunity that many along the Kokoda track take advantage of, to the detriment of other sectors. High wages for porters act as a disincentive for local people to pursue careers in health or education.

Professionals in both areas work very much in isolation, given the remoteness, and this has to be a deterrent to remaining in these locations. Endeavours to develop professional networks for salaried staff in both education and health could prove to be of value in overcoming the isolation that is a feature of remote appointments.

Greater targeting of women to work in health and education may also be worthwhile as they usually don’t work as porters. In the health sector women in both areas spoke to the review about wanting better access to female health workers for their health care. This may in turn have a positive effect on increasing supervised deliveries.

Second prong: improving implementation capacity

Developing community demand and support for services

For the provision of fully effective services beyond a donor intervention, provincial support is required. Once effective provincial support is adequate, capacity at the community level is required to deliver some services and ensure that others are delivered adequately.

There are some discrete activities that communities can do for themselves, and this is evident in Obura-Wonenara where initiatives under the LIVE component are being implemented solely by the community (e.g. walking tracks, resource centre construction).

While the KDP appears to take a much more top-down approach to program delivery, it has required community contribution. The community initially identified the schools it needed and the achievements of the education component is testimony to this (e.g. eight new elementary schools). Community contributions were required for infrastructure activities and have been


documented in agreements. Membership of school boards and health facility committees are all community contributions encouraged and supported through KDP.

Both programs have trained BOMs for schools (SLIP program), which is the avenue for communities to demand and support improved education services for their children. Sustainability for schools is the ability to receive education subsidies (paid directly to schools). Functioning school boards (i.e. boards that hold meetings, have bank accounts, etc) make this happen.

Both programs encourage community ownership and support for health.

Follow-through to outcomes

Any hope of sustainability is not possible where an intervention has not yet completed its activities. For example, some activities under the KDP have not been completed, such as infrastructure of schools and aid posts and some training – see Annex 6 (A, B-f).

For some of these (construction, renovations, supervision of volunteers) local solutions by communities or relevant authorities are needed, and may take time. However, for training, even where all modules of initial training are completed, there is limited follow-up or refresher training. Human resources best practice is very clear about the need for this if practice is to be entrenched.42

Strengthened volunteer networks

If embedded and active within communities, the volunteer networks of VHVs and VBAs have great potential, even recognising that these positions are not intended to be a full time role.

In the ICDP the development of these VHV and VBA networks was strong and a sense of cohesion in the groups was evident.

VBAs are only required where villages are a certain distance from an Aid Post / Health Centre and as such they are not required along the Kokoda Track. The development of a cohort of VHVs under the KDP however suffered some weaknesses, for example, preparedness to only work in activities for which there was an allowance paid, thus not fulfilling the mandate of the VHV of working in the community, albeit in a part time capacity.43 Concern was expressed by a number of informant groups. Better supervision and coordination seems in order.

However, it should be noted that a successful outcome of establishing volunteer health groups has been that some volunteers in both programs are already interested in enrolling in training to become health workers.

Exit strategies within a reasonable timeframe need to be planned from the beginning

Ideally, programs will have from the beginning, in consultation with provincial governments, a clear exit strategy with appropriate timeframes. Such exit strategies are, in effect, ‘sustainability strategies’. Government informants believed this approach would have greater potential for sustainability to be realised, particularly if there is an agreed ‘phased’ approach,

42 Increasing access to health workers in remote and rural areas through improved retention: Global policy recommendations, WHO, 2010.43 As has been noted above, there is often family pressure on VHVs to earn money through the trekking industry, thus detracting from their ability to carry out volunteer activities as a VHV.


whereby the program with government identifies at certain points the expected handover of responsibilities to government that have already been commenced by the program.

The ICDP identified the need for a long term approach to reaching a sustainable situation, even conceding that CARE may still provide limited support (Annex 11 - A). While the ICDP design has a reasonable consideration of sustainability and is expecting to move toward this, the current timeframe for the project is inadequate to attain the goals that it has set. As the program is rolling out it needs to make assessments about where it is at in terms of government or communities continuing with these activities. That this will be done, and when, has not been articulated and dialogue with government and communities specifically on this issue doesn’t appear to be happening.

KDP does not adequately describe sustainability. A design document would have prompted consideration of this. The only sustainability statement for the KDP (found in the needs assessment of 2008) is presented in Annex 11-B.

The ICDP and KDP were both designed for five years or less. An adequate timeframe for a sustainable, remote, community development program would be at least 10 years.

An early assessment of the likelihood of sustainability is needed

By definition, remote areas will almost always be poor.44 There may be a lower likelihood of sustainability in remote places, with continuing dependence on outside support simply because it is so much more expensive for government to support service delivery in these areas. This will be compounded by lack of government commitment and motivation to make the effort to support these services (i.e. fly or walk in to provide support). There is thus a potential conflict with the need to focus on poverty and the likelihood of sustainability. These issues therefore need to be reconciled by both government and donor.

44 Gibson J, Datt B, Allen B, Hwang V, Bourke M, Parajuli, D ‘Mapping Poverty in Rural Papua New Guinea’, Pacific Economic Bulletin (2004) pp 1-26; See also Allen J, Bourke R M and Gibson J, ‘Poor rural places in Papua New Guinea’, Asia Pacific Viewpoint (2005) Vol. 46, No. 2, pp 201-217.


Recommendations for ICDP and KDP to improve sustainability:F. Before expanding, both programs should meet the following prerequisites:

(i) programs are ‘fine-tuned’ to improve implementation;(ii) limitations to sustainability are addressed and unfinished work completed;(iii) provincial Administrations are engaged and involved in scale-up / expansion;(iv) provincial government buy-in has been independently verified; and(v) communications should be improved to ensure the lessons on what works, and

what does not work, are available and well-understood.G. Consider more realistic timeframes to meet objectives before expanding activities to

new areas and ensure that ongoing dialogue with provincial government and communities specifically addresses this issue.

H. KDP should consider more actively targeting women to work in health and education, given that they are less likely to work as porters.

I. KDP should ensure better supervision and coordination of VHVs to improve the likelihood of VHVs continuing to work in the community beyond activities for which an allowance is paid.

Recommendations for AusAID on sustainability:8. Require a detailed analysis of the three major factors that affect sustainability

(availability of government funds to deliver services; appropriate allocation of funds for service delivery; local capacity to deliver services) is included in every design (or delivery strategy).

9. Use a supply/demand framework to guide designs (including risk analyses) and identify whether elements that are not the responsibility of the proposed program are being addressed elsewhere.

10. Jointly agree ‘exit strategies’ with partner governments, with implementation clearly outlined in design documents.

Further suggestions for AusAID:III. Use lessons from the past more effectively (potentially through a ‘lessons learned

database’). In particular, AusAID should develop a list of factors that are known (lessons from many evaluations and research) to contribute to sustainability and include this in design guidance.

IV. Given a potential conflict between the need to focus on poverty and the likelihood of sustainability, AusAID and partner governments should further examine and reconcile this complex issue.

V. Address the funding flow issue by:(i) considering this issue as a pre-requisite to working in remote areas in a province;(ii) seeking provincial agreement to include this dimension as part of a total package of

any remote services program;(iii) identifying the most effective way to make this happen; and(iv) consider limiting efforts to provinces where government engagement at all levels is

assured.


“… there was poor use of travel resources. We would bump into one another … on the track.”

KDP implementation team

“We have been given more responsibility, with more opportunities to go out to the communities … it is easier to communicate – this is now a good thing with better coordination.”

KDP implementation team

Program Delivery MechanismsThe review was tasked to: examine the strengths, weaknesses, limitations, risks, missed opportunities of the different

management models; compare management models with one another and with the typical managing contractor

model – in the context of remote service delivery and community development; and discuss the best intervention model to achieve stated objectives.

Direct management by AusAID – the KDP example

KDP is managed by a small group within AusAID across all sectors. The initial management arrangements (described in Annex 12) had sectoral advisers separately contracted and reporting to different entities (Education Adviser reported to BEDP (Coffey), Health Adviser reported to CBSC (JTAI), Community Development Adviser reported to DGTP, transport adviser reported to TSSP). The AusAID KDP program Manager’s role was to manage and coordinate work under these different contractors.45

This initial fragmented arrangement challenged the development of a cohesive team sharing a common view of the KDP.46 Team meetings were irregular which further jeopardised coordination. Where they were held, there were no meeting records. Each sub-contractor had separate requirements for reporting. Financial reporting for KDP was poor. This initial management arrangement contributed to some inefficiencies. For example, travel was often not coordinated, resulting in sectoral advisers taking separate charter flights at the same time, rather than travelling together.

There was also unclear delineation of roles and responsibilities between the AusAID program manager, the contractors and the implementation team.

Expecting cooperation between competing contractors may have been short-sighted, although the implementation team reported improvements in coordination when the team co-located.

There was a political imperative to quickly establish the KDP, and as such, some shortcomings are understandable. A number of audits47 identified the needs of communities along the track, and these audits were used as a substitute to a robust design. At commencement, no program logic

45 KDP did not manage the funds, but just reported on the expenditure. The individual reports were in different formats and often different currencies, so to provide a comprehensive report on expenditure for the whole program was difficult and time consuming.46 Note: AusAID has been advised in the past, in PNG, against management models that apply a fragmented model of contracting and reporting akin to the KDP – see Independent Evaluation of Tingim Laip 2007.47 Joint Papua New Guinea – Australia Kokoda Development Program Needs Assessment Mission May 2008 Report; Naoro Catchment Audit Mission Report: Health Nov 2008; Naoro Basin: Mini Audit Report – Kairuku and Hiri Districts 2008; Kokoda Development Program Audit Mission Report of Alola, Isurava, Hoi and Kovelo Villages 17th – 24th February 2009


with clear objectives was developed, nor a monitoring framework. The absence of these key steps in a design process also meant that the program did not undergo a peer review process that would have picked up on potential weaknesses.

Given the less-than-ideal management arrangements, it would have been prudent to appoint a full-time team leader with strong leadership, management and M&E expertise to rectify some of the initial weaknesses in the program.

The new management arrangement (at time of review), with the KDP falling under SNS, provides an opportunity for the KDP to benefit from the management support of a single managing contractor48 including better coordination, consolidated reporting, improved monitoring and oversight. There is however a risk that under the SNS the KDP will be further removed from the other KI activities, which may contribute to continuing weak coordination and communication.

Developing strong relationships with stakeholders is paramount to the continuing success of this program. While this has been challenging for the KDP, it is now showing more promise with closer consultation with both provincial administrations. Further strengthening strategic engagement at all levels of government is needed, with clear responsibilities for this.

NGO management model – the ICDP example

The ICDP concept was developed by CARE based on its existing ANCP funded activities. CARE submitted this concept to AusAID and in dialogue with AusAID developed it into a program design which it would manage in consultation with AusAID as a partner.

The design of the ICDP program was developed by CARE in consultation with AusAID from a concept that emerged out of CARE’s experience with its existing ANCP funded activities. CARE manages the program through a close partnership with AusAID. CARE has a contract with AusAID to provide annual plans and report six-monthly and annually against achievements as set out in those plans. In this respect it is not unlike managing contractor models. It has all the systems in place to undertake efficient management of the program and applies standard management processes in implementation and reporting.

CARE has a cohesive team working under good leadership. It uses a team of field workers from the program’s geographic area who have the respect of local communities and are in the field more often than in the CARE office.

CARE’s comparative advantage is in its ability to relate extremely well to these remote populations, gaining easy entrée to these locations and the trust of the communities.

However, M&E for the ICDP is weak. The peer review process should have identified the design as flawed in terms of its logic and articulation of SMART indicators in the M&E framework. An ‘evaluability’ assessment would have identified these weaknesses. The program has been implemented with a flawed program logic.

48 From July 2012 KDP is part of the new PLGP contract, which will continue on from the SNS program, and which underwent a robust peer review as part of the design process. Under this arrangement, all KDP finances are in one place and all requests for expenditure come through the AusAID Manager’s office for checking (that it is against the program of activities) and sign off, which SNS/PLGP then process.


Features of strong program delivery mechanisms

The management model, whether through a managing contractor or INGO, or managed directly by AusAID, does not in itself determine the quality of a program. Nor does it determine that the cost of one model is greater or lesser than another. Management fees are involved in all of them, and contrary to popular opinion, INGOs do not necessarily cost less than other models. There is significant variability across INGOs. The management model should also not influence the extent to which AusAID is engaged and has influence, with the exception of when a program is directly managed by AusAID where by definition AusAID takes a hands-on management approach. Years of experience show us enormous variation in the relationship between managing contractors and AusAID, ranging from very good to very bad, reflecting levels of trust between the two entities.

There are some criteria in design and implementation however that will influence quality. Key elements that underpin a good management, regardless of the model, are:

Element Comments

Design

Program Design Document This needs to be developed by a team with design and M&E expertise complemented by program and sectoral knowledge. Audits, baseline studies and other information should guide the design, but not substitute a rigorous design process.

Program Logic /Theory of Change

A design team needs to have expertise in developing program logic. A logframe will enhance understanding of all key aspects and the logical progression from inputs through to outcomes and impact. This will guide the M&E Framework.

M&E Framework A sound M&E Framework is crucial to good program management, as is adequate resourcing for ongoing monitoring and reporting against the M&E framework.

Peer Review A robust peer review process should use expertise in the country program, sector, program logic and M&E, at a minimum.

Implementation – management, monitoring, reporting

Program / team management

Sound management and leadership skills are essential for a team leader. A program should include an officer dedicated to M&E, or a team leader with M&E expertise.

Contractual arrangements A single head contract will support accountability, team coherence, coordination of activities, streamlined financial management and reporting. Sub-contracting, where required, can occur under this head contract.

Financial management A single, robust system is required.

Human resources practice Sound systems and procedures are required.


Element Comments

Ongoing M&E In line with AusAID’s focus on demonstrating results, the implementation team must be adequately resourced to undertake robust M&E.

There are elements above that are not traditionally or necessarily strengths of (I)NGOs or AusAID (e.g. design, M&E). In both these programs, the design phase was rushed and AusAID’s robust quality processes were not able to be fulfilled. A managing contractor would normally have sound systems for financial management, monitoring and reporting – areas that the review team found to be weak for KDP 49 and ICDP’s M&E was also weak (financial systems in place for ICDP are sound).

Nevertheless, these risks are not insurmountable. There are some key benefits to program implementation being led by an INGO and by AusAID directly, which may be: Compared to a managing contractor model, the benefit of an AusAID or INGO managed

program is that they have a better understanding of what is happening on the ground. Program staff for both ICDP and KDP are frequently in the field.

There may be reduced management costs with INGO and AusAID managed programs, though not necessarily.

If the focus is using a community development approach then an NGO may have a comparative advantage.

Greater oversight of a sensitive and important program by AusAID, with national interest concerns.

These advantages need to be weighed up against potential disadvantages such as weaker management skills, monitoring, measuring and reporting of results.

If AusAID is to pursue direct management of development programs, the agency should ensure strong management skills in key personnel and robust financial and monitoring systems. There is scope for strengthened monitoring, coordination and communication skills by the implementation team. Clear delineation of roles and responsibilities of the AusAID program manager and implementation personnel is needed.

Training in these skills should be provided if deemed necessary, and close supervision and follow-up at key stages throughout implementation undertaken.

Recommendations for AusAID on program delivery mechanisms:11. Ensure AusAID management models adhere to streamlined contracting arrangements.12. Regardless of the management model, ensure all programs have independent M&E

expertise to support implementation.

49 The review team notes however that in the case of KDP, financial management was done through respective managing contractors and the difficulty arose in relation to drawing this reporting together.


CONCLUSIONAustralia, the largest donor to Papua New Guinea, has historically prioritised education and health for assistance to improve access to these services and achieve the Millennium Development Goals (MDGs) of these sectors by 2015. Over $600 million has been invested by Australia in these two sectors over the last 12 years (see table at Annex 13).

Despite this huge investment, remote areas throughout PNG remain neglected by their provincial governments, and communities continue to be denied their right to basic priority services. While funding is continually used as an excuse for inaction, it is also clear that commitment and motivation of government is a key factor for this continued neglect. This highlights the importance of the two programs under review. These programs are being implemented in areas of varying geographical remoteness, and the review considers these programs, in the main, successful. Many positive findings are apparent.

Lessons Learned from the two activities

In both programs the many successes can be attributed to: working with government and their procedures and processes, increasing the opportunity for people from the local areas to become service providers

(health, education, law and justice), providing opportunities for communities to participate in processes to identify broadly

their communities’ needs (KDP audit process, ICDP ward planning), and increasing/continuous analysis of the outcomes of initiatives to examine strengths and

respond to weaknesses. The results of these latter endeavours, however, need to be better shared with government to encourage government to use the learning elsewhere in the provinces.

ICDP has specifically addressed governance and in that process is giving communities the opportunity to demand more and better quality services through ward planning, as well as identifying areas where the community can make its own contribution to improving local services and amenities, encouraging self-reliance. The ward planning process needs to be wary of raising expectations as to what the government might deliver. Re-visiting the ward planning template could help to make the requests more realistic. The volunteer programs (VHV, VBA) have been effective, but they present challenges, particularly in areas where there are other opportunities for income. The important experience of the KDP has highlighted this. Greater enticement of women into the VHV program may improve the situation, though it would have to overcome the lack of support of husbands to their women engaging in volunteer activities.

The review has identified numerous challenges to supporting remote communities to have better access to priority services. These challenges are not necessarily limited to remote area programs, but they are exacerbated by the remoteness of the locations where access is challenging and expensive, thus throwing up additional barriers not necessarily present when programs are being delivered in more easily accessed locations. But they do provide lessons for AusAID as it considers further support to population groups in PNG that have for too long been neglected.


Some of the issues noted by the review – some slow progress with infrastructure, developing cohesive volunteer networks – have highlighted useful lessons and challenges to working with donor models of implementation where the program is trying to make the relevant authority take responsibility. For example, managing and monitoring the work of the VHVs is the responsibility of the health workers; managing school infrastructure is the responsibility of school boards. While time needs to be allowed for “local solutions” to problems, better monitoring and reporting of these problems by the program would help to bring these issues to the attention of appropriate groups/individuals to prompt more timely responses.

While achievements in the education sector are evident, the review was concerned that gender achievements in education would not continue. Nationally, gender disparities are quite alarming. Girls comprise forty-five per cent of children in elementary and primary schools. They start off on a par with boys, but then a fifth fall by the wayside before the end of primary school. Gender disparities widen as students move up the education ladder. This is much more likely to be exacerbated in remote areas where families are more reluctant, for many reasons, to send their girls away for continuing education – small populations cannot support higher levels of schooling. These programs need to monitor the attendance of girls through to the highest grades. While both programs have reported attendance of girls that is above the national average in some of their education components (all schools in KDP - Annex 6-F, and for child literacy programs in Simogu in ICDP - Annex 7-A), ICDP needs to monitor female/male ratios in all schools that they are working with to know that the national trend of decreasing attendance by girls at higher grades is not occurring in these programs.

Applicability of lessons beyond these two activities

It is time to return to basics, in some respects. From the design perspective a sound, informative design document is essential that is subjected to peer review, no matter the imperative to get a program up and running, with tangible deliverables to show at an early stage. Much better articulation of a program is needed that must encompass, but not be limited to: design that has a greater analytical component, with greater specificity on how remote

area barriers are to be addressed – this needs to be reflected at activity and output levels; improved consideration and more specificity of what is required for sustainability; identification of the full spectrum of factors for successful service delivery, i.e. identifying

what/who is addressing each side of the supply/demand aspects of service delivery; ensuring much greater quality of inputs; describing engagement with government that is carefully considered and regularly

assessed with, if necessary, innovative approaches to achieve this; and associated with that but going much further, an articulated communications strategy that

considers the broad spectrum of communications needed. Inclusion of communications as a discrete component in the design and M&E framework with improve accountability by all stakeholders.

It is important to acknowledge that the least expensive option may not always be the best, or lead to the desired outcomes. It can be quite to the contrary. AusAID needs to encourage/support assessment of alternative options for interventions (e.g. for sanitation), and support without qualification accepted best practice (e.g. training/capacity development which incorporates follow-up and support) and nothing less. A much greater emphasis on quality is required, especially in remote areas where ‘fixing things up’ is going to cost much more because of access costs.


Important factors that affect the likelihood of sustainability have been identified and discussed. The flow of provincial government funds to support services at the LLG and Ward level is a key challenge, and requires specific interventions. Other important precursors (e.g. engagement with provincial administrations and lower levels of government) are necessary but are not sufficient to achieve this. Community development programs can ‘kick start’ the process of getting top-down funding flowing, and generating community demand, but this alone is not going to be enough. Interventions of direct service delivery on its own will have limited impact in the longer term.

AusAID must ensure that blockages to the funding flow are being addressed, either within or alongside remote services programs themselves. Doing this alongside a program of intervention can achieve more leverage, and not distract the remote services implementers from the core business of building the capacity of services and the providers in these communities. This joint approach must gain provincial agreement to work willingly and actively to address these often contentious and sensitive issues. AusAID can use whatever it deems is the most effective avenue of support, and think laterally about how this type of development assistance can be contracted differently.

Program management arrangements need to be carefully considered, particularly if undertaken by AusAID. While an AusAID management model can work as well as any other arrangement, it clearly has to use some form of contracting to engage the implementation team. The initial management arrangement for the KDP was ill-informed, and did not consider lessons from other AusAID projects which had shown similar fragmented arrangements to be seriously detrimental to the program. It is very clear that the arrangement has to: have one single point of accountability, and one single contractor; be very clear about roles and responsibilities of the contractor, the AusAID manager and

the implementation team; and ensure regular team meetings take place to support team cohesion, efficiencies, and

monitoring.

The two programs that have been the subject of the review are still works in progress. But they have demonstrated that improving services and opportunities in remote areas is possible, though not without its challenges. Many of the challenges are common elsewhere but are significantly magnified because of remoteness. The lessons from this review should be seriously considered for further programs in remote areas in PNG.


ANNEXES

Annex 1 Distribution of Service Providers

Annex 2 Review Terms of Reference

Annex 3 Review Plan

Annex 4 People Consulted

Annex 5 Documents Consulted

Annex 6 Achievements of KDP

Annex 7 Achievements of ICDP

Annex 8 Education in Simogu Story

Annex 9 Papu’s Story

Annex 10 The Viability of Ward Development Planning

Annex 11 Exit Strategies

Annex 12 Initial KDP Management Arrangements

Annex 13 Australian Support to PNG Health and Education


EXECUTIVE SUMMARY - Home - Department of...

Documents

Transcript of EXECUTIVE SUMMARY - Home - Department of...