Asian Development Bank. 6 ADB Avenue, Mandaluyong City, 1550 Metro Manila, Philippines Tel +63 2 632 4444; Fax +63 2 636 2163; evaluation@adb.org; www.adb.org/evaluation Evaluation Approach Paper

Project Performance Evaluation Report for Loans 2064/2065 and Grant 4299-INO: Participatory Irrigation Sector Project March 2015

Team Leader: Emile Gozali, Principal Evaluation Specialist (email: egozali@adb.org)

Contact: evaluation@adb.org

A. Introduction

1. In line with the approved Independent Evaluation Department (IED) work plan, IED annually undertakes a number of project/program performance evaluations. Preparation of a project performance evaluation report (PPER) for the Participatory Irrigation Sector Project (PISP) contributes to the requirement.1 In addition, this evaluation presents an opportunity for IED to jointly work with the Policy and Operations Evaluation Department (IOB), Ministry of Foreign Affairs of the Government of the Netherlands (GON) to conduct a more rigorous impact evaluation. GON was the project’s grant co-financier. Deeper impact evaluation is a form of assessment that IED would like to get more involved in. For Asian Development Bank, this evaluation is timely since ADB is processing a follow-on project to PISP due for approval in 2015 and the evaluation may provide inputs to the IED’s ongoing corporate evaluation on ADB’s partnerships. The evaluation will also provide inputs to a water policy review under preparation by the GON relating to its development assistance worldwide. A short mission was fielded in 17-19 December 2014 to (i) firm up the rationale (mutual benefits) for the joint evaluation, and (ii) lay the groundwork for the project evaluation mission. 2. This evaluation approach paper presents the proposed evaluation design for the PISP. It contains background information of the country and project, evaluation objectives, methodology, process and timeframe.

B. Country and Project Background 3. Country context. Broad-based income improvement and poverty reduction are long standing objectives of the Indonesian Government. These objectives are particularly important for the rural areas. The contribution of rural areas to the total nationwide deprivation was 59% in 2010 and 57% in 2014, while the contribution of the urban areas was 41% and 43% in 2010 and 2014, respectively. The Government’s priority continues to rehabilitate and build new rural infrastructure because shortage of rural infrastructure has been identified as a critical constraint to job creation, productivity and income improvement. Delivery of infrastructure and other public services especially since the early 2000s has been paired with community and local government empowerment as a way to enhance the effectiveness of these services and cost effectiveness and efficiency. 4. Income growth from farming and food production has been affected by different factors such as inadequately maintained irrigation/water resource facilities, transport, extension services, farm access to inputs, credit and market information. There are also challenges arising from conversion of farm lands to non-agricultural lands and a growing disparity of land

1 The project completion report (PCR) for PISP was completed in June 2014 and the PCR validation report

(PVR) in January 2015.

2

Box: Participatory Irrigation Sector Project

Project Approval Date: 19 December 2003 Loan Signing Date: 2 February 2005 Project Effectiveness Date: 2 May 2005 (actual 2 June 2005) Closing Date: 30 June 2011 (actual 31 December 2012) Total Cost: $126.0 million (actual $127.5 million) ADB Financing Loan: $73.0 million (OCR + ADF, actual $74.3

million) Government of the Netherlands $15.0 million (grant, actual $15.0 million) Contribution of Government, and Beneficiaries: $38.0 million (actual $38.2 million) Executing Agencies: Ministry of Public Works and Human

Settlements, Ministry of Agriculture, Ministry of Home Affairs

Source: Asian Development Bank.

and asset ownership. To better sustain the irrigation/water resource facilities (and reduced capital costs of rehabilitation), the government in 1987 adopted the irrigation operations and maintenance (O&M) policy. Subsequent studies had pointed to the following persistent issues in irrigation management: (i) insufficient policy and institutional arrangements; (ii) low level of stakeholder participation, and (iii) inadequate assessment and funding of O&M. The government launched the farmer managed irrigation system project in 1995 with direct involvement of local governments and beneficiaries.2 This was accompanied by a World Bank-assisted program. With the growing impetus for decentralized government management in early 2000s, the government made further efforts to strengthen local level ownership of irrigation and agriculture initiatives. These underpinned the rationale for the Participatory Irrigation Sector Project (PISP)3 approved in 2003. 5. ADB priorities and country strategies. ADB developed country strategies for periods starting in 2001, 2003, 2006, and 2012. In all four strategy periods, cumulatively the agriculture and natural resource sector accounted for about 4% of total disbursements during 2001–2014, behind public sector management, finance, transport, and energy. Support to agriculture, natural resources and rural development has been linked to the broader agenda of poverty reduction and improved public sector management/governance over the years. The Country Strategy and Partnership (CSP) for 2003-2005, when the project was approved, identified rural development (including rural infrastructure, marketing and support services as well as microfinance) as a priority focus area for ADB operations.4 ADB’s latest CSP for 2012–20145 continues to focus on agriculture and natural resources along with five other sectors (education, energy, finance, transport and water supply and sanitation). 6. Project Impacts. The expected impact of PISP were increased economic growth and reduced poverty in 25 districts of rural Indonesia in the provinces of Banten, Central Java, East Java, Lampung, South Sulawesi, and West Java. Impact indicators were an increase in the income by 15%–20% of 1.5 million farm families by 2012, and a one-third reduction in poverty incidence among project beneficiaries by 2012. See box for the project basic data.

2 ADB. 1995. Report and Recommendation of the President to the Board of Directors on a Proposed Loans to

the Republic of Indonesia for the Farmer-Managed Irrigation Systems Project. Manila (Loan No. 1378-INO). 3 ADB. 2003. Report and Recommendation of the President to the Board of Directors: Proposed Loans to the

Republic of Indonesia for the Participatory Irrigation Sector Project. Manila. 4 ADB. 2002. Country Strategy and Program 2003–2005. Indonesia. Manila. para. 58 and Table 4.

5 ADB. 2012. Country Strategy and Program. Indonesia. 2012–2014. Manila. para. 24.

3

7. Project Outcomes. The envisaged project outcomes were development of a sustainable decentralized management of irrigation systems and increased yields in irrigated crops. The indicators and targets for these outcomes were (i) to provide 456,000 hectares (ha) with an adequate level of O&M by 2012 (ii) to avoid deferred maintenance for 456,000 ha by 2012, and (iii) to increase average paddy yield from 3.5 metric tons per ha in 2002 to 4.0 tons per ha (14.3% increase) by 2012. At appraisal, the original coverage area for the project was 625,000 ha, but this scope was reduced to 456,000 ha (73% of original target) because of delays in project implementation caused largely by changes in the broad policy and regulatory environment in Indonesia. Slow progress and price inflation from 2004 until 2008 rendered the available project funds inadequate to cover the original scope. Three outcome indicators used in the design and monitoring framework (DMF) did not monitor the decentralization progress of irrigation management (e.g., delegation of decision making or in the use of budget), but this should also be an area of PPER attention. 8. Outputs. The project activities were grouped into four output groups: (i) district/provincial capacity building for management of devolved irrigation responsibilities; (ii) improved irrigation performance at scheme level; (iii) rationalization and development of district agricultural services, strengthening and empowerment of water users’ associations (WUAs), and improvement of farmers’ access to agricultural services and inputs; and (iv) improved water resource management information system. Corresponding to these outputs, a number of targets were identified in the design monitoring framework (DMF), which are briefly described below. Project activities were grouped into three components: A–irrigation management, B–water resources information and asset management, and C–project coordination and monitoring. However, there were 4 output areas under the project (i) upgraded district capacity for management of devolved irrigation, (ii) improve irrigation performance at the scheme level, (iii) district agricultural services rationale, WUAs strengthened and farmer access to agricultural services/inputs improved, and (iv) improved water resource management information system. Appendix 1 provides diagrammatic relationships between the various result levels from inputs to impact. Indicators related to the project outcomes and outputs are shown in the evaluation framework/matrix in Appendix 2. 9. Special Focuses–Impact Evaluation, Decentralization, and Water Resource Planning. The joint evaluation will take advantage of additional resources available to carry out a deeper impact assessment. This will involve gathering of existing and new survey data on project impacts and beneficiary welfare. Data processing will be accompanied by follow-up focused group discussion. This exercise will be largely led and done by IOB/Netherlands team with support from the IED/ADB. During the evaluation mission on April 2015, the impact assessment method and household surveys will be tested and refined. Follow-up focused group discussions would pursue the survey to clarify its findings. As part of the follow-on activity, longer (approximately 1 week) immersed observations are tentatively planned in few villages for the impact assessment to understand the dynamics of WUA capacity and societal changes that may have occurred. Depending on the quality of findings, a separate monograph can be prepared from this assessment. The second special focus area for the evaluation is the project’s contribution in the institution building of local governments, particularly the planning, water resource, and agriculture agencies. This would be in the context of assessing the project’s effectiveness in achieving capacity building outcomes and project contribution to a sustaining the more decentralized government operations. Evaluation in this area will be done by IED through agency interviews. The third special area of focus will be on the progress of improving water resource plans and their implementation. This assessment will contribute to the larger study by the IOB on the water policy review and will focus on (i) integration of water management strategies at the user level (irrigation, urban, and energy) with those at the provincial and watershed levels, and (ii) progress in implementing the agreed upon water management plans/strategies. The latter study component will be done by the IOB/Netherlands team.

4

C. Issues Raised During Design 10. The project was designed in 2002–2003, with the appraisal mission fielded in November 2002 and loan approval in December 2003. The design review focused on the need to simplify project scope coverage (reducing national level institutions, activities, and provinces/districts involved). This was in view of the potential implementation challenges the project might face because of the then ongoing government-wide decentralization process, which encountered many teething problems. The project team responded by excluding a proposed basin water resource management component. Complementary scope coverage in the sector was further harmonized with development partners, primarily the World Bank which designed and approved the Water Resources and Irrigation Sector Management Program (WISM) in June 2003, six months prior to PISP’s approval. 11. Other key areas reviewed were to improve output indicators–these were largely qualitative and measuring input rather than outputs–and a continuing call on PISP to address the persistent problem of poor maintenance of irrigation structures. The PISP validation commented that the project indicators, which were finally adopted, were largely appropriate but also indicated better indicators were needed to progress in capacity building, particularly at the local level. Improved O&M largely relied on the design intent to promote greater participation by beneficiaries/farmers (in construction work and monitoring) and local governments. There were other design issues discussed and clarified (assuring the capacity of districts, training focus on decentralization, loan funds to be provided as transfer grants to districts, and arrangement to finalize co-financing with the Netherlands. D. Findings and issues from Independent Evaluations and Self Evaluations 12. The Project Completion Report (PCR) and Project Completion Report Validation (PCRV) were completed in June 2014 and January 2015, respectively. Both rated the project successful. The 9 years of project implementation (December 2003 to December 2012) helped enacted regulations on irrigation/agriculture management and local capacity building that were necessary to pursue devolution of management tasks and authorities to sub-national governments. The project was validated as relevant strategically and from the design perspective. Although relatively well designed and mindful of past lessons learned, project implementation could not overcome broader challenges posed by the transition of government structures, system and procedure towards a greater decentralization. The delay was mainly in 2003–2005 due to uncertainties of the channeling of funds and authority over budgets between the central ministries and local governments. The delay allowed unit costs to inflate and thus the targeted irrigation area coverage was reduced to about 73% of the original target along with similar reduction in the WUAs to be established/strengthened. The proposed evaluation will further consider how much decentralization should be considered an unavoidable/exogenous risk to project implementation, given this risk was known at appraisal and discussed at MRM with a view to reducing the area coverage of the project. 13. The project was rated effective largely as it met the outcome and revised output targets at completion. The PCR validation however pointed out that achievements in institutional capacity building were still gauged mainly by input indicators (e.g., number of persons trained, sessions held). This assessment needs to be strengthened, if possible, by available data on quality indicators (e.g., number of staff meeting qualification standards). 14. The project was assessed as efficient mainly on the strength of the projected economic returns, despite the project closing about 18 months behind schedule. The PCR estimated the project’s economic internal rate of return at 74%. However, the validation cautioned against possible underestimation in the repair cost of the smaller tertiary systems

5

and an over-estimation of rice price (economic price was based on the 2013 market price which may be historically high). In terms of process efficiency, the project was assessed as less than efficient. However, combining the economic re-evaluation and process efficiency, the PCR validation rated the project efficient. 15. As regards sustainability, institutional improvements under the project are being sustained, including at the district governments and beneficiary associations. The PCR rates the project likely sustainable. However, resource provision remains a concern. Budget allocations to schemes that are under district accountability (i.e., smaller than 1,000 ha) remained inadequate (PCR, Appendix 1). Further, it is not known if the full cost of O&M for the tertiary canal systems (under responsibility of WUAs) were adequate. The validation rated the project less than likely sustainable.

16. At completion the impact indicators associated with the project were satisfactory (family incomes and reduction in poverty rates in the project. The PCR did not rate the impact criteria, while the PVR rated it significant. The validation nonetheless re-emphasized the concern over sustainability of project benefits. The project enabled participation of more women through the WUAs, although not to the extent envisaged by the gender action plan (GAP). Women were still under-represented in decision-making related to water management.6 Participation of women had been incorporated in the legal documents (local government irrigation regulations and decrees on establishment and functions of local irrigation commissions). 17. CAPE and CPSFRV. The CAPE for Indonesia circulated in 20057 reported the loan effectiveness was delayed for 18 months due to the late loan signing and reorganization of the three executing agencies. This postponed compliance with certain loan conditions, the setting up of project organization and consultant recruitment. There was also a 15% shortfall in counterpart funds reported. However, it was too early to assess outputs or outcomes. The CPS Final Review Validation for 2006–20098 did not report on the PISP, while the draft validation for 2012–20149 provides summary findings and ratings of the PCR.

E. Evaluation Objectives and Methodology 18. Objectives. The main objectives of the PPER are to (i) assess the performance and impact of the project based on IED’s methodology for evaluating sovereign loans, and (ii) generate a series of findings, lessons and recommendations for future operations. Special areas of attention are given to (i) a deeper impact evaluation assessment, (ii) PISP contribution to a broader effort to devolve national government’s mandate and resources to sub-national governments, (iii) review the economic/financial performance of the project, and (iv) project contribution to improved water resource management (para. 9). Item (i) will deepen the PISP impact assessment; items (ii) and (iv) will support assessment of PISP effectiveness and sustainability, while item (iii) contributes to the assessment on efficiency. 19. Methodology. The evaluation will adopt the latest IED PPER methodology (2006 Guidelines as amended in 2013),10 focusing on three areas: (i) the performance of the project measured in terms of relevance, effectiveness, efficiency, and sustainability (REES);

6 The PCR noted that while the representation of women on the boards of 931 WUAs and 235 WUAFs

exceeded the project target of 15%, the remaining 4,145 WUAs and 408 WUAFs did not have women on their boards. Under the project women were invited to participate in training, but their participation remained below the target of 33%.

7 ADB. Operations Evaluation Department. December 2005. Country Assistance Program Evaluation for

Indonesia. Manila. 8 ADB. June 2011. Indonesia: Country Strategy and Program, 2006–2009 Final Review Validation. Manila.

9 ADB. 2015. Indonesia: Country Strategy and Program, 2012–2014 Final Review Validation. Manila. Under

preparation. 10

Including the guidance provided in the Director General’s Memorandum on the IED Approach. April 2012.

6

(ii) institutional development (improved governance practices, skills) and impact; (iii) the performance of partners, including ADB and the Government. For each evaluation criterion, the final report will provide a detailed analysis and a rating on a four-point scale,11 according to the PPER guidelines. The evaluation framework, linking criteria to the key evaluation questions is presented in Appendix 2. This Appendix also provides the linkages on how five work packages/groups (general evaluative interviews and document review, decentralization analysis and spot check of structure, study on water resource planning, economic/financial analysis, and impact evaluation) are organized to provide inputs to the evaluation performance assessment. F. Process: Collecting Data and Evidence 20. The credibility and quality of each PPER is based on the robustness of its analysis, including attention to ensuring clear evidences. For this reason, the evaluation will have to assemble the available secondary data, collect primary data (surveys and interviews) and review existing documents as required, and triangulate between these sources. All four work packages (Appendix 2) will rely on these methods. 21. Secondary data (desk review). This will be a desk review of the available documents. The review will include (i) general background documentation on the country; (ii) acquired data from the Central Bureau of Statistics (mainly on household and farm performance data) and agency data (on irrigation system performance and extension services standards), (iii) ADB policies and strategies (CPSs and Strategy 2020); (iv) project documentation, such as RRP, back-to-office reports, midterm review, and project completion report, RRP/PCR of other agriculture/irrigation projects by ADB and other partners; and (v) available evaluation reports (e.g., PCR Validation Report and Country Assistance Program Evaluation, and CPS Final Review Validation draft of 2015).

22. Primary data through interviews and field observations. The evaluation plans to collect primary data through interviews with agency staff from the central, provincial and district levels, ADB operations staff and development partners. There will also be spot checks and field observations primarily on the completed irrigation/water control structures. These will be done for the work packages for decentralization institutional analysis, water resource planning/implementation, and qualitative evaluation assessment. For the project impact assessment, household surveys and focus group discussions will be held with farmers/ WUAs. The sampling coverage for the interviews, survey and secondary data collection will be done to ensure representativeness of the samples. Draft interview questionnaire on decentralization is in Supplementary Appendix A and draft household questionnaire is in Supplementary Appendix B.

23. Results from different data sources and field observations will be compared and cross checked so that a more accurate assessment is made possible. Evidence will be presented in the main report through tables, charts, and text boxes to highlight important/interesting findings. G. Evaluation Team 24. It is proposed that the IED team would comprise Emile Gozali, Principal Evaluation Specialist, an evaluation specialist/economist (international consultant) and an institutional/irrigation specialist (national consultant). The IOB team will comprise of Ferko Bodnar and Joep Schenk, Inspectors of the IOB, Jetze Heun (water resource engineer/consultant) and a national institute to facilitate the household data collection and impact assessment.

11

3 = highly satisfactory; 2 = satisfactory; 1 = less than satisfactory; 0 = unsatisfactory.

7

25. To generate project evaluation reports (e.g., PPER), IED and IOB will prepare their respective project evaluation reports following the formats required by each organization. Overlapping sections (e.g., on REES, impacts, borrower performance) will assigned to either teams to avoid duplication of work. Other outputs envisaged are separate impact report on beneficiary impact analysis and other write up required by the IOB for its water policy review. The IED mission will prepare a case study write up on its partnership experiences with the Netherlands on PISP to contribute to ADB’s ongoing corporate evaluation study on partnership. H. Timeline and Schedule of Activities

Activity Timing

IED’s intent to conduct project performance evaluation was conveyed to ADB and Indonesian Government

11 Dec 2014

Preparatory Mission for joint evaluation arrangement 17–19 Dec 2014

Methodology development, discussions on approach and survey format, identification of evaluation team members

Jan – Mar 2015

Evaluation Approach Paper finalized 23 –27 Mar 2015

Joint Evaluation Mission 7–21 April 2015

Continuing local government interviews and household survey by national consultant and domestic institute, respectively. First Draft of PPER and case study write up on partnership produced.

April – mid June 2015a

Joint Evaluation Mission (follow up interviews to impact household survey and refinement of reports)

mid to end June 2015

IED Internal Review July 2015

Interdepartmental Circulation July 2015

Interdepartmental comments received August 2015

Editing and report finalization August–September 2015

a This period is extended to allow time for team members to work on other projects/

evaluations.

8

LIST OF APPENDIXES

Appendix 1 - Theory of Change Appendix 2 - Evaluation Matrix: Key PPER Questions

Supplementary Appendix A - Questionnaire on Decentralization (Draft)

Supplementary Appendix B - Household Questionnaire (Draft)

A

pp

en

dix

1 9

Financial assistance

Technical assistance

Sustainable management irrigation infrastructure (A3)

Capacity building districts & provinces (A1)

Capacity building WUA’s (A2)

Irrigated agriculture support (A4)

Reduced poverty

Increased economic growth

Theory of Change

Increased yields

Improved water resources information and asset management (B)

NSC, NPCMO & PPMS (C)

Source: IOB, Netherlands

A

pp

en

dix

2 10

EVALUATION MATRIX: KEY PPER QUESTIONS

* = targets that are originally in the design monitoring framework ** = targets that are of special interest by IOB (Netherlands)

Criteria Specific Evaluation Questions/Indicators Data Sources

and Work Package I. Project Performance

A. Relevance Are project objectives consistent with national agriculture, water, and rural development strategies and policies, as well as ADB’s country strategies and sector policies, as well as the needs of the beneficiaries?

Assess ex-ante and ex-post relevance. Are the original objectives still relevant at the time of evaluation? Have significant changes in the project context or ADB policies been retrofitted to the design?

Were the project activities/outputs (i.e., component mix) appropriate for achieving the project’s core objectives? Were they supported well by other project/program activities (by the government and other development partners) to achieve the project outcomes and impacts?

Were the scale of project inputs and outputs adequate to achieve stated outcomes? Were outputs and outcomes too ambitious given the time and resources available?

Were other aspect of project design done well (financial allocations/modalities, implementation arrangements, and monitoring and evaluation (M&E))?

Were project risks identified during appraisal sufficiently mitigated in the design (e.g., government/EA reorganization, low capacities, slow release/lack of counterpart funds, resettlement complexity, extended time and resources to meet safeguard requirements)

Has the project design been participatory, taking into consideration input and needs expressed by key stakeholders, (executing agencies, planning/financing ministries, local governments, and target beneficiaries?

Has the final appraisal report integrated relevant lessons learned at country and sector levels? Was the appraisal report consistent with the PPTA report? Did they gather key information on the local conditions? Has the project benefited from available knowledge (for example, the experience of other similar projects in the area or in the country)?

Does the envisaged targeting approach facilitate access for disadvantaged groups/households (smallholders and women) as planned?

Records of project processing meetings

Report and Recommendation of the President (RRP) of the ADB on the project (appraisal report).

Loan Agreement Back to office report

(BTOR) of implementation reviews

Project Completion Report (PCR)

PCR Validation Report (PVR)

CAPE/CPS Final Review Validation

Field observations by the

evaluation mission Interviews Work Package: 1 (see

legend below)

B. Effectiveness The project outcome was (i) decentralized management of irrigation systems on a sustainable basis and (ii) increased yields of irrigated crops. Outputs: (i) upgraded district

RRP Loan Agreement

A

pp

en

dix

2 11

Criteria Specific Evaluation Questions/Indicators Data Sources

and Work Package capacity for irrigation responsibilities, (ii) WUAs strengthened and empowered. (iii) improved irrigation performance, (iii) agricultural services rationalized and developed, (iv) farmer access to agricultural services improved, (v) water resource management information system improved. To what extent has the project outcomes and outputs were or likely to be achieved? The evaluation will consider the DMF indicators (marked by asterisk *) and other indicators: Outcome 1: Decentralized and sustainable management of irrigation systems

Were the provinces/districts given greater responsibilities in irrigation and agricultural sector management?

Were provinces/districts given greater authority/independence in project management (strategy formulation, planning, budgeting/activity selection, procurement, implementation involvement)?

Were O&M of irrigation budgets and spending per hectare growing in real terms or adequate?*

Was deferred maintenance avoided?* Outcome 2: Increased yields of irrigated crops

Did average paddy yields increase (3.5 t/ha to 4.0t/ha)?*

Did other crop yields increase?

Did cropping area/cropping pattern and intensity change?

What were there changes in total outputs? To what extent outputs were achieved? Output 1: Upgraded capacity of districts/provinces for irrigation management

District/provincial staff trained (5,000 staff) in irrigation*

All 27 district irrigation management plans prepared by project completion*

District irrigation plans better linked to (i) water resource management plans/rules at watershed/provincial/national levels and (ii) to medium term/annual budget plans.**

District governments function effectively as focal points for irrigation management*

Output 2: Irrigation performance at scheme level improved.

Irrigation service quality on farm (availability of water to match planting schedules, flow control, quality of water)

Appraisal Report Mid-term review–cum-

BTORs Project Completion Report Project Completion Report

Validation Report GAD evaluation PCU documentation Survey of local government

staff Field observations by

evaluation mission Key informant interviews Focus group discussions Work Packages: 1, 2, 5 Work Packages: 2, 4 Work Packages: 1, 3, 5

A

pp

en

dix

2 12

Criteria Specific Evaluation Questions/Indicators Data Sources

and Work Package

2,500 WUAs on government-managed irrigation schemes established or strengthened.* (number of farmers trained and certified)

3.750 WUAs on farmer-managed schemes strengthened.* (number of farmers trained and certified)

WUAs are federated/ greater number of WUA federation.*

Irrigation management for 250,000 ha is transferred to WUAs through partnership agreements with local agencies.*

On the transferred schemes, full cost recovery assumed by 2,500 WUAs (no subsidies left in place).*

Output 3: District agricultural services rationalized and developed; WUAs strengthened and empowered, farmers access to agricultural services and inputs improved.

District/provincial agricultural extension training and their organizations rationalized

Adoption of improved irrigation and other farming technology packages by farmers.*

changes in the number of farms with access to extension support services;

Changes in the intensity/frequency of support services provided;

Changes in the quality of support services;

Completion of pilot studies on using WUAs to improve farmers’ bargaining power and make input providers more accountable to farmers in 5 districts (by 2005).*

Did beneficiaries have improved access to markets? Output 4: Water resources management information system improved.

Water resources data management unit at the Directorate General of Water Resource established and remains operational and useful.*

Decision support systems of the DGWR central office improved (by 2005).*

Capacity for data gathering, processing and quality control in 27 project districts improved (Indicators needed. These systems are integrated for effective water resource management (at watershed, province or national levels).

Overall, the PPER will try to explain the outcomes achieved:

The evaluation will look at complementarities with other interventions, whether they helped increased or reduced the effectiveness of project activities/outputs in achieving intended outcomes and impacts?

In particular, were there any exogenous factors (e.g., policy change, political events, institutional changes) that affected project implementation and results? If yes, what were they and did ADB and the Government make the required adjustments to project

Work Packages: 1 and 5 Work Packages: 1 and 3

A

pp

en

dix

2 13

Criteria Specific Evaluation Questions/Indicators Data Sources

and Work Package design and implementation to ensure the achievement of objectives?

C. Efficiency The PPER will consider both implementation/process and resource/economic efficiency to achieve outcomes.

What was the project EIRR? check the PCR assumptions and methods (economic prices of labor and non-traded items).

Are there any delays in project implementation and delivery that may affect benefits or costs in a significant manner?

In addition to delays and cost overruns, are there unnecessary complications in project management that could have been avoided?

What are the factors that help explain the efficiency performance?

RRP Loan Agreement Appraisal Report Mid-term review–cum-

BTORs Project Completion Report Project Completion Report

Validation Report PCU documentation Direct field observations by

the evaluation mission Key informant interviews Work Packages: 4 and 5.

D. Sustainability What are project benefits (institutional change, services) that are likely to continue after project closure? What are benefits not continuing after closure?

How resilient are these achievements to changes in institution and personnel?

Political/institutional sustainability: Is there clear indication of government commitment (both at national and local level) to supporting the project activities and reforms after the loan closing date (provision of budget, staff, policy, and institutional mandate / arrangements)?

Beneficiary ownership: Do project activities benefit from the engagement, participation, and ownership of local communities, grassroots organizations, and the rural poor?

Technical sustainability: Are adopted approaches technically viable? Do project users have access to adequate training for maintenance?

Economic and financial sustainability: Are the economic activities promoted by the project generating economic profits or losses (net of subsidies)? What is the likely resilience of economic activities to shocks or progressive exposure to competition and reduction of subsidies?

Environmental sustainability: Are the ecosystem and environmental resources (e.g., water availability, soil fertility, vegetative cover) likely to be stable or is there a depletion?

Project Completion Report Project Completion Report

Validation Report PCU documentation Direct field observations by

the evaluation mission Key informant interviews Focus group discussions Work Packages: 1–5

II. Institutional Development

A. Institutional Development

See questions under effectiveness on capacity building of (i) provincial and district governments/agencies and (ii) water users associations (WUAs) and WUA federations.

Project Completion Report PCR Validation Report PCU documentation

A

pp

en

dix

2 14

Criteria Specific Evaluation Questions/Indicators Data Sources

and Work Package

Did national sector policies / regulatory framework improve?

Field observations by evaluation mission

Survey of national agency staff.

Key informant interviews Focus group discussions Work Package 1, 2, 3

B. Impact 1.5 million farm family incomes increased by about 15-20%.

Incidence of poverty among project beneficiaries reduced by one third.

Did the composition source of household incomes change (more sources)?

Did farm households’ physical assets change (land holdings, livestock, trees, equipment)?

Did other household assets change (houses, motorcycles, radios, telephones, etc.)?

Did the natural resources base status change (water, forest, etc.)? Are there any signs of increased/decreased soil fertility?

Project Completion Report M&E/ project data Existing household data Household surveys Other interviews / focus

group discussions. Work Package: 5

III. Other Assessments

A. ADB Performance (quality-at-entry and supervision)

Did ADB mobilize adequate technical expertise in project design?

Was the design process participatory (with national and local agencies, grassroots organizations)?

Were specific efforts made to incorporate the lessons and recommendations from previous independent evaluations and PCRs in project design?

Did ADB (and the Government) take the initiative to suitably modify project design (if required) during implementation in response to any changes in circumstance or emerging opportunities?

How was ADB’s performance in procurement supervision and disbursements?

Was prompt action taken to ensure the timely implementation of recommendations from BTORs, including the mid-term review and resolve any implementation bottlenecks?

What is the role and performance of the resident mission? How was the relationship between headquarters and the resident mission?

Has ADB been active in creating an effective partnership and coordination among partners (Government of Netherlands) to ensure the achievement of project objectives?

RRP Project Completion Report Project Completion Report

Validation Report PCU documentation Direct field observations by

the evaluation mission Key informant interviews Focus group discussions Work Package 1 (with inputs from other packages)

B. Borrower Performance (quality of

Did the Government take the initiative to modify project design (if required) in response to any changes in circumstance or emerging opportunities?

Was the government actively involved in the design?

RRP Project Completion Report Project Completion Report

A

pp

en

dix

2 15

Criteria Specific Evaluation Questions/Indicators Data Sources

and Work Package preparation, implementation, and M&E)

Has government assumed ownership for the project? By its exerting initiatives and actions?

Have the flow of funds and procurement procedures been suitable for ensuring timely implementation?

Has adequate staffing/resource given for project management? Have counterpart funds been provided adequately and on time?

Has project management discharged its functions adequately, and has the Government (Bappenas and MOF) provided guidance to project management when required?

Did Government ensure adequate coordination of the national agencies (MOHA, MPW, MQA)? And between national and sub-national levels?

Has audits been undertaken in a timely manner and reports submitted as required?

Has an effective M&E system been put in place and does it generate information on performance and impact which is useful for project management to take critical decisions?

Have loan covenants and the spirit of the loan agreement been followed?

Has government contributed to planning an exit strategy and make arrangements for continued funding of certain activities?

Validation Report PCU documentation Direct field observations by

the evaluation mission Key informant interviews Focus group discussions Work Package 1 (with

inputs from other packages)

ADB = Asian Development Bank, BTOR = back-to-office report, CAPE = country assistance program evaluation, EIRR = economic internal rate of return, FIRR = financial internal rate of return, M&E = monitoring and evaluation, PCR = project completion report, PPER = project performance evaluation report, RRP = report and recommendation of the President, TA = technical assistance Source: Independent Evaluation Department.

Work Package: 1 – general stakeholder interviews, document reviews (by IED and IOB) 2 – interviews (provincial/district agencies) on decentralization and spot check of structures (by IED) 3 – interviews and document reviews on water resource strategies/plans and their implementation (by IOB) 4 – economic and financial analyses (by IED) 5 – survey of farmers/water users association members (by IOB)