RPE: 99009

ASIAN DEVELOPMENT BANK Operations Evaluation Office

REPORT OF THE PRESIDENT

TO THE

BOARD OF DIRECTORS

ON

1998 EVALUATION ACTIVITIES

AND

THE TWENTY-FIRST ANNUAL REVIEW

OF EVALUATION REPORTS

March 2000

ABBREVIATIONS ADF - Asian Development Fund AOTA - advisory and operational technical assistance APEP - annual performance evaluation program BDFC - Bhutan Development Finance Corporation BME - benefit monitoring and evaluation CAP - country assistance plan CAPE - country assistance program evaluation CCADP - Chashma Command Area Development Project COS - country operational strategy COSO - Central Operations Services Office CPRM - country portfolio review mission CRBIP - Chashma Right Bank Irrigation Project DFI - development finance institution DMC - developing member country EA - executing agency ECG - Evaluation Cooperation Group EIA - environmental impact assessment EIRR - economic internal rate of return EMM - environmental mitigation measure IES - impact evaluation study Lao PDR - Lao People’s Democratic Republic LECO - Lanka Electricity Company (Private), Limited MTR - midterm review O&M - operation and maintenance OCR - ordinary capital resources OEO - Operations Evaluation Office PCR - project completion report PEIS - Postevaluation Information System PPAR - project/program performance audit report PPMS - Project Performance Management System PPR - project performance report PPTA - project preparatory technical assistance PRC - People’s Republic of China RES - reevaluation study SES - special evaluation study TA - technical assistance TCR - technical assistance completion report TPAR - technical assistance performance audit report

NOTE

In this report, “$” refers to US dollars.

CONTENTS Page

REPORT OF THE PRESIDENT TO THE BOARD OF DIRECTORS

ON 1998 EVALUATION ACTIVITIES

I. INTRODUCTION 1

II. ASSESSMENT OF 1998 EVALUATION ACTIVITIES 1 A. Project Performance Audit Reports 1 B. Technical Assistance Performance Audit Reports 14 C. Impact Evaluation and Reevaluation Studies 15 D. Special Evaluation Studies 16 E. Country Assistance Program Evaluation 19 F. Annual Performance Evaluation Program 19

III. DMC CAPABILITIES, FEEDBACK, AND FOLLOW-UP OF EVALUATION FINDINGS 20 A. Strengthening the Performance Evaluation Capabilities of Developing Member Countries 20 B. Feedback of Evaluation Findings 21 C. Follow-up of Recommendations in Evaluation Reports 21 D. External Coordination 22

IV. KEY ISSUES 22 A. Greater Focus on Operation and Maintenance after Project Completion 22 B. Mitigation of Environmental Effects of Bank Projects 23 C. Revising Project Preparatory Technical Assistance to Improve Project Performance 24 D. Asian Financial Crisis: Adjustments in Ongoing Projects 25 E. Strengthening Design and Evaluation to Improve Project Quality 26

APPENDIXES 27

REPORT OF THE PRESIDENT TO THE BOARD OF DIRECTORS ON 1998 EVALUATION ACTIVITIES

I submit to the Board of Directors the Report on 1998 Evaluation Activities and the Twenty-First Annual Review of Evaluation Reports. In 1998, the Postevaluation Office, now the Operations Evaluation Office (OEO), undertook various activities aimed at enhancing the development effectiveness of the Bank’s operations. During the year, the Office completed 44 evaluation reports, including 21 project performance audit reports (PPARs), 4 technical assistance performance audit reports (TPARs), 3 impact evaluation studies (IESs), 4 reevaluation studies (RESs), 5 special evaluation studies (SESs), and one country assistance program evaluation (CAPE).1 Of the 21 loan projects evaluated in 1998, which included one private sector project, 9 (43 percent) were rated as generally successful, another 9 as partly successful, and 3 (14 percent) as unsuccessful. By country grouping, about 44 percent of the 16 projects in Asian Development Fund-only countries were rated as generally successful, 37 percent as partly successful, and 19 percent as unsuccessful. For blend countries, 40 percent were generally successful and 60 percent were partly successful. Except for one unsuccessful project, those in the agriculture sector (10 in all, including the private sector project) were either generally or partly successful. Three of four transport projects—all in the road subsector—were likewise either generally or partly successful, with one railway project deemed unsuccessful. Two power projects were generally successful and one was unsuccessful. Two projects each in the finance and social infrastructure sectors were partly successful. Of the 21 advisory and operational technical assistance (TA) grants evaluated in the 4 TPARs, 4 were still ongoing; of the 17 completed TAs, only 1 was unsuccessful. A major initiative during 1998 was the preparation of the first CAPE, which is the Bank’s evaluation of overall assistance operations for a country. The CAPE assessed the relevance of the Bank’s strategy in terms of the needs of the People’s Republic of China (PRC) and the constraints that narrow the choice of operational options there. As a result of dialogue between the Bank and the PRC, the lending program has become more diversified in terms of both sectors and geographical regions. Important implications identified in the CAPE include (i) the country operational strategy (COS) should be made more operational, while the country assistance plan (CAP) should fully reflect the strategic directions of operations; (ii) the Bank needs to apply due diligence in reviewing and assessing Government-planned projects; and (iii) the TA program should be more focused and better targeted. An IES of secondary science education projects in Bangladesh, Nepal, and Pakistan focused on the projects’ impacts on institutions, graduates, and the policy and management environment. An important finding from the tracer studies was that certain minimum classroom conditions (e.g., number of students per classroom, student-teacher ratio, basic classroom equipment) are essential for ensuring the intended impact of Bank assistance at the classroom level. Another IES looked at Bank assistance to the power sector in Pacific developing member countries and showed that it had limited direct impact due to institutional weaknesses. A third IES examined the adequacy, implementation experience, and operational performance of 32 project preparatory technical assistance grants (PPTAs) in the agriculture and natural resources sector in 1 The remaining reports consisted of two technical assistance completion reports, the Twentieth Annual Review of

Evaluation Reports, the Seventeenth Report of Postevaluation Abstracts, one sector synthesis of evaluation findings, and the 1998 Annual Performance Evaluation Program.

iii

Bangladesh, and suggested the need for reviewing the concept, role, and various technical aspects of PPTA. Based on a review of the evaluation reports prepared in 1998, the following key issues were identified and are discussed:

(i) Greater Focus on Operation and Maintenance (O&M) after Project Completion. The long-standing issue of inadequate provision of O&M funds for completed projects by governments and executing agencies (EAs) is a key to project sustainability. Poor maintenance of facilities leads to loss of project benefits and decreased development impact. Assurances and loan covenants do not guarantee adequate O&M. The willingness of the Bank to finance rehabilitation projects can exacerbate the problem; the cost of rehabilitation often exceeds the capitalized value of annual maintenance. To remedy this situation, funding agencies should act in concert. In particular, assistance should aim at improving the country’s overall budgetary process. Institutional arrangements for O&M need to be defined in detail before any project is approved, and the Bank should consider refraining from financing projects for which a borrower cannot afford recurrent expenditures.

(ii) Mitigation of Environmental Effects of Bank Projects. While the foreign

exchange costs of environmental mitigation measures (EMMs) may be covered under a loan, the necessary local expenditures to provide training for monitoring and maintaining the EMMs may be delayed. The Bank should consider more detailed itemizing and earmarking of portions of a loan to implement EMMs. There is a need to develop and implement an EMM evaluation and reporting system on a project-specific basis. Adequate attention needs to be given at the project planning stage to training, preparing contingency plans, and designing environmental services utilizing willingness-to-pay surveys and public promotion. Greater public participation, especially during project planning and design, through multipartite teams can improve the design, operation, and monitoring of EMMs.

(iii) Revising PPTA to Improve Project Performance. In 1998, the Office

conducted two studies that looked at PPTA from different perspectives. Both studies found that PPTAs have largely improved in quality and effectiveness. However, both concluded that many PPTAs still fall short of their objective to provide successful project designs. PPTAs should pay greater attention to institutional factors, include adequate analysis of alternatives, be fully supported by EAs, and seek beneficiary participation. The Bank should review all aspects of PPTA in the broad context of its business process cycle. More time and financial resources need to be allocated to PPTA; these should pay off in terms of more successful loan projects and consequently fewer resources needed for loan supervision.

(iv) Asian Financial Crisis: Adjustments in Ongoing Projects. The impact of the

crisis was seen in several PPARs, SESs, RESs, and IESs, particularly in the finance sector, but also in a power project in Indonesia and an education project in Thailand, among others. The Bank has provided critical assistance to crisis-hit economies through program loans and related TAs. Closer attention is needed to

iv

assessing the impact of the contagion on ongoing Bank-assisted projects and to making in-course corrections. The newly instituted project performance report (PPR)/Project Performance Management System (PPMS) process will facilitate corrective action in real time. The CAPE exercise, and self-evaluation elements of COSs and CAPs, may also prove to be early warning devices.

(v) Strengthening Design and Evaluation to Improve Project Quality. Evaluation

has become an important measure of accountability, and more generally, of corporate governance, for both borrowing countries and the Bank. As an input to strengthen the Bank’s procedures, OEO’s review of loan documents has revealed methodological problems and discrepancies. The introduction of the PPR and PPMS enforces the use of specific monitorable parameters concerning development objectives as well as outputs during project implementation. But evaluability remains an important issue. Data collection and monitoring are necessary for the PPMS. OEO and the Central Operations Services Office will continue holding training sessions for operational staff, and a help desk advises individual staff on conceptualizing impact indicators and devising data collection programs.

The 1998 report on the Annual Performance Evaluation Program concluded that the most important action the Bank can take to improve portfolio performance and project outcomes is to strengthen the quality of its project supervision. The PPR and PPMS provide the required performance measurement and monitoring system. Almost as important is the need for more thorough and comprehensive economic and sector analysis to help improve project identification and design. The findings of a TA on the evaluation of Bank assistance to developing member countries, approved in June 1995, identified the need to improve benefit monitoring and evaluation; this TA resulted in a shift to the PPMS as the Bank’s approach to monitoring and evaluating projects. The findings of a TA on the study of project planning and management in the PRC highlighted the effectiveness of workshops in which member countries share development experiences. During 1998, the Office administered three TAs aimed at developing/strengthening the PPMSs of Nepal, Philippines, and Sri Lanka; and one TA to the PRC to strengthen evaluation capacity. The Office continued close coordination with other multilateral aid agencies on evaluation activities through the Evaluation Cooperation Group. In 1998, the Group worked to promote convergence of evaluation methods and practices, and developed plans to produce a newsletter.

I. INTRODUCTION

1. In 1998, the Operations Evaluation Office (OEO)1 undertook various activities aimed at enhancing the Bank's operations. A major initiative during the year was the preparation of the first country assistance program evaluation (CAPE), which is the Bank’s evaluation of overall assistance operations for a country. It was also during the year that the project performance report (PPR) was institutionalized, to replace the project administration committee notes as the document to be used in recording implementation progress and potential achievement of development objectives for all ongoing loan projects. Together with the Central Operations Services Office (COSO), OEO acted as the focal point for monitoring progress and providing assistance in accomplishing the PPRs where necessary. Projects staff were also provided training on a limited scale. Increasingly OEO is looking at larger evaluation units than the traditional individual projects and programs: impact evaluation studies (IESs), clusters of technical assistance (TA) grants, and various special evaluation studies (SESs). These activities are in line with requests from Bank shareholders for increased accountability, and with suggestions from the operations departments for improving operations based on lessons learned. Increased emphasis is being placed on real-time feedback to meet evolving requirements from the Bank’s operations. OEO recently enhanced its feedback system by operationalizing a World Wide Web home page on evaluation findings and lessons; the home page will have links with the websites of other aid organizations and will provide an easier way of sharing evaluation information with developing member countries (DMCs) as well as the general public. All these new activities provide OEO with more capacity to respond quickly to phenomena such as the financial contagion that has spread out from certain DMCs in the Bank’s region (paras. 73-76). 2. This review, the twenty-first in the annual series prepared by OEO, presents a summary of the findings from evaluation reports prepared in 1998. During the year, the Office completed 44 evaluation reports. These comprised 21 project performance audit reports (PPARs), 4 technical assistance performance audit reports (TPARs), 3 IESs, 4 reevaluation studies (RESs), 5 SESs, 2 technical assistance completion reports (TCRs), the Twentieth Annual Review of Evaluation Reports, the 1998 Annual Performance Evaluation Program, the Seventeenth Report of Postevaluation Abstracts, 1 CAPE, and 1 sector synthesis (Box 1).

II. ASSESSMENT OF 1998 EVALUATION ACTIVITIES

A. Project Performance Audit Reports

3. While OEO’s activities covered many areas, almost half of its efforts in 1998 were directed toward PPARs. Selected to reflect the variety of the Bank’s loan portfolio, 21 projects were audited. Of these, 16 were in Asian Development Fund (ADF)-only countries and 5 were in ADF-ordinary capital resources (OCR) blend countries. Sectorwise, there were 10 projects in agriculture and natural resources including 1 private sector project for the production of fertilizer, 4 1 On 12 January 1999, the name of the Postevaluation Office was changed to Operations Evaluation Office to better

reflect the wider scope of evaluation activities currently being done by the Office.

2

in transport and communications, 2 in social infrastructure, 3 in energy, and 2 in the finance sector. The list of reports covered in this review is given in Appendix 1.

Box1: Types of Evaluation Reports Project/Program Performance Audit Report • provides a comprehensive evaluation of the effectiveness of a project or program in achieving its

objectives • gives an analytical assessment focusing on issues and lessons of operational significance Technical Assistance Performance Audit Report • assesses the need for, adequacy of, and effectiveness of technical assistance (TA) including its impact on

technical, operational, institutional, and socioeconomic aspects Impact Evaluation Study • provides insights into the extent to which the benefits of Bank lending in a particular sector/subsector are

spread and sustained Reevaluation Study • focuses on an in-depth analysis of project impact and sustainability about five years after the

postevaluation stage Special Evaluation Study • provides intensive analysis of a particular thematic issue across sectors and/or countries Country Assistance Program Evaluation • examines all Bank country operational strategy studies, country assistance programs, loans, TAs, and

economic and sector work over a number of years in a particular country

3

1. Project Performance Ratings

4. The Bank uses three broad performance categories for completed projects/programs: generally successful, partly successful, and unsuccessful. The major focus of the performance assessment is the contribution of the Bank-funded activity to a DMC’s socioeconomic development. The factors that are generally taken into consideration in the assessment of success are summarized in Box 2.

Box 2: Criteria for Assessing Project/Program Performance Factors used to assess performance include: (i) the reestimated economic internal rate of return (EIRR) based on quantifiable benefits that the

postevaluation mission considers to be reasonably reliable; (ii) achievement of a least-cost or cost-effective design; (iii) the sustainability of future operations; (iv) unquantified, but well-perceived, economic benefits and costs that cannot be captured in the EIRR

computation; (v) socioeconomic impacts that do not enter into the computation of the EIRR; and (vi) the extent to which the major purposes and goals, either set out at appraisal or as modified and

accepted subsequently, have been achieved and are considered commensurate with costs actually incurred.

A project/program is considered generally successful if it is expected to be economically viable or can generate socioeconomic benefits commensurate with original expectations and/or costs incurred. If the benefits of a project/program are believed to be sustainable at reduced levels, with reasonable prospects for improvement when remedial actions are taken, it is classified as partly successful. A partly successful project/program should not be viewed as an unsuccessful one. Furthermore, these ratings considered at the time of postevaluation may change during subsequent reevaluations, which are conducted on some projects/programs about five years after postevaluation. A project/program that is not technically and/or economically viable is rated as unsuccessful.

4

5. Of the 21 projects evaluated in 1998, 9 (43 percent) were rated as generally successful, another 9 as partly successful, and 3 (14 percent) as unsuccessful (Figure 1). Looking at performance by country grouping, about 44 percent of the 16 projects in ADF-only countries were rated as generally successful, 37 percent as partly successful, and 19 percent as unsuccessful. For blend countries, 40 percent were generally successful and 60 percent were partly successful. 6. In terms of costs, generally successful projects accounted for about 65 percent of the total actual investment cost, while the partly successful and unsuccessful projects’ shares were 19 percent and 16 percent. By loan amounts disbursed, generally successful projects accounted for 50 percent, partly successful projects for 30 percent, and unsuccessful projects for 20 percent (Figure 1). 7. Except for one unsuccessful project, those in the agriculture sector were either generally or partly successful. Three transport projects performed well because their designs were based on adequate investigations, and they employed good materials and construction practices. OEO’s evaluation of energy projects also revealed that success can be engendered by sound rationale and design developed at appraisal and appropriately adopted during implementation. Two projects each in the finance and social infrastructure sectors were classified as partly successful (Appendix 2). 8. An overall look at the 514 projects evaluated by the end of 1998 reveals about 59 percent of all completed projects classified as generally successful, 30 percent as partly successful, and 11 percent as unsuccessful. Expectedly, the evaluation findings show that projects tend to perform more successfully in more rapidly growing economies, pointing once again to the stronger institutional capacity of such countries as a major contributing factor. Within the member groupings, the highest rate of project success remained in OCR-only

Figure 1: Performance Assessment of Projects Evaluated in 1998

14%

43%

43%

Generally Successful Partly Successful Unsuccessful

19%

16%

65%

50%

30%

20%

By Loan DisbursementBy Number of Projects By Investment Cost

Source: Appendix 2.

5

economies, with 78 percent of projects classified as generally successful. For ADF-only and blend countries, 50 and 58 percent of projects were rated as generally successful (Appendix 3). 9. As in previous years, there was no pronounced trend in performance ratings of evaluated projects on an annual basis because of their highly heterogeneous country, sector, and subsector mix. In addition, the projects evaluated were approved in different years under differing Bank policy frameworks. Meaningful lessons should therefore be drawn by going beyond a simple comparison of the ratings given during particular years.

2. Major Findings

a. Lessons Learned

10. The 1998-evaluated projects highlight a number of lessons that are relevant to both current and future Bank operations, including the importance of (i) determining the appropriate project objectives based on a thorough analysis of the recipient countries’ economic, political, institutional, and cultural conditions; (ii) assessing the balance between project rationale and scope during the design stage to avoid the inclusion of unattainable objectives; (iii) basing project formulation, physical targets for various components, and implementation schedules on realistic estimates of both financial resources and time required for various project activities—keeping in view external limiting factors; (iv) strengthening Bank review missions in terms of addressing project weaknesses during implementation, especially the underlying causes; (v) examining the basis of using covenants to strengthen financial and institutional performance; and (vi) monitoring sector performance and institutionalizing systems for gathering information, especially on beneficiary and other socioeconomic impact. Several other lessons learned were highlighted in the impact evaluation, reevaluation, and special evaluation studies as well as the TPARs (paras. 37-53). b. Project Formulation and Design 11. This year’s project evalautions revealed several factors, both positive and negative, that influenced the rationale, design, implementation, and consequent prospects for sustainability of Bank-financed projects. Weak project preparation, especially in terms of formulating the design, is a major factor that can lead to serious problems during project implementation. More importantly, it can endanger the sustainability of a project’s benefits after completion. The need to address the constraints to overall project sustainability, by the Bank and the DMCs, is particularly urgent in light of the financial and/or economic crisis being experienced by some of the recipient DMCs. 12. Complementing the findings of the 1994 Report of the Task Force on Project Quality, subsequent annual reviews of evaluation findings have established the importance of conducting rigorous analysis of a country’s economic, institutional, and social conditions prior to project formulation. Indeed, one of the determinants of project success is the identification of project objectives that are in line with the country’s overall strategy, both medium and long term. For example, because livestock is an integral part of the subsistence farming system in Bhutan and the most widely held production asset, the project rationale for the Highland Livestock Development Project2 was appropriate. The Project was consistent with the high priority 2 Loan 808-BHU(SF): Highland Livestock Development Project, for $4.3 million, approved on 2 December 1986.

6

accorded to livestock by the Government in its fifth and sixth plans. The design was a well-integrated strategy with strong emphasis on dairy development and marketing in the western region and on broad-based development in the eastern region. The Project recognized the constraint arising from religious sensitivity to killing animals by expecting meat processing facilities to be done by non-Buddhist private operators. However, the strength of religious reaction was underestimated, and the completion of the facilities was delayed. Yet, the overall performance of the Project was generally successful. 13. In contrast, the fundamental weakness of the Bhutan Development Finance Corporation (BDFC) Project3 was assessed to be inappropriate objectives, given the country’s economic and political conditions. The Project was intended to promote private sector industry as well as develop BDFC into an efficient, autonomous development finance institution (DFI), capable of operating on commercial lines. The predominant role of agriculture in the economy was not considered; rather the focus was on the scarcity of skilled management and staff, as well as various policy impediments to industrialization, private sector growth, and development of efficient and financially viable banking institutions. In addition, there was no explicit linkage between the subprojects financed and the overall project benefits identified at appraisal. The subprojects continue to perform less than satisfactorily based on a review of their repayment performance and operational status. The financial sustainability of BDFC is of utmost concern given the ongoing expansion of rural lending operations. BDFC is continuously used by the Government as a tool for social and rural development, while the Bank generally assumes it is lending to a self-sustaining vehicle for industrial/commercial development. The Government maintains its view that the small size of the agriculture and industry loan portfolios does not warrant the creation of separate financial institutions at present, although at some date in the future BDFC may be split along lending lines. 14. The road subsector projects demonstrate that sound project rationale, design, and scope contribute to generally satisfactory performance. In Papua New Guinea, the Fourth Road Improvement (Sector) Project4 was appropriately targeted at implementing a medium-term action plan for the road subsector. The findings of the earlier Transport Investment Program Study provided a good foundation for the identification of subproject roads. In Nepal, the Roads Improvement Project5 reduced transport constraints by upgrading and rehabilitating selected roads and highway links. Project completion was affected by delays arising from suspension of the Government’s Trade and Transit Treaty with India, but the overall project rationale and design proved sound. An important component was the rehabilitation of Doti Road (81 kilometers [km]) in the Far West Region. The improved road stimulated development in the region and complemented other development programs that included extension of the road for a further 65 km, opening of three schools and a vocational training institute, several water supply projects, irrigation development, electrification, and agricultural extension. Hence, rehabilitation of existing roads can provide the opportunity for human development, economic growth, and poverty reduction. 15. Design of the Shanxi-Xiaoliu Railway Project6 in the People’s Republic of China (PRC) did not adequately consider the difficult geological and soil conditions of the whole length of the railway. Furthermore, the design was based on an overoptimistic traffic forecast prepared by 3 Loan 934-BHU(SF): Bhutan Development Finance Corporation Project, for $2.5 million, approved on 13 December

1988. 4 Loan 690/691-PNG(SF): Fourth Road Improvement (Sector) Project, for $33.25 million, approved on 4 September

1984. 5 Loan 806-NEP(SF): Roads Improvement Project, for $30.0 million, approved on 2 December 1986. 6 Loan 948-PRC: Shanxi-Xiaoliu Railway Project, for $39.7 million, approved on 31 January 1989.

7

the Government under the central planning system. As the country moved to a market system, the forecasts should have been thoroughly reviewed to determine their appropriateness. There has been no passenger traffic to date, and none is anticipated in the near future, making the investment in passenger facilities largely useless. Overall, the Project’s performance was classified as unsuccessful. 16. The experience of the Hexian Pulp Mill Project,7 also in the PRC, showed that inappropriate design contributed to unsuccessful performance. While the mill size of 50,000 tons/year was considered below the economic scale of production, the price assumptions used in the economic and financial analyses indicated that the Project was viable. It was also assumed that the mill’s capacity could be more than doubled without considering the availability/adequacy of wood resources. However, the layout of the mill does not lend itself to the suggested expansion, which raises the question whether plans for expansion were considered at the local level during the design stage. These shortcomings can be attributed to lack of initial sector work. Furthermore, a project in a technically sophisticated field such as pulp making requires that the consequences of technology choices on production cost, output, and prices be considered at the design stage. The experience indicates that the lack of technical design specifications may have had an effect on the cost of the Project. 17. For the projects covered by the review, an analysis of the effectiveness of associated technical assistance (TA) shows mixed results. The Bank-financed project preparatory technical assistance (PPTA) for the Second Livestock Development Project in Nepal8 provided a well-reasoned, integrated, and focused design for livestock development through improved productivity. While this PPTA was partly developed to rectify shortcomings of the (First) Livestock Development Project,9 it highlighted the importance of marketing to ensure utilization of increased production. Similarly, implementation of the PPTA for the Second Health and Population Project10 in Pakistan was generally satisfactory. The consultants’ mix of expertise was appropriate and covered health planning and management, medical personnel, equipment, and drugs. In Sri Lanka, the PPTA for the Secondary Towns Power Distribution II Project11 was sufficient and effective. The recommendations of the accompanying TA for an institutional study12 of Lanka Electricity Company (Private), Limited (LECO) evolved around strengthening the autonomy of LECO, improving tariff structures, strengthening organizational development and training programs, and making changes to the regulatory framework of the power sector. 18. In contrast, the PPTA carried out under the Chashma Command Area Development Project (CCADP)13 simply revised the scope of the ongoing Chashma Right Bank Irrigation Project (CRBIP).14 It proposed that a separate project be designed to specifically address the drainage and flood protection requirements of the first of three stages of the CRBIP. The PPTA focused on restructuring the CRBIP, but did not produce the detailed designs for CCADP per se. Consequently, the design of the Project was based on theoretical socioeconomic conditions and cropping patterns—actual cropping patterns had significantly higher water requirements

7 Loan 937-PRC: Hexian Pulp Mill Project, for $49.6 million, approved on 20 December 1988. 8 Loan 745-NEP(SF): Second Livestock Development Project, for $14.0 million, approved on 24 October 1985. 9 Loan 445-NEP(SF): Livestock Development Project, for $5.0 million, approved on 11 October 1977. 10 Loan 710-PAK(SF):Second Health and Population Project, for $16.0 million, approved on 29 November 1984. 11 Loan 870-SRI(SF): Secondary Towns Power Distribution II Project, for $34.8 million, approved on 3 December

1987. 12 TA 930-SRI: Institutional Strategy Study for LECO, for $930,000, approved on 3 December 1987. 13 Loan 723-PAK(SF): Chashma Command Area Development Project, for $40.8 million, approved on 13 December

1984. 14 Loan 330-PAK(SF): Chashma Right Bank Irrigation Project, for $31.5 million, approved on 15 December 1977.

8

than projected at appraisal, with potential water supply shortages for other stages of the CRBIP. A more in-depth analysis on the effectiveness of and issues relating to PPTA was carried out in the context of agriculture PPTAs in Bangladesh15 and is discussed in Chapter IV. c. Environmental Aspects 19. The attention given to environmental aspects of the projects was generally positive. Although the guidelines for environmental considerations of Bank operations were not in effect at the time of approval of the Shanxi-Xiaoliu Railway Project in the PRC, the Project considered the environmental effects of railway construction. The Loan Agreement required the central Government and the province to ensure that construction and operation of the railway would conform with the country’s Environmental Protection Law.16 Although the Project did not require it, follow-up actions by the Government and the Bank have resulted in the adoption of an environmental management plan for the region as a whole. In Pakistan the CCADP together with the CRBIP had a positive environmental impact. It contributed to lowering and controlling the groundwater table, reducing waterlogging and salinity, improving environmental sanitation, and reducing waterborne diseases. An ongoing Bank-financed project will be implementing an environmental management plan that includes the project area. 20. However, the lack of a watershed management and conservation component in the design of the Irrigated Command Area Development Project in Indonesia17 resulted in insufficient control and monitoring of watershed deterioration at upstream project sites. Because measures for controlling and mitigating destructive agricultural practices, illegal logging, and increasing population pressure in the project area were not established at appraisal, the consequent reactive measures to address the problems have been inadequate. The inability of the Government to address watershed management and conservation will result in inadequate water supply to the project schemes, placing enormous doubt on the sustainability of project benefits. Environmental issues were more thoroughly examined in relation to Bank-assisted infrastructure projects18 and are discussed in Chapter IV. d. Sustainability 21. For most of the projects evaluated, serious concerns were raised about the sustainability of benefits. Sustainability of physical infrastructure as well as of the benefits to be generated by the projects depends to a large extent on the financial and institutional capacity of the operating agencies. Many of the facilities and buildings under the Second Livestock Development Project in Nepal have been poorly maintained because of inadequate budgetary provisions for operation and maintenance (O&M). The feed mill suffered from lack of adequate maintenance, while the National Artificial Insemination Center was plagued with an intermittent supply of liquid nitrogen due to frequent plant breakdowns. Also in Nepal, the operational performance of the Mini-Hydropower Project19 has been hampered by inadequate routine maintenance.

15 See IES: BAN 98016: Impact of Bank Project Preparatory Technical Assistance on the Agriculture Sector in

Bangladesh, December 1998. 16 The Law requires an environmental impact assessment and includes other regulations pertaining to coal

production, washing, desulfurization, and transportation. 17 Loan 818-INO: Irrigated Command Area Development Project, for $28.8 million, approved on 11 December 1986. 18 SST: REG 99002: Environmental Mitigation Measures in Selected Bank-financed Projects, December 1998. 19 Loan 512-NEP(SF): Mini-Hydropower Project, for $8.3 million, approved on 21 April 1981.

9

22. In Pakistan, the risks to the sustainability of the Second Health and Population Project are largely attributable to low budgetary support resulting in irregular and inadequate supplies of medicine, contraceptives, and equipment; insufficient financial resources for O&M; and shortage of qualified health personnel, particularly female health workers. Unless alternative systems for increased sustainability are eventually installed with the necessary built-in mechanisms to protect the poor and socially disadvantaged, and/or unless scarce Government resources can be channeled to the more needy rural areas, accessibility to quality public health care services and family planning services will be a continuing concern. The current lack of financing for O&M of the subsurface drains under the CCADP will eventually lead to siltation and waterlogging, which the Project was supposed to resolve. Failure to maintain the irrigation system to design standards will reduce irrigation efficiencies significantly below project targets and will substantially limit the area that can be effectively irrigated. The most important threat to the sustainability of the Highland Agriculture Development Project20 in the Philippines comes from inadequate maintenance of physical facilities. While some routine maintenance and emergency repairs have been done, periodic maintenance has yet to commence. The surfaces of all the project roads have deteriorated to a very rough state. As the deterioration progresses, the roads may revert to seasonal use, and agricultural output will eventually decline. An analysis of the experience of other projects revealed that the backlog of maintenance work suggests a basic institutional inadequacy for maintenance works, rather than simply the current shortage of funds for upkeep. 23. On a positive note, the introduction of the farmer group approach was one of the institutional successes of the Second Livestock Development Project in Nepal. The approach not only was a vehicle for channeling extension services by the Department of Livestock Services but also allowed farmers to manage joint milk collection, marketing, resource mobilization, and the credit applications of group members. In Indonesia, the potential for sustainability of the benefits under the Irrigated Command Area Development Project is good because of actions by the provincial government. The provincial water resources services continue to provide funds for O&M of all schemes at the project sites, particularly for cleaning, repairing, and desilting primary and secondary canals. At the same time, farmer members of the water user associations exhibited strong ownership of the tertiary and farm-level irrigation facilities. Recognizing that improper maintenance of the facilities and lack of cooperation in scheduling water distribution will adversely affect the delivery of water—with negative consequences for crop yields—these farmers have been responsible for carrying out O&M at the tertiary level. In Pakistan, the institutional strengthening at the level of water user associations helped existing user groups during construction. However, it did not work thereafter, primarily due to lack of a clear mechanism for beneficiary participation. O&M issues were especially examined under an SES for the road sector,21 and the effects of O&M on this and other sectors are discussed in Chapter IV. 24. Unquestionably, beneficiary participation is important to ensure that project benefits are sustained. However, while all projects cite numerous expected socioeconomic impacts at the time of appraisal, the 1998-evaluated projects reveal that there was usually an inadequate basis for assessing such benefits. Frequently there was an absence of baseline and subsequent benefit monitoring and evaluation (BME) data on socioeconomic and sociocultural conditions of the project areas concerned. Sometimes, the broad socioeconomic benefits expected at appraisal did not materialize because the design of the projects did not include any

20 Loan 802-PHI: Highland Agriculture Development Project, for $18.8 million, approved on 25 November 1986. 21 SST: REG 98014: Operation and Maintenance of Road Facilities and Their Impact on Project Sustainability,

December 1998.

10

program to achieve these. It was also demonstrated that measures of socioeconomic factors during appraisal, implementation, and completion of projects were often not readily available, thus constraining a thorough assessment of the socioeconomic impacts of projects. 25. Instead, loan covenants that directly impact on a project’s sustainability such as BME and financial/institutional strengthening are incorporated in the project agreement as a way of justifying the project. However, the use of loan covenants to ensure project success revealed major shortcomings. Often the major covenants were not complied with. Key areas of noncompliance with loan covenants that affected the performance of the Nepal Mini-Hydropower Project were tariff setting, project staffing, and load promotion. In Pakistan, the inadequate budgetary provision by the Government, lack of competent staff, and delayed procurement of land had detrimental effects on the timely implementation of various components of the Second Health and Population Project. Instead of simply including a budgetary provision in the covenants, measures to reduce exclusive dependence on the Government for health care resources—e.g., a comprehensive health insurance program, and participation of nongovernment organizations and health volunteers—should have been made part of the Project. 26. In Nepal, covenants specifying provision of an O&M budget for the facilities and buildings of the Second Livestock Development Project, and the setting up of a BME system were not complied with. The system was eventually instituted after project completion with funding from another agency, but was dismantled when the funding ended. The Shanxi-Xiaoliu Railway Project in the PRC also has limited prospects for the immediate future because of its weak operating performance. The executing agency (EA) continued to exhibit poor compliance with outstanding loan covenants designed to improve the financial soundness of the Project.

e. Private Sector Projects

27. By the end of 1998, seven private sector projects had been evaluated; all were rated generally successful (Appendix 4).22 An important factor in their success was strong management capability, including effective financial control and technical capability. Furthermore, marketing prospects proved to be one of the most important determinants of project sustainability.23 However, because market conditions may change according to the level of consumption and other external factors, future proposals for private sector lending should have built-in flexibility.24 The review highlights the need for the Bank to (i) prepare specific guidelines for the divestiture of equity investments; (ii) strengthen its implementation monitoring and review of private sector loans; and (iii) define the functions and responsibilities of Board membership of Bank representatives, as well as considering the risk of exposure to possible litigation and liabilities.25 There is also a need to monitor adherence to the policy guidelines for loan recipients to avoid concentration of the loans in a small number of industry subsectors.26

22 Because the Bank has yet to establish a monitoring mechanism for completed projects except when there is a

follow-on project, the impact of the recent financial/economic turmoil on these projects is not available. A review of the performance of some of the private sector projects will provide insights into the Bank’s overall private sector portfolio.

23 See PPA: BAN 20712: Padma Textile Mill, September 1992. 24 See PPA:INO 21710: P.T. Gunung Garuda Project, October 1993. 25 See PPA: PHI 20714: Planters Development Bank Equity Investment Project, December 1993. The covenant

stated that disbursement was to be made at ADB’s request after 31 December 1993. To date, ADB has not divested of its equity investment.

26 See PPA: PAK 20716: National Development Leasing Corporation Limited, July 1997.

11

28. The evaluated projects included the Fauji Fertilizer Company Ltd.’s Fertilizer Expansion Project.27 This Project was well implemented and achieved its objectives of augmenting the domestic supply of urea and contributing to the country’s foreign exchange savings. It has contributed to increasing the private sector’s share of fertilizer production, building a domestic support industry for fertilizer production, and broadening listings on the Karachi Stock Exchange. The effectiveness of the Bank in fulfilling its catalytic role and in maximizing the impact of its limited resources was mixed, though. The Project appears not to have required the Bank’s assistance at the level extended and could have supported sizeable credit on commercial terms. Overall, however, the Project was rated as generally successful because of its strong operational and financial performance, its satisfactory economic return, the apparent sustainability of its benefits, and its generally positive impact on Pakistan’s private sector.

3. Implementation Arrangements

29. The efficiency of the implementation arrangements of the projects—the manner of execution of the various components in terms of time and cost, and the overall supervision provided by the Bank—comprises the major factor to be examined in assessing overall project performance.

a. Delays 30. Other than the Second Cook Islands Development Bank Project,28 which was completed some seven months earlier than the original schedule, all projects experienced implementation delays ranging from four months for the private sector loan project to 6.5 years for the Nepal Mini-Hydropower Project (Figure 2). The subprojects under the Nepal energy project experienced implementation delays from 3.5 to 6.5 years because of (i) delayed completion of site surveys and detailed designs; (ii) lengthy tendering processes; (iii) inadequate technical investigations contributing to the need for substantial design modifications during construction, e.g., soil instability discovered during excavation of one of the headrace tunnels necessitated a new and more costly tunnel alignment; and (iv) delays in transportation of materials to the sites because of remote locations and monsoons. The original implementation schedule at appraisal was unrealistic. A review of other small hydropower projects then under way, and the problems and delays they encountered, would have provided a more realistic basis for project scheduling at appraisal. 31. Excessive delays were experienced in the Farm-to-Market Roads Project29 in Pakistan because of poor performance of the contractors. A number of roads in Punjab had to be retendered, with delays in the procurement of construction/maintenance equipment varying from 2 to 7 years. Another factor was the ambitious estimation of completion times for all project activities. The slow processing and other procedural difficulties in the Government agencies were not adequately anticipated at appraisal. Similar experience was highlighted in two other projects in Pakistan: the Second Health and Population Project and the CCADP.

27 Loan 7047/1003-PAK: Fauji Fertilizer Expansion Project, for $30.0 million, approved on 19 December 1989. 28 Loan 1155-COO(SF): Second Cook Islands Development Bank Project, for $1.5 million, approved on 14 January

1992. The loan was closed ahead of schedule and the undisbursed amount of $200,000 was canceled because of lack of demand for Bank funds.

29 Loan 758-PAK(SF): Farm-to-Market Roads Project, for $29.3 million, approved on 26 November 1985.

12

32. The delay in the Highland Agriculture Development Project in the Philippines was attributed to weak management of the EA; inappropriate project design; and external factors, notably the typhoons and earthquake that caused significant damage to project facilities. Implementation of the Shanxi-Xiaoliu Railway Project was delayed by two years because of difficult terrain and soil conditions, which hindered construction. However, it may be questioned why the appraisal implementation schedule did not anticipate such delays, as section 1 was already under construction and crossing some very rugged areas.

b. Project Costs 33. Excluding the private sector loan and the two DFI projects, 8 of the projects evaluated during 1998 had cost underruns averaging about 25 percent of the total project cost, and 10 projects incurred cost overruns that averaged about 12 percent (Figure 3). As in previous years, factors that contributed to the cost underruns were scaling down of original scope, overestimation of costs during appraisal, depreciation of the local currency against the dollar, and underutilization of credit lines. Seven projects in ADF-only countries had cost overruns averaging 13 percent. On the other hand, cost underruns averaging 12 percent were experienced by two projects in blend countries. Cost overruns were experienced because of implementation delays; changes in design and scope; and large increases in the cost of civil works, consulting services, and land acquisition.

Figure 2: Average Implementation Period of Projects Evaluated in 1998 Estimate vs. Actual

0 1 2 3 4 5 6 7 8 9

Finance

Social Infrastructure

Transport andCommunications

Energy

Agriculture and NaturalResources

Number of Years

ActualEstimate

Source: Appendix 5.

AgriculturalSupportServices

Figure 3: Average Cost Underrun/Overrun of Projects Evaluated in 1998by Sector/Subsector

-40

-30

-20

-10

0

10

20

30

Agriculture andNatural Resources Energy

Transport andCommunications

SocialInfrastructure

Ave. (%)

Industrial Crops& Agro-Industry

Irrigation & RuralDevelopment

AgriculturalSupportServices

ElectricPower

Road Transport

Railways

Education

Health andPopulation

13

c. Bank Supervision 34. As in previous years, the 1998 evaluation of projects revealed a mixed record concerning the Bank’s role in monitoring project implementation and providing support to the EAs. The effectiveness of the Bank’s supervision was an important factor in satisfactory performance, i.e., identification of problems and remedial actions were timely. Overall, the experience of these projects underscored the importance of fielding frequent and timely review missions so that problems with the scope and design, as well as implementation-related matters, could be addressed in the early stages. 35. Good supervision is illustrated by the Irrigated Command Area Development Project in Indonesia. The Bank sent 12 review and loan administration missions with the appropriate mix of expertise. The missions were helpful in their recommendations for overcoming problems contributing to the initial implementation delays. Guidance on the Bank’s procurement and disbursement procedures was provided to the two EAs; this improved disbursements. The midterm review (MTR) mission, fielded two and a half years after project commencement, was effective in resolving major issues concerning procurement, interagency coordination, civil works construction, and land development activities. The 17 supervisory and monitoring visits made by the Bank under the Second Livestock Development Project in Nepal facilitated its implementation. A significant result of these visits was the relatively speedy establishment of an imprest fund to overcome payment difficulties that arose because of differing Bank and the Government approval processes. 36. The experience of the Nepal Mini-Hydropower Project demonstrates the importance of addressing project weaknesses when found during implementation, and addressing their underlying causes rather than simply reducing scope to meet the appraised budget. Despite COSO’s recommendation to field an inception mission, no such mission was sent. Although the Project was the EA’s first involvement with a Bank loan, the progress of critical activities such as site survey/design and the promotion of productive uses of electricity were not closely monitored. The first Bank review mission, fielded two years into implementation, also had to review five other ongoing projects and four TAs in the country. While the mission recognized several problems that were to adversely affect project performance, they were not addressed. Effective follow-up to the review mission was not carried out. Subsequent missions, typically once a year during implementation, had to deal with several other larger projects. In fact, the Bank did not conduct site visits during the nearly 10 years of implementation. The Project was rated unsuccessful. Similarly, Bank on-site supervision during construction of the Shanxi-Xiaoliu Railway Project was limited. Three brief review missions took place over three years, but all

14

involved review of other projects as well. Progress was reviewed without visiting the project site. Instead, project officials met the missions in the capital city, often in combination with work on other loan projects or TAs. An assessment of the Bank’s MTR process was the subject of a SES30 conducted in 1998 (para. 50).

B. Technical Assistance Performance Audit Reports

37. In line with OEO’s recent initiative to assess the impact and development effectiveness of the Bank’s TA operations by evaluating clusters of TAs, four TPARs were prepared in 1998. They relate to three advisory and operational TAs (AOTAs) on capacity building in the finance sector of Viet Nam; nine AOTAs in the power sector in India—the first evaluation report prepared for that sector in the country; four AOTAs for strengthening the capacity of the Bhutan Ministry of Finance, in particular the National Budget and Aid Coordination Division; and five AOTAs supporting the development of financial intermediation through institutional strengthening or subsector studies in the PRC. The degree of impact on the four countries was mixed. 38. In Viet Nam, the three TAs as a group contributed significantly to capacity building and institutional development in the finance sector. However, some of the specific objectives of two TAs, Strategic Development Plan for a Money Market, and Operationalization of the Debt Management and Financial Analysis System, were not fully met. The concepts and approaches adopted by the TAs were not all appropriate, primarily because many aspects of the basic framework for the finance sector were not in place or were inadequate. The key issues revolve around the sustainability of training benefits, the delay in implementing reform measures recommended by the TAs because of the Asian crisis, duplication of TAs in the same field by other funding agencies, and the need for an external debt management strategy and plan. 39. In Bhutan, except for the TA aimed at modifying a previous TA, the others covered by the review performed less satisfactorily. Given the complexity of public expenditure management, the Bank should have thoroughly examined the existing capacities in the country and applied a phased approach providing assistance over the medium to long term. The lack of institutional analysis and performance criteria impeded both the design and effectiveness of the TAs. 40. In the PRC, two TAs were rated as generally successful and three as partly successful. Performance varied depending on the initial conditions of the targeted institutions and subsectors, appropriateness of TA design, and performance of the consultants. To further improve TA operations in the PRC finance sector, the Bank needs to respond to the rapid evolution of finance sector development, require consultants to have country-specific knowledge and enhanced client orientation, and improve the monitoring framework. 41. The TAs in India form part of the Bank’s current strategy for India’s power sector, which focuses on a few states. Seven of the TAs fully achieved their intended objectives in terms of policy/plan development, institutional support, and capacity building. Of these, four currently administered TAs are contributing substantially to the ongoing reform process in Gujarat—demonstrating the Bank’s leverage in accelerating the reform process. The TPAR recommends follow-up of the TA recommendations by the concerned projects department and the Office of the Environment and Social Dimensions, and biannual reports by the EAs on the progress of implementation. 30 See SST:REG 98028: Midterm Review Process, December 1998.

15

C. Impact Evaluation and Reevaluation Studies

42. During the year, OEO carried out three IESs and four RESs (Appendix 7). An IES of secondary science education projects in Bangladesh, Nepal, and Pakistan focused on the projects’ impacts on institutions, graduates, and the policy and management environment. A second IES looked at Bank assistance to the power sector in the Pacific DMCs. An IES in Bangladesh assessed the adequacy, implementation experience, and operational performance of 32 PPTAs in the agriculture and natural resources sector, as well as their impact and effectiveness in contributing to the performance of ensuing projects. 43. The study on science education projects showed that, while the projects improved the institutional capacities of science education centers and secondary schools, such improvement did not always filter down to the students in terms of better learning achievements in science and related subjects. The main constraint continued to be suboptimal classroom conditions. Despite their shortcomings, the projects helped make the administrative and policy environment more conducive to science education, and provided important building blocks for the long-term development of the secondary education system as a whole. Issues like overcrowded classrooms and heavy teaching loads are being addressed under follow-up projects. The second study showed that, other than in Fiji Islands, power projects in the Pacific DMCs have had limited direct development impact largely due to institutional weaknesses; the utilities are generally poorly managed and have financial problems. The use of financial covenants alone to secure reforms has been the wrong approach to deal with structural problems such as tariffs and lack of autonomy. The third study found that PPTAs tend to produce a project design based on a predetermined project concept, rather than being a feasibility study to develop a project to achieve certain outcomes. There was minimal transfer of skills through most PPTAs, and the EAs generally did not improve their capacity to design projects themselves. Suggestions for improving PPTA are given in para. 72.

44. All three studies point to the need to strengthen TA, both project preparatory and advisory, in order to strengthen loan projects. Specifically, PPTA should undertake rigorous institutional capacity studies, develop EAs, be participatory, identify and analyze project risks, and incorporate project performance indicators. Advisory TA should focus on improving (i) O&M, (ii) standards of safety and environmental protection, (iii) efficiency—including lowering costs, (iv) management capacity for both administration and planning, (v) reporting and transparency of operations, and (vi) measurement of performance. 45. The four RESs exhibited mixed results. The reevaluation of the Second Vocational Education Project31 in Thailand confirmed the generally successful rating in the PPAR. The Project not only contributed to Thailand’s industrial development over the past two decades, but also expanded the technical and vocational education system in the country. In the Philippines, the Technical and Vocational Education Project32 was selected, though not earlier evaluated in a PPAR, to assess its long-term sustainability and impact. While the Project improved the physical and human resources of technical education institutes, it did not contribute to better management of the subsector. Project performance was rated as partly successful. 31 Loan 441-THA: Second Vocational Education Project, for $15.0 million, approved on 14 December 1979. 32 Loan 531-PHI: Technical and Vocational Education Project, for $27.0 million, approved on 8 October 1981.

16

46. The Palawan Integrated Area Development Project33 retained the PPAR’s partly successful rating. Its socioeconomic impact, though positive, has been less than expected, and the prospects for its sustainability are not bright. Overall, the Project has failed to provide the intended integrated development impacts. The rating of the Sabah/Sarawak Fisheries Infrastructure Project34 was downgraded from generally successful in the PPAR to unsuccessful during reevaluation. The earlier rating was based on expectations that all project facilities would be fully utilized by 1997. However, the Project’s overall contribution to increases in fisheries output was at most limited. By the time of reevaluation, the facilities were still underutilized.

D. Special Evaluation Studies

47. During the year, OEO carried out five SESs.

1. Evaluation of Port Projects

48. The study of port projects in India, Indonesia, Malaysia, and Philippines reviewed the Bank’s operational policies and strategies in the subsector by assessing the effectiveness of past Bank port projects in these countries. It identified changes in the transport sector and the port subsector, particularly those that would affect future Bank lending. The results showed that the considerable amount of infrastructure provided has allowed the ports to grow in response to rapidly increasing traffic. However, the need for Bank support of capital investment in port infrastructure has declined as the private sector has demonstrated a willingness to make capital investments where the returns are reasonable. The study highlights the need for a new approach to provide Bank assistance to the port subsector. Such an approach should look beyond the provision of physical infrastructure, facilities, and equipment to the efficiency with which these assets are used and the ability of port management to provide services that facilitate trade through the ports. Port services constitute one vital link in a chain of trade services that are part of a country’s production process; the trade service view should lead to a significantly different examination and valuation of project benefits for new port projects.

2. Operation and Maintenance of Road Facilities and Their Impact on Project Sustainability

49. The study attempted to derive generic issues relating to O&M by examining the experience in the road subsector. It reviewed 40 Bank-assisted road projects in five countries, namely PRC, Fiji Islands, Lao People’s Democratic Republic (Lao PDR), Nepal, and Thailand. The Bank’s experience supports the general observation of other aid agencies that the widespread lack of systematic and appropriate road maintenance in developing countries is mainly the result of lack of funds. Paucity of funds leads to weak institutional capacities reflected in lack of adequate management information systems and shortage of skilled staff. Even

33 Loan 528/529-PHI: Palawan Integrated Area Development Project, for $47.0 million, approved on 29 September

1981. 34 Loan 563-MAL: Sabah/Sarawak Fisheries Infrastructure Project, for $20.6 million, approved on 22 December

1981.

17

Thailand, which has traditionally financed all its routine maintenance needs, finds it difficult to have a workable maintenance program. The country’s pavement management system, on which plans and budgets are based, runs on the assumption of relatively unconstrained budgets, leaving little scope for flexible response to accommodate smaller allocations by using less costly repair methods. With the recent regional financial/economic crisis, repairs that cannot be implemented in the manner specified in the system are left untouched until funds become available. The study reveals, however, that even with insufficient funds, O&M receives better attention in countries with relatively strong economies and/or with a better developed road network, as in the case of PRC, Fiji Islands, and Thailand. In countries with weak economies like Lao PDR, an option for dealing with future O&M costs is to construct overdesigned roads that will require less maintenance. O&M can then focus more on operational aspects, particularly road safety and traffic management, that can easily gain public support as their impacts are immediate and highly visible. The study recommends that the DMCs and the Bank need to strengthen their efforts in the areas of road financing and cost recovery measures, institutional programs, standardization of maintenance equipment, linking disbursement with adherence to loan covenants, and participation of road users and beneficiaries. More importantly, the study highlights that the problem of inadequate O&M can be attributed to the weak link between planning and budgeting in most DMCs. National planning focuses on capital investment expenditures and is normally managed by the planning agency or its equivalent. Recurrent expenditures (of which O&M is a component) on the other hand are often determined by the ministry of finance in collaboration with the budget agency. Assistance in addressing the problem of O&M, therefore, must also be aimed at improving the country’s overall budgetary process, and not just at the individual project level.

3. Midterm Review Process

50. A review of the Bank’s MTR process for project lending was conducted to determine whether the process should be retained. The findings of the study show that the MTR is an effective mechanism, fully within the Bank’s control, for taking corrective action during project implementation, thus playing an important role in the Bank’s efforts toward improving project quality and streamlining its business processes. However, this potential was found to be underutilized. At the root of this unsatisfactory situation is the problem of limited staff resources and mission budgets. The study notes that, while a properly structured MTR requires significantly more resources than a regular review, this can be offset by reductions in delays and other implementation cost-escalating factors. More importantly, the potential exists for greater project benefits. While not all projects would necessarily have to undergo MTR, the Bank would need to allocate additional resources for project administration over and above the current allocation if an effective MTR system is to be established. The study recommends that, for all projects, the loan agreement should specify a milestone or date at which a decision should be made on whether or not the MTR is needed. A position paper would be prepared by that time assessing progress on the project and identifying relevant issues, including terms of reference for the MTR. Preparatory work for the MTR, starting with the position paper, should involve the executing/implementing agencies, ministries of finance and planning, and other stakeholders. In case the MTR is considered unnecessary, the justification for waiving it should be included in the position paper. The study also recommends that MTRs question the relevance of project objectives in the light of changing conditions, review social and environmental aspects, and compare them with the original environmental impact assessment (EIA) and related covenants specified in the loan documents.

18

4. Factors Affecting Project Performance in the Agriculture and Social Sectors

51. A study of the factors affecting project performance in the agriculture and social sectors was OEO’s first attempt at using statistical procedures to identify potential explanatory variables for success and failure specific to a sector. It examined 141 PPARs done between 1991 and 1997 on projects approved between 1981 and 1992. Because the study was limited by data and information constraints, caution was exercised in applying the results of the statistical tests that confirm and quantify the relationships between measures of project performance and causative factors. In particular, information constraints made it difficult to analyze the impact of such variables as use of a participatory approach and environment, as projects referring to them were limited. The major findings of the study, which were based on statistically significant variables and were concurred with by operational departments based on their experience, are as follows: (i) ensuring timely project implementation and disbursement levels is insufficient to achieve the intended project benefits in the agriculture and social sectors—many other factors appear to have more substantial impacts on project performance; (ii) both project-specific and external factors that impact significantly on project performance should be given equal attention; (iii) good borrower performance—implying a policy environment favorable to project operation, government commitment to a project, and provision of counterpart funds—can improve project performance significantly; and (iv) project success in the sectors depends substantially on the EA’s capability to supervise project implementation and properly utilize facilities, and on its commitment to maintenance and repair. The study emphasizes that the Bank’s foremost contribution can come at the PPTA stage, when its profile is very high, reinforcing the conclusion of the IES on agriculture PPTAs in Bangladesh. Thus, there is a need to undertake a critical review of Bank policies and practices relating to PPTAs with a view to enhancing their positive contribution to project performance (para. 72).

5. Environmental Mitigation Measures in Selected Bank-financed Projects

52. A study of the environmental mitigation measures (EMMs) in selected Bank-financed projects focused on two thermal energy generation projects each in PRC, India, Indonesia, and Philippines, and two urban development projects in Indonesia. The study found that the EIA process required by the Bank has often resulted in successful identification and mitigation of major environmental problems. The study projects generally complied with environmental protection regulations and guidelines of the respective DMCs, though the mandates and capacities of their environmental management agencies to regulate and monitor EMMs seemed low. The study concludes that environmental protection needs greater attention at the planning and feasibility study stage, including broader consideration of technologies and systems for pollution prevention, by-product utilization, waste minimization, energy efficiency, water conservation, system cost-effectiveness, and O&M requirements. Site location, technology, and management expertise were the most important factors affecting the success of EMMs. The scope and timing of the EIAs of the study projects indicated that EMMs were often designed after the site and the technologies for the plant had been selected. 53. Some of the study’s major recommendations are as follows: (i) the EIA, when necessary, should be done during the feasibility stage prior to finalizing the site and type of technology—including strategic, regional, and cumulative studies, as well as risk assessment and emergency response planning for some projects; (ii) the Bank should consider more detailed itemizing and earmarking of greater portions of a loan or project financing to implement EMMs; (iii) the Bank

19

needs to develop and implement an EMM evaluation and reporting system as part of its operational procedure for sensitive projects but not as a stand-alone system; and (iv) the Bank should promote regular external audits of large power plants to ensure compliance with regulations and continue to develop the institutional capacity, as well as the enforcement, compliance, and monitoring capabilities of environmental management agencies, depending on each country’s needs. Furthermore, there is a need to review the application of proposed external audits to ongoing projects. The implications, including the costs of the multipartite monitoring groups, should also be examined.

E. Country Assistance Program Evaluation

54. In 1998, OEO reviewed the Bank’s assistance program in the PRC from 1987 to 1997. While the CAPE’s main concerns are the relevance, efficacy, and effectiveness of past Bank operations, the approach and methodology appropriate for such an evaluation are still evolving. This CAPE assessed the relevance of the Bank’s strategy in terms of the needs of the PRC and the constraints that narrow the choice of operational options. During the early years of PRC borrowing (1986-1990), Bank operations were guided by the Government’s intention to borrow mainly for industrial development, and generally in the more developed eastern coastal region. As a result of dialogue between the Bank and the PRC, the lending program has become more diversified in terms of both sectors and geographic regions. Five of the eight completed projects approved in the early years were evaluated, of which three were judged generally successful and two unsuccessful. Of the completed projects from the later period, those in energy and infrastructure were all generally successful, while those in agriculture, industry, and finance were mostly partly successful. 55. Based on OEO’s assessment and on broad agreements reached with the recipient authorities and the Bank’s regional staff, several major implications were identified, namely, (i) country operational strategy (COS) should be made more operational and provide a better basis for evaluation of success, and the country assistance plan (CAP) should more fully reflect the strategic directions of proposed operations; (ii) in light of the need to improve development impact and project quality, as well as in the broader context of good governance, the Bank should support the work of the recently established Inspectorate under the State Development Planning Commission; (iii) regardless of the dominance of PRC authorities in the project planning and preparation process, the Bank needs to apply due diligence in reviewing and accepting projects; (iv) the Bank’s TA program in the PRC should be more focused, better targeted, and managed more effectively; (v) high-level dialogue between the Bank and the PRC is needed to agree on a program of TAs and loans to make greater progress in this area, as in the case of the finance sector reforms, where the Bank focused on strengthening the infrastructure for enterprise reform; (vi) the Bank should consider putting safety net projects under the poverty classification; and (vii) the Bank should continue its approach in the environment sector and extend this to green issues, which focus on preservation of the natural environment, especially living natural resources.

F. Annual Performance Evaluation Program

56. Following the format of the earlier annual performance evaluation program (APEP), the 1998 report focused on the activities of various departments and offices as they relate to project

20

and portfolio performance, with a view to providing more effective feedback from Bankwide self-assessment of project performance for improving project quality. It provided both a review of activities in 1997, together with activities planned for 1998, and an update of actions on recommendations made under the earlier APEPs. A key conclusion of the 1998 APEP is that the single most important action the Bank can take to improve portfolio performance and project outcomes is to strengthen the quality of its own project supervision. This requires an improved performance measurement and monitoring system, which the introduction of the PPR and the Project Performance Management System (PPMS) targets. Almost as important is the need for more thorough and comprehensive economic and sector analysis to help improve project identification and design. The other recommendations are to (i) provide support for the interdepartmental working group recommendations on improving the COS/CAP process and country portfolio review missions (CPRMs), including the establishment of performance targets; (ii) systematically evaluate the COSs and reassess their relevance on a yearly basis through the CAP exercise; and (iii) support the proposed guidelines for the CPRM, which cover the need for a review of sector and policy issues affecting portfolio performance, and for developing appropriate performance indicators, an area where more work is needed.

III. DMC CAPABILITIES, FEEDBACK, AND FOLLOW-UP OF EVALUATION FINDINGS

A. Strengthening the Performance Evaluation Capabilities of Developing Member Countries

57. During 1998, OEO administered three TAs aimed at developing/strengthening the PPMSs in Nepal, Philippines, and Sri Lanka; and one TA to the PRC for strengthening the evaluation capacity of the China National Audit Office, State Development Planning Commission, and State Development Bank. The findings of a TA on the evaluation of Bank assistance to DMCs for BME35 highlighted the need to improve BME by establishing conceptual clarity and changing the terminology itself to reflect the revised concept. The TA resulted in a shift to the PPMS as the Bank’s approach to monitoring and evaluating projects. A handbook on the PPMS as a practical guide for Bank staff and DMC counterparts was developed under the TA to facilitate the shift to the new system. It is now mandatory that all new projects under processing have a project framework, which constitutes the basis of the PPMS, properly built into them. The findings of a TA on the study of project planning and management in the PRC highlight the effectiveness of conducting workshops to share development experiences of best practices in project planning and management among member countries. Such workshops not only enhance DMC project quality but also strengthen partnerships between DMCs, and between the Bank and its DMCs.

35 TA 5636-REG: Evaluation of Bank Assistance to DMCs for Benefit Monitoring and Evaluation, for $450,000,

approved on 22 June 1995.

21

B. Feedback of Evaluation Findings

58. The computerized Postevaluation Information System, which was made accessible on-line to all Bank staff in 1997, was expanded. By the fourth quarter of 1998, OEO had operationalized a World Wide Web home page on evaluation findings and lessons, enhancing the current feedback system. This intranet/internet system enables downloading documents and conducting advanced searches for information in project completion reports (PCRs), PPARs, TCRs, TPARs, IESs, RESs, and other reports prepared by OEO. It will have links with the websites of other aid organizations and will provide an easier way of sharing evaluation information with the DMCs as well as the general public. 59. As part of the effort to enhance the effectiveness of the feedback system within the Bank, OEO attends loan management review meetings, as well as staff review committee and loan and technical assistance coordination committee meetings. OEO likewise comments on project and TA documents prepared by various departments and offices. These documents include concept and/or position papers; project briefs; various drafts of the report and recommendation of the President and TA reports; PCRs; TCRs; CAPs; as well as policy, strategy, and other Board papers. This close interaction of OEO with the other departments in the Bank facilitates the application of lessons learned to improve new as well as ongoing projects and development activities.

C. Follow-up of Recommendations in Evaluation Reports

60. Over the years, evaluation reports have generally identified both project-specific and broad issues for appropriate follow-up actions to be taken by the Bank, borrowing DMCs, and EAs. Following established procedures, the proposed follow-up actions and recommendations contained in the reports prepared by OEO in 1998 were brought to the attention of the projects departments and the concerned agencies in the borrowing DMCs. The actions taken on these recommendations, as monitored by the projects departments, are summarized in Appendix 8. 61. Several reevaluation reports and annual reviews have emphasized that the performance of many partly successful and even unsuccessful projects has improved when remedial actions are taken to resolve problems identified in evaluation reports. The Bank is able to monitor actions taken on recommendations made in project completion and postevaluation reports only when there is a follow-up project or when a similar project is being implemented by the same EA in the same sector. While initial steps have been taken by some projects departments, beginning in 1997, to field review missions on an ad hoc basis to discuss the implementation status of follow-up actions on completed projects with concerned agencies, the Bank has yet to establish systematic monitoring of follow-up actions once projects are completed, particularly for those where subsequent investments are not planned. Only a limited number of such missions can be fielded because of staff and budget constraints. It is again suggested that a study of the budgetary and staffing implications of such monitoring and follow-up be undertaken.

22

D. External Coordination

62. OEO continued close coordination with other multilateral aid agencies on evaluation activities through the Evaluation Cooperation Group (ECG) of the multilateral development banks. The purposes of the ECG are to strengthen evaluation for greater effectiveness and accountability, share lessons learned, harmonize performance indicators and methodologies, enhance professionalism, and facilitate the involvement of borrowing countries in evaluation. In 1998, the ECG worked to promote convergence of evaluation methods and practices, especially in examining private sector operations. The Group also developed plans to produce a biannual newsletter devoted to the evaluation activities of its members.

IV. KEY ISSUES

63. While a review of all evaluation reports since the establishment of OEO would show many recurring issues, the goal of evaluation efforts each year is to discuss problems and issues in the light of new information and procedures, and then to point the way to solutions. The issues thus identified and discussed here are

(i) greater focus on O&M after project completion; (ii) mitigation of environmental effects of Bank projects; (iii) revising PPTA to improve project performance; (iv) Asian financial crisis: adjustments to ongoing projects; and (v) strengthening design and evaluation to improve project quality.

Proper consideration of these issues in ongoing and future projects/programs will increase beneficiary participation, enhance government and EA ownership, and improve project/program quality and development impact.

A. Greater Focus on Operation and Maintenance after Project Completion

64. The inadequate provision by governments and EAs of funding for O&M of completed Bank-assisted project facilities is a long-standing issue leading to diminished effectiveness of Bank assistance. The 1998 SES on Operation and Maintenance of Road Facilities and Their Impact on Project Sustainability (para. 49) is by no means the only voice pointing out the critical nature of O&M to Bank projects. Infrastructure projects, and in fact projects in practically all sectors, are often adversely affected by poor maintenance of facilities, leading to loss of project benefits and decreased development impact. 65. The need to properly maintain assets may be well recognized during project preparation and even formalized through loan covenants and other assurances, but it is generally soft-pedaled because it will occur only after a project is complete. Despite the assurances, the necessary expenditures may not be forthcoming after loan closing for a number of reasons. In the public sector, frequent mention is made in evaluation reports of difficulties with government procurement and budget procedures, inappropriate staff deployment, low salaries, and lack of

23

other incentives. However, it is more often a problem of government priorities and of affordability. The willingness of the Bank and other funding agencies to provide financing for rehabilitation of long-neglected facilities has tended to mask this issue. Many of the Bank’s road projects demonstrate that governments have opted for one-time road rehabilitation instead of annual routine maintenance, and in many cases this has been an expensive choice; the cost of rehabilitation often exceeds the capitalized value of annual maintenance interventions. Bank loans for rehabilitation have thus sometimes not only supported a relatively costly alternative, but have also substituted for governments’ annual maintenance expenditures. The promise of Bank financing for rehabilitation may also have provided a disincentive for governments to allocate internal funds in some cases. 66. Some sectors are not able to utilize allocated O&M funds, while others do not receive enough. To improve O&M policies and systems, the Bank and its DMCs need to strengthen their efforts in several areas. Recognizing that the issue of limited funds for O&M is a country concern and not project specific, the practice of aid agencies individually seeking guarantees of adequate recurrent funding for particular projects needs to be reviewed. Policy discussions with the government, if carried out jointly by major aid agencies in a country, could have a better impact than if carried out individually. In particular, assistance in addressing the problem must be aimed at improving the country’s overall budgetary process—this will assure the provision of adequate resources for O&M of priority projects in all sectors. Other areas that need further attention include cost recovery measures, increasing the participation of beneficiaries, and privatizing all or part of the facilities and services. Institutional arrangements for O&M and for takeover of facilities by appropriate agencies need to be defined in detail before any project is approved. The Bank should continue to explore effective means for funding O&M, and should consider refraining from financing projects for which the borrower cannot afford recurrent expenditure. One way to assess affordability is for appraisal missions for follow-on projects to visit similar, already completed projects for a look at their O&M situation.

B. Mitigation of Environmental Effects of Bank Projects

67. Environmental protection is one of the Bank’s strategic objectives. Yet infrastructure projects and those with infrastructure components, by their very essence involving alterations of the natural setting, are prone to creating environmental problems. The SES on Environmental Mitigation Measures in Selected Bank-Financed Projects (paras. 52-53) found that the Bank’s EIA process is working reasonably well at the project design stage, and that the projects audited were generally in compliance with the environmental protection regulations and guidelines of the respective DMCs. Also, most of the PPARs prepared in 1998 found no adverse environmental effects due to the projects. But full-scale environmental assessment is usually not within the scope of a PPAR and would be limited to projects highly susceptible to environmental effects. Even in these cases, however, limited Bank resources would not always permit full-fledged environmental audits. Also, while the foreign exchange costs of EMMs (often for equipment) may be covered under a loan, the requisite local expenditures to provide training for monitoring and maintaining the EMMs may be delayed. The Bank should consider more detailed itemizing and earmarking of portions of a loan to implement EMMs. Under its operational procedures, the Bank has not systematically monitored a project’s environmental compliance during project implementation, much less during operation. Such assessment is necessary, though, as even occasional noncompliance can harm the environment. There has thus been a need to develop and implement an EMM evaluation and reporting system,

24

especially for sensitive projects, on a project-specific basis. At the request of the Board, COSO has revised the PPR to include a field to indicate the status of compliance with environmental covenants. 68. The SES found that some of the plants studied had experienced difficulties due to improper planning, such as waste disposal problems, inadequate space to install equipment required by new regulations, and overestimation of demand for environmental services. Adequate attention therefore needs to be given at the project planning stage to training, preparing contingency plans, using forward-looking approaches to formulate environmental regulations, and designing environmental services utilizing willingness-to-pay surveys and public promotion. 69. Furthermore, although the role of public awareness and public participation in effecting EMMs in the DMCs is recognized and usually built into projects, this role needs to be strengthened. Greater public participation, especially during project planning and design, including during the development of detailed terms of reference for EIA, can improve the design, operation, and monitoring of EMMs. Increased transparency is also expected to provide greater flexibility in adjusting the EMMs to suit changing circumstances in a particular country, sector, or project. The SES highlights a mechanism for beneficiary participation in environmental monitoring of utility projects that is employed in the Philippines: Multipartite teams consisting of representatives of the local community, local government, local nongovernment organizations, the environmental management agency, and the utility company are assembled prior to construction and meet regularly well into the operations stage to address the implementation of EMMs. This system has wide applicability even beyond utility projects and should be encouraged in environmental category A projects.

C. Revising Project Preparatory Technical Assistance to Improve Project Performance

70. The Task Force on Improving Project Quality (1994) raised concerns about the impact of PPTA on project performance. In 1998, OEO conducted two studies that looked at PPTAs from different perspectives and largely followed up on the issues first raised by the Task Force. An IES examined the effectiveness of PPTAs in the agriculture and natural resources sectors in Bangladesh. The SES on Factors Affecting Project Performance in the Agriculture and Social Sectors, among other aspects, attempted to assess the contribution of PPTAs to the quality of the ensuing loan projects. The studies found that PPTAs have largely improved in quality and effectiveness. However, both studies concluded that many PPTAs still fall short of their intended objective to provide successful project designs. Specific findings were that PPTAs

(i) should pay greater attention to institutional factors such as the institutional capacities of EAs, and recommend appropriate assistance;

(ii) must include adequate analysis of relevant project alternatives; (iii) were often not fully supported by EAs so that ownership and commitment

presented problems; and (iv) did not seek adequate beneficiary participation, and many designs were

produced without participation of the affected population.

25

71. As these observations may be made concomitantly about many PPTAs, the PPTAs themselves may have become the very cause for less successful loan projects. This seems to be suggested by statistical analysis, which shows a relatively high correlation between the large negative gaps in EIRRs (at the postevaluation stage compared with the appraisal stage) and the existence of preceding PPTAs. This hypothesis needs further testing.

72. There is a need for the Bank to review the concept, role, and various technical aspects of PPTA. Such a review should be conducted in the broader context of the Bank’s business process cycle, and not in isolation as a segmented, nonlending operation. The requirement for greater participation by the stakeholders in PPTA, especially the beneficiaries of the ensuing project, calls for a significant change in PPTA formulation and implementation. The terms of reference for PPTA consultants should clearly indicate the consultants’ responsibility to work closely with the EAs and beneficiaries. Users (EAs and beneficiaries) need to be involved from the identification of their needs through the planning of mechanisms to implement the project. Clearly, more time and financial resources need to be allocated for this part of the cycle. The additional resources invested in strengthening PPTA are expected to pay off in terms of more successful loan projects, and consequently, fewer resources needed for loan supervision. Such an approach to PPTA might be tried on a pilot basis. Apart from such a fundamental change in the concept and modality of PPTA, several measures could be implemented to increase, to a certain extent, its effectiveness, for example, (i) making sure that PPTAs for similar and follow-on projects critically judge the outcomes and experiences of earlier completed as well as ongoing projects, and not just replicate previous designs; (ii) strengthening the PPTA process by allowing iterations in decision making and by providing decision makers with design alternatives; and (iii) setting out clear and comprehensive guidelines for PPTA, covering its formulation, implementation, completion, and evaluation, to improve overall PPTA operations.

D. Asian Financial Crisis: Adjustments in Ongoing Projects

73. The regional economic/financial crisis, with its currency devaluations, stock and commodity market slumps, budgetary deficits, and high inflation and interest rates, raises concern about the performance and sustainability of current projects. In fact, the experience of the 1998-evaluated projects highlights some aspects of the crisis. Also, SESs and IESs examine ongoing projects, and so their connection with the crisis is closer than PPARs and RESs. As an example of OEO findings, the economic crisis in Indonesia and the resulting reduction in demand for construction materials made it difficult for the Suralaya power plant to utilize the ash it generates, with implications for its financial projections and for the environment. The need to provide social safety nets in the light of the impact of the crisis on the poor is a further drain on the already limited budgets of governments. Continuation of the contagion across and within DMCs would raise further doubts about project sustainability, as O&M of most project facilities will become increasingly limited. 74. In the education sector, as discussed under the RES on the Second Vocational Education Project in Thailand, concerns over the social impact of the economic and financial crisis call for a reexamination of policies and strategies in human resource development such that quick adjustments can be made in the training of technical personnel to meet rapidly changing demands. Technical/vocational education projects must be designed to increase efficiency and effectiveness in matching labor demand and supply through better selection and improvement of relevant course offerings and through imparting multiple, broad-based skills to

26

workers. The best route to this end is a more effective partnership between schools and industry. The orientation of technical education from a supply-driven system based on a large social need must shift to a demand-driven system guided by labor market exigencies. 75. The impact of the crisis is even more apparent in the finance sector. Experience has shown that weakening of the growth performance of several DMCs in the region, together with devaluation of their currencies, has highlighted the risks associated with fragile banking systems. In Viet Nam, for example, the contagion has caused a slowdown of reforms in the finance sector, while bringing out problem areas in need of immediate attention such as poor standards of corporate governance and management. The crisis has reinforced the belief that the lack of careful monitoring of financial institutions in many countries has contributed to the situation. The crisis has strengthened the view that the performance of financial institutions needs to be assessed in terms of capital adequacy, loan loss reserves relative to assets, profitability, and the magnitude of nonperforming loans. The Bank has provided critical assistance to crisis-hit economies to address these issues through program loans and related TAs for reviewing and assessing weakened financial institutions, and for strengthening the framework for finance sector regulation and supervision. 76. Closer attention is needed to assess the impact of the contagion on ongoing Bank-assisted projects and, as necessary, to making in-course corrections. The newly instituted PPR/PPMS process is a mechanism that will facilitate corrective action in real time. The CAPE exercise, and self-evaluation elements of COSs and CAPs, may also prove to be early warning devices. As the Bank moves into the new millenium, it must pursue all avenues of rapid feedback in order to anticipate problems and work for their resolution before becoming overwhelmed by them.

E. Strengthening Design and Evaluation to Improve Project Quality

77. The Bank’s and other aid agencies’ increasing concern about the impact and effectiveness of their assistance has added a new dimension to evaluation. While learning through evaluation remains a key function, evaluation has also become an important measure of accountability, and more generally, of corporate governance. This applies to the Bank’s clients and to the Bank itself. In 1998, the Bank implemented evaluation capacity-building TA to the PRC and Nepal, and it will pursue this governance objective in other DMCs in the future. The Bank is, however, aware that its own procedures also need to be strengthened. The broader significance that evaluation has assumed means that more rigorous standards and practices must be applied. If evaluation is to meet the heightened expectations, it will have to produce results that are transparent and operationally useful. Technical skills and analytical rigor are necessary prerequisites. But equally important conditions are project objectives and designs that lend themselves to unbiased monitoring and verification. In a word, projects must be evaluable. 78. Experience in this regard in 1998 was mixed. OEO’s review of appraisal documents and discussions at management review meetings revealed common shortcomings: Project objectives in some cases tended to be vague, monitorable success indicators were often absent, and the use of logical frameworks was not rigorous. Evaluation of projects had to rely heavily on intuition and subjective judgement. As a result, a common basis for self-evaluation through PCRs and for independent evaluation through PPARs did not necessarily exist. It is

27

therefore hardly a surprise that the PCR and PPAR exercises show different success rates, with the former’s ratings sometimes being downgraded by the latter. Of course, factors other than lack of evaluability contribute to this discrepancy. An important one is the difference in vantage points of the two exercises, the PCR normally occurring within a year of closing and the PPAR being a few years later. It is, however, the vagueness of project designs, allowing different interpretations of outcomes, that enables variation in the basis for evaluation. 79. In 1998, important inroads were made to improve this situation. The use of the logical framework is now mandatory, and the introduction of the PPR and PPMS enforces the use of specific monitorable parameters concerning development objectives as well as outputs during project implementation. The logical framework and the PPMS are complementary to the extent that the PPMS depends on the objectives, milestone events, and monitorable indicators determined in the logical framework. However, while the new approach will improve the evaluability of projects, it must not be taken as a solution per se. If done properly at the beginning of project planning, the logical framework imposes desirable discipline on the thought processes that can generate rational project designs, but this does not ensure that such designs will emerge in all cases. OEO’s observations to date indicate that the quality of logical frameworks is mixed. They frequently tend to avoid the required degree of specificity and the succession of cause and effect, and sometimes appear to have been constructed after the project was designed, thus defeating their intended purpose. They do not indicate expected outcomes and impacts, only outputs. OEO can help strengthen the awareness of and consensus on the “objectively verifiable indicators” column of logical frameworks. Particular attention needs to be paid to program loans and their policy matrices. 80. Evaluability therefore remains an important issue. Data collection based on impacts identified in a project’s logical framework needs to be planned prior to project approval. Sufficient monitoring needs to be carried out during implementation to provide the data needed for the PPMS. Such data collection should continue during project operation as well, to support project management. Whether the observed gap between success rates can be narrowed will depend on the efforts of Bank staff and DMC counterparts to monitor development impacts during implementation and beyond. OEO, in cooperation with COSO, will continue holding training sessions for operational staff, and a help desk has been established to advise individual staff on conceptualizing impact indicators and devising data collection programs.

28

APPENDIXES

Cited on

No. Title Page (page, para) 1 Profile of Projects Evaluated in 1998 28 1, 3 2 Overall Performance Assessment of Projects Evaluated in 1998 31 3, 7 3 Historical Performance of Evaluated Projects by Economy Group/Economy 35 4, 8 4 Profile of Evaluated Private Sector Loans 36 9, 27 5 Average Implementation Period of Projects Evaluated in 1998, Estimate Versus Actual 37 10, 30 6 Average Cost Underrun/Overrun of Projects Evaluated in 1998 39 11, 33 7 Summary of Findings from Impact Evaluation and Reevaluation Studies 41 14, 42 8 Follow-up Actions on Major Findings and Recommendations of Evaluation Reports Completed during 1998 44 20, 60

PROFILE OF PROJECTS EVALUATED IN 1998

Project/TA CostReport Loan/TA Date Loan/TA Amount ($mn) Expected Actual Overrun/ Time Overrun/

Type/Title No. Approved ($ mn) ($ mn) (Underrun) Expected Actual Performance(%)

A. Project Performance Audit Reports

Mini-Hydropower Project PE-503 512 NEP(SF) 21-Apr-81 8.30 8.30 15.15 14.94 (1.39) 31-Dec-84 30-Jun-91 USShanxi-Xiaoliu Railway Project PE-504 948 PRC 31-Jan-89 39.70 37.29 55.04 57.62 4.69 31-Dec-91 31-Dec-93 USIrrigated Area Command Development Project PE-505 818 INO 11-Dec-86 28.80 18.39 52.10 42.89 (17.68) 30-Jun-92 09-Oct-93 GSSecond Hill Irrigation Project PE-506 596 NEP(SF) 04-Nov-82 20.00 11.08 24.30 8.85 (63.58) 31-Dec-89 31-May-95 PSHighland Agriculture Development Project PE-507 802 PHI 25-Nov-86 18.80 18.43 26.90 25.51 (5.17) 31-Mar-92 31-Jul-94 GSFarm-to-Market Roads Project PE-508 758 PAK(SF) 26-Nov-85 29.30 36.40 44.00 54.00 22.73 30-Sep-90 31-Aug-94 GSSecond On-Farm Water Management Project PE-509 871 PAK(SF) 08-Dec-87 28.50 27.57 55.16 51.78 (6.13) 30-Jun-93 31-Dec-94 GSSecondary Towns Power Distribution Project PE-510 870 SRI(SF) 03-Dec-87 34.80 34.30 47.90 52.20 8.98 30-Jun-91 30-Jun-94 GSSecond Livestock Development Project PE-511 745 NEP(SF) 24-Oct-85 14.00 6.90 17.50 8.80 (49.71) 30-Jun-91 30-Jun-94 PSSecond Health and Population Project PE-512 710 PAK(SF) 29-Nov-84 16.00 10.80 20.00 14.50 (27.50) 30-Dec-90 30-Sep-93 PSChashma Command Area Development Project PE-513 723 PAK(SF) 13-Dec-84 40.00 37.46 71.50 52.68 (26.32) 30-Jun-90 30-Apr-94 PSSeventh Power Project PE-514 751 BAN(SF) 31-Oct-85 40.50 44.60 66.00 82.80 25.45 30-Jun-90 30-Sep-94 GSRoads Improvement Project PE-515 806 NEP(SF) 02-Dec-86 30.00 31.30 42.50 44.40 4.47 30-Jun-90 30-Jun-95 GSAgricultural Research and Extension Project PE-516 1110 PNG(SF) 29-Oct-91 22.00 22.11 27.52 28.62 3.99 30-Jun-94 25-Apr-96 PSHexian Pulp Mill Project PE-517 937 PRC 20-Dec-88 49.60 49.60 82.95 91.57 10.39 31-Dec-91 31-Jul-95 USHighland Livestock Development Project PE-518 808 BHU(SF) 02-Dec-86 4.30 4.68 7.88 9.24 17.30 31-Dec-91 31-Mar-95 GSFourth Road Improvement Sector Project PE-519 690 PNG 04-Sep-84 33.25 38.32 48.50 55.93 15.32 30-Sep-89 30-Mar-93 PS

691 PNG(SF)Bhutan Development Finance Corporation PE-520 934 BHU(SF) 13-Dec-88 2.50 2.26 2.50 2.26 (9.76) 03-May-93 15-Nov-94 PSFauji Fertilizer Expansion Project PE-521 7047/1003 PAK 19-Dec-89 30.00 30.00 330.00 310.00 (6.06) 31-Oct-92 31-Mar-93 GSSecond Cook Islands Development Bank Project PE-522 1155 COO(SF) 14-Jan-92 1.50 1.50 1.50 1.50 0.00 25-Jun-96 02-Oct-95 PSAgricultural Technology Education Project PE-523 884 PHI(SF) 08-Mar-88 15.86 14.76 24.76 25.53 3.11 31-Mar-94 30-Sep-95 PS

B. Technical Assistance Performance Audit Reports

Advisory and Operational TAs in the Power TE-23 21-Nov-89 1.00 - 2.05 - - - - PSSector in India 21-Nov-89 0.66 - 0.69 - - 31-Jul-92 30-Sep-95 PS

31-Nov-89 0.74 - 0.77 - - 31-Jul-92 30-Sep-95 GS30-Aug-90 0.49 - 0.51 - - - - GS29-Sep-92 0.60 0.60 0.62 - - 30-Jun-93 10-Feb-94 GS17-Dec-96 0.60 - 0.65 - - 28-Feb-99 on-going -17-Dec-96 0.30 - 0.37 - - 30-Sep-98 on-going -17-Dec-96 0.24 - 0.26 - - 31-Mar-99 on-going -17-Dec-96 0.58 - 0.67 - - 30-Sep-98 on-going -

GS= generally successful; PS= partly successful; TA= technical assistance; US= unsuccessful.Source: Postevaluation Information System.

(Years)

-

3.2--

0.6

-

-

Approved Disbursed

--

3.2

No.

3.02.83.8

(Underrun)Completion Date

1.5

3.5

1.50.4(0.7)

5.01.83.6

4.3

6.52.01.35.42.33.91.53.0

2741-IND

1756-IND2738-IND2739-IND2740-IND

1228-IND1229-IND1365-IND1366-IND

PROFILE OF PROJECTS EVALUATED IN 1998 (continued)

Project/TA CostReport Loan/TA Date Loan/TA Amount ($mn) Expected Actual Overrun/ Time Overrun/

Type/Title No. Approved Approved Disbursed ($ mn) ($ mn) (Underrun) Expected Actual Performance(%)

Institutional Strengthening of the National Budget TE-24 30-Apr-88 0.23 0.20 0.26 0.20 (21.32) - 31-Dec-90 USand Aid Coordination Division of the Ministry of 28-Feb-91 0.60 0.58 0.60 0.58 (2.83) 31-Dec-93 PSFinance

31-Aug-93 0.10 0.09 0.10 0.09 (6.00) - 30-Jun-95 GS30-Nov-96 0.10 0.10 0.11 0.10 (13.64) 31-May-98 PS

Capacity Building in the Financial Sector in the TE-25 27-Dec-93 0.60 0.51 0.64 0.51 (20.31) 28-Jul-95 GSSocialist Republic of Vietnam 17-Feb-94 0.60 0.39 0.64 0.45 (29.29) 30-Apr-95 PS

05-Dec-94 0.42 0.34 0.45 0.34 (24.61) 31-Jul-97 PSDevelopment of Financial Intermediation in the PRC TE-26 01-Dec-94 0.60 0.52 0.70 - - 30-Jun-97 GS

12-Jan-95 0.83 0.81 0.93 - - 30-Jun-98 PS16-Aug-96 0.45 0.31 0.49 - - 28-Feb-98 PS07-Oct-96 0.60 0.47 1.20 - - 31-Jul-98 GS

16-Oct-96 0.50 0.49 0.56 - - 28-Feb-98 PS

C. Technical Assistance Completion Reports

Evaluation of Bank Assistance to DMCs for BME 22-Jun-95 0.45 - - - - - GSPRC: Study of Project Planning and Management 08-Dec-95 0.23 0.21 0.56 - - - GS

D. Reevaluation/Impact Evaluation Studies

Technical and Vocational Education Project IE-52 08-Oct-81 27.00 20.20 38.50 35.50 (7.79) 30-Sep-89 PSSabah/Sarawak Fisheries Infrastructure Project IE-53 22-Dec-81 20.60 17.63 41.00 38.51 (6.07) 31-Mar-91 USPalawan Integrated Area Development Project IE-54 29-Sep-81 47.00 32.89 85.00 58.02 (31.74) 31-Dec-90 PS

Bank Project Preparatory TAs in the Agriculture IE-55Sector in BangladeshSecondary Science Education Projects IE-56Second Vocational Education Project IE-57 14-Dec-79 15.00 15.00 21.31 23.82 11.78 30-Jun-83 31-Dec-86 GSBank Assistance to the Power Sector in the Pacific IE-58Developing Member Countries

E. Special Evaluation Studies

Evaluation of Port Projects SS-30Factors Affecting Project Performance in theAgriculture and Social Sectors SS-31

BME = benefit monitoring and evaluation; GS= generally successful; PRC= People's Republic of China; PS= partly successful; TA= technical assistance; US= unsuccessful.

31-Mar-8630-Jun-8831-Dec-88

31-Dec-92

31-Mar-9731-Dec-9431-Oct-9431-Jan-9731-May-9731-May-98

28-Feb-9830-Sep-97

31-Dec-9631-Dec-96

1.2

1.0

Years

-

-

2039-VIE2063-VIE2218-VIE2214-PRC2288-PRC2626-PRC

2664-PRC2658-PRC

30-Apr-97

Completion DateNo. (Underrun)

970-BHU1487-BHU

1932-BHU2691-BHU

5636-REG2462-PRC

531-PHI563-MAL528-PHI529-PHI

5793-REG3.5441-THA(SF)

5734-REG

5715-REG

0.50.50.1

0.6

0.10.8

2.0

-

0.4

-

3.52.8

0.4

PROFILE OF PROJECTS EVALUATED IN 1998 (continued)

Project/TA CostReport Loan/TA Date Loan/TA Amount ($mn) Expected Actual Overrun/ Time Overrun/

Type/Title No. Approved Approved Disbursed ($ mn) ($ mn) (Underrun) Expected Actual Performance(%)

Operation and Maintenance of Road Facilities SS-32and Their Impact on Project SustainabilityBank's Midterm Review Process SS-33Environmental Mitigation Measures in Selected SS-34Bank-Financed Projects

Country Assistance Program Evaluation in thePeople's Republic of China

1998 Annual Performance Evaluation Programand Review of 1997 Activities

Sector Synthesis of Evaluation Findings in the Fisheries Sector

(Underrun)

5793-REG5762-REG

Years

5762-REG

Completion DateNo.

Table 1. Overall Performance Assessment of Projects by Sector/Subsector in 1998Generally Partly

Sector/Subsector Successful Successful Unsuccessful AllNumber % Number % Number % Number %

A. Agriculture and Natural ResourcesIndustrial Crops and Agro-Industry 0 0.0 0 0.0 1 100.0 1 100.0Irrigation and Rural Development 3 60.0 2 40.0 0 0.0 5 100.0Livestock 1 50.0 1 50.0 0 0.0 2 100.0Agricultural Support Services 0 0.0 1 100.0 0 0.0 1 100.0Fertilizer Production 1 100.0 0 0.0 0 0.0 1 100.0

Subtotal (A) 5 50.0 4 40.0 1 10.0 10 100.0

B. EnergyElectric Power 2 66.7 0 0.0 1 33.3 3 100.0

Subtotal (B) 2 66.7 0 0.0 1 33.3 3 100.0

C. Transport and CommunicationsRoads and Road Transport 2 66.7 1 33.3 0 0.0 3 100.0Railways 0 0.0 0 0.0 1 100.0 1 100.0

Subtotal (C) 2 50.0 1 25.0 1 25.0 4 100.0

D. Social InfrastructureEducation 0 0.0 1 100.0 0 0.0 1 100.0Health and Population 0 0.0 1 100.0 0 0.0 1 100.0

Subtotal (D) 0 0.0 2 100.0 0 0.0 2 100.0

E. Finance Development Finance Institution 0 0.0 2 100.0 0 0.0 2 100.0

Subtotal (E) 0 0.0 2 100.0 0 0.0 2 100.0

Total 9 42.9 9 42.9 3 14.3 21 100.0

Source: Postevaluation Information System.

OVERALL PERFORMANCE ASSESSMENT OF PROJECTSEVALUATED IN 1998

Generally PartlyCountry Group/Country Successful Successful Unsuccessful All

Number % Number % Number % Number %

A. ADF-onlyBangladesh 1 100.0 0 0.0 0 0.0 1 100.0Bhutan 1 50.0 1 50.0 0 0.0 2 100.0PRC 0 0.0 0 0.0 2 100.0 2 100.0Cook Islands 0 0.0 1 100.0 0 0.0 1 100.0Nepal 1 25.0 2 50.0 1 25.0 4 100.0Pakistan 3 60.0 2 40.0 0 0.0 5 100.0Sri Lanka 1 100.0 0 0.0 0 0.0 1 100.0

Subtotal (A) 7 43.8 6 37.5 3 18.8 16 100.0

B. ADF-OCR blendIndonesia 1 100.0 0 0.0 0 0.0 1 100.0Papua New Guinea 0 0.0 2 100.0 0 0.0 2 100.0Philippines 1 50.0 1 50.0 0 0.0 2 100.0

Subtotal (B) 2 40.0 3 60.0 0 0.0 5 100.0

Total 9 42.9 9 42.9 3 14.3 21 100.0

ADF = Asian Development Fund; OCR = ordinary capital resources; PRC = People's Republic of China.Source: Postevaluation Information System.

Table 2. Overall Performance Assessment of Projects by Country Group/Country in 1998

Total No. Generally Successful Partly Successful Unsuccessful TotalCountry Group/Sector of Projects Amount Percent Amount Percent Amount Percent Amount

Evaluated ($ mn) ($ mn) ($ mn) ($ mn)

ADF-only 16 604.4 70.5 88.6 10.3 164.1 19.1 857.1Agriculture 7 371.0 69.6 70.3 13.2 91.6 17.2 532.9Energy 3 135.0 90.0 0.0 0.0 14.9 0.0 149.9Finance 2 0.0 0.0 3.8 100.0 0.0 0.0 3.8Social Infrastructure 1 0.0 0.0 14.5 100.0 0.0 0.0 14.5Transport and Communications 3 98.4 0.0 0.0 0.0 57.6 0.0 156.0

ADF-OCR blend 5 68.4 38.3 110.1 61.7 0.0 0.0 178.5Agriculture 3 68.4 70.5 28.6 29.5 0.0 0.0 97.0Social Infrastructure 1 0.0 0.0 25.5 100.0 0.0 0.0 25.5Transport and Communications 1 0.0 0.0 55.9 100.0 0.0 0.0 55.9

Overall 21 672.8 65.0 198.7 19.2 164.1 15.8 1035.6Agriculture 10 439.4 69.8 99.0 15.7 91.6 14.5 629.9Energy 3 135.0 90.0 0.0 0.0 14.9 10.0 149.9Finance 2 0.0 0.0 3.8 100.0 0.0 0.0 3.8Social Infrastructure 2 0.0 0.0 40.0 100.0 0.0 0.0 40.0Transport and Communications 4 98.4 46.4 55.9 26.4 57.6 27.2 212.0

Total 21 672.8 65.0 198.7 19.2 164.1 15.8 1035.6

ADF = Asian Development Fund; OCR = ordinary capital resources.Source: Postevaluation Information System.

Table 3: Performance of Projects Evaluated in 1998 by Investment Cost

Total No. Generally Successful Partly Successful Unsuccessful TotalCountry Group/Sector of Projects Amount Percent Amount Percent Amount Percent Amount

Evaluated ($ mn) ($ mn) ($ mn) ($ mn)

ADF-only 16 208.8 55.8 70.0 18.7 95.2 25.4 374.0Agriculture 7 62.2 37.2 55.4 33.1 49.6 29.6 167.3Energy 3 78.9 90.5 0.0 0.0 8.3 0.0 87.2Finance 2 0.0 0.0 3.8 100.0 0.0 0.0 3.8Social Infrastructure 1 0.0 0.0 10.8 100.0 0.0 0.0 10.8Transport and Communications 3 67.7 0.0 0.0 0.0 37.3 0.0 105.0

ADF-OCR blend 5 36.8 32.9 75.2 67.1 0.0 0.0 112.0Agriculture 3 36.8 62.5 22.1 37.5 0.0 0.0 58.9Social Infrastructure 1 0.0 0.0 14.8 100.0 0.0 0.0 14.8Transport and Communications 1 0.0 0.0 38.3 100.0 0.0 0.0 38.3

Overall 21 245.7 50.5 145.2 29.9 95.2 19.6 486.0Agriculture 10 99.1 43.8 77.6 34.3 49.6 21.9 226.2Energy 3 78.9 90.5 0.0 0.0 8.3 9.5 87.2Finance 2 0.0 0.0 3.8 100.0 0.0 0.0 3.8Social Infrastructure 2 0.0 0.0 25.6 100.0 0.0 0.0 25.6Transport and Communications 4 67.7 47.2 38.3 26.7 37.3 26.0 143.3

Total 21 245.7 50.5 145.2 29.9 95.2 19.6 486.0

ADF = Asian Development Fund; OCR = ordinary capital resources.Source: Postevaluation Information System.

Table 4: Performance of Projects Evaluated in 1998 by Loan Disbursement

Appendix 3

UnsuccessfulNo. % No. % No. %

ADF-only 217 109 50.2 74 34.1 34 15.7Bangladesh 38 16 42.1 18 47.4 4 10.5Bhutan 3 1 33.3 1 33.3 1 33.3China, People's Republic of 5 3 60.0 — — 2 40.0Cook Islands 2 — — 2 100.0 — —India 2 1 50.0 1 50.0 — —Kiribati 2 1 50.0 — — 1 50.0Lao PDR 9 5 55.6 3 33.3 1 11.1Maldives 4 3 75.0 — — 1 25.0Mongolia 1 1 100.0 — — — —Myanmar 8 5 62.5 2 25.0 1 12.5Nepal 29 15 51.7 6 20.7 8 27.6Pakistan 54 34 63.0 16 29.6 4 7.4Solomon Islands 7 1 14.3 3 42.9 3 42.9Sri Lanka 25 12 48.0 10 40.0 3 12.0Tonga 8 7 87.5 1 12.5 — —Vanuatu 3 — — 1 33.3 2 66.7Viet Nam 5 — — 5 100.0 — —Western Samoa 12 4 33.3 5 41.7 3 25.0

ADF-OCR blend 190 111 58.4 59 31.1 20 10.5Indonesia 76 44 57.9 26 34.2 6 7.9Papua New Guinea 18 5 27.8 12 66.7 1 5.6Philippines 57 27 47.4 17 29.8 13 22.8Thailand 39 35 89.7 4 10.3 — —

OCR-only 107 83 77.6 20 18.7 4 3.7Fiji 8 6 75.0 1 — 1 12.5Hong Kong, China 5 4 80.0 1 20.0 — —Republic of Korea 43 37 86.0 5 11.6 1 2.3Malaysia 42 27 64.3 13 31.0 2 4.8Singapore 9 9 100.0 — — — —

Total 514 303 58.9 153 29.8 58 11.3

— = magnitude zero; ADF = Asian Development Fund; Lao PDR = Lao People's Democratic Republic;OCR = ordinary capital resources.Source: Postevaluation Information System.

HISTORICAL PERFORMANCE OF EVALUATED PROJECTSBY ECONOMY GROUP/ECONOMY

(as of December 1998)

Economy Group/ Economy

Total No. of

Projects

Generally Successful

Partly Successful

(as of December 1998)

Report Loan/Inv Date Sector/Title No. No. Approved Expected Actual

Padma Textile Mills PE-367 858-BAN (SF) 10-Nov-87 3.53 3.53 31-Dec-89 30-Sep-89 Industry7018-BAN (non-agri)

P.T. Gunung Garuda PE-404 905-INO 27-Sep-88 15.00 15.00 31-Mar-90 10-Aug-91 Industry7026-INO (non-agri)

Planters Development Bank Equity Investment PE-408 7008-PHI 25-Nov-86 0.49 0.50 b b DFI

P.T. BBL Dharmala Finance PE-421 979-INO 26-Oct-89 15.00 15.00 29-May-93 29-May-93 Leasing

Philippine Long Distance Company PE-465 885-PHI 29-Mar-88 24.00 23.75 30-May-90 31-Dec-93 Tele-7020-PHI communications

National Development Leasing Corporation Ltd. PE-485 7003-PAK 13-Dec-84 1.19 1.19 - - Leasing7011/814-PAK 09-Dec-86 5.60 5.60 11-Oct-88 11-Oct-887027/913-PAK 27-Oct-88 15.00 15.00 28-Dec-90 28-Dec-907077/1132-PAK 26-Nov-91 10.00 10.00 29-Oct-92 29-Oct-92

Fauji Fertilizer Exoansion PE-521 7047/1003-PAK 19-Dec-89 30.00 30.00 31-Oct-92 31-Mar-93 Fertilizer

a All generally successful.b Divestment was to be made at the Bank's request after 31 December 1993.Source: Operations Evaluation Office reports.

PROFILE OF EVALUATED PRIVATE SECTOR LOANS a

Subsector

Production

ApprovedCompletion Date

DisbursedLoan Amount ($mn)

Appendix 5, page 1

AVERAGE IMPLEMENTATION PERIOD OF PROJECTS EVALUATED IN 1998,ESTIMATE VERSUS ACTUAL

Time Overrun/Sector/Subsector (Underrun)

Years %

Agriculture and Natural Resources 4.58 7.47 2.89 70.70Agricultural Support Services 2.33 4.15 1.82 78.00Industrial Crops and Agro-Industry 2.10 5.68 3.58 171.10Irrigation and Rural Development 5.40 8.28 2.87 51.60Livestock 4.87 8.00 3.13 64.50

Energy 3.63 8.22 4.59 127.70Electric Power 3.63 8.22 4.59 127.70

Transport and Communications 3.63 7.11 3.48 98.30Roads and Road Transport 4.04 8.01 3.97 103.30Railways 2.42 4.42 2.00 83.00

Social Infrastructure 5.72 7.84 2.13 37.00Education 5.70 7.20 1.50 26.00Health and Population 5.73 8.48 2.75 48.00

Finance 3.96 4.36 0.41 10.50Development Finance Institution 3.96 4.36 0.41 10.50

Total 4.30 7.24 2.94 75.40

a Excludes one private sector project in Pakistan (PAK-7047/1003: Fauji Fertilizer Expansion Project), where the average delay was five months or 41 percent.Source: Postevaluation Information System.

Implementation Period (Years)

Table 1: Average Implementation Period of 1998-evaluated Projects by Sector/Subsector a

Estimate Actual

Appendix 5, page 2

Table 2: Average Implementation Period of 1998-evaluated Projectsby Country Group/Country a

Time Overrun/Country Group/Country (Underrun)

Estimate Actual %

ADF-only 4.21 7.44 3.22 83.70Bangladesh 4.32 8.58 4.26 98.00Bhutan 4.22 6.62 2.40 55.50China, People's Republic of 2.26 5.05 2.80 127.00Cook Islands 4.00 3.27 (0.73) (18.25)Nepal 4.58 9.44 4.86 118.50Pakistan 5.12 8.12 3.00 60.00Sri Lanka 3.14 6.14 3.00 96.00

ADF-OCR blend 4.55 6.63 2.09 50.80Indonesia 5.15 6.43 1.28 25.00Papua New Guinea 3.43 6.09 2.66 78.00Philippines 5.37 7.28 1.92 36.50

Total 4.30 7.24 2.94 75.40

ADF = Asian Development Fund; OCR = ordinary capital resources.a Excludes one private sector project in Pakistan (PAK-7047/1003: Fauji Fertilizer Expansion Project), where the average delay was five months or 41 percent.Source: Postevaluation Information System.

Implementation Period (Years)

Years

Appendix 6, page 1

Number NumberCountry Group/Country

ADF-only 6 29.9 7 13.3

Bangladesh 0 0.0 1 25.5Bhutan 0 0.0 1 15.4China, People's Republic of 0 0.0 2 8.1Nepal 3 38.2 1 4.4Pakistan 3 21.6 1 22.7Sri Lanka 0 0.0 1 9.0

ADF-OCR blend 2 11.6 3 7.9

Indonesia 1 18.0 0 0.0Papua New Guinea 0 0.0 2 10.4Philippines 1 5.3 1 3.1

Total 8 25.4 10 11.7

ADF = Asian Development Fund; OCR = ordinary capital resources.a Excludes one private sector and two development finance institution projects.

Source: Postevaluation Information System.

Table 1: Average Cost Underrun/Overrun of 1998-evaluated Projects by Country Group/Country a

AVERAGE COST UNDERRUN/OVERRUN OF PROJECTSEVALUATED IN 1998

Average

Projects with Cost Underrun

Projects with Cost Overrun

(%) Average

(%)

Sector/Subsector Number Number

Agriculture and Natural Resources 6 29.0 3 10.2

Agricultural Support Services 0 0.0 1 3.6Industrial Crops and Agro-Industry 0 0.0 1 11.5Irrigation and Rural Development 5 24.8 0 0.0Livestock 1 49.7 1 15.4

Energy 1 1.4 2 17.2

Electric Power 1 1.4 2 17.2

Transport and Communications 0 0.0 4 12.2

Roads and Road Transport 0 0.0 3 14.8Railways 0 0.0 1 4.7

Social Infrastructure 1 27.6 1 3.1

Education 0 0.0 1 3.1Health and Population 1 27.6 0 0.0

Total 8 25.4 10 11.7a Excludes one private sector and two development finance institution projects. Source: Postevaluation Information System.

Average(%)

Average(%)

Projects with Cost Underrun Projects with Cost Overrun

Table 2: Average Cost Underrun/Overrun of 1998-evaluated Projects by Sector/Subsector a

SUMMARY OF FINDINGS FROM IMPACT EVALUATION AND REEVALUATION STUDIES

A. Impact Evaluation Studies 1. Impact evaluation studies review projects and programs, both ongoing and completed, in a given sector in one member country or in a group of countries to assess the Bank’s contributions to the countries’ development. During the year, the Operations Evaluation Office carried out three such studies. 1. Secondary Science Education Projects 2. The evaluation study of secondary science education projects in Bangladesh, Nepal, and Pakistan focused on the projects’ impacts on institutions, graduates, and the policy and management environment. The findings show that, while the projects improved the institutional capacities of science education centers and secondary schools, such improvement did not always filter down to the students in terms of better learning achievements in science and related subjects. The main constraint continued to be the actual classroom conditions: (i) average class size of 60-80 students in a classroom designed for 40 students; (ii) teaching load of 26-36 periods per week; (iii) topic overload; and (iv) high-stakes external examinations that encourage memorization as the primary mode of learning instead of practical exercises, experiments, and demonstrations. In spite of their shortcomings, however, the projects, when viewed from a long-term perspective and in the context of the socioeconomic and cultural settings of the concerned countries, have made significant contributions to science education. The projects have provided important building blocks for the long-term development not only of science education but also of the secondary education system as a whole. The projects have also contributed significantly to making the administrative and policy environment more conducive to science education.

2. Bank Assistance to the Power Sector in the Pacific Developing Member Countries (PDMCs)

3. This study showed that, other than those in Fiji Islands, Bank-financed power projects have had limited direct development impact on the PDMCs largely due to institutional weaknesses. The utilities in general are poorly managed and have financial problems. The successful projects are highly correlated with strong and adaptable institutions, and the failures with weak institutions. The use of financial covenants alone to secure reforms has been the wrong approach to dealing with structural problems such as tariffs and lack of autonomy. Recognizing that lack of institutional strength and not of capital is the principal constraint facing the PDMC power sector, technical assistance (TA) to the countries should focus on improving (i) operation and maintenance (O&M), (ii) standards of safety and environmental protection, (iii) efficiency—including lowering costs, (iv) management capacity for both administration and planning, (v) reporting and transparency of operations, and (vi) measurement of performance.

3. Bank Project Preparatory TAs in the Agriculture Sector in Bangladesh 4. A study in Bangladesh assessed the adequacy, implementation experience, and operational performance of 32 project preparatory TAs (PPTAs) in agriculture and natural resources, as well as their impact and effectiveness in contributing to the performance of ensuing projects. It found that PPTAs tend to produce a project design based on a predetermined project

concept, rather than being a feasibility study to develop a project to achieve certain outcomes. Interviewees revealed that there was minimal transfer of skills through most PPTAs, and that the executing agencies (EAs) generally did not improve their capacity to design projects themselves. Despite increasing recognition of the importance of participatory planning in PPTAs and loan projects, performance remains low. The study’s recommendations include (i) the need to formally develop the country operational strategy, and the agriculture and natural resources sector strategy in a way that provides balance between the technical products and mechanisms to deliver such strategies; (ii) the need to undertake rigorous institutional capacity studies; (iii) consideration by the Bank of strategic action to improve EA performance or develop alternative mechanisms outside of the immediate public sector to prepare and implement projects; (iv) the need for the Bank to provide a more participatory approach to PPTA; (v) the need to identify and analyze project risks to enhance Bank Management’s capacity to make informed decisions on the merits of a project, and to realistically evaluate how the risks are to be addressed; and (vi) incorporation of project performance indicators into a management information system, which needs to be developed for each PPTA during appraisal. B. Reevaluation Studies 5. Reevaluation studies were undertaken for two projects each in the agriculture and education sectors to assess their performance and sustainability several years after their original evaluation.

1. Sabah/Sarawak Fisheries Infrastructure Project

6. Reevaluation of this project in Malaysia was conducted six years after its project performance audit report (PPAR), which was done within 12 months of project completion. At the time of the PPAR, both subprojects were assessed as generally successful. This was based on the expectation that there would be no adverse environmental or social impacts, and that the project would be sustainable and all project facilities fully utilized by 1997. These positive expectations did not materialize, and by the time of reevaluation the facilities were still underutilized. Overall, the Project’s contribution to observable increases in fisheries output in Sabah have been limited, and in Sarawak very small. In view of the low achievements so far of the major parts of the Project, and their weak relevance to future sector operations, its rating on reevaluation was unsuccessful. The failure of the original evaluation to correctly identify trends highlights the importance of having several years of operational results available before evaluation.

2. Palawan Integrated Area Development Project

7. This Philippine project was classified as partly successful both in the PPAR and at reevaluation. The lower than anticipated project impact and the low sustainability of the benefits of some project components reflect important design weaknesses. These weaknesses include inadequate information and database, inadequate consideration of existing farming systems, limited consultation with intended beneficiaries and other stakeholders, and failure to recognize sociological and institutional constraints and the risk of a sudden shift to high-technology, multistory cropping systems. Except for the ports and malaria eradication subcomponents, which were by and large successful, the operational success and the impact on farm incomes of the other components such as irrigation, agricultural intensification and diversification, and livestock production were below targets and continue to decline. The socioeconomic impact of the Project, though positive, has been less than expected, and the prospects for its sustainability are not bright. Overall, the Project has failed in providing the intended integrated development impact.

3. Technical and Vocational Education Project

8. This Project had not been evaluated in a PPAR but was selected for study to assess its long-term sustainability and impact. It aimed to improve planning and management of the subsector, promote research and curriculum development, institute in-service training for teaching and nonteaching staff, and enable technical education institutions (TEIs) to deliver an improved training program for middle-level technicians. It was completed in 1989 and was assessed at that time to have contributed to upgrading the quality of middle-level technical human resources required for social and economic development. However, the rating of the Project at reevaluation was only partly successful. It improved the physical and human resources of the TEIs but did not contribute to better management of the subsector. Reevaluation highlighted the importance of locating TEIs close to relevant industries. This increases interaction between the TEI and industry, ensuring the relevance of the program content, minimizing the cost of on-the-job training on the part of the students, and facilitating eventual employment as well.

4. Second Vocational Education Project

9. Reevaluation of this Project in Thailand confirmed the generally successful rating in the PPAR. The success and quality of upgrading technical education and vocational training in the country underline the importance of providing assistance and policy guidelines over the long term in the education sector. The Project contributed not only to Thailand’s industrial development in the past two decades, but also expanded the technical and vocational education system in the country. It helped the Government to provide equitable access to quality technical education for more balanced national growth. While the institutional and policy framework in Thailand favors the sustainability of technical and vocational education, the recent economic crisis in the country and the region brought to light important issues that need to be addressed, such as the need (i) for a safety net in the form of steady and adequate yearly allocation for recurrent expenditure, to ensure proper O&M and continuing improvement in quality; (ii) to broaden funding sources for research relating to policy, technical/vocational education administration, and media; and (iii) to monitor and control the cost of Government-provided training in relation to its social and individual rates of return.

Appendix 8, page 1

FOLLOW-UP ACTIONS ON MAJOR FINDINGS AND RECOMMENDATIONS OF EVALUATION REPORTS COMPLETED DURING 1998

Recommendations for Follow-Up Action Action Taken/Proposed A. Project/Program Performance Audit Reports PE-503: Mini-Hydropower Project (Loan 512-NEP[SF])

The Nepal Electricity Authority (NEA) Small Hydropower Department should (i) implement training programs in the service and maintenance of electronic governors; (ii) expand its training in rudimentary maintenance of power plants and civil works, specifically for small hydro projects; (iii) establish radio links, as envisaged at appraisal, from powerhouses to the nearest NEA headquarters; and (iv) install fault protection and isolation equipment on transmission lines. Any further Bank assistance should recognize the availability of these six subprojects, which may be financially sustainable with proper attention to load promotion and efficient staffing, to support integrated development efforts in the hill areas.

NEA is contracting out operation and maintenance (O&M) of the mini hydropower project to private operators, who are more efficient. The planned Bank loan for a distribution and transmission project will also include a special component to finance expansion of local distribution systems for grid power stations.

PE-504: Shanxi-Xiaoliu Railway Project (Loan 948-PRC)

A market assessment of major traffic generators should be undertaken to determine future traffic and rolling stock requirements. The forward commitment of the Ministry of Railways (MOR) and the necessary permits for coal transportation should be sought to underwrite the Shanxi-Xiaoliu Railway Company's (SXRC) marketing plan and as a basis for implementing a rehabilitation program.

MOR has adopted several measures to improve the performance of the Project. These include management changes and strengthening, and increased coordination with local governments for marketing.

An engineering audit of the line should be carried out to confirm the need for the rehabilitation program proposed by SXRC, identify other necessary investment as well as major engineering risks that could interrupt operations, and propose remedial measures as required. The future level of operations should be based on economic and financial evaluations of proposed investments.

The Shanxi Provincial Government is supporting the urgent investment needs of SXRC. The performance of freight loaded is showing improvement as mentioned below.

Careful consideration should be given to implementing the organizational and financial management changes recommended in technical assistance (TA) 1117-PRC, in particular (i) improving management accountability, (ii) strengthening the top management structure, (iii) transferring the infrastructure assets and liabilities to SXRC's account, (iv) clarifying the relationship between the parent company and its subsidiaries, and (v) computerizing and developing a management information system (MIS).

Several concrete steps have been taken to implement the recommendations in TA No. 1117: (i) and (ii) A new general manager has been appointed and the number of deputy general managers increased from one to three for monitoring of operations; (iii) the infrastructure assets and liabilities have been handed over to SXRC’s account, an internal auditor has been appointed, and a daily cash position statement is being submitted to the management; (iv) appropriate steps for subsidiary activities have been taken; and (v)

Appendix 8, page 2

Recommendations for Follow-Up Action Action Taken/Proposed under TA 1117 computerization of the MIS has been accomplished.

MOR should consider classifying Xiaoyi as a division station to permit through waybilling from SXRC to MOR stations.

MOR has made the Project an area of focus for improving performance. This is reflected in the management changes. The supply of freight wagons has improved.

A study should be carried out to (I) review institutional rigidities in the functioning of coal transport and their impact on the performance of railway projects in the People’s Republic of China (PRC), and (ii) review the criteria used by MOR to allocate wagons and their impact on the overall efficiency of the railway system. Compliance, to the extent now possible, with the outstanding loan covenants designed to improve the financial soundness of the Project should be monitored closely, with Bank staff time allocated for the purpose. To the extent possible, this should be carried out until the operational performance of SXRC is more financially sound, and action on the recommendations set out in the project performance audit report (PPAR) is initiated.

The performance of the Project improved in 1997 with 663,626 tons loaded compared with 374,256 tons loaded in 1996. Performance in 1998 was about 800,000 tons loaded. As from 1 July 1998, the freight tariff was increased from 16-18 fen to 22 fen per ton-km. Further substantial improvement will be necessary for compliance with financial covenants.

Several Bank-financed railway projects in the PRC are dependent for traffic growth on developments outside the control of the railway. Bank review missions should examine the progress in such developments and reconfirm the validity of achieving the traffic forecasts.

Bank review missions are looking into developments in the project areas in consultation with executing agencies (EAs) and provincial authorities. Areas of attention are being flagged.

PE-505: Irrigated Command Area Development Project (Loan 818-INO)

The provincial government through the provincial water resources services needs to continue and intensify its efforts toward the improvement, conservation, and management of watershed areas in each of the project sites to minimize erosion and siltation in the irrigation schemes.

With recent initiatives for devolving the National Government’s functions and responsibilities to the provincial and other local government institutions, the provincial water resources services are being strengthened to play a lead role in coordinating relevant activities in the provinces.

The provincial government needs to reassess the current allocations for O&M for each of the irrigation schemes in both Bengkulu and Lampung provinces, continue with the training of water users association (WUA) farmer-members, and give WUA farmer members greater role in operating and maintaining the irrigation system. The provincial agriculture services should intensify the training of farmers in both provinces on integrated pest management and on the effective use of agrochemicals and fertilizers, and increase their awareness of the negative effects of increased use of fertilizers and chemicals on the environment and on human health.

Steps are being taken by the provincial government to strengthen the WUAs and devolve irrigation and drainage system O&M to them. The Government has already issued instructions allowing the WUAs to collect irrigation service fees and use them for maintaining the systems.

The provincial water resources services and provincial agriculture services should collaborate in improving monitoring and evaluation of Project impacts as well as

There is a greater awareness among the provinces’ service agencies regarding mutual cooperation and collaboration in the monitoring

Appendix 8, page 3

Recommendations for Follow-Up Action Action Taken/Proposed determine and monitor key factors (e.g., destruction of catchment area, water supply trends, population growth, and pollution trends) that may have significant impacts on project performance and sustainability.

and evaluation of parameters affecting the environment.

The Bank should continue dialogue with the Directorate General of Water Resources Development and the provincial governments on the need to provide adequate O&M budgets to ensure sustainability of project benefits.

Generation of adequate funds for O&M continues to be a major issue in the Bank’s policy dialogue for the sector.

PE-506:Second Hill Irrigation Project (Loan 596-NEP[SF])

Agriculture projects need to take into account policy changes, particularly those relating to agricultural inputs and outputs, that affect the economic environment in which the projects will operate. The new participatory approach to project design should include the planning of maintenance within the capacity of farmers. Cost ceilings to screen out excessively costly hill irrigation projects need to be systematically applied.

The Agriculture Perspective Plan now provides overall direction for the agriculture sector based on the overall policy framework in the agriculture sector. To screen excessively costly projects, the Department of Irrigation (DOI) switched from fixing cost ceilings to economic internal rate of return criteria.

The borrower needs to continue building capacity to plan and implement hill irrigation projects. Less frequent personnel transfers would help staff take more responsibility for their roles in project implementation. There is an urgent need to allocate sufficient funds to ensure that project facilities and system operation will be sustained.

The Nepal Resident Mission (NRM) holds regular dialogue with respective project directors/managers on the issue of frequent transfers of project personnel. DOI also supports the view of having a mandatory minimum financial requirement and time period for maintenance by the EAs after project completion, especially in hill areas. DOI suggested that the necessary budgetary provision be under the loan agreement between the borrower and the aid agency.

PE-507:Highland Agriculture Development Project (Loan 802-PHI)

Further strengthening of farmer organizations is needed to enable them to take responsibility for as much maintenance of the Project's physical facilities as possible. This particularly includes the seed potato stores and communal irrigation systems (CISs). Greater clarity in the role of the beneficiaries and the Government in maintenance of the CISs is also required.

As of March 1999, seed potato storage houses were being adequately maintained by farmer organizations. CISs were being maintained, and more resources were provided to maintain watersheds under the ongoing Cordillera Highland Agricultural Resource Management Project (Loan 1421/1422-PHI)

Further strengthening of local government units (LGUs) is required to enable them to take over full responsibility for local physical infrastructure including the roads constructed under the Project, provide adequate maintenance for this infrastructure, and ensure an adequate level of agricultural support services.

The quality of maintenance of completed roads is moderate to good. Clarification of the LGU taxation mandate is being pursued to enhance LGU budgetary resources to undertake infrastructure maintenance.

Appendix 8, page 4

Recommendations for Follow-Up Action Action Taken/Proposed PE-508:Farm-to-Market Roads Project (Loan 758-PAK[SF])

The Government should prepare and implement an action plan for road maintenance as soon as possible, to provide separate funds and deploy maintenance staff from the provincial communication and works department under the existing mechanism. The funds earmarked for farm-to-market roads should not be utilized for other priority roads until the responsibility of maintaining farm-to-market roads is totally transferred to the district councils and provincial department of local government and rural development to which these roads originally belonged. For future projects, the EA should improve the collection and availability of baseline information on traffic and socioeconomic conditions. The Bank should follow up on the effective implementation of maintenance programs.

Some additional maintenance funding has been provided. Future strategy focuses on provincial level interventions, with specific attention given to reforming institutions responsible for road network O&M.

PE-509:Second On-Farm Water Management Project (Loan 871-PAK[SF])

The major follow-up actions required from the borrower are to ensure (i) proper O&M of irrigation and drainage facilities other than at the project on-farm level; and (ii) early desilting of major drains to permit removal of water from the pilot on-farm drainage scheme, thereby preventing rapid reduction of project benefits.

The issues concerning O&M for the upstream works are being addressed by the National Drainage Program and other Bank-financed projects.

The borrower and the Bank together need to further promote the emergence of effective farmer institutions as envisaged under the proposed Bank-supported Punjab Farmer-Managed Irrigation Project.

In addition to WUAs formed under other ongoing Bank-financed projects, which can be organized into federations to undertake O&M of distributary canals, there will be federations of WUAs under the Punjab Farmer-Managed Irrigation Project.

Further revision of water and agriculture pricing policies is needed for economically efficient allocation of resources.

It is proposed under the National Drainage Program and the Punjab Farmer-Managed Irrigation Project that farmers pay the true cost of water. However until the procedures are under way and the mechanism for collecting and using the water tax is worked out, no changes in government water pricing policy can be made. On the broader agriculture pricing policy, however, the Bank will be financing TA this year to prepare the Agriculture Sector Strategy Study. This TA study will examine the pricing policies in the sector and recommend policy changes.

PE-510:Secondary Towns Power Distribution Project II (Loan 870-SRI[SF])

There is a need for additional investment in rehabilitation to avoid previous technical overloading problems. To facilitate the attraction of investment into Lanka Electric Company (Private), Limited (LECO) on a free-market basis, several interdependent measures could be taken, including (i) strengthening the management autonomy of

The Bank conducted an energy sector strategy study for Sri Lanka that was completed in September 1998. The study recommended a comprehensive program for restructuring the power sector; and for Bank assistance for planning, restructuring, and financing some of

Appendix 8, page 5

Recommendations for Follow-Up Action Action Taken/Proposed LECO to make timely and independent investment decisions; (ii) improving the market competitiveness of the power sector; (iii) encouraging the further separation and privatization of generation, transmission, and distribution operations; (iv) moving toward the setting of tariffs according to long-run marginal cost principles; and (v) divesting the Government's ownership in LECO to private sector interests and the public. Promotional aims set out in LECO's Corporate Plan for 1998-2002 should be followed to strengthen public support for the envisaged structural reforms. The Government is encouraged to (i) support separation of the generation, transmission, and distribution operations, and privatization of the power sector; and (ii) deepen the autonomy of LECO, bearing in mind the need for additional investment in rehabilitation and for assurances concerning supply and an improved connection system.

the related transmission and distribution development to improve sector efficiency. The study report was discussed with the Government in September 1998; it was agreed that the Bank would provide TA to develop detailed plans for power sector restructuring and to prepare a loan to finance restructuring. The TA was approved by the Bank’s Management in January 1999 and is progressing as scheduled.

PE-511:Second Livestock Development Project (Loan 745-NEP[SF])

The Government needs to review the facilities provided under the Project and ensure that they are fully functional. The Bank could monitor and provide the necessary advice during its review or project administration missions for the Third Livestock Development Project.

The facilities developed by the Second Livestock Development Project (SDLP) are in operation at different levels of production depending upon the need of the Department of Livestock Services (DLS). DLS will submit the actual production levels to the Bank in the second quarter of 1999.

The Government needs to take concrete steps to commercialize the operations of the vaccine production unit and the Hetauda Feed Mill. To encourage the Government to move toward privatization, Bank follow-up through policy dialogue during the implementation of the Third Project is suggested for the vaccine production unit—its commercialization was a loan covenant under the Second Project. In addition, the Government should ensure that adequate steps are taken to observe environmental standards in these operations.

Privatization of the vaccine production facilities did not happen because of the lack of experience and capability of the private sector. DLS will develop a viable modality and plan for privatization of these facilities including the Hetauda Feed Mill by 15 July 1999.

The benefit monitoring and evaluation (BME) system, or more appropriately in the context of current Bank initiatives, the Project Performance Management System (or local equivalent thereof), needs to be institutionalized in DLS as part of an organizational setup that will monitor and evaluate benefits from projects implemented by DLS. Equipment and data that were dispersed with the dismantling of the BME system after completion of the Second Project need to be regrouped or replaced.

The BME system was incorporated into DLS and Third Livestock Development Project (TLDP) operations in February 1999.

The pricing of milk needs to be further liberalized by the Government to provide adequate incentives for greater milk production, and the Dairy Development Corporation should be privatized as agreed upon under the 10-year Dairy Development Plan of the Government prepared with help from Danish International Development Assistance. The Bank could further this process through its policy

Regular market outlets have become a problem for producers rather than the price structure. DLS and the TDLP are working closely with the National Dairy Development Board, the Dairy Development Corporation, and private dairies to provide farmers with reliable markets. The Ministry of Agriculture

Appendix 8, page 6

Recommendations for Follow-Up Action Action Taken/Proposed dialogue with the Government during implementation of the Third Project.

has submitted its plans for the privatization of the Pokhara Milk Supply Scheme and is now awaiting the Ministry of Finance’s decision.

Given its budgetary constraint, the Government needs to explore ways of providing adequate resources for O&M of the project facilities and equipment.

The Government has allocated NRs2.5 million this year for O&M of facilities developed by SDLP and will continue to allocate the required budget in the future.

PE-512:Second Health and Population Project (Loan 710-PAK[SF])

The Government should restore the facilities of the Electro Medical Workshop to their original purpose.

The Education, Health, and Population Division West (AWEH) requested the Pakistan Resident Mission (PRM) through a memo on 4 February 1999 to follow this up with the Government.

Remedial measures to improve the efficiency and effectiveness of operations, discussed during the Postevaluation Mission (PEM), included (i) an action plan to fill vacancies in the basic health units and other rural health outlets; (ii) survey of the use of residential bungalows and staff quarters; (iii) incentives to attract and retain youth in health care studies; (iv) funds for the immediate rehabilitation of the Regional Training Institute (RTI), Lahore; (v) a database system on the beneficiary population; (vi) a preventive maintenance system; (vii) health promotion, education, and community participation in the areas benefited by the Project; and (viii) coordination of the various health and population service workers.

Remedial measures to improve the efficiency and effectiveness of operations for follow-up actions (i), (ii), (iii), (v), (vi), and (viii) are being carried out under Loan 1493-PAK(SF): Second Social Action Program. With regard to remedial action (iv), AWEH has requested PRM (memo of 4 February 1999) to assist the Government with the immediate rehabilitation of RTI, Lahore under the ongoing Loan 1277-PAK(SF): Population Project. With regard to remedial action (vii), AWEH has requested the PRM to help the Government under the ongoing Loan 1200-PAK(SF): Health Care Development Project.

Future projects of a similar nature should consider (i) intensive sector analysis and social assessment, with community participation; (ii) a lead agency or central coordination unit for multiagency projects; (iii) the absorptive capacity of the ministry/department; (iv) relevant sector policy issues; (v) institutional capability of EAs and implementing agencies as well as beneficiary groups; (vi) substantial staff development and training; (vii) a flexible project implementation system based on the process-oriented approach; and (viii) inclusion of facilities for medical waste disposal and sewage treatment.

These follow-up actions will be given due consideration during the preparation of future projects of a similar nature.

PE-513:Chashma Command Area Development Project (Loan 723-PAK[SF])

The Borrower should ensure provision of adequate funds for O&M, particularly for subsurface drainage sump pumps, expectedly through the National Drainage Sector Project, and for monitoring groundwater levels to objectively determine the impact of O&M conditions on water tables and waterlogging in the project area. Both the Borrower and the Bank need to examine the consequence on total irrigated crop output of the present cropping pattern.

An O&M performance contract for the Project’s drainage facilities is expected to be awarded by 31 December 1999. Monitoring of groundwater levels will be undertaken under the Government's National Drainage Program.

Appendix 8, page 7

Recommendations for Follow-Up Action Action Taken/Proposed There is a need for further revision of the Government's policies on agricultural input and output prices (under the National Drainage Sector Project and other Government programs) to reduce or eliminate price distortions. Achievement of benefits, sustainability of desired crop production levels, and poverty reduction would be aided by a more efficient use of resources.

The proposed Agriculture Sector Strategy TA will assist the Government in reviewing sector issues including those related to input and output prices.

PE-514:Seventh Power Project and East Zone Thermal Feasibility Project (Loan 751-BAN[SF])

The Bank should ensure that projects are planned (i) to strengthen financial management and the transparency of financial statements, (ii) with more rigorous attention to devising technical designs consistent with modern practice and the least-cost investment option, (iii) to take into account institutional requirements for strengthening the operational management of control systems, and (iv) with stronger conditionalities for tariff adjustments.

To strengthen financial management and the transparency of financial statements, the Bank approved TA 2004-BAN: Financial Management Upgrade for the Bangladesh Power Development Board (BPDB) and the Dhaka Electricity Supply Authority (DESA). This TA has developed and implemented a complete set of financial and accounting management systems in pilot areas of BPDB and DESA that are being replicated in more parts of BPDB and DESA with the use of loan funds. The designs being developed for Bangladesh for the Eighth Power Project have been reviewed by international consultants and are deemed to be consistent with modern engineering practices. Institutional changes are being initiated through corporatization of sector entities leading to strengthening of the operational management and control systems. Tariff adjustments have been made automatic through the introduction of a formula that takes into account variations in foreign exchange and inflation rates.

The Government should undertake studies/measures aimed at (i) rationalizing investment costs associated with existing transmission designs and operations; (ii) providing BPDB with legislative support and recovery mechanisms for dealing with outstanding customer accounts, including those of Government departments and agencies; (iii) promoting public awareness and support for improving services; and (iv) introducing a system (including consideration for privatization) for meeting repairs on malfunctioning and disabled equipment. BPDB should aim to provide more focused leadership and management to (i) reduce power system losses; (ii)

The Bank has accorded TA for preparing a power system master plan for Bangladesh that rationalizes investments associated with generation, transmission, and distribution. TA 2743-BAN: Review of Electricity Legislation and Regulations and TA 3129-BAN: Support for the Energy Regulatory Authority Project have been accorded to provide legislative support for reorganization and restructuring of the power sector. Recovery of dues from customer accounts including those of Government departments and agencies is ongoing, and significant improvements have been made in transferring Government dues to BPDB. Public interaction is now being sought to increase awareness of and support for improvement. Customer interactions were scheduled in May 1999 by DESA and the Dhaka Electricity Supply Company. A start has been made to repair distribution transformers and consumer meters on a systematic basis. The Overseas Economic Cooperation Fund

Appendix 8, page 8

Recommendations for Follow-Up Action Action Taken/Proposed develop a corporate culture for improving operational efficiencies and achieving service targets; and (iii) initiate reforms for upgrading skills of operational staff and improving maintenance of facilities, customer relations, billing, and recovery mechanisms. In addition, BPDB should bring to the manufacturers' attention the defective 33-kV circuit breakers.

(OECF) of Japan has provided assistance to conduct studies on a pilot basis to reduce the power system losses of BPDB and DESA. In the areas where this has been introduced, there have been significant reductions in losses. The Bank has initiated a dialogue between the E-7 Group of electricity companies and the sector entities in Bangladesh with a view to developing the corporate culture for improving operational efficiencies and achieving improvement targets. On the generation side, reforms for upgrading skills of operational staff and improving maintenance of facilities are planned through corporatization of two BPDB power stations and technical support for operation and maintenance from Kreditanstalt für Wiederaufbau of Germany and OECF.

PE-515:Road Improvement Project (Loan 806-NEP[SF])

The Bank should continue to support road improvement projects but should ensure that (i) roads intended for opening development and reducing poverty adequately address how poverty relief for targeted beneficiaries will be achieved; and (ii) progress on civil works is monitored, and revisions in scope are in keeping with the Project's design objectives. The Government should ensure that reforms aimed at developing administrative and technical competencies and accountability at the central level are also transferred to the divisional and district levels. The reforms should include a commitment from the Government to enable regional and divisional administrators to approve and enter into timely and obligatory contract payments. In addition, it is recommended that studies be initiated to address the seriousness of overloading and to devise regulations and cost-recovery measures. The Government should also ensure that roads selected with political considerations for new construction or improvement are referred for the Department of Road's (DOR) endorsement that the proposed designs and expenditures are justified using incremental analysis techniques. In the interests of sustaining the pavement life of the Doti project road, DOR is encouraged to ensure that the lineman system is fully implemented as soon as possible so that drainage obstructions are routinely cleared.

Formulation of the follow-on Third Road Improvement Project and Fourth Road Improvement Project considered poverty reduction. The transfer of responsibilities to the divisional and district levels is proceeding, but not yet effectively. New roads have been selected based on the World Bank-assisted Priority Investment Plan. The current situation is being determined.

PE-517:Hexian Pulp Mill Project

Appendix 8, page 9

Recommendations for Follow-Up Action Action Taken/Proposed (Loan 937-PRC) The Government should review pricing and taxation policies, which at present place the mill at a disadvantage vis-a-vis imported pulp. After a thorough reevaluation, the Government can consider whether it wishes to continue operating the mill. The mill should reduce operating cost in the areas of staffing and consumption of raw materials as well as consider switching to eucalyptus. It should follow up on environmental recommendations in the PPAR and introduce safety standards for workers. When designing similar projects, technical design specifications have to be sufficiently detailed. When project proponents and Bank consultants present differing cost estimates, a careful analysis of the reasons for such differences and adjustments should be made. Whenever considerable delays are experienced in starting implementation or during implementation, the Bank should analyze the implications on the project's commercial viability and debt-servicing capacity.

Copies of the report containing the Bank’s recommendations for follow-up actions were provided to the EA in February 1999. The deputy director, Agriculture and Social Sectors Department (East) visited the Project in April 1999 and included the PPAR recommendations in his discussion with the EA.

PE-518:Highland Livestock Development Project (Loan 808-BHU[SF])

The Government needs to ensure that the developed pastureland will be maintained. It may lease more public lands with proper safeguards for pasture development, because small farms do not have enough pastureland.

Fodder development has been built into the ongoing Eighth Five-Year Plan. The Crop and Livestock Services Division has submitted the revised Pasture Development Policy (jointly prepared by all sections within the Ministry of Agriculture) to the Government.

The Government should encourage greater acceptance among farmers of the need to cull livestock to relieve the pressure on the environment.

The culling of livestock is a by-law in the Livestock Act. Private entrepreneurs are being advised to adopt various means such as exchange of good heifers for unproductive animals.

Follow-up projects should be implemented by the Government, perhaps with external assistance to ensure the sustainability of benefits from the Project.

Ongoing subprojects in the Highland Livestock Development Project use lessons learned from completed projects as guidance. Future project activities will be formulated based on such lessons.

The Government needs to resolve Bhutan Dairy Limited's (BDL) management difficulties to ensure that the milk processing plant will continue to provide a ready market for raw milk from the farmers and an adequate supply of processed milk to the urban centers.

The Ministry of Agriculture has already taken initiatives to resolve the issue of BDL’s management difficulties; the Bank will be informed of the final decision.

PE-519:Fourth Road Improvement (Sector) Project (Loan 690-PNG/691-PNG[SF])

The Government should (i) undertake immediate emergency maintenance on the Umsis-Mumeng and Wariman-Hawain River roads;

Necessary works on Wariman-Hawain are being prepared. Work on Umsis-Mumeng is more substantial and will be included as a priority in the next investment project.

Appendix 8, page 10

Recommendations for Follow-Up Action Action Taken/Proposed (ii) maintain funding for road maintenance at a level to sustain the four project road systems; (iii) improve the capacity of the Office of Transport (a) to regulate axle loads and vehicle dimensions, and (b) to comply with the newly developed environmental requirements; (iv) implement the recommendations formulated through the Bank TA on road safety, and on road user revenues and expenditures; and (v) undertake a project performance evaluation on the four subprojects as prescribed in the loan covenant on BME (Schedule 5 of the Loan Agreement).

Maintenance budgets have been low because of overall fiscal constraints. However, for 1999, the appropriations and expenditures have greatly improved. The Government has recently renewed its agreement with Queensland Transport for weigh-in-motion equipment. Environmental concerns are being addressed. Further assistance on institutional strengthening is being provided through both TA and loans. The Government is reviewing the cost recovery policy with further TA assistance, including possible establishment of a road authority.

The Bank should follow up in particular actions (i) to (iii) to be taken by the Government.

PE-520:Bhutan Development Finance Corporation (Loan 934-BHU[SF])

The Government should not rely on Industrial Lending Department operations to cross-subsidize Agricultural Lending Department operations, thereby undermining the Bhutan Development Finance Corporation's (BDFC) role as an industrial development lender. If Government social policy requires a subsidy, this should be provided in a transparent manner.

This was brought up by the Project Completion Review Mission. The Government is still considering what action to take.

The Government should expedite the adoption of a bankruptcy law as well as procedures for financial institutions to facilitate loan recovery and collateral collection.

The laws on bankruptcy are being written.

The Royal Monetary Authority (RMA) should revise and improve its guidelines on loan loss provisioning, and introduce interest rates based on assessment of specific business or project risk rather than sector risk.

A Bank supervisory TA to look at these problems is under processing.

Steps should be taken to allow the market to determine compensation levels and delink salaries from the Government system to strengthen the autonomy of financial institutions. Representation on boards should also be widened to include more diverse private sector participation.

No action has been taken.

BDFC's financial reports should be in full conformity with general accounting standards acceptable to the Bank and with RMA prudential guidelines.

BDFC is taking action on this problem.

Appendix 8, page 11

Recommendations for Follow-Up Action Action Taken/Proposed PE-522:Second Cook Islands Development Bank Project (Loan 1155-COO[SF])

The Government should review the Cook Islands Development Bank's (CIDB) operating conditions and revisit its goal in relation to its competitors. This should take into account the relatively narrow market potential and the less favorable cost structure of CIDB.

The review is scheduled later this year.

Provided that CIDB’s operations can be based on a more viable foundation, additional TA may be justified to further streamline operational procedures, and to provide training to existing, and especially to new staff, in a situation of rapid staff turnover.

A decision on further TA will be based on the findings of this review.

Funds need to be raised through increased mobilization of local deposits, seeking equity or term financing as in the past, or, as suggested in the Economic Restructuring Plan Matrix (3.4.1), issue of medium-term securities in the local market, if conditions permit.

Action is ongoing.

Staff resources need to be augmented to ensure effective subproject monitoring, supervision, and data collection with a view to improving the operational and repayment performance of subprojects. CIDB should continue to streamline its operations and improve overall efficiency, especially the quality of its loan portfolio and loan monitoring and supervision; continue to exercise rigid cost controls; and make concerted efforts for the collection and management of its arrears. A special effort should be made to tidy up the record of the multiple-subloan subprojects.

Streamlining of CIDB's operations is in progress.

Special efforts need to be made to obtain performance data and financial information from the borrowers on a regular basis. The services of an accountant can be retained for specified periods in the year to (i) train staff, (ii) train subborrowers, and (iii) prepare accounts for some borrowers on a rotating basis. Strong adherence to recognized accounting standards should be a prerequisite for access to loans in the future.

The Bank may consider a TA after this review.

PE-523:Agricultural Technology Education Project (Loan 884-PHI[SF])

Government attention is required in several areas, such as (i) correction of the information deficiency by institutionalizing a system of information gathering on graduates’ performance and the job market; (ii) strengthening the guidance counseling offices at each provincial technical institute of agriculture; and (iii) further rationalization of the agricultural education system. At the provincial technical institutes of agriculture (PTIAs), clearer definition and separation of the teaching and income-earning roles of the vast farm areas should be made. Proper budgeting and accounting systems should be established for the commercial production areas. The PTIAs should also improve the maintenance of equipment

Based on the submission of the Education, Health and Population Division East (AEEH), the Government has been requested to submit a report on the progress of actions taken with respect to these recommendations by 30 September 1999.

Appendix 8, page 12

Recommendations for Follow-Up Action Action Taken/Proposed and include equipment replacement in their financial plans. B. Technical Assistance Performance Audit Reports

TE-23: Advisory and Operational TAs in the Power Sector in India (TAs 1228/1229/1365/1366/1756/2738/ 2739/ 2740/2741-IND)

The Bank's policy should focus initially on corporatizing the distribution subsector beginning with urban distribution, leading to its eventual privatization.

While the Energy Division West (IWEN) agrees with this comment in principle, practical considerations may dictate variations in this approach. The Bank is in dialogue with the governments of the states of Gujarat and Madhya Pradesh to implement this policy.

The Bank should continue to encourage the states to divide their distribution networks into manageable units and convert them into business units with separate financial accounting systems.

This is being done in Gujarat and Madhya Pradesh.

The Bank should encourage the states to adopt dual tariffs, particularly in rural areas—one based on normal supply (at a higher commercial price), and the other based on interruptible supply (at a lower price, consistent with the principle of affordability)—with a view to minimizing financial losses.

Tariff rationalization studies based on various commercial principles, including the one suggested by the Operations Evaluation Office (OEO), have been done for Gujarat. These are planned for Madhya Pradesh also. The studies are being discussed with the Government for implementation.

Future assistance by funding agencies should focus on assisting the states to formulate projects for system improvement, energy conservation, and demand-side management; develop dual tariffs; and corporatize and eventually privatize the distribution subsector. A sector approach should be adopted for any future public sector lending.

This suggestion has been incorporated in the policy dialogue between the Bank and the states of Gujarat and Madhya Pradesh.

There is a need for the Bank's projects department and the Office of Environment and Social Dimensions (OESD) to follow up, as part of the Bank's long-term country strategy, on the recommendations in TA 1229 for the development of a national environmental policy cutting across all sectors. Similarly, the recommendations in TA 1366 should be followed up as part of the Bank's long-term environmental strategy in Tamil Nadu.

This suggestion, although welcome, is not energy sector specific and is left out for initiation by the Programs Department (West) in consultation with OESD.

The India Resident Mission (INRM) should be utilized for TA administration and to follow up with the concerned EAs regarding the implementation of TA recommendations.

Most ongoing loans are being administered by INRM with the assistance of IWEN. However, per the policy of the Bank, loans for the first year after their approval and TAs have to be administered by IWEN. Changes in this procedure, if required, may be sought from the Strategy and Policy Office by OEO.

Future TAs should provide for submission of periodic reports by EAs on the progress of implementation of the TA recommendations.

This will be incorporated in subsequent operations.

Appendix 8, page 13

Recommendations for Follow-Up Action Action Taken/Proposed The Bank should, through more intensive dialogue with other funding agencies, ensure that the projects funded by the Central Government as far as possible directly benefit the states chosen for Bank assistance in order to increase the Bank's leverage in getting the desired reforms implemented.

Although not explicitly stated during the Bank's dialogue with the Government of India, this suggestion has been kept in mind while finalizing the subprojects of the central sector entities that are being assisted by the Bank.

TE-24: Institutional Strengthening of the National Budget and Aid Coordination Division of the Ministry of Finance (TAs. 940/1487/1932/2691-BHU)

The PEM recommended additional long-term assistance in the form of cluster TAs to address pending tasks. Such assistance should address skill deficits at different levels of government and focus on capacity building.

The Bank will consider favorably the provision of necessary inputs in a long-term perspective to complement and support the reform agenda of the Government and the overall development objectives in public expenditure management.

TE-25:Capacity Building in the Financial Sector in the Socialist Republic of Viet Nam (TAs 2039/2063/2218-VIE)

The Government should continue to seek external assistance for training to build on what has already been achieved not only in project financing but also in related subjects including general banking operations. Qualified staff from relevant agencies should be selected and fully utilized appropriately after undergoing training.

The State Bank of Viet Nam has been asking several funding agencies for this. So far, there are many ad hoc-based assistance projects by many agencies.

The Government should speed up the establishment of the regulatory framework and the institutional mechanisms for the introduction of the various financial markets. The Bank should continue to engage in policy dialogue with the necessary supporting TA to encourage the smooth implementation of finance sector reforms.

The Financial Sector and Industry Division West will process project preparatory TA for the Finance Sector Program for 2000.

TE-26:Selected TAs for the Development of Financial Intermediation in the People’s Republic of China (TAs 2214/2288/2626/2658/2664-PRC)

Close monitoring of the State Development Bank of China (SDB) is critical for the Bank to understand the sector's evolution in the PRC.

SDB has requested further TA, which will cover corporate reengineering and risk management. A joint Bank/World Bank mission is also planned to evaluate SDB as a major channeling institution for Bank and World Bank loans.

The Bank should carefully monitor the policy implications of recent restrictive measures, such as prohibitions on foreign banks from extending standby foreign exchange guarantees over yuan lending by domestic banks, and trade finance (letter of credit) to projects outside of the regions where the branches are located, as a basis for sector policy dialogue.

The Bank continues to monitor the Government policy regarding foreign banks. The overall policy direction is to open the market for foreign banks. For example, the People's Bank of China (PBC) announced recently that further foreign bank offices would be set up in more cities in the PRC, and that they could use foreign exchange as collateral to borrow yuan.

Everbright Bank of China should improve its management The Financial Sector and Industry Division

Appendix 8, page 14

Recommendations for Follow-Up Action Action Taken/Proposed information system. The Agricultural Bank of China needs to monitor the effectiveness of a pilot approach, being taken under an ongoing TA, to implementing some of the recommendations of TA 2288. Improvement of internal communication and establishment of a better feedback system may be necessary at SDB. PBC should work to maintain the momentum of rural credit cooperative reform while allocating appropriate staff to the newly established department.

East Review Mission in January 1999 for TA No. 2658 found that Everbright Bank's MIS has improved significantly, but that it will need to be upgraded to fully meet all the demands of operational departments.

C. Impact Evaluation Studies IE-52:Reevaluation of the Technical and Vocational Education Project (Loan 531-PHI)

Future assistance to the subsector has to address more than weaknesses of the supply side; it should be based on an analysis of (i) the education sector as a whole; (ii) structural and management issues, the role of private sector institutions, and how the Technical Education and Skills Development Authority (TESDA) can perform its role to guide and ensure the quality of technician programs; (iii) demand-side concerns; and (iv) cost and benefit comparison of technician training against various forms of postsecondary education. The Government needs to review institutional and managerial issues, and, in collaboration with industries and parent associations, to discuss and agree on measures to improve the quality and reputation of the technician profession.

AEEH is processing the Technical Education and Skills Development Project, which is scheduled for Board consideration in October 1999. The suggestions will be considered to strengthen project design in addressing these issues and concerns. TESDA recognizes the need to rationalize and strengthen its management capacity and relations with industries to address the issues. This aspect will receive special attention in the new project.

The matrix structure of the subsector's management system, with the Commission on Higher Education (CHED) in charge of organizational supervision and TESDA providing functional oversight, has created confusion resulting in technician education suffering in terms of quality and reputation. These issues should be examined to assess the viability of the matrix-type of structure, and whether the existence of technician training programs within tertiary education institutions is appropriate. TESDA should play a major role in creating a different image of the technician profession, working with industries, technical education institutions (TEIs), parents, and students.

A component of the TA on Rationalizing and Strengthening Management Capacity of TESDA, which accompanies the new loan, will address the issues. Clear, distinctive, and complementary, rather than competing, roles of TESDA and CHED will be addressed, including coordination and cooperation between the two agencies and their respective institutions.

The cost of technician education requires the Government to consider new strategies such as (i) encouraging TEIs to introduce more income-generating projects, guidelines to ensure that income-generating projects do not replace teaching activities, and accounting systems and standards such that resources are actually reinvested into teaching programs; (ii) allocating resources from the national budget to reflect the relatively higher cost of technician training; (iii) introducing an accounting system that allows TEIs to calculate the actual cost of programs; and (iv) involving private TEIs to reduce the burden of financing technician training from Government funds.

The suggestions will be incorporated in the project design, and mechanisms will be incorporated to implement them in the field. In the new project, there is a major component on rationalizing and strengthening the role of private providers. The suggestions will be taken into consideration during the design and implementation of the project.

Appendix 8, page 15

Recommendations for Follow-Up Action Action Taken/Proposed The Government should focus its efforts and concentrate TEIs near industries so that close cooperation is possible. Special programs that sponsor participants from remote areas or low-income households could be introduced.

These suggestions are in line with the new project's objectives and will be included among its activities.

The Government should coordinate projects funded by several agencies to increase the impact of assistance and to ensure that objectives and strategies are shared by all.

The Government has been advised, and the new project will ensure that loan funds are judiciously used to increase the impact of the project.

IE-53: Reevaluation of the Sabah/Sarawak Fisheries Infrastructure Project (Loan 563-MAL)

The Government should divest itself of the Sabah Fish Marketing Sdn. Bhd. (SAFMA) and the Sarawak facilities operated by the Fisheries Development Authority of Malaysia. It should upgrade the information base for the sector and review the policies governing fishing licenses. The parallel introduction of arrangements to change fishers from hunters to resource managers would also be beneficial.

The Government does not plan to divest of the SAFMA and Sarawak facilities. The Department of Fisheries (DOF) has upgraded its computerized fisheries information system so that the databases of the states and the Central Government are integrated. DOF conducted a fisheries resource survey in 1998, which updated the maximum sustainable yield in five major fishing grounds. The updated database will be used for revising the licensing policy for the fishing boats. Moreover, DOF is implementing a vessel tracking management system. DOF is also initiating a community-based fisheries resource management program.

IE-54: Reevaluation Study of the Palawan Integrated Area Development Project (Loan 528-PHI/529-PHI[SF])

The first priority in allocating water use fees should be O&M of existing facilities. Amortization payments should be made only after adequate resources are allocated to sustain irrigation facilities. There is a case for reviewing and possibly renegotiating a reduced amortization schedule for the irrigators associations (IAs). Consideration should be given to directly involving IAs in improving and strengthening watershed management.

Beneficiary repayment of the chargeable costs of irrigation system investment is scheduled over a 50-year period with interest. This is a gradual transition of the capital investment program of the agency. The Palawan Forestry Program is now assisting the LGUs in improving watershed management, which should ensure water availability.

More clearly defined policies are needed to take action against farmers who do not pay dues and on reallocation of scarce water resources. Responsibility for ownership and O&M of project facilities needs to be clarified. There is a need to reassess the role of government in supporting livestock development in Palawan and strategies that are most likely to benefit the poor. Effective arrangements need to be made to deliver institutional credit to the rural poor.

This entails policy issues and is presently under study by the National Irrigation Administration.

Concrete bridgework is needed, not bailey bridges. Plans for further expansion of ports should include a review of navigation and marine safety arrangements, an environmental impact analysis (EIA), a review of arrangements for handling ship wastes, and the formulation of contingency plans to deal with emergencies such as oil spills. Port development plans should be

The Department of Agriculture and the provincial government are negotiating for the final turnover of seven rural agriculture centers in response to the clamor of the concerned LGUs. The Rural Agricultural Center of Puerto Princesa, which was turned over in 1992, is being properly maintained by the city

Appendix 8, page 16

Recommendations for Follow-Up Action Action Taken/Proposed formulated within the framework of a provincial transport development strategy. Dysfunctional water supply schemes should be restored to operation where possible with effective institutional arrangements. The supply of antimalarial drugs should be continued in adequate quantities, and the proper use of these drugs should be monitored. Issuance of land titles should be monitored along with other important project benefits. Attention needs to be paid to the effects of in-migration on the environment of Palawan, and the policy on in-migration needs to be reconsidered.

government. The Provincial Agricultural Center and the Livestock Resource Center in Irawan and the Abo-abo Satellite Research Center have an appropriation of about P14.0 million for 1999 operation. The Government has commissioned the Land Bank of the Philippines to deliver institutional credit not only in Palawan but throughout the country through rural cooperatives. The Department of Public Works and Highways has assured the Palawan Council for Sustainable Development that the recommendation of the Bank’s Reevaluation Mission has been considered in their programming for 2000. Budgeting for 1999 was complete when the Reevaluation Mission was conducted. The Palawan Council for Sustainable Development (PCSD) must evaluate projects before the Department of Natural Resources can issue an environmental compliance certificate. PCSD will ensure that all the concerns of the Reevaluation Mission are addressed, such as the review of navigation and marine safety arrangements, EIA, review of arrangements for handling ship waste, and formulation of contingency plans to deal with emergencies such as oil spill. The Department of Public Works and Highways will include in its budget the funds to be used in restoring the defective water supply systems. The Department of Health (DOH) is continuously intensifying its program on mosquito net impregnation. DOH is likewise conducting research on drug resistance by mosquitoes as part of its regular program. However due to the low budget ceiling for procurement of drugs, an adequate supply of antimalarial drugs cannot be ensured. The Land Management Division of the Provincial Environment and Natural Resources Office has established linkages with the Assessor's Office and the Register of Deeds to ensure the continuous issuance/release of land titles. On the other hand, the Land Resource Information System was established to monitor the progress of land title application and releases of land titles. The Environmentally Critical Areas Network (ECAN) Guidelines for Terrestrial and Coastal Marine Areas have been finalized and have been adopted by the LGUs. Delineation of alienable and disposable, and forest areas is complete, and preparation of ECAN maps for terrestrial areas is in process. The maps will be the basis for the formulation of the LGUs' land-use plans.

IE-57:Reevaluation of the Second Vocational

Appendix 8, page 17

Recommendations for Follow-Up Action Action Taken/Proposed Education Project (Loan 441-THA[SF]) Adherence should be ensured to the specific reform measures and various activities as agreed upon in the Social Section Program loan to address policy priorities and systemic problems relating to technical education and vocational training.

The Government has already taken steps to comply with the relevant conditions relating to technical education and vocational training.

A high-level national body should be established to (i) coordinate and update policies for technical education among Government institutions at different levels; and (ii) facilitate implementation through networking among agencies, schools, and industry.

The recent mission to Thailand asked the Government to convene the National Vocational Training Coordination Committee meetings quarterly.

Institutional linkages with industries should be established to (i) improve competency-based curricula, (ii) provide for on-the-job training for both students and teachers, (iii) facilitate employment marketing in line with regional needs and priorities, and (iv) provide a venue for maintaining and sustaining research relating to policies and strategies specific to the technical/vocational subsector and to the education sector as a whole.

The Bank has encouraged the Government to form an advisory council for each institution to reflect local skills requirements in its curriculum. This is being implemented by the project schools of Loan 1494-THA: Skills Development Project.

Private sector sponsorships or scholarships for teacher development should be explored, and formal cooperative (twinning) arrangements with local and foreign institutions intensified to maintain and regularly upgrade teacher qualifications.

Twinning arrangements with overseas schools are being sought by a few institutions.

A comprehensive employment promotion system should be established for labor market monitoring, career guidance, job placement, and feedback to institutions/schools for curriculum review.

The system is being tested under Loan 1494-THA and will be expanded to the education sector if proven successful.

A proper regulatory framework for skill development and vocational training, and incentives for enterprises to promote these activities, should be included in the Vocational Education and Occupational Training Act (VEOTA).

The VEOTA was appropriately amended and renamed the Skills Development Promotion Act, which was approved by the Cabinet on 23 February 1999.

More gender-neutral industry courses, such as those in the service industry, which are in high demand, should be established.

Such courses as catering, textile design, computer operation, accounting, etc. are being introduced.

Appropriate review mechanisms should be set up for curriculum and textbook improvement or development on a regular basis, placing emphases on entrepreneurial training, work attitudes and ethics, industry linkages, and better foundations in science and mathematics.

A systematic mechanism to update curricula and training materials is being tested by the Department of Skill Development with emphasis on work attitudes and ethics. With the recent economic crisis, emphasis is being given to entrepreneurship training.

The MIS should be upgraded to include continuous monitoring and auditing measures to determine a project's impact and effectiveness on educational development throughout the project period and after project completion.

Some essential baseline data have been collected, and action plans have been developed to measure the progress and impact of the project (Loan 1494-THA). Future loan projects will be required to follow this

Appendix 8, page 18

Recommendations for Follow-Up Action Action Taken/Proposed procedure.

A review is needed of the strategy to support Government efforts in preparing quality technical personnel for the country's revised industrialization program. The Government needs further assistance to ensure the realization of the full potential of past investments.

The Government plans to borrow from bilateral and multilateral agencies to improve technical education and vocational training facilities.